Run your first eval

prima-bench never touches your model. You run the model in your own process and hand the library predictions in the expected shape; it validates them against the task contract, runs the metric panel, and returns a structured EvalResult. This walkthrough takes you from an empty checkout to your first scored result.

1. Install and set credentials

Datasets are fetched on demand from object storage. Credentials are a GCS HMAC keypair carried in prima-bench’s own environment variables; a gitignored, project-local .env is loaded automatically. Copy the example and fill it in:

# install the library (uv resolves the environment)
uv sync

# copy the credentials template and fill in your HMAC keypair
cp .env.example .env

HMAC_BOTO_PRIMA_MENTE_BENCHMARK_KEY_ID=<gcs-hmac-access-id>
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_SECRET=<gcs-hmac-secret>
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_REGION=auto

These are HMAC keys, not literal AWS keys — prima-bench passes them to boto3 as explicit credentials and never falls back to AWS_* or ~/.aws. See the Guide for generating a keypair with gcloud.

2. Discover what is registered

Treat these calls — not hardcoded strings — as the source of truth; the registered tasks, datasets, and metrics evolve over time.

import prima_bench as pb

pb.list_tasks()
# -> ["batch_integration", "cell_type_annotation", "perturbation_prediction", ...]

# Dataset keys are "{publication}/{slug}@{version}".
pb.list_datasets(task="perturbation_prediction")
pb.list_metrics(task="perturbation_prediction")

The prima-bench CLI mirrors the same registry — it is discovery and docs only. There is no score subcommand; scoring runs through the Python API.

prima-bench list tasks
prima-bench list metrics --task perturbation_prediction
prima-bench list datasets --task perturbation_prediction

# regenerate the Markdown task/metric/dataset cards
prima-bench docs build

3. Fetch data, predict, evaluate

pb.fetch_data returns a (train_path, test_path) pair — no per-task polymorphism, and the solution bytes are never returned. pb.load invokes the registered per-split loader and returns EvalData; you may also load the path yourself with scanpy, polars, a PyTorch dataloader, or whatever fits.

import prima_bench as pb

# Downloads on demand into the local cache; returns local Paths.
train_path, test_path = pb.fetch_data(task="perturbation_prediction")

train = pb.load(train_path, task="perturbation_prediction", split="train")
test  = pb.load(test_path,  task="perturbation_prediction", split="test")

model.fit(train)
predictions = model(test)   # an EvalData aligned to the eval input

run = pb.Evaluation(task="perturbation_prediction")  # loads censored solution + metric panel
run.update(predictions=predictions)                  # one-shot == a single update
result = run.compute()                               # per-metric compute() -> EvalResult
print(result)

Here is the exact, non-fabricated invocation for a real registered eval — switch tabs for the Python flow and the CLI discovery commands:

import prima_bench as pb

# Downloads on demand into the local cache; returns local Paths.
train_path, test_path = pb.fetch_data(task="perturbation_prediction", dataset="hao_2024_nature_methods/norman_2019_science@0.0.1")
train = pb.load(train_path, task="perturbation_prediction", split="train", dataset="hao_2024_nature_methods/norman_2019_science@0.0.1")
test = pb.load(test_path, task="perturbation_prediction", split="test", dataset="hao_2024_nature_methods/norman_2019_science@0.0.1")

# Your model produces predictions aligned to the eval input.
predictions = my_model(test)

# Score against the censored solution over this task's metric panel:
#   l2_pseudobulk, l2_pseudobulk_top_1000_control_genes
run = pb.Evaluation(task="perturbation_prediction", dataset="hao_2024_nature_methods/norman_2019_science@0.0.1")
run.update(predictions=predictions)   # one-shot; or stream shard-by-shard
result = run.compute()                # -> EvalResult
print(result)

4. Streaming and evaluation-only datasets

pb.Evaluation is dual-mode. For large or distributed runs, feed predictions shard-by-shard; the result is identical to a single one-shot update of the same data.

run = pb.Evaluation(task="cell_type_annotation")
for shard in prediction_shards:   # produced incrementally / per rank
  run.update(predictions=shard)
result = run.compute()

Some datasets (e.g. test-only variant-effect sets) expose no train split, so pb.fetch_data raises KeyError. Use pb.fetch_test to get just the censored eval input:

test_path = pb.fetch_test(
  task="variant_effect_classification",
  dataset="avsec_2026_nature/clinvar_missense@0.0.1",
)
test = pb.load(
  test_path,
  task="variant_effect_classification",
  split="test",
  dataset="avsec_2026_nature/clinvar_missense@0.0.1",
)

What you get back

EvalResult is structured and comparable across models: per-metric scores plus SHA-256 hashes of the canonicalized predictions, truth, and dataset ref. What you do with the numbers — log to W&B, write a CSV, publish — is yours; the library has no leaderboard and no opinion.

Next: browse the glossary to see every task and metric, or learn how the bytes get curated in Add a recipe end-to-end.