Good models need great benchmarks.

Documentation

• GUIDE.md — the full guide, and the best place to start: why prima-bench exists, how to evaluate a model against a task (consumer), and how to add tasks, metrics, datasets, and recipes (developer).
• ARCHITECTURE.md — directory map and the layer-by-layer separation of concerns.
• AGENTS.md — contributor conventions: code reuse, data licensing, worktrees, testing.

Usage Example

import prima_bench as pb

# Discover what's registered — treat these calls (not hardcoded strings) as the
# source of truth; the registered tasks/datasets/metrics evolve over time.
pb.list_tasks()                                   # -> all task names
pb.list_datasets(task="perturbation_prediction")  # -> dataset "name@version" keys for a task
pb.list_metrics(task="perturbation_prediction")   # -> metric names for a task

The prima-bench CLI mirrors the same API:

prima-bench list tasks
prima-bench list datasets --task perturbation_prediction
prima-bench list metrics --task perturbation_prediction

Eval on a Tasks default Dataset:

train_path, test_path = pb.fetch_data(task="perturbation_prediction")

# Optional: invoke the registered per-artifact loader. 
# Converts to AnnData or Tensor depending on task
train = pb.load(train_path, task="perturbation_prediction", split="train")
test  = pb.load(test_path,  task="perturbation_prediction", split="test")

model.fit(train)
preds = model(test)

run = pb.Evaluation(task="perturbation_prediction")  # loads solution; builds the metric panel
run.update(predictions=preds)                        # one-shot == a single update
result = run.compute()                               # per-metric compute() -> EvalResult

pb.Evaluation is the evaluation API. It loads the (censored) solution internally and fans each update out to the task’s metric panel via the dual-mode metrics’ update / compute cycle. For large or distributed runs you can feed predictions shard-by-shard instead of materializing them all at once:

run = pb.Evaluation(task="cell_type_annotation")   # loads solution; builds the metric panel
for shard in prediction_shards:                    # produced incrementally / per rank
    run.update(predictions=shard)
result = run.compute()                             # per-metric compute() -> EvalResult

A single-update (one-shot) run produces the same result as streaming the same data across several shards.

Benefits

1. Task curators can easily define Tasks, Metrics, and Datasets.
2. Documentation for Tasks, Metrics, and Datasets is autogenerated as Markdown cards (prima-bench docs build).
3. Evals are reproducible and idempotent. Same predictions + same dataset version + same benchmark package version + same task / metric selection = same scores, always.
4. Results are structured and comparable across models. A standardized results format that supports cross-model leaderboards and per-perturbation drill-down.
5. Datasets are download on demand and are cached locally.

Storage backends & credentials

Datasets are fetched on demand from object storage. prima-bench is bucket-agnostic: it reads s3:// and gs:// URIs through a single boto3, S3-compatible client (AWS S3, Google Cloud Storage via its S3-interop / XML API, Cloudflare R2, MinIO). There is no gcloud dependency.

Credentials are a GCS HMAC keypair carried in prima-bench’s own environment variables — and a project-local .env is loaded automatically (it is gitignored). The canonical datasets live on GCS, so the HMAC keypair is all you need:

# .env  (copy from .env.example)
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_KEY_ID=<gcs-hmac-access-id>
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_SECRET=<gcs-hmac-secret>
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_REGION=auto

These are HMAC keys, not literal AWS keys. prima-bench reads them directly and passes them to boto3 as explicit credentials — it never uses the AWS_* environment variables, ~/.aws/credentials, or an instance role, so set the keypair only under these names. With no HMAC keypair configured, a dataset fetch raises rather than silently falling back.

Generating a GCS HMAC keypair

If you don’t already have an HMAC keypair, create one with gcloud (a one-time setup; the secret is shown only once). Replace YOUR_PROJECT_ID and YOUR_BUCKET with your own — Prima Mente uses project dna-fm and bucket prima-mente-benchmark.

# 0. Confirm/set your project
gcloud config get-value project
# If wrong: gcloud config set project YOUR_PROJECT_ID   # dna-fm for us

# 1. Create a dedicated service account
gcloud iam service-accounts create gcs-s3-interop \
    --display-name="GCS S3 Interop"

# 2. Grant least-privilege object access on the bucket
gcloud storage buckets add-iam-policy-binding gs://YOUR_BUCKET \
    --member="serviceAccount:gcs-s3-interop@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectUser"

# 3. Create the HMAC key (prints accessId + secret — secret shown ONCE)
gcloud storage hmac create \
    gcs-s3-interop@YOUR_PROJECT_ID.iam.gserviceaccount.com

The printed accessId and secret go into .env as HMAC_BOTO_PRIMA_MENTE_BENCHMARK_KEY_ID and HMAC_BOTO_PRIMA_MENTE_BENCHMARK_SECRET respectively.

gs:// URIs default to the GCS interop endpoint automatically; point PRIMA_BENCH_S3_ENDPOINT_URL at R2/MinIO (or PRIMA_BENCH_GCS_ENDPOINT_URL to override the GCS endpoint) when reading other S3-compatible stores.

Design Decisions

1. The benchmark never touches model or training internals. Users run their model externally and hand in predictions in a standardized format. The benchmark scores them. The eval contract is defined by the expected output format (e.g., an expression matrix with matching gene space), not by a model API. Any model that produces the right shape can be scored, regardless of framework, language, or hardware.
2. The benchmark uses hosted canonical datasets (S3/GCS/HF) that are versioned and downloadable on demand, fetched through a bucket-agnostic boto3 client.
3. Metrics run on a torch backend (Metric subclasses with update/compute) and use the GPU when one is available, CPU otherwise. Device selection is automatic via resolve_device(), overridable with the PRIMA_BENCH_DEVICE env var. Metrics are dual-mode: they buffer RAW inputs in torchmetrics add_state declarations ("cat" for per-sample tensors, "sum" for counters) and run all math in compute, so the same code path supports BOTH single-process and distributed evaluation via torchmetrics state reduction. The caller may either gather inputs to one rank and score single-process, OR shard update calls across ranks and let torchmetrics all-gather/sum the state before compute — both give identical results (see the dual-mode recipe in prima_bench.metrics.base).

See ARCHITECTURE.md for more details.