prima·bench
Repo ↗

Prima Mente · Model evals

Every eval we run, in one ledger.

prima-bench grades what a model actually knows about biology — across genes, cells, sequences, and whole samples. This is the internal map: which evals exist, what each one measures, and the exact way to run it.

Evals
179
task × dataset
Tasks
15
registered
Families
10
with ≥1 eval
Datasets
182
in the registry
Recipes
59
curation dirs

catalog fb70fe0 · 2026-07-21

01

Coverage

Where the 179 live evals land today — by bucket and task family, and how many reach across paired modalities.

Evals per bucket179 total
  • Sequence-level evals
    102
  • Gene-level evals
    42
  • Cell-level evals
    24
  • Sample-level evals
    11
  • sequence classification68
  • variant effect31
  • sequence clustering3
  • gene function29
  • gene regression5
  • gene classification4
  • sample classification4
  • integration15
  • cell type annotation8
  • perturbation1
  • sample classification11
Modality mix179 arms
  • single-modality164
  • gene expression9
  • chromatin accessibility5
  • joint (paired)1
02

Five buckets

03

Run one

prima-bench never touches your model. Fetch a censored eval input, hand back predictions in the expected shape, and score against the held-out solution — any framework, any language. The prima-bench CLI is discovery + docs; scoring runs through the Python API.

example gene_function_classification · ashburner_2000_nature_genetics/go_bp@0.0.3 

import prima_bench as pb

# Downloads on demand into the local cache; returns local Paths.
train_path, test_path = pb.fetch_data(task="gene_function_classification", dataset="ashburner_2000_nature_genetics/go_bp@0.0.3")
train = pb.load(train_path, task="gene_function_classification", split="train", dataset="ashburner_2000_nature_genetics/go_bp@0.0.3")
test = pb.load(test_path, task="gene_function_classification", split="test", dataset="ashburner_2000_nature_genetics/go_bp@0.0.3")

# Your model produces predictions aligned to the eval input.
predictions = my_model(test)

# Score against the censored solution over this task's metric panel:
#   gene_function_term_auprc, gene_function_term_auroc, gene_function_micro_auprc, gene_function_fmax, gene_function_smin
run = pb.Evaluation(task="gene_function_classification", dataset="ashburner_2000_nature_genetics/go_bp@0.0.3")
run.update(predictions=predictions)   # one-shot; or stream shard-by-shard
result = run.compute()                # -> EvalResult
print(result)