Core · concepts
The registry, in words.
Every family, task, and metric prima-bench knows about — 10 families, 12 tasks, 58 metrics — generated straight from the eval catalog. Add one to the registry and its card lands here on the next build.
Families 10
-
integration cell
Embedding-quality families — batch and multimodal integration, scored by the scIB panel.
-
cell_type_annotation cell
Per-cell identity labels graded against a curated atlas.
-
perturbation cell
Predict a cell's expression response to a perturbation.
-
gene_function gene
Ontology-term membership per gene — the CAFA-style function-prediction family.
-
sample_classification gene
Whole-sample phenotype / disease classification.
-
gene_regression gene
One gene in, one scalar out (essentiality, constraint, dosage).
-
gene_classification gene
One gene in, one categorical class out (e.g. protein family or a TF axis).
-
sequence_classification sequence
Regulatory readouts from genomic sequence (the Nucleotide Transformer suite).
-
variant_effect sequence
Effect of a single-base variant — classification (pathogenicity, hits) and regression (effect size).
-
sequence_clustering sequence
Zero-shot clustering quality of genomic-region embeddings against known labels.
Tasks 12
-
batch_integration cell
Score a cell embedding for biology preservation and batch mixing over the 11-metric scIB panel; batch and cell-type labels join by cell_id on the solution side.
-
cell_type_annotation cell
Score per-cell cell-type label predictions against ground truth, paired by cell_id. Panel matches OpenProblems label_projection plus CL-ancestor-aware headline metrics.
-
multimodal_integration cell
Score a joint embedding of two weakly-paired modality streams: the scIB panel (batch = modality) plus FOSCTTM and kNN-AUC alignment.
-
perturbation_prediction cell
Predict post-perturbation gene expression from baseline plus perturbation identity, over an exactly-aligned gene space (a 'ctrl' row is required).
-
gene_function_classification gene
Predict a scalar membership score per (gene, ontology-term) pair; truth is 0/1, paired by the composite gene_id|term key. Source-agnostic across ontology families.
-
sample_classification gene
Score sample-level disease-classification predictions (per-class scores) against held-out truth, paired by sample_id. Primary metric AUROC with a bootstrap CI.
-
gene_regression gene
Predict a scalar continuous value per gene (e.g. DepMap gene-essentiality); paired by gene_id, with the target censored from test.
-
gene_classification gene
Predict a per-class score vector per gene over a K-way single-label class space (e.g. PANTHER family); truth is one class index per gene, paired by gene_id.
-
sequence_classification sequence
Score genomic sequence-classification predictions, paired on read_id. Serves the 18 Nucleotide Transformer downstream tasks; primary metric MCC.
-
variant_effect_classification sequence
Predict a per-variant binary outcome probability (pathogenic/benign, hit/no-hit); scalar per variant, paired by variant_id.
-
variant_effect_regression sequence
Predict a per-variant continuous effect of a single-base substitution (e.g. QTL effect size); scalar per variant, paired by variant_id.
-
zeroshoot_clustering sequence
Zero-shot clustering of genome-partition class embeddings: Leiden-cluster the sequence embedding and score label recovery (ARI/NMI) and separation (ASW).
Metrics 58
-
accuracy
Fraction of cells with label_pred == label_true (OpenProblems); misleading on imbalanced atlases — pair with f1_macro.
-
accuracy_ancestor_aware
Accuracy crediting predictions whose CL term is an ancestor/descendant of the truth (granularity-tolerant). The CTA headline.
-
ari
Adjusted Rand Index between Leiden clusters of the integrated embedding and ground-truth cell types. Higher → better label recovery (scIB).
-
batch_asw
Batch average silhouette width — within-cell-type batch mixing. Higher → better mixing (scIB).
-
cell_type_asw
Average silhouette width on cell-type labels, rescaled to [0,1]. Higher → cell types stay compact and separable (scIB).
-
clisi
Cell-type LISI — effective number of cell types in each cell's kNN neighbourhood, rescaled to [0,1]. Higher → better label preservation (scIB).
-
clustering_ari
ARI between Leiden clusters (res 1.0) of the sequence embedding and genome-partition class labels. Higher → organises by functional class.
-
clustering_asw
Average silhouette width of the sequence embedding on genome-partition class labels, rescaled to [0,1].
-
clustering_nmi
NMI between Leiden clusters (res 1.0) of the sequence embedding and genome-partition class labels.
-
f1_macro
Unweighted mean of per-class F1 across cell types — imbalance-honest headline (OpenProblems).
-
f1_macro_ancestor_aware
Macro-F1 crediting ancestor/descendant predictions (granularity-tolerant); falls back to flat f1_macro without CL ids.
-
f1_per_class
Per-class F1 across cell types, surfaced as a per-class table — reveals which cell types the model fails on.
-
f1_weighted
Support-weighted mean of per-class F1 — stable headline (OpenProblems).
-
foscttm
FOSCTTM — mean fraction of other-modality cells closer than a cell's true partner. Lower → better cross-modality alignment.
-
gene_class_accuracy
Top-1 multi-class accuracy for gene_classification (argmax vs truth index). Pair with f1_macro under imbalance.
-
gene_class_accuracy_ancestor_aware
Top-1 accuracy crediting predictions on the truth class's lineage. Headline on hierarchical class spaces.
-
gene_class_balanced_accuracy
Mean per-class recall over truth-present classes — imbalance-robust (sklearn).
-
gene_class_f1_macro
Unweighted mean of per-class F1 over the K-way gene class space — imbalance-honest headline.
-
gene_class_f1_macro_ancestor_aware
Macro-F1 crediting lineage predictions — imbalance-honest hierarchical headline; falls back to flat f1_macro.
-
gene_class_f1_weighted
Support-weighted mean of per-class F1 for gene_classification — stable headline.
-
gene_class_mcc
Multiclass Matthews correlation coefficient for gene_classification (sklearn).
-
gene_class_top_k_accuracy
Top-k accuracy — correct when the truth class is among the k highest-scored (default k=5).
-
gene_function_fmax
Protein-centric CAFA F-max over a shared threshold grid for the gene-function ontology-term task. Higher → better.
-
gene_function_micro_auprc
Micro-averaged AUPRC — every present (gene, term) pair is one binary example; weights all pairs equally.
-
gene_function_smin
Information-weighted CAFA semantic distance (S-min). A distance — lower is better, 0 is perfect.
-
gene_function_term_auprc
Macro-averaged per-term area under the precision-recall curve for gene-function ontology terms (true-path propagated).
-
gene_function_term_auroc
Macro-averaged per-term ROC-AUC for gene-function ontology terms (true-path propagated).
-
gene_reg_mse
Mean squared error between scalar per-gene predictions and continuous truth (sklearn).
-
gene_reg_pearson
Pearson correlation of scalar per-gene predictions vs continuous truth (aligned by gene_id).
-
gene_reg_r2
Coefficient of determination (R²) of per-gene predictions vs truth; NaN when the truth is constant.
-
gene_reg_rmse
Root mean squared error between scalar per-gene predictions and continuous truth.
-
gene_reg_spearman
Spearman rank correlation of scalar per-gene predictions vs truth. PRIMARY for gene_regression — ranking beats calibration.
-
graph_connectivity
Mean fraction of each cell type's cells in the largest connected component of its kNN subgraph. Higher → cell types stay connected (scIB).
-
ilisi
Integration LISI — effective number of batches in each cell's kNN neighbourhood, rescaled to [0,1]. Higher → better mixing (scIB).
-
isolated_labels_asw
Silhouette (rescaled) averaged over labels present in the fewest batches. Higher → rare labels stay separable.
-
isolated_labels_f1
Best F1 between Leiden clusters and each isolated label's indicator. Higher → rare labels recoverable as clusters.
-
kbet
kBET acceptance rate (1 − rejection), stratified by cell type. Chi-squared test of local batch composition. Higher → better mixing.
-
knn_auc
kNN-AUC — per-cell overlap of within-modality-A and within-modality-B neighbours (partner-matched). Higher → better neighbourhood preservation.
-
l2_pseudobulk
L2 distance per perturbation on pseudobulk vectors (mean-collapsed by perturbation).
-
l2_pseudobulk_top_1000_control_genes
L2 per perturbation on pseudobulk, restricted to the top-1,000 control-expressed genes (Ahlmann-Eltze 2025 Fig 1A).
-
nmi
Normalised Mutual Information between Leiden clusters of the integrated embedding and ground-truth cell types. Higher → better label recovery (OpenProblems).
-
pcr_batch
1 − principal-component regression of the embedding on batch — variance not batch-driven. Higher → less batch variance (scIB).
-
sample_accuracy
Accuracy at the operating point (Youden threshold fixed on train, or argmax). Secondary — AUROC is primary.
-
sample_auprc
Average precision (area under PR curve); binary at K=2, macro OvR at K>2. Threshold-free; informative when positives are rare.
-
sample_auroc
PRIMARY sample metric: AUROC with a mandatory bootstrap 95% CI (binary at K=2, macro OvR at K>2).
-
sample_balanced_accuracy
Balanced accuracy (mean per-class recall) at the operating point. Secondary to AUROC.
-
sample_f1
F1 at the operating point (Youden threshold at K=2, macro argmax at K>2). Secondary to AUROC.
-
sample_mcc
Matthews correlation at the operating point — a balanced confusion summary, robust to imbalance. Secondary to AUROC.
-
sample_score_correlation
Pearson + Spearman of the predicted positive-class score vs a continuous clinical target. No-op when absent.
-
sample_sensitivity
Sensitivity (recall) at the operating point (Youden threshold fixed on train). Compare to the WHO TB triage TPP (≥90%).
-
sample_specificity
Specificity (TNR) at the operating point (Youden threshold fixed on train). Compare to the WHO TB triage TPP (≥70%).
-
seq_accuracy
Fraction with label_pred == label_true — imbalance-sensitive; read with seq_mcc / seq_f1_macro (sklearn).
-
seq_f1_macro
Unweighted mean of per-class F1 — imbalance-honest secondary to seq_mcc (sklearn).
-
seq_mcc
Matthews correlation of predicted vs true sequence-class labels — the Nucleotide Transformer PRIMARY metric, robust to imbalance.
-
vep_auprc
Area under the PR curve for binary VEP tasks (ClinVar, dsQTL, MFASS, ENCODE-rE2G). Scalar prediction per variant.
-
vep_auroc
Area under the ROC curve for binary VEP tasks — class-balance insensitive; pair with vep_auprc on positives-rare sets.
-
vep_pearson_r
Pearson correlation of scalar predictions vs continuous QTL effect sizes (aligned by variant_id).
-
vep_spearman_rho
Spearman rank correlation of scalar predictions vs continuous QTL effect sizes.