The operational “how to work in this repo” guide, for coding agents (Claude Code auto-loads CLAUDE.md, which points here; Codex and others read this file directly) and human contributors alike. Humans wanting the narrative — why prima-bench exists, and the consumer + developer walkthroughs — should read GUIDE.md, the human guide; this file is its working companion.

prima-bench is a benchmarking library for biological AI models: model predictions in (EvalData), scores out (EvalResult). It never imports a model framework and never runs or trains models.

Read before you change anything

• ARCHITECTURE.md — the directory map, layer separation, invariants, and coordinate conventions. Read it before making or planning changes, and don’t make changes that alter the decisions recorded there. If a request can’t be met within the current architecture, update ARCHITECTURE.md and the relevant README.md in the same PR.
• The per-layer src/prima_bench/<layer>/README.md for whatever you’re touching, and GUIDE.md Part 3 for the full add-a-task/metric/dataset/recipe walkthrough.

Where things live (edit in the right layer)

Invariants you must not break

Full detail is in ARCHITECTURE.md; this is the guardrail checklist.

• solution is never reachable from consumer code — no fetch_* returns it; pb.Evaluation loads it internally.
• pb.fetch_data returns (train_path, test_path) — no per-task polymorphism; test-only datasets expose no train split and raise KeyError (use pb.fetch_test).
• Censorship is by NaN, never zero or absence.
• Versioned bytes are immutable — any byte change requires a new v{version}/.
• Metrics are dual-mode — state holds raw inputs; compute does all the math, so single-process and sharded/distributed runs give identical results.
• Coordinates: a variant pos is 1-based; an index / slice is 0-based half-open. Off-by-ones here are silent correctness bugs.

General Principles

• State assumptions clearly and explicitly
• Write well-designed, modular, extensible code
• Avoid writing code or plans that would change decisions in ARCHITECTURE.md. If user requests cannot be satisfied with the current architecture, ensure that ARCHITECTURE.md and corresponding README.md files are updated to reflect changes.

Code reuse, DRY & conciseness

• Search before you write. Before adding any function, helper, class, or constant, grep/glob for something that already does it. Check core/ (EvalData, Contract, results), metrics/ (the Metric base class), io/ (download/cache, hashing), and the registry decorators (register_metric / register_task / register_dataset) first. Reuse existing bases and registered components rather than re-deriving them; don’t hand-roll a metric torchmetrics already provides.
• Reuse / extend / create. Existing code covers ≥~80% of the need → reuse it (import; never copy-paste). Close but not exact → extend it without breaking callers. Write new only when nothing fits — with a one-line note on why. Rule of three: the third duplicate gets factored into a shared helper; define constants once (UPPER_SNAKE), no repeated magic values.
• Be concise. Minimum code that solves the problem: no speculative generality, unused params, or defensive scaffolding for impossible cases. Match the surrounding file’s density; skip docstrings on obvious helpers; prefer composition / expression chains over long imperative blocks. Keep comments concise too — explain why when it’s non-obvious, not what the code already says, and skip full background-context dumps.

Logging & output

• Diagnostics go through logging, not print. Use a module logger (logger = logging.getLogger(__name__)) for progress, warnings, and errors; call logging.basicConfig(...) once at the CLI entry point. These land on stderr.
• Reserve print() (stdout) for the tool’s actual data product — e.g. a report table a caller may pipe or parse. Never mix diagnostics into stdout.
• Recipe helpers raise, they don’t sys.exit. In recipe code (recipes/**/recipe.py and the shared modules — utils.py, storage.py, obo.py, qtl_*.py, …), a validation/precondition failure inside a helper function must raise an exception (ValueError, FileNotFoundError, …), not sys.exit(...). Helpers are imported by main(), by explore.py preflights, and by unit tests — a sys.exit there can’t be caught or pytest.raises-tested and tears down the whole process. Only the script boundary owns the exit code: a main(), explore.py preflight control flow, and the shared utils.fail_if_mismatches verdict may sys.exit. A guard test (recipes/tests/test_no_sysexit_in_helpers.py) enforces this in CI.
• Joining gs:// paths: os.path.join(base, *parts) works fine for gs:// URIs — as long as base carries the scheme (don’t pass a bare "gs://" as its own component). No dedicated helper needed.

Data licensing (read before adding any dataset)

Every dataset curated into prima-bench must be released under a license that permits commercial use, redistribution, and derivatives — we run commercial models against these benchmarks. “Academic / non-commercial research only” data cannot enter the repo.

• The acceptable / not-acceptable classification is recipes/LICENSING.md — the single source of truth.
• Licensing maintainers: Francisco M. Martín-Zamora (francisco@primamente.com) and Pouya Niki (pouya@primamente.com). All licensing questions and approvals go through them.
• Before writing a new recipes/{publication}/{dataset}/ recipe, confirm the upstream license is on the Acceptable list. If it isn’t clearly acceptable, stop and ask the user before proceeding — re-curating around a bad license later is expensive.
• Each dataset README must carry a ## License section with an SPDX token — or an approved fallback (public-domain / US-Gov-PD) where no SPDX identifier exists (see the per-dataset template in recipes/README.md).
• Flag this to the user whenever a PR adds or changes a dataset/recipe and the license is missing, non-commercial, no-derivatives, controlled-access, or unclear — don’t let it slip through silently. A not-acceptable license is not an automatic no: tell the developer to raise it with the maintainers (Francisco / Pouya) to evaluate the risk and the exact use case, since testing is sometimes allowed. CI (.github/workflows/license-check.yml) enforces the same gate, but call it out at authoring time.

External systems

• Linear: issues for this repo live in the prima-bench project on the Tech team (https://linear.app/prima-mente/project/prima-bench-8b623afc8174). File new tickets there — not against the Tech team root, and not in any other project.

Worktrees

Before making code changes, create a new worktree for the task. worktrees live under .worktrees/.

Testing

• CI (.github/workflows/ci.yml) gates a PR on four things, so run them locally before pushing: uv run ruff check src tests, uv run pyright, uv run pytest --cov=prima_bench, and an import-smoke of every recipe (uv run --project recipes pytest recipes/tests/ -q).

• The standard suite runs with uv run python -m pytest.

• Some integration tests are skip-by-default: they cross-check our code against an external reference library that is not a project dependency, so they importorskip and are silently skipped in CI. Run them by overlaying the dependency, e.g. the indel/sequence cross-check against kipoiseq:

  uv run --with kipoiseq python -m pytest tests/integration/test_indel_matches_kipoiseq.py

  Re-run the relevant differential test whenever you touch the code it validates (the source it covers carries a NOTE pointing back here).

Adding things — quick index

• A metric on an existing task → drop a file in tasks/<task>/metrics/, decorate it with @register_metric, import it from that dir’s __init__.py, and add its name to the task’s metrics tuple. (tasks/README.md → “Adding a metric”.)
• A new task → tasks/README.md → “Writing a new task”: contract → metrics → split preparers → register_task.
• A new dataset → datasets/README.md: register identity + artifact URIs only; per-task wiring is the split preparer’s job, not the dataset’s.
• A new curation recipe → recipes/README.md. One (publication, dataset) per recipe; ship recipe.py (plus explore.py — strongly recommended); confirm the upstream license against recipes/LICENSING.md before writing it.

One subdirectory per task. A task is a Contract plus a metrics panel, with per-dataset split preparers that bridge from a registered Dataset to the consumer-facing (train, test, solution) paths.

What lives here

Everything a task needs to score predictions: its data contract, its metric panel, and the per-dataset adapters that get truth onto disk. Domain logic that is specific to one task lives in that task’s subdirectory; code shared across tasks moves out to transforms/ or core/.

Shared scaffolding (Task and SplitPreparer value types, register_task / register_split_preparer) lives in base.py. Reusable Contract bases that several tasks build on sit at the tasks/ root behind a leading underscore (so they stay out of the per-task layout below): _paired_scalar.py (ScalarPairedContract) and _variant_effect.py (VariantEffectContract).

Writing a new task

A task is four things: a name, a Contract, a list of metric names, and a SplitPreparer for each dataset it can run against. Build them in this order — each step depends on the one above it.

The expected layout:

tasks/<task>/
├── __init__.py           registers the Task; imports submodules below
├── contract.py           the Contract dataclass + any shared projection helper
├── metrics/
│   ├── __init__.py       imports each metric module so decorators fire
│   ├── <metric>.py       one file per metric (or closely-related pair)
│   └── utils.py          (optional) task-local numerical helpers
├── split_preparers/
│   ├── __init__.py       imports each preparer module so registrations fire
│   └── <dataset>.py      one file per dataset this task can score against
└── aggregation.py        (optional) panel-level overall score

1. Define the data contract

Create <task>/contract.py with a @dataclass(frozen=True) that implements the Contract protocol. It validates (predictions, solution) before any metric runs: shapes, required metadata columns, feature alignment, any task-specific pairing rules. If multiple metrics share a projection (e.g. joining predictions and solution on a key), put that helper here so the metrics can call it directly.

2. Write the metrics

One file per metric (or closely-related pair) under <task>/metrics/, each decorated with @register_metric(name=..., description=..., references=...) (description is required — doc autogen depends on it). Two metric shapes are supported, and the registry handles both transparently:

• A Metric subclass — the dominant pattern. A torchmetrics-style update/compute class from metrics/base.py (Metric). State holds raw aligned inputs, update appends a shard, compute does all the math globally, so the same code is correct single-process and under distributed evaluation. See metrics/README.md for the dual-mode recipe.
• A plain function (pred, solution) -> MetricResult — fine for a simple one-shot scorer with no accumulation.

Shared numerical helpers live one layer out: kNN in transforms/, and device/stats helpers in metrics/. Note summarize exists in both — the numpy transforms.summarize and the torch metrics.summarize (a matching port); inside a torch Metric.compute use from prima_bench.metrics import summarize, as the class metrics do. Genuinely task-local primitives go in <task>/metrics/utils.py. The decorators only fire when each metric module is imported, so re-export them from <task>/metrics/__init__.py.

3. Wire each dataset

One module per dataset this task can score against, at <task>/split_preparers/<dataset>.py:

• a prepare function that fetches artifacts via dataset.fetch_artifact(...), derives whatever splits the bucket doesn’t ship (e.g. NaN-censored test from solution), and returns the consumer-facing dict.
  • Conventional keys: train, test, solution. For unsupervised tasks set test = train.
  • If the recipe already ships a censored test + a solution parquet (recipe-side censoring, e.g. sequence_classification), prepare is a thin mapper — no eval-time derivation. A dataset may also ship a labelled val artifact; leave it out of the returned dict if the task doesn’t score it.
• per-split loaders (Callable[[Path], EvalData]). Every key in the preparer’s return dict must have a matching loader entry. Registration validates this.
• a register_split_preparer call at module top level.

4. Register the task

In <task>/__init__.py:

register_task(Task(
    name="<task>",
    contract=YourContract(...),
    metrics=("metric_a", "metric_b"),
    default_dataset="<name>@<version>",  # optional
    description="...",
))

…then import the metrics/ and split_preparers/ submodules so their side-effect registrations fire.

Adding a metric to an existing task

1. Drop a file under tasks/<task>/metrics/.
2. Decorate the function with @register_metric(name=..., description=..., ...).
3. Import it from tasks/<task>/metrics/__init__.py so the decorator fires.
4. Add the metric name to the task’s metrics tuple in tasks/<task>/__init__.py.

What does not live here

• Dataset descriptors — those are in datasets/. A task only refers to a dataset by its name@version key.

Curation recipes for public datasets persisted to gs://prima-mente-benchmarks/. Each recipe is a one-shot script that downloads from upstream, runs preprocessing, and uploads the artifact to the canonical benchmarks bucket.

This folder is not part of the installable prima_bench package. Recipes have their own dependency graph (GEARS, scanpy, boto3, etc.) and are run offline by curators, not at benchmark runtime.

prima-bench consumers use pb.fetch_data(...) to read the bucket; recipes write it.

Once a recipe has written bytes, two more steps make them scorable — both live in the installable package, not here: register the (name, version) → artifact URIs descriptor in src/prima_bench/datasets/<dataset>/__init__.py, then wire the per-task splits in src/prima_bench/tasks/<task>/split_preparers/<dataset>.py (see src/prima_bench/tasks/README.md).

Credentials

All bucket I/O goes through boto3 via the shared recipes/storage.py helper (upload_dir, download_dir, cat, ls, copy_tree) — there is no gcloud dependency. Credentials come from the environment or a project-local .env (loaded automatically; .env is gitignored). For the canonical gs://prima-mente-benchmarks/ bucket, supply GCS HMAC keys:

HMAC_BOTO_PRIMA_MENTE_BENCHMARK_KEY_ID=<gcs-hmac-access-id>
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_SECRET=<gcs-hmac-secret>
HMAC_BOTO_PRIMA_MENTE_BENCHMARK_REGION=auto
# These are HMAC keys, not literal AWS keys; prima-bench reads them directly and
# passes them to boto3 as explicit credentials (no AWS_* / ~/.aws fallback).
# gs:// defaults to the GCS S3-interop endpoint; override only for other stores:
# PRIMA_BENCH_S3_ENDPOINT_URL=https://<r2-or-minio-endpoint>

The HMAC identity must have read access to every upstream source bucket a recipe pulls from and write access to the destination bucket. See .env.example at the repo root for the full set of variables. MDS writes (build-dataset) use the same credentials — for a gs:// --out, set GCS_KEY / GCS_SECRET so mosaicml-streaming uses its S3-interop path.

Directory layout

One directory per publication, then one subdirectory per dataset. The publication directory name follows the bucket’s top-level identifier; the dataset subdirectory name matches the {dataset} segment in the bucket path. Each (publication, dataset) pair is one recipe.

recipes/
├── README.md                        # this file
├── pyproject.toml                   # one shared env; also packages `utils`
├── uv.lock
├── utils.py                         # shared helpers, installed editable so recipes `import` it
├── bed_to_mosaic.py                 # shared curation step: any BED/bedGraph -> sequence mosaic
└── {first_author}_{year}_{journal}/
    └── {dataset}/
        ├── README.md                # this dataset's provenance
        ├── recipe.py                # download + preprocess + upload all views/splits for this dataset
        └── explore.py               # scratch EDA / provenance for the choices recipe.py bakes in

recipe.py is required; explore.py is strongly recommended. recipe.py writes the bucket bytes; explore.py is the scratch EDA / provenance that justifies the choices recipe.py bakes in — keep it in the PR so a reviewer (and the next curator) can retrace them.

Layout variants

The two-level {publication}/{dataset}/ form above is the default, but two sanctioned variants exist in the tree:

• Shared publication-level recipe. One recipe.py + explore.py at the publication root can cover several dataset subdirs that are README-only (e.g. ashburner_2000_nature_genetics/, lambert_2018_cell/, thomas_2022_protein_science/). Use it when one script naturally emits several related datasets.
• Flattened single level. A publication shipping exactly one dataset may put README.md + recipe.py + explore.py directly under the publication dir with no {dataset}/ (e.g. chang_2026_biorxiv/, johansen_2025_biorxiv/, linder_2025_biorxiv/). ⚠️ The license-check gate does not see these — it matches only two-level recipes/*/*/README.md (scripts/check_dataset_licenses.py), so a flattened README is never license-checked automatically. Confirm the license by hand and record it in the README as usual.

Naming rules for the publication directory follow the existing company-wide sequencing data conventions:

• {first_author} — surname of the first author or consortium name. Lowercase, no underscores.
• {year} — 4-digit publication year.
• {journal} — journal or preprint source. Lowercase. Internal underscores are allowed for multi-word journals (e.g. nature_genetics).
• Single underscore between the three components.
• For datasets without a publication, follow the conventions doc’s edge case: {source}_{dataset_name} (e.g. 10x_genomics_human_brain_3k).
• Benchmarking publications may republish existing datasets with light processing updates. The top-level directory must correspond to the publication from which the data was downloaded — often the derivative publication rather than the original.

Examples: hao_2024_nature_methods/norman_2019_science (Norman 2019 redistributed via scFoundation’s Figshare bundle — the publication directory tracks the scFoundation paper since that’s what we download from, while {dataset} keeps the original publication’s {first_author}_{year}_{journal} slug to identify the upstream source), madan_2022_nature_genetics/{dataset}, 10x_genomics_human_brain_3k/{dataset}.

The bucket’s deeper structure under {dataset} (v{version}/{view}/{split}/) is not mirrored in the recipes tree. There is a recipe.py per dataset that produces every version, view, and split for that dataset.

Bucket layout

Recipes write to:

gs://prima-mente-benchmarks/
  {first_author}_{year}_{journal}/
    output/                           # canonical contents read by prima-bench
      {dataset}/                      # a publication may present many datasets
        v{version}/                   # v0.0.1 — semantic version of the dataset's curated content
          manifest.json               # ties this version's bytes to the recipe execution that produced them
          {view}/                     # counts_view | sequence_view | ... (controlled vocab)
            {split}/                  # named slice of the dataset (e.g. train | val | test | fold0_train)
              <artifact>              # filename depends on view (h5ad, fasta, parquet, ...)
          solution/                   # held-out truth for test (if recipe-censored); view-independent
            labels.parquet            # primary_key -> target; loaded only internally by pb.Evaluation
    processed_author/                 # optional; upstream-author-provided artifacts

The output/ subtree is the canonical content prima-bench reads. Other top-level subfolders (e.g. processed_author/) are optional and follow the same publication-rooted naming when present.

Recipe-side censoring (solution privacy). A recipe may ship the held-out test answers censored: write test under {view}/test/ with the target set to null/NaN, and write the truth once to v{version}/solution/labels.parquet — a parquet keyed by a row primary key ({primary_key, target}) that maps 1:1 onto the test rows. solution sits at the version root (it is view-independent) and is never returned by pb.fetch_data / pb.fetch_test; pb.Evaluation loads it internally and the task Contract pairs predictions to it on the primary key. A labelled val split (same format as train) is optional — a dataset may ship one as an artifact without task-wiring it, or omit it entirely (don’t invent a validation split the upstream benchmark doesn’t define). The NT benchmark (recipes/niki_2025_biorxiv/nt_benchmark/) is the reference implementation of recipe-side censoring; it ships train/test/solution and no val.

Shared curation step: bed_to_mosaic.py

Turns any BED/bedGraph whose first three columns are chrom, start, end (0-based half-open) into a sequence mosaic. A dataset’s recipe.py imports it rather than re-implementing the tiling and the record contract:

from bed_to_mosaic import bed_to_mosaic

bed_to_mosaic(
    bed_uri="gs://.../cpg_islands.hg38.bed",
    bucket="gs://prima-mente-benchmarks/{publication}",
    dataset_slug="cpg_islands",
    extra_columns={"name": 3},   # optional: mosaic column <- 0-based BED field
)

Each region is tiled into chunk_size (default 1000 bp) chunks; the final chunk keeps the remainder, so every base of the region is covered. min_chunk (default 30 bp) then drops any chunk shorter than the threshold — an inclusive floor, so a 30 bp chunk survives and a 29 bp chunk does not. It applies to whole regions as well as trailing remainders, so a region shorter than min_chunk yields no rows. Dropped bases are counted and logged, never silently discarded.

3500 -> 1000, 1000, 1000, 500
3030 -> 1000, 1000, 1000, 30
3029 -> 1000, 1000, 1000        (29 bp tail dropped)
3000 -> 1000, 1000, 1000
  20 -> <nothing>               (whole region below min_chunk)

Output is the mosaic alone — sequence_view/{partition}/dna/ MDS shards plus manifest.json, and no solution sibling, since a plain BED carries no held-out value. One row per surviving chunk, each an alexandria GenomeSequencingRecord. read1_sequence is the chunk exactly (ragged, at most chunk_size bases), not a fixed-length window centred on the region.

Reference sequence is read in place, never downloaded

The reference FASTA is not fetched to disk. GENOME_FASTA_URIS points at the pre-indexed copy in object storage (gs://cfdx-research-data/reference-genomes/hg38/hg38_ref.fa), whose sibling .fai records where each contig’s bases begin and how they are line-wrapped. RemoteFasta uses that to convert a base interval into a byte interval and issues one ranged read per region — so the CpG-island smoke pulls ~9 kB instead of 3.27 GB, with no faidx build and no local cache. That file’s 455 contig names and lengths are identical to UCSC’s hg38.chrom.sizes, so BED coordinates mean exactly what they mean upstream.

Override the source with --genome-uri (a sibling .fai must exist). Programmatic callers can pass build_mosaic_rows(..., fasta=…) any chrom -> sliceable contig mapping — a pyfaidx.Fasta on local disk, a stub contig in a test, or a RemoteFasta with a different byte-fetching leg (reader / range_reader) when the ambient credentials aren’t storage’s HMAC keypair.

One mosaic row

Row 0 of the CpG-island smoke (cpg_islands.hg38.bed, 10 sampled regions -> 13 chunks), read straight back off the published shard:

from mds import read_mds

row = read_mds("output/cpg_islands/v0.0.1/sequence_view/smoke/dna")[0]
for column, value in row.items():
    print(f"{column:<14}  {value!r}")

assay           'reference_genome'
chromosome      'chr17'
end_position    50130237
modality        'genome_sequencing'
name            'CpG_island'
read1_cigar     '938M'
read1_sequence  <938 bp, printed below>
read1_strand    '+'
read_id         'chr17:50129299-50130237'
record_type     'genome_sequencing'
sample_id       'cpg_islands_hg38'
source          'bulk'
start_position  50129299
technology      'DNA-seq'

read1_sequence in full — 938 bases, exactly the reference at chr17:50129299-50130237:

CGGGCGTCGGGCGGGAGGGAGGGGATTAGGGCCATAAATCCTATCTCCGCCCCCTCCCCTGACAGCCCGGCACCCC
AGCCCCGTGCGGTCCCCCTAGGGATAGGGGAAGAAAGGCCCTTGTCGCTTAGCCCTGCCCAACCCCACTCAGGGTC
CCACCGCGGGCCCGAGTGGCCCTAGCCCCAGTTCCCGGTCCACCCCAGGGCGCCCGGCCCCTCACCTCCGGCATCG
GTCATCTTCAGCGTCTGGCGCCCCGGGAACCCCTCAGGCCCATGGTGCTCGAGGCGGGGGCCGGGCAGGAGAGGGG
TGGGGGGACCGTCACCCCGGCCGCGGGGGGCTCCGGGCGGGCTCGAGCCGCTCCATCCCGGGGGTGGGGGTGAGGT
CGGCGGGAAGGGTCCCAGAGCCGGGGGAGATGGAGGGCGGGGGATGTGGGGAGAGGGAACAGGTTCGCTGGAAGCA
GGAGGCGCGGGCGGGGCTGGACCAGGAGACCTGACGGGAGGATGCTCCGTGTGTGTGTGCGTGTGCGTGTGCGTGT
GTGTGTGTGTGACAGAGAGAGAGAGAGAGAGAAGAGCGAAGAGCGAGCGCCACCTCCCTCCCCGCCCCTGCCGCCC
GCCAGGCCTGGCCGGACGCGGGGCCCCCGCCTGGCAGGGCCGATTCGCGCGGCGTCGCGTCGGCAGGGGTCCGCGG
GGGTCTCCGGGGAAGGGGCGTCGGGGTGGGGGGCGCGTGGCGGAGAAGCCGGGCGCCCAGGAGACAAAGAGCGGGA
GGGAGCGAGGGGGCGGGGGCGGGGGAGGGCGGCGGGGAGGAGGCGGCTGGGGCCGGGGAGCGGAGTTGCAGCTACT
TCTCTGCCCGCGGGAGACGCGGCGCACCGGAGTCTGCGCGGGTGGGAGCGGCTCCGGGGCGGTCCCGCCACGGGAC
CCACGGGGCTTCCGCCTCCTCTGCCG

The 14 columns are the 13 GenomeSequencingRecord contract fields plus the one caller-supplied extra, name:

This region is 938 bp — under chunk_size — so it stayed a single ragged row. A longer region becomes several rows sharing everything but read_id / start_position / end_position / read1_sequence / read1_cigar; the extras (name here) are copied onto every chunk of their region.

cell_id, barcode and cell_type are absent rather than null: the record is dumped with exclude_none, and the first two are forbidden outright under source=bulk.

Temporal-discovery evals (T0→T1 “new biology”)

A discovery recipe (discovery.py + discovery_explore.py, either at the publication root as in ashburner_2000_nature_genetics/discovery.py, or inside the base dataset dir as in gargano_2024_nucleic_acids_research/hpo_phenotypes/discovery.py and schriml_2025_nucleic_acids_research/disease_associations/discovery.py) turns two dated releases of the same annotation source into a “predict the newly-curated annotations” eval, reusing the gene_function_classification task, its annotation_view split preparer, and its metrics unchanged. It reads two release-tagged endpoints — an old T0 and the pinned current T1 — staged out-of-repo under

gs://prima-mente-artifacts/gene_annotations/<source>/{raw,processed}/<release-tag>/<name>.parquet

(the untagged current files feed the interpolation evals and are left in place). Both endpoints share the same GENCODE v49 crosswalk, so the T0→T1 diff reflects genuinely new biology rather than gene-ID drift. The shared builder recipes/temporal_eval.py:build_discovery_tables encodes the diff: train = T0 positives; test/solution = the T0-negative frontier with target=1 where an edge flipped positive at T1; term IDs are normalized through the T1 ontology’s alt_id/replaced_by map (recipes/obo.py:parse_obo_alt_ids) so a term rename is never miscounted as a discovery; IC is computed over the T1 corpus; solution also carries a paralog_had_term_at_t0 stratification flag. The pinned T0/T1 tags per source live in each recipe and in the dataset READMEs’ ## Source / provenance. See TECH-4884 (staging) and TECH-4883 (evals).

Provenance

Dataset identity is the (dataset, version) tuple. Versioned bucket paths are immutable by convention — any change to output bytes requires a new v{version}/ — so the path itself is the integrity guarantee, backed by GCS object versioning and bucket-level access controls.

Per-curation audit metadata (who ran the recipe, when, with what reviewed PR landed the recipe, and resolved tool versions) lives in the per-dataset README’s Curator history table. Each recipe execution that produces uploaded bytes appends one row.

Each v{version}/ directory also carries a manifest.json that ties the uploaded bytes to the recipe execution that produced them:

{
  "recipe_pr_url": "https://github.com/cfdx-ai/prima-bench/pull/18",
  "curated_at": "2026-05-02T01:20:49Z",
  "curator": "jg@cfdx.io"
}

The manifest is deliberately minimal — its only job is to point a reviewer from a versioned bucket path back to the exact reviewed PR (and via that PR, the merged commit) and the curator that produced the bytes. Recipes derive recipe_pr_url from the squash-merged commit at HEAD (subject ends in (#N)); curators are expected to run --upload against a clean post-merge main. Richer audit detail lives in the per-dataset README’s Curator history.

Per-dataset README template

Each dataset directory carries a README.md documenting:

• Source — citation, DOI, upstream download URL.

• License (required) — a ## License section declaring the upstream license as an SPDX token (or an approved fallback such as public-domain / US-Gov-PD when no SPDX identifier exists), e.g.

  ## License
  
  - **License (SPDX):** `CC0-1.0`
  - **Commercial use:** permitted
  - **Reference:** https://example.org/path/to/upstream/license

  The license must be on the Acceptable list in LICENSING.md — prima-bench data must permit commercial use, redistribution, and derivatives. CI fails any PR that adds a dataset without one (see scripts/check_dataset_licenses.py). If the upstream license isn’t clearly acceptable, ask before writing the recipe.

• What this recipe produces — list of (dataset, view, version, split) tuples written to the bucket.

• Preprocessing summary — what the recipe does between download and upload, with rationale for any non-obvious choices.

• Version rationale — why the current version is what it is; what would force a bump.

• Curator history — append a row per recipe execution that produced uploaded bytes: date, curator, recipe PR (URL), version produced, resolved tool versions, notes.

See niki_2025_biorxiv/nt_benchmark/README.md for a worked example — a conformant dataset README with a ## License section and both recipe.py + explore.py.