Add a recipe end-to-end

A recipe produces the immutable benchmark bytes for one (publication, dataset) pair. Recipes live in the separate recipes/ package (their own dependency graph — GEARS, scanpy, boto3) and are run offline by curators. Consumers only ever read the bucket through pb.fetch_data. This walkthrough goes from an unlicensed source dataset to a registered eval.

1. Confirm the license first

Every dataset curated into prima-bench must be released under a license that permits commercial use, redistribution, and derivatives — we run commercial models against these benchmarks. The acceptable / not-acceptable classification in recipes/LICENSING.md is the single source of truth.

If the license is missing, non-commercial, no-derivatives, controlled-access, or unclear, stop and raise it with the licensing maintainers (Francisco M. Martín-Zamora and Pouya Niki) before writing a line — re-curating around a bad license later is expensive. Record the SPDX token (or an approved fallback like public-domain / US-Gov-PD) in the dataset README’s ## License section.

2. Scaffold the recipe directory

One directory per publication, one subdirectory per dataset. Naming follows the company sequencing conventions: lowercase first_author, 4-digit year, lowercase journal. Every new recipe PR ships both recipe.py and explore.py.

recipes/{first_author}_{year}_{journal}/{dataset}/
├── README.md     provenance: source/DOI/license, what it produces, version rationale, curator history
├── recipe.py     download + preprocess + upload all views/splits for this dataset
└── explore.py    scratch EDA / provenance for the choices recipe.py bakes in

3. Write recipe.py — deliberately dumb

Recipes describe a publication, not an API. They write the censored test plus the per-item solution; they do not split train/test, know about regimes (supervised / zero-shot), or invent views — that is the SplitPreparer’s job at eval time, over the same bytes. Canonical content lives under output/:

gs://prima-mente-benchmarks/{pub}/output/{dataset}/v{version}/{view}/{split}/<artifact>
# plus a manifest.json per version tying the bytes to the reviewed PR/commit + curator

Versioned paths are immutable — any byte change requires a new v{version}/. Recipe helpers raise on precondition failures (they never sys.exit); only the main() script boundary owns the exit code. Diagnostics go through logging; reserve print (stdout) for the tool’s actual data product.

4. Register the dataset descriptor

Under datasets/<dataset>/, register identity plus artifact URIs only — no train/test knowledge. That is a task-level concept.

from prima_bench.registry import register_dataset
from prima_bench.core import Dataset

register_dataset(Dataset(
  name="{publication}/{slug}",
  version="0.0.1",
  artifacts={
      "test": "gs://prima-mente-benchmarks/.../v0.0.1/default/test/data.parquet",
      "solution": "gs://prima-mente-benchmarks/.../v0.0.1/default/solution/labels.parquet",
  },
))

The same dataset is wired to multiple tasks by registering more preparers, with no change to the descriptor.

5. Wire a SplitPreparer onto a task

The SplitPreparer is the per-(task, dataset) adapter: it fetches artifacts via dataset.fetch_artifact(...) and returns the consumer-facing split dict with a per-split loader for every key (registration validates this). It may censor a golden solution into a NaN-censored test, or — when the recipe already ships an already-censored test — just map the bytes through.

from prima_bench.registry import register_split_preparer

def prepare(dataset):
  test_path = dataset.fetch_artifact("test")
  solution_path = dataset.fetch_artifact("solution")
  return {
      "train": test_path,          # passthrough where there is no train fold
      "test": test_path,
      "solution": solution_path,
      # loaders: Callable[[Path], EvalData] for every key above
  }

register_split_preparer(task="<task>", dataset="{publication}/{slug}@0.0.1", prepare=prepare)

Registration is by import side-effect: re-export the new modules from the relevant __init__.py so the register_* decorators fire when import prima_bench runs. Adding a metric instead? Drop a file under tasks/<task>/metrics/, give it a non-empty description (doc autogen depends on it), import it, and add its name to the task’s metrics tuple.

6. Verify, then upload from clean main

CI gates a PR on four checks — run them locally before pushing:

uv run ruff check src tests
uv run pyright
uv run pytest --cov=prima_bench
uv run --project recipes pytest recipes/tests/ -q   # import-smoke every recipe

Run recipe.py --upload against a clean post-merge main, then append one row to the dataset README’s Curator history table (who / when / which PR / tool versions). The versioned path is the integrity guarantee.

See the Contributing page for the full task and recipe READMEs, and Run your first eval for the consumer side of the same bytes.