Data pipeline

The pipeline/ directory holds the scripts that fetch, build and check the data the apps ship, together with the independent reference values the unit tests compare the engines against. This page explains what each part produces, where the output lands, how snapshot expiry is enforced, and how to add or refresh a data set.

Principles

Nothing is fetched at build time. Pipeline scripts are run by hand or by a scheduled workflow, and their outputs are committed. npm run build reads only files already in the repository, so a build is the same on every machine.
Oracle before engine. Reference values for the scientific engines come from named external tools (Biopython, Python RDKit, R nls and drc, SciPy, primer3-py, exact rational arithmetic), never from the engine under test. Each fixture records the script, its SHA-256, the oracle and its version.
Build-time tools never ship. Python, R, Blender and GPL oracles run only here. Their numbers enter the repository as fixture records; their code is never a dependency or ported into src/. scripts/licences.mjs scans src/ and content/ to enforce this (see Build, test and CI).
TypeScript scripts reuse the app's own modules. .ts and .mts scripts run directly on Node 22.18 or later with type stripping, importing the page's modules by relative path with their extension, so the shipped data cannot drift from the code that reads it.
Python runs in a virtual environment outside the tracked tree, with the pinned requirements.txt beside each script. The READMEs use /tmp/... environments or the gitignored .venv/. The Assay oracles, R and Python, also run in the oracle image (pipeline/oracle-env/), which pins R, the CRAN snapshot and the Python build so a fixture can name its environment by digest.

What each subdirectory produces

Subdirectory	Produces	Lands in
`drug-pipeline-explorer/`	`snapshot.ts`: example snapshots of ClinicalTrials.gov and openFDA for the seven preset searches. `precedent/`: five FDA precedent snapshots and test fixtures. `verify_stems.py`: a check, no output	`public/data/drug-pipeline-explorer/snapshots/`, `public/data/drug-pipeline-explorer/precedent/`
`protein-explorer/`	`verify.ts` live-service checks; independent alignment, substitution and retained-image reference scripts; `evidence/publish.py`	Oracle fixtures under `pipeline/protein-explorer/fixtures/`; retained-image references under `public/verification/protein-explorer/0.3.0/`
`molecule-workbench/`	ChEMBL fetch and library build, the precomputed opening example, OpenChemLib tables, RDKit reference values and PAINS data, and a full-library comparison against Python RDKit	`public/data/molecule-workbench/`, `public/tools/molecule-workbench/qed-notice.txt`, `src/components/tools/molecule-workbench/`
`inside-a-cell/`	`verify_sources.py` (`verified.json`), `build_structures.py` (trimmed structures, scene data, `facts.json`), `render.py` (Blender stills)	`public/data/inside-a-cell/`, plus `verified.json` and `facts.json` in the pipeline folder
`education/`	`molecular.py`: scenes, posters and guide videos for the Education workspaces; `verify.py` checks them; `structure-sources/` holds the coordinate records and preparation scripts	`public/data/education/`
`lab-calculators/`	`curate-reagents.py` fills PubChem identifiers in `reagents.csv`; `build-reagents.mts` writes the reagent table; `reference.py` writes oracle fixtures and publishes them	`public/data/lab-calculators/reagents.<hash>.json` with `src/components/tools/lab-calculators/reagents-manifest.ts`; fixtures under `src/lib/*/fixtures/`; `public/verification/lab-calculators/<version>/`
`assay-analyser/`	`generate_curves.py` and `reference.R` (fitting oracle), `relative-irls-reference.py`, `make-plate-fixtures.mts` (synthetic plate exports), `make-prism-example.mts` (the guided walk's real plate readings)	`pipeline/assay-analyser/fixtures/` and `plates/`, read in place by the tests
`sequence-toolkit/`	`reference.py` oracle fixtures from saved inputs; `build-codes.mts` genetic code tables from NCBI's `gc.prt`; `publish.py`	`pipeline/sequence-toolkit/fixtures/`; `src/lib/seq/codes-data.ts`; `public/verification/sequence-toolkit/<version>/`
`plasmid-map/`	`fetch-records.mts` saves NCBI and UniProt records, including the candidates in `CANDIDATE_RECORDS` whatever their review verdict; `reference.py` and the other Python oracles write fixtures (`specificity` writes the controls, the short-feature cases and the all-against-all cross-check; `display` writes the per-locus display cases from it); `build-enzymes.mts`, `build-features.mts` (with `features.manifest.csv`, `library-gaps.csv` and `library-pairs.csv`), `build-primers.mts` and `build-examples.mts` write versioned data; `reviews/` holds a review record per source record; `publish.py`; `verify.ts`	`pipeline/plasmid-map/records/` and `fixtures/`; `public/data/plasmid-map/` with manifests in `src/lib/seq/`; `public/verification/plasmid-map/<version>/`
`bioreactor-simulator/`	`reference.py` (SciPy oracle), `export-presets.mts` (oracle inputs), `write-golden.mts` (golden runs), `record-presets.mts` (preset version hashes)	`pipeline/bioreactor-simulator/fixtures/`; `src/lib/hosts/golden/`; `src/lib/hosts/presets/manifest.ts`
`oracle-env/`	The oracle image (`Dockerfile`): R 4.4.3 on a dated CRAN snapshot with the R oracles' packages, Python 3.11.15 in five virtual environments (the three Assay requirements files, Sequence's and Calc's), MAFFT 7.526 and EMBOSS 6.6.0; `run-oracle.sh` picks the interpreter; `record-environment.sh` records what resolved. See Oracles and verification	the image in the repository's GitHub container registry; `R-packages.csv` and `python-packages.txt` here
`patent-exclusivity/`	`capture.py`: the daily capture of the FDA Orange Book data files, the Purple Book download and the Paragraph IV list (Stage I-0, D-0077), run by `capture-patent.yml`; its `--watchdog` mode is run by `scheduled.yml`	The orphan branch `data/exclusivity` (`current/`, `changes/`, `log.ndjson`); `build-store.py` writes the app's store (`public/data/patent-exclusivity/store.<hash>.json`, `changes.<hash>.json`) and the manifest `src/lib/exclusivity/store-manifest.ts`, captured store from the pinned 17 October archive
`walks/j3/`	Candidate dose-response data sets and the sources behind the J3 walk's step text; nothing here ships. The landing page's RDKit depiction of erlotinib hydrochloride (`landing-depiction.py`, `erlotinib-hydrochloride-2d.svg`, 20 October 2026) was removed with its reader when the guided example was simplified at the CEO's request the same day; the CSO's ruling allowed removing the structure	stays in the folder
`cellosaurus/`	`fetch-release.py` downloads a Cellosaurus release file and records its size and hash (`releases/<version>.json`); `build-snapshot.py` writes the served cell line snapshot and its TypeScript manifest; `oracle.py` writes the resolution fixture (E10, D-0099)	`public/data/cell-lines/cellosaurus-<n>/` and `src/lib/cells/snapshot-manifest.ts`; `pipeline/cellosaurus/fixtures/`

Most subdirectories have a README with the exact commands, sources, retrieval dates and licences; read it before rerunning anything. public/data/bioreactor-simulator/preset-runs.json is not a pipeline output: the prebuild step scripts/prebuild/bioreactor-simulator-preset-runs.mjs writes it from the model on every build.

Search catalogues

The offline search indexes are built by two scripts under scripts/ rather than pipeline/. scripts/build-search-catalogs.mjs takes downloaded source files as arguments (the Disease Ontology OBO file, a UniProt TSV and ChEMBL JSON) and scripts/build-search-details.mjs computes formulas and masses with RDKit from the shipped structures. Both write content-hashed files to public/data/search/<name>.<hash>.json through scripts/lib/search-catalog-files.mjs, which also generates src/lib/search/catalog-urls.ts. Source URLs, licences and generators are recorded in public/data/search/sources.json. Hashed names get an immutable cache rule (SUITE_CACHE_RULES in content/redirects.ts).

Snapshot scripts and verify scripts

The pipeline has two kinds of script:

Snapshot and build scripts download or compute data and write files that the app ships or the tests read. They change the repository, so their output is reviewed as a diff and committed.
Verify scripts re-check live public services against what the app expects: that identifiers are still active, API shapes still parse, saved records are unchanged and the app's request counts stay within published limits. They write nothing to the repository and exit non-zero on a failure.

scheduled.yml runs the verify scripts weekly (Mondays 05:17 UTC), one job per check so one provider's outage never hides another's result:

Check	Command	Needs
Protein (UniProt, AlphaFold DB, PDBe, Open Targets)	`node pipeline/protein-explorer/verify.ts`	Node
Cell sources (RCSB PDB, Europe PMC, UniProt)	`python pipeline/inside-a-cell/verify_sources.py`	Python 3.12, standard library
FDA precedent (openFDA fields, fda.gov table headers, request count)	`node pipeline/drug-pipeline-explorer/precedent/verify.ts`	Node
INN stems (WHO stem book)	`python pipeline/drug-pipeline-explorer/verify_stems.py`	`pypdf`
Plasmid records (NCBI E-utilities, the shipped records and primers)	`node pipeline/plasmid-map/verify.ts`	Node
Molecule library (Python RDKit against RDKit.js)	`rdkit_reference.py library` then `verify-library.mts`	`rdkit==2026.3.6`, the release the app ships

pipeline/education/verify.py is not in the matrix; it checks the Education media files and is run by hand after a rebuild. Scheduled workflows run from the default branch only.

The Pipeline example snapshots

The Pipeline app calls ClinicalTrials.gov and openFDA live from the browser. When ClinicalTrials.gov is down or slow it falls back to a dated snapshot of the same preset search. pipeline/drug-pipeline-explorer/snapshot.ts writes one file per preset to public/data/drug-pipeline-explorer/snapshots/<id>.json, plus index.json listing the files with their sizes. Each snapshot records retrievedAt, ClinicalTrials.gov's own data timestamp and a terms block with each source's attribution, terms and this site's modifications. The script stops rather than save a partial snapshot if any page or lookup fails.

ClinicalTrials.gov asks that republished data be kept current, so the app hides a snapshot more than 60 days old (SNAPSHOT_MAX_AGE_DAYS and snapshotIsStale in src/components/tools/drug-pipeline-explorer/snapshot.ts). Once the snapshots expire, the landing page's examples and the Pipeline welcome show nothing offline. The snapshots keep their file names when regenerated, so they are served under the short-lived /data cache rule (ten minutes fresh, a day of background revalidation) rather than an immutable one.

FDA precedent snapshots

pipeline/drug-pipeline-explorer/precedent/ writes five snapshots of FDA's own files (cder-nme.json, breakthrough-approvals.json, project-confirm.json, surrogate-endpoints.json, crl.json). A precedent snapshot becomes stale after 90 days; the view keeps showing it and says so, because FDA's files carry no currency term. The precedent README documents the five commands, the download cache and the draft test fixtures built by the scripts in precedent/fixtures/.

Cellosaurus: the cell line snapshot

E10 of the enterprise-standard actions (D-0099; the CTO's ruling 4(c), the CSO's ruling F5, both 17 October 2026). pipeline/cellosaurus/ holds three standard-library Python scripts; the README there has the commands, the schema and the counts.

fetch-release.py --out DIR downloads https://ftp.expasy.org/databases/cellosaurus/cellosaurus.txt with resume and writes pipeline/cellosaurus/releases/<version>.json (URL, bytes, SHA-256, retrievedAt, the version and date from the file's header, the header's licence sentence). The raw file is never committed. Release 56.0 (25 June 2026): 122,297,334 bytes.
build-snapshot.py --release-file FILE --replace refuses a file that differs from its fetch record, keeps the CSO's fields of every record (primary name, synonyms, accession, species, disease, sex, age, "Problematic cell line" comments verbatim, the STR profile with its sources, and the category) and writes public/data/cell-lines/cellosaurus-<n>/: snapshot.json (release, date, retrievedAt, licence, attribution, the raw file's hash, the schema, counts, the name shards' first keys and the hashes of every other file), names/NNN.json (normalised name to accession, key-ordered runs of about 40 KB), records/NNN.json (384 shards by FNV-1a 32 of the accession), diseases/N.json, species/N.json, secondary.json, and names/hashes-N.json and records/hashes-N.json. It also writes src/lib/cells/snapshot-manifest.ts (CELL_LINE_SNAPSHOT: release, date, retrievedAt, directory, shard counts, the hash of snapshot.json), which the registry's dataset record and the engine read. --replace removes any other release's directory, so only the current release stays in the repository; saved records keep their quoted fields by value, so nothing saved depends on an old release's files (the CTO's ruling).
oracle.py --release-file FILE writes the write-once fixture (see Oracles and verification).

Limits, by src/lib/cells/snapshot.test.ts: every file at most 64 KB raw and 16 KB gzipped, the whole snapshot at most 8 MB gzipped (read as transfer bytes; every record is about 22 MB raw, recorded for the CTO in docs/progress/enterprise-actions/cell-line.md) and at most 24 MB raw. Release 56.0: 558 files, 21,990,345 bytes raw, 6,339,900 bytes gzipped, largest shard 48,247 raw and 14,130 gzipped. The test also checks the hash chain from snapshot.json to every shard, that no other file is served, the provenance fields, that the registry's record names the same release, and that the served release has a resolution fixture. The files keep their names within a release, so they take the short-lived /data cache rule.

Staleness and freshness checks

Several checks watch the snapshots' age so a refresh lands before anything expires:

Check	Reads	Fails when	Where it runs
`scripts/snapshot-staleness.mjs`	each example snapshot's `retrievedAt`	a snapshot is within 14 days of its 60-day expiry (`SNAPSHOT_WARN_DAYS`), has expired, or has no readable date	weekly `staleness` job (fails); CI `check` job with `--warn` (warning annotation only)
`scripts/snapshot-freshness.mjs`	`retrievedAt` in `snapshots/index.json`	the snapshots are 60 or more days old; warns from 45 days	CI `check` job, before the build
`src/components/tools/drug-pipeline-explorer/precedent/snapshot-age.test.ts`	every example and precedent snapshot	an example snapshot was saved more than 46 days ago (60 less 14), or a precedent snapshot is within 14 days of its 90-day limit; also checks the precedent envelope	`npm test`, so CI and the snapshot workflow
`scripts/snapshot-staleness.mjs`, Cellosaurus family (`cellLineState`)	`retrievedAt` in `src/lib/cells/snapshot-manifest.ts`	the cell line snapshot is 180 days old or has no readable date (`CELL_LINE_FAIL_DAYS`); a warning annotation from 120 days (`CELL_LINE_WARN_DAYS`)	the same weekly `staleness` job and CI `--warn` run; the rule is pinned with fixed clocks by `scripts/snapshot-staleness.test.mjs`
`pipeline/cellosaurus/fetch-release.py --check-newer`	the first 4 KB of the current release file (version, "Last update") and the served `snapshot.json`	a Cellosaurus release newer than the served one has been out for 30 days or more (`NEWER_FAIL_DAYS`); a warning annotation before that	weekly `cellosaurus-release` job in `scheduled.yml` (needs the network)

snapshot-staleness.mjs is kept out of npm test on purpose, so a calendar date alone does not turn every pull request red. Note that snapshot-age.test.ts does compare against today, so npm test fails once the example snapshots pass 46 days. The monthly refresh is meant to land well before that. snapshot-staleness.mjs --json prints every snapshot's state.

The snapshot workflow

.github/workflows/snapshots.yml runs on the 1st of each month at 05:17 UTC and on demand:

npm ci.
NODE_USE_ENV_PROXY=1 node pipeline/drug-pipeline-explorer/snapshot.ts.
Set up Python 3.12 and install pipeline/drug-pipeline-explorer/precedent/requirements.txt.
Run fetch-cder-nme.py, fetch-breakthrough.py, fetch-surrogate.py, fetch-project-confirm.mts and snapshot-crl.mts.
Build the FDA precedent examples fixture for the new snapshots: a new write-once file, examples-<retrievedAt date>.json, by build_examples.py from FDA's spreadsheet in precedent/.cache/, after checking its SHA-256 against cder-nme.json. The step fails if that dated file already exists, because a second refresh on one day would replace a published fixture (the CTO's memo docs/board/memos/2026-10-17-cto-snapshots-examples-step.md).
node scripts/check-write-once.mjs --base against origin/main, since the pull request the workflow opens gets no CI of its own.
npm test, so the new snapshots must pass their tests (examples.test.ts compares them with the new fixture).
Open a pull request from the branch snapshots/refresh containing only public/data/drug-pipeline-explorer/snapshots, public/data/drug-pipeline-explorer/precedent and the new precedent/fixtures/examples-*.json.

Nothing merges itself. The reviewer checks the index's retrievedAt and per-condition counts, and the precedent row counts, dates and checksums, then merges and redeploys. The repository must allow GitHub Actions to create pull requests.

A second job, cellosaurus (E10), refreshes the cell line snapshot on the same schedule: it runs fetch-release.py, build-snapshot.py --replace and, when the release has no fixture yet, oracle.py; runs the cell line tests (src/lib/cells/*.test.ts, content/services.test.ts, scripts/snapshot-staleness.test.mjs) without letting a failure stop it, since a new release needs the methods entry's counts (content/apps/cell-lines.methods.ts) updated by hand from the new fixture; and opens a pull request from snapshots/cellosaurus with public/data/cell-lines, src/lib/cells/snapshot-manifest.ts, pipeline/cellosaurus/releases and pipeline/cellosaurus/fixtures, its body saying whether the tests passed. An unchanged release still refreshes retrievedAt, which resets the staleness clock.

The patent and exclusivity capture (Stage I-0)

.github/workflows/capture-patent.yml runs pipeline/patent-exclusivity/capture.py daily at 10:23 UTC and on demand (D-0077, 13 October 2026; docs/design/brief-patent.md §5.4). The FDA publishes the Orange Book's patent and exclusivity rows as a current file only, so the capture keeps the history that the Patent app's change log, watch and fixtures will read:

Check out the default branch, then fetch the orphan branch data/exclusivity into a worktree (shallow), or start it as an orphan on the first run.
Download each source (the Orange Book zip and its three tilde-delimited files; the Purple Book monthly CSV; the Paragraph IV list found by scanning its fda.gov page for the download link), normalise it to text with a stable row order, and keep current/<source>/<file> only when its content hash changes, with a .meta.json beside it (URL, hashes, bytes, Last-Modified, served file name, header, row count, capture time). A Paragraph IV list served as a PDF is kept as the file itself, on change only, for Stage I-1 to parse. A list served as an XLSX is turned to tab-separated text with each cell placed by its column reference (an XLSX leaves empty cells out), and its .meta.json records "textFrom": "xlsx-cells-by-reference-v1"; captures before 17 October 2026 read cells by order, so their text lost the columns after every empty cell (REL-229), and the next capture rewrites it.
Write changes/<date>.json with the rows added, removed or changed, keyed by application, product number, patent number and use code, or exclusivity code (the whole row when a key column is missing), only on change.
Append one heartbeat line per source to log.ndjson, commit whatever was captured with the Actions bot identity, and push to data/exclusivity; then fail the job if any source failed, so the failure notifies after the successful sources were kept.

The branch is written only by that workflow and never merged into main; fast.yml and ci.yml never run on it; shallow checkouts of main never fetch it, so CI's checkout time does not grow with the series (the CFO's condition). Standard-library Python only; no npm ci; estimate 1 to 2 runner-minutes a day and under 10 MB a year packed (to be measured after 30 days and recorded here). scheduled.yml's capture-watchdog job fails when a source's last successful capture is older than its allowance (Orange Book 3 days, Paragraph IV list 10, Purple Book 45) or the capture has never run. npm test runs the script's self-test on synthetic files (capture.test.ts). Every URL, file name, header and cadence in the script is to be checked on the first runs (fda.gov was unreachable when it was written); the README in the directory lists what to confirm. Kill terms: checkpoint 13 April 2027, kill date 13 October 2027 (D-0077).

The Patent store (Stage I-1)

pipeline/patent-exclusivity/build-served.mts builds the Patent app's served store (REL-197; the CTO's ruling of 14 October 2026, docs/board/memos/2026-10-14-cto-patent-store-split.md). It runs build-store.py (standard-library Python) into a working directory, which packs the branch's current/<source>/<file> text with each .meta.json and the last 90 days of changes/<date>.json (--data-dir DIR --commit SHA), or the synthetic fixture (--synthetic); --codes, --codes-as-of and --built-at pass through. The TypeScript engine then parses that text, joins it and traces every application at build time (src/lib/exclusivity/served.ts) and writes, each named by its content hash, meta.<hash>.json, index.<hash>.json, the shards sNN.<hash>.json (application number modulo 100) and changes.<hash>.json to public/data/patent-exclusivity/, with the manifest src/lib/exclusivity/store-manifest.ts (META_FILE, INDEX_FILE, CHANGES_FILE, SHARD_FILES, STORE_KIND, STORE_BUILT_AT). It refuses to write any file over its gzipped ceiling (the index 350 KB, a shard 50 KB) and prints every file's size; --served-out and --served-manifest write a trial build elsewhere. Measured on the capture of 14 October 2026 with all three sources (28,292 applications): the index 337 KB, shards median 33 KB and largest 44 KB, meta 0.9 KB, changes 115 KB gzipped. The 18 October candidate ships the pinned real capture with independent regulatory review still pending (SCI-124); this is not completion of the original F1–F7 acceptance set. Refreshing is a pull request with the new files and manifest, never a hand edit; no workflow runs it yet (a later ruling, with a Node step).

First real store, 18 October 2026

Source data/exclusivity commit 2d0b7be911499788d58924e6a16b145eeab5cb01; generator commit 97ae27ce; Node 22.23.3 and Python 3.9.6. The fixed --built-at 2026-10-18T00:00:00Z makes two independent builds byte-identical, including the manifest. Exact file bytes, SHA-256 and base/head/limit sizes are recorded in docs/progress/patent-store-2026-10-18.json.

The store has 28,292 applications and 103 served files. It retains 48,865 product rows, 22,438 patent rows, 2,421 exclusivity rows, 2,251 Purple Book rows and all 1,647 PIV rows (1,571 assigned and 76 withheld with reasons). The test reconstructs all five original normalised members byte for byte from the shipped files and verifies their SHA-256, then compares all opened application dates with the catalogue. No budget was widened: index 338.6 KiB of 350; largest shard 43.6 of 50; meta 4.5 of 8; changes 0.2 of 256.

The pinned log has 27 heartbeats over 14–17 October: nine per source. Orange Book has nine successes; Purple/PIV each have eight and an initial 404. Each source retains one raw-download hash throughout. This measures retrieval attempts, not a daily FDA publication cadence or a fortnight history. September Orange/Purple filenames and the PIV workbook's 9-28-2026 label remain filenames; no precise dataAsOf date is invented. The 14/17 October change archives are initial capture/normalisation repairs and are excluded with visible receipts; zero FDA change events are claimed. Thirty-day cadence and storage-growth measurement remain due under DAT-15.

Provenance and licence metadata

The 18 October capture-build preflight (verify_capture.py) pins all 15 archived files of 2d0b7be911499788d58924e6a16b145eeab5cb01 in fixtures/capture-2026-10-17-inputs.json. A capture build refuses another commit, missing/extra/tampered files or inconsistent metadata before replacing served files. It independently checks member digest, bytes, header, row count, source identity and shared download metadata; it never substitutes build time for missing capture time. The raw ZIP/XLSX binaries are absent, so their metadata hashes cannot be recomputed from this archive. Source metadata now retains each normalised member's digest separately from its container digest.

The reviewed 14 and 17 October change archives contain initial capture or normalisation corrections under unchanged raw download hashes. They remain intact on the data branch and are excluded from the FDA-listing-change collection, with machine-readable excludedCaptures reasons. This does not establish an FDA change event. The existing index/shard ceilings remain 350/50 KiB; the 18 October CTO ruling adds previously absent meta/changes ceilings of 8/256 KiB, enforced before writing and by the served-file tests. The engine retains unknown source and code-list dates, each source’s last successful heartbeat and all unresolved Paragraph IV rows. The Changes view exposes the exclusion reasons. The real store is committed separately from the engine corrections, with its reproducibility receipt below.

The project rules (checked at the pre-live pass under D-0014) are that every data source carries a licence id and the date its licence page was read, share-alike sources are flagged, and excluded sources are listed with the reason. During development a new source may carry the licence line "To be checked". In practice this means:

In the registry. Each app's entry lists its sources with a stable id in sourceIds, a sourceMeta record per id (licenceId, licenceReadOn, and where relevant shareAlike, displayOnly, noticeRequired or a decision), excludedSources with reasons, and flags.shareAlikeData. scripts/licences.mjs fails on a missing sourceMeta record, a source with no reuse licence and no decision, an endpoint without a terms page, or a licence page read more than twelve months earlier. See Registry and content.
In the data files. Shipped files carry their own provenance: the Pipeline snapshots' terms block; the precedent envelope (sourceUrl, licence, licenceReadOn, retrievedOn, and each source file's URL and SHA-256, asserted by snapshot-age.test.ts); _source, _version and _licence keys in ocl-resources.json; the ChEMBL release and ids inside the CC BY-SA library file, which is kept separate from everything else; retrieved and pdbRevision in the Cell outputs.
In the fixtures. Each oracle fixture records its generating script and hash, run date, oracle, oracle version and licence. Manifests (fixtures/manifest.json, pipeline/lab-calculators/fixture-hashes.json) hold the SHA-256 of each file, and the tests recompute them, so an edited fixture fails until its script is rerun.

Sources whose terms do not allow commercial reuse are left out or used only as references, and each README says why (for example, the WHO stem book is used to check stems but its definitions are not shipped). A GPL dependency, or a feature built on a source that cannot ship commercially, means a rewrite at the pre-live pass, so flag it when proposing one.

Write-once fixtures and versioned data

Protein's python3 pipeline/protein-explorer/evidence/publish.py <version> copies the frozen v1, v2 and v3 evidence JSON and their Python scripts directly from the pipeline. It refuses an existing local destination or one already on origin/main; it writes a manifest of source paths, byte lengths and SHA-256 digests. Version 0.3.0 is the first publication. These files check receipt identity and capture boundaries, not biological validity or an offline scene. Do not rerun a reference into an existing fixture or republish over a release.

Once on main, scientific fixtures and versioned data are immutable. scripts/check-write-once.mjs, run by the CI check-write-once job against the base commit, protects:

any manifest.json or fixture-hashes.json (except the Education teaching-media manifest), and the TypeScript manifests src/lib/hosts/presets/manifest.ts, src/lib/seq/enzymes-manifest.ts, src/lib/seq/features/library-manifest.ts and src/lib/seq/features/primers-manifest.ts;
.json and .csv files under any fixtures/ or golden/ folder;
.json files under public/verification/ and public/data/plasmid-map/;
versioned preset files src/lib/hosts/presets/<name>@<n>.ts.

A protected file may not change or disappear. A manifest entry may change only by documented lifecycle transitions (current to superseded with a dated reason, or a dated withdrawal naming its replacement), and no entry may still be marked draft when merged. The plasmid and bioreactor scripts follow the same rule: a version stays a draft, rewritten in place, until --freeze marks it published; after that a change needs --bump (plasmid) or a new preset version (bioreactor). publish.py scripts copy fixtures into public/verification/<slug>/<version>/ and never replace a file main already holds.

Refreshing an existing data set

Read the subdirectory's README for the commands, environment and expected outputs.
For the Pipeline snapshots, either run the snapshot workflow on demand or run NODE_USE_ENV_PROXY=1 node pipeline/drug-pipeline-explorer/snapshot.ts locally (the variable is only needed behind an HTTPS proxy). Rerun at least monthly, and whenever the classifier or aggregations change.
For other data, rerun the build script. Some scripts pin an upstream release and refuse another: rdkit_reference.py runs only with the RDKit release the app ships, and fetch-chembl.mts stops if ChEMBL serves a different release unless given --expect-release with --refresh.
Run npm test, review the diff (counts, dates, checksums), commit, and redeploy so the new files are served.

Adding a new data set or snapshot

Put the script in pipeline/<slug>/: TypeScript that imports the app's own modules by relative path, or Python with a pinned requirements.txt. Add a README with the source, retrieval date, licence and commands.
Write shipped data to public/data/<slug>/. Files whose name stays fixed get the short-lived /data cache rule. For files that should be cached permanently, use content-hashed names and give the app an immutable cache rule in its registry entry.
Record the source in the app's registry entry (sources, sourceIds, sourceMeta, any excludedSources) and declare any host the app contacts, since the Content-Security-Policy is generated from that list (see Security and network).
Put provenance inside the file: source URL, licence id, the date the licence page was read, retrieval date and, for downloaded files, a checksum.
Add a unit test that reads the committed file and checks its shape and provenance. If it is a fixture or versioned data, give it a manifest with SHA-256s so the write-once check covers it.
If the data has a currency rule, add an age check alongside the existing ones. If it should refresh automatically, add the commands and the output path to snapshots.yml. If live services should be re-checked, add a verify script and a matrix entry in scheduled.yml.

Environment variables used by the workflows are listed in Deployment and environment.

Public reference workflows (16 October 2026)

pipeline/seo-reference contains exact ClinicalTrials.gov captures plus metadata/hashes and native-parser build scripts for selected trial/FDA reference payloads. content/seo/data contains only selected route data. Existing verification fixtures are untouched. See Public reference workflows for the shared route, state, source and export contracts.