Why mirror something that is already online
The global biodiversity record is a public good that lives on somebody else's servers. It is one funding decision, one policy change or one outage away from being harder to reach than it is today. ARK starts from the assumption that the copy should exist in more than one place, and that one of those places should be a drive you can hold.
The full occurrence snapshot came down as 9,705 files, roughly 286 GB, carrying more than 3.7 billion records. It is not a scrape and not a subset: it is the snapshot, with a per artifact manifest recording byte count, sha256 and record count, and a verifier that re-checks disk against manifest, counts against the live registry, and opens a Parquet sample to catch a download that silently truncated. The mirror either passes that check or it is not trusted.
Everything downstream carries a snapshot date. A query answered against the archive states which snapshot it was computed against, so a number in a paper stays reproducible after the upstream record has moved on.
The bloat is the part that got removed
A raw snapshot is optimised for being distributed, not for being asked questions. ARK rewrites it: occurrences are repartitioned into a country and year tree, sorted inside each file by species key and latitude so Parquet row group statistics can skip most of a partition before reading a byte, and the five aggregate rollups that answer the common questions are precomputed into their own files.
A query planner sits on top and decides, per question, whether a precomputed rollup can answer it or whether the raw table is required. That is the whole reason the thing feels instant. Filtered semantic search over the vector corpus measured 32 ms cold and 16 ms warm against the real collection; an earlier filtered search run measured about 8 ms. The gate for that path was 100 ms.
The same corpus, chunked and embedded, is the training and retrieval substrate for a purpose-built model over this data rather than a general assistant that has read a little of everything.
A companion in your pocket
The iOS companion is a native Swift 6 and SwiftUI app on iOS 18, built as a workspace plus a Swift package so the feature code is testable away from the app shell. It resolves a taxon name and a place name to real keys before it ever issues a query, then shows you the resolution steps it took, so a wrong answer is traceable to the wrong match rather than to a black box.
It talks to the same deployed warehouse the web app does, so phone and desktop cannot drift into disagreeing about a number.
In the code
"""Stage A — flat parquet to a country/year layout.
Rewrites the flat, unpartitioned occurrence snapshot into
<out>/country=US/year=2019/part-*.parquet, sorted within each output file
by (specieskey, decimallatitude) so Parquet row-group zone maps prune taxon
queries even inside a partition. Rows with a null countrycode land in
country=__NULL__.
"""
PHASE1_BATCH_SIZE = 10A single global ORDER BY before PARTITION_BY forces the engine to sort 286 GB before it can write anything, which needs 500 GB of spill and reliably fills the temp disk. Phase one partitions with no ordering, cheap and streaming. Phase two sorts each partition independently, so the sort is bounded by one country-year rather than by the dataset. The batch size exists for a second reason: the real mirror has more than 53,000 distinct partitions, the writer holds an open buffer per partition it has touched in a single copy, and one copy over all 9,705 files needed hundreds of GB of RAM regardless of the configured memory limit. Both facts were found by running it, not by reading the manual.
NAMESPACE = uuid.UUID("f4b2c9a0-6b5d-4e3c-9a1e-2d7c8b6a5f10")
UPLOAD_BATCH_SIZE = 64
def point_id(doc_type: str, doc_key: str, chunk_index: int) -> str:
return str(uuid.uuid5(NAMESPACE, f"{doc_type}:{doc_key}:{chunk_index}"))Four lines that decide whether a multi-day ingest is survivable. A random identifier per chunk means a crash halfway through leaves duplicates that have to be found and deleted. A uuid5 derived from the document type, key and chunk index means the same chunk always lands on the same point, so a re-run upserts instead of duplicating and the recovery procedure for any failure is "run the same command again." At this corpus size the resume story is not a convenience, it is the difference between a pipeline that finishes and one that does not.
# Claims that a taxon itself is absent, as opposed to records being absent.
ABSENCE_PATTERNS = [
r"does(?:\s+not|n't)\s+occur",
r"(?:is|are)\s+absent\b",
r"(?:is|are)\s+not\s+(?:found|present|recorded\s+as\s+present)\b",
r"never\s+occurred\b",
r"(?:is|are)\s+extinct\b",
r"there\s+are\s+no\s+(?!records|occurrences|observations|specimens|datasets|matching)",
]Occurrence data records where somebody looked and wrote it down. It does not record where the animal is. "Zero records in this snapshot" is true and useful; "does not occur in Texas" is a claim the data cannot support, and it is the single most common way this kind of dataset gets misread. The module header states the design rule plainly: a hard gate enforced only by asking the model nicely is not a gate. So the rule is checked mechanically on the way out, alongside a citation check that every identifier the answer quotes actually resolves. Three of the four release gates measured 100 percent against live services.
How this differs from the ordinary version
It is an archive first and a product second
The ordinary version of this is a search box in front of somebody else's database, and it stops working the day the database does. ARK inverts that: the bytes are local and verified, the index is rebuilt from them, and the web app, the phone app and the model are all readers of the same local artifact.
Speed came from layout, not from a bigger machine
Partitioning by country and year, sorting inside each file by species key, and precomputing the five rollups people actually ask for is why a question over 3.7 billion records returns in milliseconds. None of that requires a cluster. It requires deciding, in advance, what shape the questions have.
Every answer names its snapshot
Live services quietly change their answers as data is corrected and added. Anything computed here states the snapshot date it came from, so a figure cited in one document still reproduces later. That is a research requirement, not a nicety.
The known gaps are written down
The vector corpus reached about 136,000 chunks against a 500,000 target, held back by a measured embedding throughput ceiling on the development machine rather than by a defect. Bloom filters were dropped because the installed engine has no per column option. Both facts live in the repository next to the code they constrain.
In the field
A regional record that outlives its host
Border ecology, water, and land use arguments all lean on occurrence data, and they get made over years. A local mirror with a stated snapshot means an El Paso or Santa Teresa study can be re-run and checked in five years without depending on an upstream service still being reachable and still returning the same number.
The model that comes after the archive
A clean, deduplicated, citation-bearing corpus is the expensive half of training something specialised. ARK exists partly to be that corpus, so the resulting model answers from a body of evidence it can point at rather than from what it happened to absorb.
Questions
- How much data is actually mirrored?
- More than 3.7 billion occurrence records. The snapshot came down as 9,705 files, roughly 286 GB, and passed the verifier: pending count zero, byte and record counts matching the manifest, and Parquet spot checks opening cleanly. The registry, taxonomy and literature mirrors sit alongside it.
- Why is it fast?
- Layout, not hardware. Occurrences are repartitioned by country and year and sorted inside each file by species key, five aggregate rollups are precomputed, and a planner routes each question to the rollup that can answer it. Filtered vector search measured 32 ms cold and 16 ms warm against the real collection.
- Is there a mobile app?
- Yes. A native Swift 6 and SwiftUI companion for iOS 18, built as a workspace plus a Swift package, talking to the same deployed warehouse as the web app. It resolves taxon and place names to keys first and shows the resolution steps, so a wrong answer is traceable.
- Can the assistant say a species is gone from somewhere?
- No, and that is enforced in code rather than in the prompt. A pattern check rejects answers claiming a taxon is absent, extinct or no longer present, because the underlying data records observation effort, not organism presence. Record counts are the correct phrasing and the only one allowed.