1. Concepts¶

Datavo has seven nouns and two mechanisms. This chapter defines all of them, with no code. Every later chapter is one of them in practice.

Sample and key¶

A sample is one unit of data — an utterance, a recording, a take. Datavo mints its identity: a canonical sample_id that the server owns. You never compute it, and nothing downstream identifies a sample any other way.

A sample carries keys. A key is a named payload with an extension that says how to read it: audio.wav, transcript.txt, f0.npy, speaker.cls. Keys are the columns of the data; the sample is the row.

A key can be written more than once, and each write is a revision. Datavo keeps the history and resolves reads to one revision, so a key is a stable name rather than a specific set of bytes.

Sample store¶

A sample store holds samples. It is the ownership boundary for everything about them: their identity, their keys, their statistics, and their deletion.

A store is append-only, and it is not a folder — it is a graph of the operations that built it. Two kinds of operation exist:

Grow operations create sample rows. A source import brings raw tar archives in and normalizes them into canonical samples (chapter 4). An add brings samples that are already in Datavo in from another store, dataset or workset (chapter 5).
Attach operations write new keys onto samples that already exist (chapter 6).

So a store grows in rows and is enriched in columns, and its history is legible: every sample and every key traces back to the operation that put it there (chapter 7).

sample store "voice_corpus_v1"
  ├─ source_import   ──▶  6 samples, keys: audio.wav, transcript.txt   (rows)
  ├─ add             ──▶  4 more samples from another store            (rows)
  └─ attach          ──▶  f0.npy on every sample, by producer f0_v1    (columns)

Producer¶

A producer is the identity that attached a key: a model version, a feature extractor, a decoding setting. Attaching is the act; the producer is who performed it.

Producer identity is what makes derived data safe to work with. A derived key is owned by a (store, producer) pair, so one store can hold the same key name written by several producers — f0.npy from f0_v1 and from f0_v2 — without either overwriting the other or being confused for the other. A read that wants a derived key names the producer it means (chapter 9).

A producer is a caller-owned id, plus an optional display name and metadata — the same three things you own on a store, and nothing more. There is no lifecycle to manage and no version to bump; a new model version is simply a new producer id. Producers are scoped to a store, so f0_v001 in two stores is two producers.

Idempotency: replay and convergence¶

Running the same thing twice is normal — a job retries, a response is lost, a pipeline re-runs over a store that grew. Datavo answers that two ways, and they are not interchangeable:

Replay asks has this operation already happened? Every operation carries an operation id you choose. Submit the same one again and you get the existing operation back, with no new work. Applies to all three operation kinds. (Chapter 4 teaches it.)
Convergence asks is this work already done? An attach selects its work by subtracting what its producer already wrote, so it drains to empty — and a run that covered half a store leaves exactly the other half. Attach only, and the reason partial progress resumes. (Chapter 6 teaches it.)

Chapter 7 puts them side by side once you have seen all three operation kinds.

Workset¶

A workset is a selection of samples and keys — "these samples, these keys, in shards of this size" — over one or more stores.

A workset is live: it stores the selection, not its result. Every read re-runs it against the store as it is right now. Three consequences follow:

Creating one costs nothing. There is nothing to materialize, so select freely and throw them away.
It never goes stale. A store that grew yields more samples on the next read.
It is not reproducible. It can answer differently tomorrow, by design.

A workset is unnamed and ephemeral. It lives as long as it is being used and is released once it goes idle. You iterate one directly (chapter 8) — reading needs nothing else.

Dataset¶

A dataset is a named, frozen workset. Converting resolves the workset's selection once, at that instant, and materializes the result: fixed membership, fixed order, fixed shards, a name.

The freeze is the point: a dataset trained on last quarter opens byte-identical today, however much its stores have grown. It is the source to train, evaluate, publish and audit against (chapter 11).

The two nouns are one substrate and one verb apart:

	Workset	Dataset
Membership	resolved per read	frozen at conversion
Name	none	required
Lifetime	while in use	permanent, until deleted
Cost to create	none	materialization on a worker
Reproducible	no	yes

Naming a workset is converting it. That is the only way to freeze one, and the only way to keep one.

Split set¶

A split set partitions a dataset into child datasets — train, val, test — from a seed, deterministically, honouring rules like "never put one speaker in two splits".

Splits are a dataset feature, and only a dataset feature: a partition needs a member set that cannot change underneath it. Each child is an ordinary dataset you can open, add from, or split again (chapter 12).

Where the bytes are¶

Nothing in this list copies payload bytes. A key revision is a pointer into the tar it arrived in — blob path, offset, size — so an add is metadata, a workset is metadata, and a dataset materializes a plan rather than a copy of the data.

What you download, when you finally iterate, is a shard: a tar of {sample_id}.{key} members assembled on demand and cached locally. Shards are content-addressed, so the second read of the same thing is a local disk hit (shard cache).

Next: 2. Setup — get a client talking to a server.