11. Datasets

Runnable companion: examples/08_workset_to_dataset.py

Everything so far read through a live workset, which is what exploring wants. It is also what training does not want: a source that can answer differently tomorrow cannot be reproduced.

A dataset is a named, frozen workset. Converting resolves the membership once, at that instant, and freezes the result under a name.

Convert

dataset = client.convert_workset_to_dataset(
    workset.workset_id,
    "voice_train_v1",
    shard_plan=ShardPlan(shard_size=1000),
)
ready = client.wait_for_dataset("voice_train_v1", timeout_seconds=1800)
ready.lifecycle_state          # "ready"

for sample in SampleStream.from_dataset(
    "voice_train_v1", keys=["audio.wav", "transcript.txt"], client=client
):
    ...

Sources, filters and key groups come from the workset, so the dataset is that workset frozen — you are keeping it, not redefining it. Override key_groups= only for a different download grouping, and scalars= to record constants alongside the data ({"sample_rate": 24000}).

Conversion is worker-deferred: it returns a planning dataset and a worker materializes membership, order and shard plan. wait_for_dataset polls to ready. The workset stays live and usable afterwards; nothing consumed it.

While it plans, the dataset's planning field says where it is: the phase (queued, waiting_for_store_lock, writing_members, …), done/total for the sample-scale phases, and attempt — above 1 when an earlier attempt's worker was stopped or killed. The dataset page and datavo dataset get show the same.

What freezing buys

	Workset	Dataset
Membership	re-resolved per read	fixed at conversion
A store that grows	shows up	does not
Name	none	required, unique
Reproducible	no	yes
Splits	not possible	yes (chapter 12)
Reading	SampleStream.from_workset	SampleStream.from_dataset

Import 1000 more samples into the store, and the workset yields them on the next read while the dataset does not. That is the guarantee, not staleness: a dataset opened a year from now yields the same samples in the same order.

from_dataset also has less to do than from_workset: a dataset's shard plan is already materialized, so opening one is a couple of indexed reads and the first network cost lands at the first sample, not at open. keys= is optional here — omit it to read every key the dataset defines, and client.get_dataset(name).key_groups lists them.

Which producer each key reads

A dataset reads one revision of each key per member, fixed when it froze. For a derived key that a store holds from several producers (chapter 9), that is:

• a key pinned with producer_pins reads its producer;
• an unpinned key reads, per member, whichever producer attached it most recently before the freeze. A newer producer that covered only some members leaves the dataset reading a mix.

The dataset tells you which:

provenance = client.get_dataset_key_provenance("voice_train_v1")
for group in provenance.key_groups:
    for key in group.keys:
        for producer in key.producers:
            print(key.key, producer.producer_id or "imported", producer.member_count)
        print("  pinned:", key.pinned_producer_ids, "unpinned:", key.reads_unpinned)

Producers come most members first, each with the range of times its revisions were attached. An imported key is one imported entry that counts its source imports. A producer that attaches after the freeze never appears, because the dataset does not read it. datavo dataset get prints the same table, and the dataset page shows it on each key in the Keys panel and in Segmentation.

Naming a workset is converting it

There is one freeze verb and one way to keep a workset. There is no "save this workset", no pinning, no TTL to extend: worksets are unnamed and ephemeral by construction, and the moment you want a name, you want a dataset.

The working order is therefore: select freely, read, refine, convert the one you keep.

create_workset ──read──▶ refine ──read──▶ convert_workset_to_dataset ──▶ train
   (free)                  (free)              (materialized, named)

Retention

A dataset holds a tracked dependency on the stores it reads, which is what keeps its bytes alive. That dictates teardown order: delete the dataset before the store it reads. Deleting the store first returns 409 while a protecting dataset depends on it. Likewise a dataset another dataset is built from — a dataset converted from a union with it as a part — cannot be deleted while that dataset exists: delete the one built from it first.

Downloading it

Datasets can be pulled to local tars — for an offline machine, an archive, or a non-Python consumer:

datavo dataset get voice_train_v1
datavo dataset pull voice_train_v1 --output-dir ./downloaded/train

That writes one tar per shard, one directory per key group; add --key-group to pull only some. For normal training you do not need it: SampleStream.from_dataset streams and caches the same shards on demand (shard cache).

Pitfalls

• Opening a planning dataset. Conversion is worker-deferred; wait for ready.
• Reusing a name. Dataset names are unique and datasets are frozen. A changed workset is a new dataset name — version them (voice_train_v1, _v2).
• Converting to escape a slow read. Both paths stream the same cached shards; convert for reproducibility, not for speed.
• Deleting the store first. Archive and delete the dataset, then the store.
• Deleting a dataset another dataset is built from. It returns 409 naming the dependents; delete those first.
• Assuming an unpinned derived key reads one producer. Check the dataset's key provenance, or pin the producer before converting.
• Converting too early. A dataset is permanent and named. While you are still changing your mind, stay on the workset — it costs nothing.

---

Next: 12. Split sets — cut a dataset into train/val/test with rules and deterministic assignment.