11. Datasets

Runnable companion: examples/08_workset_to_dataset.py

Everything so far read through a live workset, which is what exploring wants. It is also what training does not want: a source that can answer differently tomorrow cannot be reproduced.

A dataset is a named, frozen workset. Converting resolves the membership once, at that instant, and freezes the result under a name.

Convert

dataset = client.convert_workset_to_dataset(
    workset.workset_id,
    "voice_train_v1",
    shard_plan=ShardPlan(shard_size=1000),
)
ready = client.wait_for_dataset("voice_train_v1", timeout_seconds=1800)
ready.lifecycle_state          # "ready"

for sample in SampleStream.from_dataset(
    "voice_train_v1", keys=["audio.wav", "transcript.txt"], client=client
):
    ...

Sources, filters and key groups come from the workset, so the dataset is that workset frozen — you are keeping it, not redefining it. Override key_groups= only for a different download grouping, and scalars= to record constants alongside the data ({"sample_rate": 24000}).

Conversion is worker-deferred: it returns a planning dataset and a worker materializes membership, order and shard plan. wait_for_dataset polls to ready. The workset stays live and usable afterwards; nothing consumed it.

What freezing buys

	Workset	Dataset
Membership	re-resolved per read	fixed at conversion
A store that grows	shows up	does not
Name	none	required, unique
Reproducible	no	yes
Splits	not possible	yes (chapter 12)
Reading	SampleStream.from_workset	SampleStream.from_dataset

Import 1000 more samples into the store, and the workset yields them on the next read while the dataset does not. That is the guarantee, not staleness: a dataset opened a year from now yields the same samples in the same order.

from_dataset also has less to do than from_workset: a dataset's shard plan is already materialized, so opening one is a couple of indexed reads and the first network cost lands at the first sample, not at open. keys= is optional here — omit it to read every key the dataset defines, and client.get_dataset(name).key_groups lists them.

Naming a workset is converting it

There is one freeze verb and one way to keep a workset. There is no "save this workset", no pinning, no TTL to extend: worksets are unnamed and ephemeral by construction, and the moment you want a name, you want a dataset.

The working order is therefore: select freely, read, refine, convert the one you keep.

create_workset ──read──▶ refine ──read──▶ convert_workset_to_dataset ──▶ train
   (free)                  (free)              (materialized, named)

Retention

A dataset holds a tracked dependency on the stores it reads, which is what keeps its bytes alive. That dictates teardown order: delete the dataset before the store it reads. Deleting the store first returns 409 while a protecting dataset depends on it.

Downloading it

Datasets can be pulled to local tars — for an offline machine, an archive, or a non-Python consumer:

datavo dataset get voice_train_v1
datavo dataset pull voice_train_v1 --output-dir ./downloaded/train

That writes one tar per shard, one directory per key group; add --key-group to pull only some. For normal training you do not need it: SampleStream.from_dataset streams and caches the same shards on demand (shard cache).

Pitfalls

• Opening a planning dataset. Conversion is worker-deferred; wait for ready.
• Reusing a name. Dataset names are unique and datasets are frozen. A changed workset is a new dataset name — version them (voice_train_v1, _v2).
• Converting to escape a slow read. Both paths stream the same cached shards; convert for reproducibility, not for speed.
• Deleting the store first. Archive and delete the dataset, then the store.
• Converting too early. A dataset is permanent and named. While you are still changing your mind, stay on the workset — it costs nothing.

---

Next: 12. Splits and evaluation — cut train/val/test and compare runs.