3. Download data¶

The fastest way to real data on your disk: no SDK, no build, two commands. Find a dataset, pull it. Everything here is the datavo CLI from chapter 2 — you need only the reader role.

Find a dataset¶

A dataset is a named, frozen selection someone published — the unit you download. List what exists, and search by name:

datavo dataset list                 # name, state, sizes
datavo dataset list --query voice   # search — matches the name

Each row is a dataset's name, lifecycle state, backing store, and sample and shard counts. Only a dataset in state ready is materialized and pullable; a planning one is still being built.

Before you pull, check a dataset actually carries the keys your run reads. dataset get lists its key groups and the keys in each:

datavo dataset get voice_train_v1

name: voice_train_v1
lifecycle_state: ready
sample_count: 42188
group  keys
-----  --------------------------
audio  audio.wav
text   transcript.txt,speaker.txt

--query searches names; --key searches contents. Filter to the datasets that contain a key — repeat --key to require several at once (a dataset must contain every key you pass):

datavo dataset list --key audio.wav --key transcript.txt

Each row's keys column shows what that dataset contains. datavo sample-store list browses the raw stores behind datasets, but stores are not pulled directly — read those live from Python (chapter 8).

Pull it to disk¶

datavo dataset pull voice_train_v1 --output-dir ./data

That writes one tar per shard under ./data/<key-group>/, every key group in the dataset. Narrow it, or re-pull, with:

datavo dataset pull voice_train_v1 --output-dir ./data --key-group audio --overwrite

--key-group (repeatable) restricts which groups download; --overwrite replaces files already on disk. Each shard is a separate download, so re-running into a directory that already has shards needs --overwrite — without it the pull stops at the first file that already exists.

What you got¶

The tars are WebDataset shards: each holds many samples, one file per key (<sample_id>.audio.wav, <sample_id>.transcript.txt, …) plus a __meta__.json. Any WebDataset-aware loader reads them as-is.

To iterate decoded samples in Python instead of handling raw tars — with row filters, key selection and prefetch — read them live with the SDK: that is chapter 8. To pre-warm the local cache ahead of a training run rather than copying files, see datavo warm.

Pitfalls¶

dataset pull needs a dataset name, not a store — take one from dataset list. If what you want lives only in a store, read it from Python (chapter 8) or ask its owner to publish a dataset.
A planning dataset has nothing to pull yet — wait for ready.
A 403 is your role, not your setup (chapter 2's role table). Reader is enough to list and pull.

Next: 4. Ingest a source — the other direction, when you have data to bring in.