Datavo User Guide

A fifteen-chapter path from "I have a tar of data" to "two training runs scored on the same held-out split" — with a shortcut at chapter 3 for when you just want existing data on your disk. Read it in order the first time: every chapter introduces exactly one new concept and uses only what earlier chapters introduced.

Most chapters have a runnable companion in ../../examples/. The scripts are self-contained and self-cleaning — run the chapter's script while you read it. (The intro and CLI chapters — 1, 2, 3 — have nothing to run beyond the commands inline.)

#	Chapter	You learn	Runnable companion
1	Concepts	store, sample, key, operation, producer, workset, dataset; replay and convergence	—
2	Setup	install the SDK, authenticate, point at a server; the tokens involved	—
3	Download data	find a dataset and pull it to disk with the CLI — no SDK, no build	—
4	Ingest a source	create a store; import a tar into canonical samples; replay	01_ingest_source.py
5	Add samples to a store	bring existing samples into another store	02_add_to_store.py
6	Attach derived keys	compute a derived key group and write it back under a producer; convergence	03_attach_keys.py
7	Store operations	read the operation graph; remove an operation; the two idempotency mechanisms	04_operations.py
8	Read samples	select a workset and iterate decoded samples	05_read_samples.py
9	Consume derived keys	pin a producer to read a derived key; compare two producers	06_consume_produced_features.py
10	Combine sources	union stores with different schemas; OR and optional keys	07_union_sources.py
11	Datasets	convert a workset into a frozen, reproducible dataset	08_workset_to_dataset.py
12	Splits and evaluation	cut train/val/test; score and compare runs	09_train_eval_compare.py
13	Staged pipelines	chain producers; move an intermediate; compare options	10_nested_pipeline_compare.py
14	Teams and sharing	who owns what; share and publish	11_share_a_dataset.py
15	Shard caching and the cache hierarchy	cache shards locally: tiers, fall-through, warming — the second read is free	12_warm_and_read_cache.py

Two shapes of work

If you just want existing data, chapter 3 pulls it to disk with the CLI and you are done. To build or select data yourself, the rest splits into a write side and a read side, and you can stop after either one:

 chapters 4-7            chapters 8-10             chapters 11-13
 build a store    ──▶    read it live       ──▶    keep it and train
 import / add /          workset + iterate         convert to a dataset,
 attach / inspect        (nothing frozen)          split it, score runs

Everything you build that way is private to you. Chapter 14 is how you let other people at it.

If you just want data on your disk, chapter 3 (CLI) is enough. If you read data other people built in Python, chapters 1, 2, 8, 9 and 11 are enough. If you only attach derived keys, chapters 1, 2 and 4–7 are enough. Chapter 14 stands on its own once you have something to share.

Beyond the guide

• Reference — CLI, the shard cache, idempotency, deletion, teams and sharing, local shard IO.
• Internals — the architecture, what one sample costs in storage, the read path, the split engine.