Datavo User Guide

A sixteen-chapter path from "I have a tar of data" to "two training runs scored on the same held-out split" — with a shortcut at chapter 3 for when you just want existing data on your disk. Read it in order the first time: every chapter introduces exactly one new concept and uses only what earlier chapters introduced. Chapters 17 and 18 are advanced branches off chapter 12, read when you need them rather than in sequence.

Most chapters have a runnable companion in ../../examples/. The scripts are self-contained and self-cleaning — run the chapter's script while you read it. (The intro and CLI chapters — 1, 2, 3 — have nothing to run beyond the commands inline.)

#	Chapter	You learn	Runnable companion
1	Concepts	store, sample, key, operation, producer, workset, dataset; replay and convergence	—
2	Setup	install the SDK, sign in, point at a server	—
3	Download data	find a dataset and pull it to disk with the CLI — no SDK, no build	—
4	Ingest a source	create a store; import a tar into canonical samples; replay	01_ingest_source.py
5	Add samples to a store	bring existing samples into another store	02_add_to_store.py
6	Attach derived keys	compute a derived key group and write it back under a producer; convergence	03_attach_keys.py
7	Store operations	read the operation graph; remove an operation; the two idempotency mechanisms	04_operations.py
8	Read samples	select a workset and iterate decoded samples	05_read_samples.py
9	Consume derived keys	pin a producer to read a derived key; compare two producers	06_consume_produced_features.py
10	Combine sources	union stores with different schemas; OR and optional keys	07_union_sources.py
11	Datasets	convert a workset into a frozen, reproducible dataset	08_workset_to_dataset.py
12	Split sets	cut a dataset into train/val/test: selectors, rules, determinism	09_split_a_dataset.py
13	Evaluation	score and compare runs on the same held-out split	10_train_eval_compare.py
14	Staged pipelines	chain producers; move an intermediate; compare options	11_nested_pipeline_compare.py
15	Access: teams, sharing and tokens	your role; who owns what; share and publish; run tokens for programs	12_share_a_dataset.py
16	Shard caching and the cache hierarchy	cache shards locally: tiers, fall-through, warming — the second read is free	13_warm_and_read_cache.py
17*	Manual split sets	(advanced, branches off 12) materialize a partition you already have: id-list splits, staged upload, partial coverage	14_manual_split.py
18*	Reuse a membership	(advanced, branches off 12) read a dataset's or split's samples with other keys, in a store they were added into, or everything else: MemberOf	15_member_of.py

Two shapes of work

If you just want existing data, chapter 3 pulls it to disk with the CLI and you are done. To build or select data yourself, the rest splits into a write side and a read side, and you can stop after either one:

 chapters 4-7            chapters 8-10             chapters 11-14
 build a store    ──▶    read it live       ──▶    keep it and train
 import / add /          workset + iterate         convert to a dataset,
 attach / inspect        (nothing frozen)          split it, score runs

Everything you build that way is private to you. Chapter 15 is how you let other people — and your programs — at it.

If you just want data on your disk, chapter 3 (CLI) is enough. If you read data other people built in Python, chapters 1, 2, 8, 9 and 11 are enough. If you only attach derived keys, chapters 1, 2 and 4–7 are enough. Chapter 15 stands on its own once you have something to share, or a program to run.

Beyond the guide

• Reference — CLI, the shard cache, idempotency, deletion, teams and sharing, local shard IO.
• Internals — the architecture, what one sample costs in storage, the read path, the split engine.