Skip to content

5. Add samples to a store

Runnable companion: examples/02_add_to_store.py

An add is the other way a store grows. Where a source import brings raw bytes in, an add brings samples that are already in Datavo into another store — a whole store, a filtered slice of one, a dataset, or a workset.

Use it when the goal is a separate store rather than more columns on an existing one: a scratch store to prototype against, an evaluation store seeded from a held-out split, a store holding one model's outputs away from the ground truth.

The two steps

client.create_store(name="voice_eval_scratch", display_name="Eval scratch")

client.add_to_store(
    "voice_eval_scratch",
    operation_id="seed-from-corpus",     # yours; a stable id makes the call replayable
    from_store="voice_corpus_v1",
    key_subset=["audio.wav"],            # bring only this key across
    limit=50,                            # deterministic slice
    display_name="50 samples from the corpus",
)

client.wait_for_add("voice_eval_scratch", "seed-from-corpus", timeout_seconds=180)

The source is one of from_store, from_dataset or from_workset. A store source can be narrowed with filters= (the predicate list from chapter 8) and limit=; key_subset= projects which keys come across, in any case.

Adds are worker-deferred: the call returns a planning operation and the worker materializes it (planning → committing → ready). wait_for_add polls the operation log for that operation_id.

What an add actually does

It is a disjoint, zero-copy union:

  • Fresh ids. Every added sample is a new row with a new sample_id. It is not a reference to the original, so the two stores are independent: delete either one and the other is untouched.
  • Zero-copy bytes. Key revisions keep pointing at the tar the data already lives in. Adding 100k samples writes catalog rows, not payload bytes, so cost scales with row count and not with sample size.
  • origin_key lineage. Each new row carries a stable origin_key back to the sample it came from, plus its immediate parent. That is how you ask "where did this row come from" across stores (client.origin_lineage(...)).
  • Disjointness is enforced. Adding the same origin twice into one store is a conflict, not a silent duplicate, so a re-run cannot double the store.

Because ids are fresh, an add is a branch, not a view. Later attaches into the new store are invisible to the source store: the store boundary is the isolation boundary.

One-call scratch stores

For the common "give me a small disposable store seeded from real data" case, there is a single call that creates the store and adds in one step:

from datavo_sdk import CopySamplesToNewStoreRequest

client.copy_samples_to_new_store(
    CopySamplesToNewStoreRequest(
        target_sample_store="eval-scratch",
        source_sample_store="voice_corpus_v1",   # or source_dataset="my_dataset"
        keys=["audio.wav"],                      # omit for all keys
        limit=50,
        randomize=True,                          # seeded shuffle instead of first-N
        seed=0,
    )
)

It fails if the target already exists (create-or-fail), and returns samples_matched / samples_copied / keys_copied. The same thing from the CLI and the UI:

datavo sample-store copy eval-scratch --source-store voice_corpus_v1 --limit 50

--limit is the only required option; add --key to project keys and --randomize (with --seed) for a repeatable random subset.

In the UI, "Copy to new store" on a store or dataset detail page opens the same flow and lands you on the new store.

Pitfalls

  • Expecting a live view. An add captures its source at the moment the worker materializes it. Samples that arrive in the source afterwards do not appear; add again for those.
  • Forgetting the wait. A store read right after add_to_store shows the pre-add count. Wait for the operation, or the numbers will confuse you.
  • Re-running with a fresh operation_id. A random id makes a second add, which then trips the disjointness check. Pass the same operation_id to replay one — that is the same replay mechanism chapter 4 used for imports.
  • Projecting away a key you need. key_subset is a projection: keys left out are simply not in the new store, and an attach later cannot invent them.

Next: 6. Attach derived keys — add columns instead of rows.