5. Add samples to a store

Runnable companion: examples/02_add_to_store.py

An add is the other way a store grows. Where a source import brings raw bytes in, an add brings samples that are already in Datavo into another store — a whole store, a filtered slice of one, a dataset, or a workset.

Use it when the goal is a separate store rather than more columns on an existing one: a scratch store to prototype against, an evaluation store seeded from a held-out split, a store holding one model's outputs away from the ground truth.

The two steps

client.create_store(name="voice_eval_scratch", display_name="Eval scratch")

client.add_to_store(
    "voice_eval_scratch",
    operation_id="seed-from-corpus",     # yours; a stable id makes the call replayable
    from_store="voice_corpus_v1",
    key_subset=["audio.wav"],            # bring only this key across
    limit=50,                            # deterministic slice
    display_name="50 samples from the corpus",
)

client.wait_for_add("voice_eval_scratch", "seed-from-corpus", timeout_seconds=180)

The source is one of from_store, from_dataset or from_workset. A store source can be narrowed with filters= (the predicate list from chapter 8) and limit=; key_subset= projects which keys come across, in any case.

Adds are worker-deferred: the call returns a planning operation and the worker materializes it (planning → committing → ready). wait_for_add polls the operation log for that operation_id.

A large add materializes a chunk of 1,000 samples at a time, so it lands in the target store progressively: every sample that is there is complete, and ready means all of it is. Its operation reports materialized_sample_count of sample_count as it goes. An add of tens of thousands of samples takes an hour, so bound the wait on progress rather than on the clock:

client.wait_for_add("voice_eval_scratch", "seed-from-corpus", stall_timeout_seconds=900)

stall_timeout_seconds fails only once no chunk has landed for that long; timeout_seconds stays the absolute deadline.

What an add actually does

It is a disjoint, zero-copy union:

• Fresh ids. Every added sample is a new row with a new sample_id. It is not a reference to the original, so the two stores are independent: delete either one and the other is untouched.
• Zero-copy bytes. Key revisions keep pointing at the tar the data already lives in. Adding 100k samples writes catalog rows, not payload bytes, so cost scales with row count and not with sample size.
• origin_key lineage. Each new row carries a stable origin_key back to the sample it came from, plus its immediate parent. That is how you ask "where did this row come from" across stores (client.origin_lineage(...)).
• Disjointness is enforced, per origin and key. A recording is one sample in a store, however many adds bring it. An add that reaches a sample already there brings keys onto it instead (next section), and a key the sample already holds is a conflict, not a silent duplicate, so a re-run cannot double the store.

Because ids are fresh, an add is a branch, not a view. Later attaches into the new store are invisible to the source store: the store boundary is the isolation boundary.

Find one import's samples after an add

A source import's operation_id keeps naming its samples wherever an add takes them. A store lists the imports whose samples it holds, whether each import ran there or its samples were added in:

for row in client.list_store_imports("voice_eval_scratch").imports:
    print(row.operation_id, row.sample_count)

Each row is an operation_id and how many of the store's samples it names. prefix= narrows the listing by operation id, and offset= / limit= page it. From the shell it is datavo sample-store imports voice_eval_scratch.

ImportedBy selects one import's samples in the store you read, and selects exactly the listed count:

from datavo_sdk import ImportedBy, ShardPlan, SourceSpec

workset = client.create_workset(
    sources=[SourceSpec(sample_store="voice_eval_scratch", keys=["audio.wav"],
                        filters=[ImportedBy("demo-batch-001")])],
    shard_plan=ShardPlan(shard_size=256),
)

The ids it selects are the target's own, so listing and reading them needs read access to the target store only, not to the store the import ran in. The listing names no store and no sample beyond the one you read. It matches on origin_key, so it follows any number of adds, and Not(ImportedBy(...)) is everything else in the store.

client.list_source_import_samples_by_operation_id(store, operation_id) answers a different question: the samples an import introduced into that store. It is how an importer checks for a replay, so for an import that ran elsewhere it is a 404. Read an added session through ImportedBy on the store you train on.

If two stores each imported something under the same operation_id, ImportedBy matches the copies of both: they are all samples an import of that name introduced. The listing shows that name once, with both counts summed.

Bring more keys later

key_subset is not final. To give the samples already in a store another of their source's keys, add the same selection again and name the new keys:

client.add_to_store(
    "voice_eval_scratch",
    operation_id="seed-transcripts",
    from_store="voice_corpus_v1",
    key_subset=["transcript.txt"],       # only keys the samples do not hold yet
    limit=50,
)
client.wait_for_add("voice_eval_scratch", "seed-transcripts", timeout_seconds=180)

A sample whose origin is already in the store is extended: it keeps its id, its lineage and everything attached on it, and gains transcript.txt zero-copy. A sample not yet in the store is added as usual, so one add can do both. The operation reports how many samples it extended as extended_sample_count.

Two rules keep this exact:

• Name the keys. An add that reaches samples already in the store with no key_subset is refused (409). "All keys" onto an existing sample would mean whatever it happens to lack.
• Only keys they lack. A key already on one of those samples, whether an earlier add, an attach or the import put it there, rejects the whole add, and the error names the key.

A dataset frozen before the add does not see the new keys: they land above its watermark.

Removing the add that brought transcript.txt removes that key from those samples, and the samples stay. The add that put the samples there is frozen while another add's keys are on them, so remove in reverse order. Both follow the usual rule for grow operations: once keys are attached in a store, its adds are no longer removable.

One-call scratch stores

For the common "give me a small disposable store seeded from real data" case, there is a single call that creates the store and adds in one step:

from datavo_sdk import CopySamplesToNewStoreRequest

client.copy_samples_to_new_store(
    CopySamplesToNewStoreRequest(
        target_sample_store="eval-scratch",
        source_sample_store="voice_corpus_v1",   # or source_dataset="my_dataset"
        keys=["audio.wav"],                      # omit for all keys
        limit=50,
        randomize=True,                          # seeded shuffle instead of first-N
        seed=0,
    )
)

It fails if the target already exists (create-or-fail), and returns samples_matched / samples_copied / keys_copied. The same thing from the CLI and the UI:

datavo sample-store copy eval-scratch --source-store voice_corpus_v1 --limit 50

--limit is the only required option; add --key to project keys and --randomize (with --seed) for a repeatable random subset.

In the UI, "Copy to new store" on a store or dataset detail page opens the same flow and lands you on the new store.

Pitfalls

• Expecting a live view. An add captures its source at the moment the worker materializes it. Samples that arrive in the source afterwards do not appear; add again for those.
• Forgetting the wait. A store read before the add is ready shows none of it, or — for a large add — part of it. Wait for the operation, or the numbers will confuse you.
• Re-running with a fresh operation_id. A random id makes a second add, which then trips the disjointness check. Pass the same operation_id to replay one — that is the same replay mechanism chapter 4 used for imports.
• Attaching a key the source already has. An attach uploads new bytes. If the key exists in the source store, bring it with another add instead (see "Bring more keys later"): zero-copy, and on the same samples.

---

Next: 6. Attach derived keys — add columns instead of rows.