10. Combine sources

Runnable companion: examples/07_union_sources.py

Spanning stores is adding a source. A workset takes one SourceSpec per store and its membership is the row-wise union of them — public corpus plus private corpus, this year's batch plus last year's, ground truth from one store and predictions from another.

workset = client.create_workset(
    sources=[
        SourceSpec(sample_store="voice_public",  keys=["audio.wav", "transcript.txt"]),
        SourceSpec(sample_store="voice_private", keys=["audio.wav", "transcript.txt"]),
    ],
    shard_plan=ShardPlan(shard_size=256),
)

That is the whole of it when the stores agree on key names: no key_groups, because one all group over the sources' keys is derived. A single store is the same shape with one element — sources is the only way a workset says where its samples come from.

Each source carries its own keys, its own row filter and its own producer pins, so the arms need not be symmetric:

workset = client.create_workset(
    sources=[
        SourceSpec(sample_store="voice_public",  keys=["audio.wav", "transcript.txt"],
                   filters=[Attr("language", "en")]),
        SourceSpec(sample_store="voice_private", keys=["audio.flac", "transcript.txt"]),
    ],
    key_groups=[
        KeyGroupSpec(name="audio", keys=["audio.wav", "audio.flac"], mode="any"),
        KeyGroupSpec(name="text",  keys=["transcript.txt"]),
    ],
    shard_plan=ShardPlan(shard_size=256),
)

"English only from the public store, everything from the private one" is one workset rather than two reads you stitch together. The key_groups here are spelled out because the two stores name their audio differently — the next section is about that.

When schemas differ

Real corpora disagree about names. One batch delivers audio.wav, another audio.flac. A key group has a coverage mode that says what the group guarantees:

• mode="all" (the default) — every member key is present on every sample. A missing member is a data gap and fails loudly rather than streaming silently.
• mode="any" — at least one member key is present on every sample. Which one varies by sample.
• optional_keys=[...] — members that are present only for the samples that have them, in either mode.

Datavo does not coalesce, rename or unify the alternatives. A row from an any group carries whichever member that sample actually had, so the consumer reads either:

for row in SampleStream.from_workset(workset.workset_id,
                                     keys=["audio.wav", "audio.flac", "transcript.txt"],
                                     client=client):
    audio = row.get("audio.wav") or row.get("audio.flac")     # never assume which
    transcript = row["transcript.txt"]                        # `all` group: always there

Read optional keys the same way — row.get(key), expecting None.

Not coalescing is deliberate: a canonical name would hide which encoding you got, and decode paths usually care.

Which shape do you want?

Three ways to end up reading data from more than one place, and they are not interchangeable:

You want	Use	Result
One workset over several stores	multiple sources (this chapter)	nothing is written; the stores stay separate
One store holding samples from several places	add_to_store (chapter 5)	new rows, new ids, permanent
One sample carrying keys from several producers	producer_pins (chapter 9)	the same rows, more columns resolved

A union reads across stores; an add moves samples into one. Use the union when the arrangement is per-read, and the add when the arrangement should itself be a durable object.

Combine selections

A union need not be written source by source. sources_of returns the sources that reproduce an existing selection — a workset's own sources, or one dataset_scope source for a dataset or a split — so combining selections is concatenating them:

from datavo_sdk import sources_of

both = client.create_workset(
    sources=[*sources_of(wav_workset), *sources_of(flac_workset)],
    shard_plan=ShardPlan(shard_size=256),
)

Each part keeps its own keys. A part selects exactly what it selects on its own, gated on all of the keys it lists, and each sample streams with its own part's keys. So parts whose keys differ combine as an OR across parts with no key_groups at all: the audio.wav part streams audio.wav, the audio.flac part streams audio.flac. The derived all group spans every part's keys, but it is applied per part — it never asks a WAV sample for FLAC. Spell out an any group (above) when you want the OR declared to readers as one field; the membership is the same either way.

Four rules make a union:

• One store per union. Two parts over the same store are refused — convert each to a dataset (chapter 11) and union the datasets; datasets and splits of one store combine freely.
• Samples are identified by store and id. A sample and its Add-copy share an origin_key but are two samples, and a union holds both.
• A union is a set. A sample two parts select is one member, reading the keys of both; where both read one key under different producers, the earlier part's reading stands. Converting a union freezes exactly what it reads. To read one sample under two readings, make them two samples with an Add (chapter 5).
• A dataset part reads the dataset as frozen. A dataset or split in a union streams the revisions it streams on its own, at its freeze: a key attached to its store after the freeze, by any producer, does not reach the union. A dataset converted from the union records that reading, so it streams exactly what the union streamed.

Ordering and shards

ShardPlan decides how the unioned membership is laid out: shard_size samples per shard, seed to shuffle deterministically, sort_by to order by attributes. The plan applies to the union as a whole, so samples from both stores interleave according to it rather than arriving store by store.

Pitfalls

• Assuming a fixed key from an any group. row["audio.wav"] raises for a FLAC sample. Use .get(...) or .get(...).
• Listing alternatives on one source. A source is gated on every key it lists that an all group holds, so a source naming both audio.wav and audio.flac selects only samples that have both. Give each source the key its store carries, or put the alternatives in an any group.
• Trying to attach across a union. An attach writes into one store, so an attach workset must address exactly one — a union carrying Not(AttachedBy(…)) is refused. Union to read; attach per store (chapter 6).
• Expecting a union to dedupe. Two stores holding the same underlying data contribute both rows; their ids are genuinely different samples. Use origin_key lineage if you need to tell that story.
• Different keys meaning different things. A union lines up keys by name, so a transcript.txt that is verbatim in one store and normalized in another becomes one column of mixed content. Attach a normalized key under a producer and select that instead.

---

Next: 11. Datasets — freeze a workset.