10. Combine sources

Runnable companion: examples/07_union_sources.py

Spanning stores is adding a source. A workset takes one SourceSpec per store and its membership is the row-wise union of them — public corpus plus private corpus, this year's batch plus last year's, ground truth from one store and predictions from another.

workset = client.create_workset(
    sources=[
        SourceSpec(sample_store="voice_public",  keys=["audio.wav", "transcript.txt"]),
        SourceSpec(sample_store="voice_private", keys=["audio.wav", "transcript.txt"]),
    ],
    shard_plan=ShardPlan(shard_size=256),
)

That is the whole of it when the stores agree on key names: no key_groups, because one all group over the sources' keys is derived. A single store is the same shape with one element — sources is the only way a workset says where its samples come from.

Each source carries its own keys, its own row filter and its own producer pins, so the arms need not be symmetric:

workset = client.create_workset(
    sources=[
        SourceSpec(sample_store="voice_public",  keys=["audio.wav", "transcript.txt"],
                   filters=[Attr("language", "en")]),
        SourceSpec(sample_store="voice_private", keys=["audio.flac", "transcript.txt"]),
    ],
    key_groups=[
        KeyGroupSpec(name="audio", keys=["audio.wav", "audio.flac"], mode="any"),
        KeyGroupSpec(name="text",  keys=["transcript.txt"]),
    ],
    shard_plan=ShardPlan(shard_size=256),
)

"English only from the public store, everything from the private one" is one workset rather than two reads you stitch together. The key_groups here are spelled out because the two stores name their audio differently — the next section is about that.

When schemas differ

Real corpora disagree about names. One batch delivers audio.wav, another audio.flac. A key group has a coverage mode that says what the group guarantees:

• mode="all" (the default) — every member key is present on every sample. A missing member is a data gap and fails loudly rather than streaming silently.
• mode="any" — at least one member key is present on every sample. Which one varies by sample.
• optional_keys=[...] — members that are present only for the samples that have them, in either mode.

Datavo does not coalesce, rename or unify the alternatives. A row from an any group carries whichever member that sample actually had, so the consumer reads either:

for row in SampleStream.from_workset(workset.workset_id,
                                     keys=["audio.wav", "audio.flac", "transcript.txt"],
                                     client=client):
    audio = row.get("audio.wav") or row.get("audio.flac")     # never assume which
    transcript = row["transcript.txt"]                        # `all` group: always there

Read optional keys the same way — row.get(key), expecting None.

Not coalescing is deliberate: a canonical name would hide which encoding you got, and decode paths usually care.

Which shape do you want?

Three ways to end up reading data from more than one place, and they are not interchangeable:

You want	Use	Result
One workset over several stores	multiple sources (this chapter)	nothing is written; the stores stay separate
One store holding samples from several places	add_to_store (chapter 5)	new rows, new ids, permanent
One sample carrying keys from several producers	producer_pins (chapter 9)	the same rows, more columns resolved

A union reads across stores; an add moves samples into one. Use the union when the arrangement is per-read, and the add when the arrangement should itself be a durable object.

Ordering and shards

ShardPlan decides how the unioned membership is laid out: shard_size samples per shard, seed to shuffle deterministically, sort_by to order by attributes. The plan applies to the union as a whole, so samples from both stores interleave according to it rather than arriving store by store.

Pitfalls

• Assuming a fixed key from an any group. row["audio.wav"] raises for a FLAC sample. Use .get(...) or .get(...).
• Putting alternatives in an all group. all means every listed key on every sample, so an alternative-carrying group must be any or the workset narrows to samples that have both encodings. This is also why the derived group does not fit a union of differing schemas: it is one all group over every key the sources name, so leaving key_groups out there selects nothing. Omit it when the stores agree on names; spell it out when they do not.
• Trying to attach across a union. An attach writes into one store, so an attach workset must address exactly one — a union carrying Not(AttachedBy(…)) is refused. Union to read; attach per store (chapter 6).
• Expecting a union to dedupe. Two stores holding the same underlying data contribute both rows; their ids are genuinely different samples. Use origin_key lineage if you need to tell that story.
• Different keys meaning different things. A union lines up keys by name, so a transcript.txt that is verbatim in one store and normalized in another becomes one column of mixed content. Attach a normalized key under a producer and select that instead.

---

Next: 11. Datasets — freeze a workset.