Skip to content

9. Consume derived keys

Runnable companion: examples/06_consume_produced_features.py

Chapter 6 attached f0.npy under producer f0_v001. A chapter-8 workset does not return it. This chapter is the one extra thing a derived key needs: a pin.

The rule

A derived key is owned by a (store, producer) pair. A workset that wants one names the producer it means. A workset that does not gets, per sample, whichever producer attached that key most recently.

This is a correctness rule, not a style rule. With one producer the two readings agree. Attach the same key under a second producer — a new model version, a rerun over part of the store — and an unpinned read mixes them: each sample carries the latest attach, some from one producer and some from the other. The read succeeds with no error and no missing key, and a run trains on two models' outputs. The pin is what makes that impossible.

Pinning a producer

producer_pins is a {key: producer_id} map on the source:

from datavo_sdk import SampleStream, SourceSpec, KeyGroupSpec, ShardPlan

workset = client.create_workset(
    sources=[
        SourceSpec(
            sample_store="voice_corpus_v1",
            keys=["audio.wav", "transcript.txt", "f0.npy"],   # base + derived
            producer_pins={"f0.npy": "f0_v001"},              # who wrote the derived one
        )
    ],
    key_groups=[
        KeyGroupSpec(name="inputs", keys=["audio.wav", "transcript.txt"]),
        KeyGroupSpec(name="features", keys=["f0.npy"]),
    ],
    shard_plan=ShardPlan(shard_size=64),
)

for sample in SampleStream.from_workset(
    workset.workset_id, keys=["audio.wav", "f0.npy"], client=client
):
    ...

Ground-truth keys stay unpinned — they resolve from the store itself. Only keys that a producer attached need naming.

A pin resolves the key to that producer's revisions and nothing else, so a later attach by a different producer cannot change what this workset sees. Pins carry into the dataset when you convert (chapter 11).

A pin picks the revision; it does not choose the members. When the pinned producer covers only part of the store, add AttachedBy(producer_id, [key]) to the source's filter so the selection holds only the samples it covered:

from datavo_sdk import AttachedBy, HasKeys

SourceSpec(
    sample_store="voice_corpus_v1",
    keys=["audio.wav", "f0.npy"],
    producer_pins={"f0.npy": "f0_v002"},
    filter=[HasKeys(["f0.npy"]), AttachedBy("f0_v002", ["f0.npy"])],
)

In Explore

Explore shows the same rule. With one store in scope, each selected derived key has a producer choice: Latest per sample is the unpinned read, marked when it mixes producers, and choosing a producer pins the key. Explore always says which producer a derived-key value came from. An unpinned column can mix producers, so each of its cells names its own. A pinned column reads one producer, so its header names it once. A pinned required key counts only that producer's samples, so the count and the rows agree, and the pin rides along into the .tar download and into Create dataset. A producer's panel on the store page links straight to Explore pinned to it.

A key two or more producers attached can also be compared. Compare side by side, in the same producer choice, adds a column per chosen producer next to the key's own, labeled with the key and the producer. Each of its cells is that producer's revision for the row's sample, or says the producer did not attach it — never another producer's revision. Compare columns are for looking only: the count, the rows, the .tar download and Create dataset read the key's own column, whether or not any are shown.

Two producers, one key

Because the pin is on producer identity rather than key name, one store can hold the same semantic key from several producers — two model versions, two decoding settings — side by side. Attaching under f0_v002 does not overwrite f0_v001: idempotency is per (producer_id, key), so the second producer sees every sample as its own work.

Comparing them is then two worksets that differ only in the pin:

def pinned(producer_id: str):
    return client.create_workset(
        sources=[SourceSpec(sample_store="voice_corpus_v1",
                            keys=["audio.wav", "f0.npy"],
                            producer_pins={"f0.npy": producer_id})],
        key_groups=[KeyGroupSpec(name="features", keys=["f0.npy"])],
        shard_plan=ShardPlan(shard_size=64),
    )

v1, v2 = pinned("f0_v001"), pinned("f0_v002")

Same samples, same key, different bytes. To look at a few samples, compare the two producers side by side in Explore; to score them, read these two worksets. Chapter 13 turns this into scored run comparison, and chapter 14 chains it.

Isolation by store, not by flag

A derived key is visible only inside the store it was attached to. To keep model outputs away from ground truth entirely, put them in their own store: add the base store's samples into a fresh store (chapter 5) and attach there (chapter 6). The store boundary is the isolation — there is no separate mode or type to opt into.

Which of the two you want is a scoping decision:

  • Attach into the same store when the derived key is part of the corpus — features everyone reads, computed once.
  • Add into a new store, then attach when the derived key belongs to one run — predictions, per-run metrics, anything you want to delete as a unit.

Building further on a derived key

A producer that reads a derived key and writes another one is just another producer in the same store (metrics over predictions, a decoder over a signal). Pin the upstream key in its attach workset, declare it as the required input, and attach the new key as usual. Nothing about the second hop is special — chapter 14 walks a three-stage chain.

Discovery

client.list_producers("voice_corpus_v1")           # who has written here
client.get_producer("voice_corpus_v1", "f0_v001")  # one producer, in full
client.producer_coverage("voice_corpus_v1", "f0_v001")

get_producer shows the split that matters: producer_id, display_name and metadata are yours — you chose the id and recorded the rest — while output_keys is what datavo recorded from that producer's attaches. You never declare the keys; each attach adds what it wrote.

Producers are scoped to a store, so f0_v001 in two stores is two producers with independent metadata. There is no store-wide default producer: resolution is always the pin you write, which is why a read without one gets base resolution.

Pitfalls

  • Reading a derived key without a pin. You silently get base resolution. If a feature "did not appear", this is why.
  • Pinning a key nobody attached. The pin names a (store, producer) that has to exist; check with list_producers.
  • Pinning ground truth. Unnecessary, and it fails if the ground-truth key came from an import rather than a producer.
  • Assuming a new checkpoint overwrites the old. It does not. Two producers, two sets of bytes, pinned separately — which is the feature.

Next: 10. Combine sources — read across stores.