9. Consume derived keys¶

Runnable companion: examples/06_consume_produced_features.py

Chapter 6 attached f0.npy under producer f0_v001. A chapter-8 workset does not return it. This chapter is the one extra thing a derived key needs: a pin.

The rule¶

A derived key is owned by a (store, producer) pair. A workset that wants one names the producer it means. A workset that does not resolves only the store's own ground-truth keys.

This is a correctness rule, not a style rule. Point a trainer at an unpinned workset and expect a freshly attached feature to appear, and the read succeeds with base bytes — no error, no missing key, a run trained on the wrong data. The pin is what makes that impossible.

Pinning a producer¶

producer_pins is a {key: producer_id} map on the source:

from datavo_sdk import SampleStream, SourceSpec, KeyGroupSpec, ShardPlan

workset = client.create_workset(
    sources=[
        SourceSpec(
            sample_store="voice_corpus_v1",
            keys=["audio.wav", "transcript.txt", "f0.npy"],   # base + derived
            producer_pins={"f0.npy": "f0_v001"},              # who wrote the derived one
        )
    ],
    key_groups=[
        KeyGroupSpec(name="inputs", keys=["audio.wav", "transcript.txt"]),
        KeyGroupSpec(name="features", keys=["f0.npy"]),
    ],
    shard_plan=ShardPlan(shard_size=64),
)

for sample in SampleStream.from_workset(
    workset.workset_id, keys=["audio.wav", "f0.npy"], client=client
):
    ...

Ground-truth keys stay unpinned — they resolve from the store itself. Only keys that a producer attached need naming.

A pin resolves the key to that producer's revisions and nothing else, so a later attach by a different producer cannot change what this workset sees. Pins carry into the dataset when you convert (chapter 11).

Two producers, one key¶

Because the pin is on producer identity rather than key name, one store can hold the same semantic key from several producers — two model versions, two decoding settings — side by side. Attaching under f0_v002 does not overwrite f0_v001: idempotency is per (producer_id, key), so the second producer sees every sample as its own work.

Comparing them is then two worksets that differ only in the pin:

def pinned(producer_id: str):
    return client.create_workset(
        sources=[SourceSpec(sample_store="voice_corpus_v1",
                            keys=["audio.wav", "f0.npy"],
                            producer_pins={"f0.npy": producer_id})],
        key_groups=[KeyGroupSpec(name="features", keys=["f0.npy"])],
        shard_plan=ShardPlan(shard_size=64),
    )

v1, v2 = pinned("f0_v001"), pinned("f0_v002")

Same samples, same key, different bytes. Chapter 12 turns this into scored run comparison, and chapter 13 chains it.

Isolation by store, not by flag¶

A derived key is visible only inside the store it was attached to. To keep model outputs away from ground truth entirely, put them in their own store: add the base store's samples into a fresh store (chapter 5) and attach there (chapter 6). The store boundary is the isolation — there is no separate mode or type to opt into.

Which of the two you want is a scoping decision:

Attach into the same store when the derived key is part of the corpus — features everyone reads, computed once.
Add into a new store, then attach when the derived key belongs to one run — predictions, per-run metrics, anything you want to delete as a unit.

Building further on a derived key¶

A producer that reads a derived key and writes another one is just another producer in the same store (metrics over predictions, a decoder over a signal). Pin the upstream key in its attach workset, declare it as the required input, and attach the new key as usual. Nothing about the second hop is special — chapter 13 walks a three-stage chain.

Discovery¶

client.list_producers("voice_corpus_v1")           # who has written here
client.get_producer("voice_corpus_v1", "f0_v001")  # one producer, in full
client.producer_coverage("voice_corpus_v1", "f0_v001")

get_producer shows the split that matters: producer_id, display_name and metadata are yours — you chose the id and recorded the rest — while output_keys is what datavo recorded from that producer's attaches. You never declare the keys; each attach adds what it wrote.

Producers are scoped to a store, so f0_v001 in two stores is two producers with independent metadata. There is no store-wide default producer: resolution is always the pin you write, which is why a read without one gets base resolution.

Pitfalls¶

Reading a derived key without a pin. You silently get base resolution. If a feature "did not appear", this is why.
Pinning a key nobody attached. The pin names a (store, producer) that has to exist; check with list_producers.
Pinning ground truth. Unnecessary, and it fails if the ground-truth key came from an import rather than a producer.
Assuming a new checkpoint overwrites the old. It does not. Two producers, two sets of bytes, pinned separately — which is the feature.

Next: 10. Combine sources — read across stores.