18. Reuse a membership

Runnable companion: examples/15_member_of.py

Advanced — a branch off chapter 12. Read that first; this chapter assumes you know what a dataset and a split set are.

A dataset freezes which samples. Often you want that exact membership again, but not the dataset's keys or not its store: the test split, now with a feature attached after it froze; the test split, in the store you added the corpus into for a new inference run; everything except the test split, for a leakage check. Each of these is one row predicate on a store source — the membership predicate, MemberOf:

from datavo_sdk import MemberOf, Not

in_test = MemberOf("train_v3", split_set="default", split_name="test")

workset = client.create_workset(
    sources=[SourceSpec(sample_store="voice", keys=["wavlm.npy"], filters=[in_test])],
    shard_plan=ShardPlan(shard_size=256),
)

MemberOf names a dataset, or a split with split_set + split_name. It never names a workset: a membership you reuse must be frozen, so convert a workset first (chapter 11).

Which samples match

A sample of the source's store matches when it is a member or descends from one through an Add. Matching is on origin_key, the lineage every Add propagates (chapter 5), and a store holds at most one sample per origin, so each member matches at most one sample in any store:

The source's store	What matches
one of the dataset's own stores	each member itself — read it with keys the dataset never froze
a store the members were added into, in any number of hops	each member's copy, under its own id
a store with no lineage to the members	nothing

A store imported separately from the same recordings has no lineage to them, so it matches nothing: MemberOf follows Adds, not content.

Composes like any predicate

MemberOf is one more predicate in the row filter (chapter 8), so it conjoins with the rest and negates with Not:

# Everything in the store that is not in the test split.
filters=[Not(in_test)]

# Attach work on exactly the test split: its samples the producer has not covered yet.
filters=[in_test, Not(AttachedBy("wavlm_v1", ["wavlm.npy"]))]

The second is the shape an inference run over one split takes (chapter 6): the attach workset converges on the split's samples and nothing else.

Partial matches say so

If only some members were added into the store, the workset holds fewer samples — and says so. The workset response carries one membership_matches entry per MemberOf:

match = client.get_workset(workset.workset_id).membership_matches[0]
match.members         # the reference's members
match.member_origins  # their distinct origins
match.matched         # the store's samples that matched

Compare matched with member_origins, not members: a dataset that holds a sample and its copy has two members of one origin, and one store can hold only one of them. The counts are taken when datavo first resolves the reference against the store, so they read None on a workset nothing has read yet. datavo dataset create warns when a spec it builds through a workset matches partially.

Combine splits

A frozen membership also combines with others. sources_of (chapter 10) turns a dataset or a split into one dataset_scope source, so a union of splits — or of datasets — is one workset over their sources:

from datavo_sdk import sources_of

fit = client.get_dataset("default/fit")
test = client.get_dataset("default/test")
both = client.create_workset(
    sources=[*sources_of(fit), *sources_of(test)],
    shard_plan=ShardPlan(shard_size=256),
)

Datasets and splits of one store union freely, and a union is a set: a union of a dataset and its own test split holds each test sample once, reading every part's keys, and freezes into a dataset exactly as it reads.

To train on the same samples twice — once with an original key, once with a key a model attached later — make the second reading its own samples: Add the dataset's samples, with only the new key, into a store of their own, then union the dataset with that store filtered by MemberOf(dataset). The copies have their own ids, so the union holds every sample twice, once per key.

Access and lifecycle

MemberOf reveals which of a store's samples a dataset holds, so reading it needs read access to the referenced dataset as well as to the store — checked for every caller, run tokens included, on every read. The reference must name a ready, unarchived dataset or split; archive or delete it and a live workset over it fails its next read, naming the dataset. A dataset you convert from such a workset keeps its members: it holds them itself and no longer depends on the reference.

Pitfalls

• Naming a workset. MemberOf("wrk_…") is refused; convert the workset and name the dataset.
• Expecting a content match. Two imports of the same recordings are unrelated samples; only an Add links them.
• Putting it on a dataset read. A dataset_scope source is a dataset read, so it takes no MemberOf; put the predicate on a store source instead.
• Reading members as the target. A full match is matched == member_origins.

---

Back to: 12. Split sets.