12. Split sets

Runnable companion: examples/09_split_a_dataset.py

A split set cuts one dataset into named parts — usually train / val / test. You hand the engine a frozen parent dataset and a small spec, and it assigns every canonical sample to exactly one split, then materializes each split as an ordinary child dataset. This chapter is the engine: what you can ask for, how it decides, and what it guarantees. Scoring runs on a split is the next chapter.

Splits are cut over a dataset, never a live workset. The engine needs a member set that cannot move underneath it, and a workset's does — so asking for a split over a workset is rejected, with a pointer to convert first (chapter 11).

The spec, and the two phases

A split set is one canonical object: an assignment_seed, a list of splits, and a list of rules. Everything the engine does falls into two phases, and keeping them apart is the key to reasoning about it:

• Assignment decides which split each sample lands in. Driven by the rules and the assignment_seed.
• Output ordering decides the order of samples inside each generated child dataset. Driven by each split's shuffle flag.

You hand that object to create_dataset_split_set. splits and rules are the two lists you fill in — everything else in this chapter is what goes in them:

from datavo_sdk import (
    DatasetSplitSetCreateRequest, SplitOutputSpec, SplitRatioSelector,
    ExclusivityRule, ShardPlan,
)

client.create_dataset_split_set(
    "voice_train_v1",                    # the frozen parent dataset
    DatasetSplitSetCreateRequest(
        name="voice_v1",                 # names the split set (and its child datasets)
        assignment_seed=7,
        splits=[
            SplitOutputSpec(name="train", selector=SplitRatioSelector(ratio=0.6)),
            SplitOutputSpec(name="val",   selector=SplitRatioSelector(ratio=0.2)),
            SplitOutputSpec(name="test",  selector=SplitRatioSelector(ratio=0.2)),
        ],
        rules=[ExclusivityRule(keys=["speaker.txt"])],
        shard_plan=ShardPlan(shard_size=256),
    ),
)

The CLI takes that same spec as a YAML (or JSON) --spec file. Two things shift from the Python: the parent dataset is the command argument, not a field; and each selector and rule spells out its kind, since in a file that tag is the only way to tell the shapes apart (in Python the class carries it):

# voice_split.yaml — the same split set as above
name: voice_v1
assignment_seed: 7
splits:
  - name: train
    selector: {kind: ratio, ratio: 0.6}
  - name: val
    selector: {kind: ratio, ratio: 0.2}
  - name: test
    selector: {kind: ratio, ratio: 0.2}
rules:
  - kind: exclusivity
    keys: [speaker.txt]
shard_plan:
  shard_size: 256

datavo split-set create voice_train_v1 --spec voice_split.yaml

Creation is worker-deferred: create_dataset_split_set returns a planning split set with no children, and the worker cuts them. When it settles, the children are ordinary datasets named <split_set>/<split>:

voice_v1/train   voice_v1/val   voice_v1/test

Anything a dataset can do, they can do — pull to disk (chapter 3), open and iterate (chapter 11), seed a store with an add (chapter 5), or split again. Poll until ready before using them (client.get_dataset_split_set(dataset, split_set); the examples wrap that in a wait_for_split_set helper).

What you can split on

The candidate set is the parent dataset's frozen canonical membership. Split rules and attribute selectors read scalar keys only — the small, per-sample values, not the payload. A key is scalar by its extension: .txt (and .text, .transcript) resolve as categorical, .cls (and .count, .index) as numeric. An opaque audio.wav is not eligible.

get_dataset_split_fields reports what is usable, and how each key resolves:

locale.txt   categorical present=48  → rules: exclusivity, stratification, representation
speaker.txt  categorical present=48  → rules: exclusivity, stratification, representation
year.cls     numeric     present=48  → rules: exclusivity, stratification, representation

Every scalar key supports every rule; a key's numeric/categorical kind changes only how its values resolve (below), not which rules it accepts. Referencing a key that isn't there is rejected at preview and create.

Three ways to define a split

Each split names a selector — the rule for which samples it gets.

Ratio — a proportional share, balanced greedily:

SplitOutputSpec(name="train", selector=SplitRatioSelector(ratio=0.6))

Ratios must not exceed 1.0; they may sum to less, which holds the remainder out of every split (a deliberate down-sample). The share is approximate, not exact — see the guarantees below.

Attribute — a declarative carve-out by value, so a split is a fixed, non-random slice of the data rather than a share of it:

SplitOutputSpec(
    name="test",
    selector=SplitAttributeSelector(
        name="year.cls",
        predicate=SplitNumericRangePredicate(gte=2024),
    ),
    shuffle=True,   # attribute splits may shuffle their output order (ratio splits may not) — see Output ordering
)

The same split as a --spec entry:

- name: test
  selector:
    kind: attribute
    name: year.cls
    predicate: {kind: numeric_range, gte: 2024}
  shuffle: true

A predicate is either text_values (categorical membership) or numeric_range (gt/gte/lt/lte). This is the shape for a temporal hold-out — "test is every recording from 2024 on" — or any split defined by what the data is.

The two mix. Attribute splits claim their samples first; whatever is left flows to the ratio splits. So test = year ≥ 2024 (attribute) alongside train = 0.75, val = 0.25 (ratio) means: test is exactly the 2024 samples, and train/val split everything older 75/25. Attribute selectors that share a key are checked so their claims can't overlap (text values are single-owner; numeric ranges may not intersect), which is what keeps every sample in exactly one split.

Manual — an id_list selector, for when you already know the exact partition (a benchmark split, a curated eval set, a reproduction of an external split) and want to hand datavo the membership directly instead of computing it:

SplitOutputSpec(name="test", selector=SplitIdListSelector())

A manual split set bypasses the assignment engine — every split is id_list, there are no ratios, attributes, or rules — and its ids arrive through a staged upload rather than in the request. It behaves differently enough (partial coverage, dropped ids, no preview) that it has its own chapter: Manual split sets.

Once a split exists, its membership travels: read the test split with keys attached after it froze, or in a store its samples were added into, with the membership predicate MemberOf — Reuse a membership.

Rules constrain the assignment

The companion example makes this concrete with a small parent — 48 samples: eight speakers of six utterances each, three locales assigned independently of speaker, and a recording year — so exclusivity, stratification, and an attribute carve-out each have something real to do.

Ratios alone leak. In the example, a ratio-only cut scatters nearly every speaker across all three splits:

→ 7/8 speakers leak across splits (train↔val↔test): spk_0, spk_1, …

Three rule kinds constrain the assignment. Each is optional.

Exclusivity (hard, the anti-leak rule). Samples that share a value never land in two splits — one speaker, one split:

ExclusivityRule(keys=["speaker.txt"])

The engine unions same-valued samples into groups and assigns each group whole. With several keys it takes the transitive closure across them (a sample linked by any key joins the group), and missing values on a key collapse into one shared group. Only one exclusivity rule is allowed; exclusivity over several keys is achieved by putting them all in its keys list. Adding it drives the leak to zero:

→ 0 speakers leak (exclusivity keeps each speaker whole)

Stratification (best-effort). Keep a key's value distribution close to the parent's, so no split is accidentally all-de-DE:

StratificationRule(keys=["locale.txt"])

Categorical keys stratify on exact values. Numeric keys are rank-bucketed first: sorted by value and cut into min(10, N) groups of roughly equal count (N is the number of samples that have the key), not equal-width value ranges — and missing is its own bucket. The distinction matters most when a key is skewed. Take a duration_ms.cls in milliseconds, and imagine three buckets:

sorted values:  100 150 180 200 220  900 1000  5000 8000 30000

equal-WIDTH bins   →  [100–10k): 9 samples    [10k–20k): 0    [20k–30k]: 1
equal-COUNT ranks  →  bucket_1: 100 150 180 200    bucket_2: 220 900 1000    bucket_3: 5000 8000 30000

Equal-width bins would pile 90% of the clips into one bin; the rank buckets stay evenly filled however uneven the gaps are, so stratifying on them keeps a similar mix of short / medium / long clips in every split. Stratification is an objective, not a constraint — the planner approximates it. Only one stratification rule is allowed, and it too may list several keys at once. In the example it holds every locale even across splits:

test  n=12 locales={'de-DE': 4, 'en-US': 4, 'fr-FR': 4}
train n=30 locales={'de-DE': 10, 'en-US': 10, 'fr-FR': 10}
val   n=6  locales={'de-DE': 2,  'en-US': 2,  'fr-FR': 2}

Representation (best-effort or strict). Require given values to appear in given split combinations — evaluated per distinct value, not per sample:

RepresentationRule(
    key="session_id.cls",
    require=[
        RepresentationClause(all_of=["train"]),
        RepresentationClause(any_of=["val", "test"]),
    ],
    enforcement=Enforcement.BEST_EFFORT,   # the default; Enforcement.STRICT makes it a gate — see below
)

Each session_id value should appear in train and in at least one of val/test. It works on any scalar key — a numeric identifier is matched by its value like a categorical one. Multiple representation rules are allowed, one per key.

Enforcement decides only what happens to a required value the planner can't place — both levels run the same assignment, so strict is not "try harder", it is a verdict on the result. best_effort (the default) always creates the split set and lists any unsatisfied values in the diagnostics for you to judge. strict is a gate: if even one required value is still unsatisfied after planning, the whole split set ends in state failed (no children) instead of ready, naming the values that fell short — so an incomplete split never slips through as a usable one. (On a large parent, where only an approximate preview exists, strict feasibility is reported as indeterminate until the async create settles.)

A key with very many distinct values relative to its samples (a continuous field) draws an advisory warning, since per-value coverage is then rarely meaningful.

One key belongs to one rule kind, and a representation key may not also be an exclusivity key.

The same three rules in a --spec file:

rules:
  - kind: exclusivity
    keys: [speaker.txt]
  - kind: stratification
    keys: [locale.txt]
  - kind: representation
    key: session_id.cls
    require:
      - all_of: [train]
      - any_of: [val, test]
    enforcement: best_effort

How assignment decides

Assignment is greedy and deterministic. It builds groups (each sample alone, or whole exclusivity groups), orders them, and places each into the split that best fits — comparing candidates lexicographically: hard validity, then ratio fit, then representation coverage, then stratification, then split name. Ratio fit outranks the softer objectives, which is why those are best-effort.

Which planner runs — and whether the seed matters — depends on the rules:

Rules present	Planner	Group order	Seed effect
none	random	shuffled by assignment_seed	changes the partition
exclusivity and/or stratification	greedy	by group size, then first id	none — sorted, not seeded

This is the one genuinely surprising thing about the engine: the seed only moves the random planner. Add exclusivity or stratification and the assignment is fixed by the data and the rules; changing the seed does nothing. Both are fully reproducible — only the random planner is seed-sensitive.

Output ordering

After assignment, each split's shuffle flag sets the order of its child dataset. The default (shuffle=False) preserves the parent's order. shuffle=True shuffles the split's members.

Shuffle applies to attribute splits only. A ratio split rejects it — its order already follows from assignment_seed, so a second seed would be redundant:

(shuffle on a ratio split is rejected — its order comes from assignment_seed)

Determinism

For a fixed parent dataset and a fixed spec, the partition is identical — on any machine, forever:

→ same seed → identical membership: True;  different seed → different membership: True

The seed is the whole reproducibility story of a random-planner split, so record it. Note the "fixed parent" clause: determinism is per (parent dataset, spec). Re-cutting over a different parent — even the "same" data re-imported into a new store — gives new sample ids and a different partition at the same seed. That is exactly why splits are a dataset feature.

What is and isn't guaranteed

Guaranteed: every candidate sample lands in exactly one split; no sample is duplicated; exclusivity groups are assigned atomically; preview and create run the same planner; children are immutable standalone datasets.

Not guaranteed: exact ratios, exact per-value stratification, a globally optimal assignment, or leak protection for keys you didn't name in an exclusivity rule. If a key can leak, it will unless you list it.

Preview before you commit

Preview runs the same planner and writes nothing. Read its diagnostics — per-split ratio error, the exclusivity groups, rule outcomes, warnings — then create:

preview = client.preview_dataset_split_set(dataset, draft)

preview: 48 candidates, 8 exclusivity groups (largest: group_1=6, group_2=6, group_3=6)
  train target=0.6  got=0.625 (n=30, |err| 0.025)
  val   target=0.2  got=0.125 (n=6,  |err| 0.075)
  test  target=0.2  got=0.250 (n=12, |err| 0.050)
  rule[exclusivity] ok: Protect speaker.txt from crossing split boundaries.
  rule[stratification] ok: Balance locale.txt across splits.

ratio_error is the absolute gap between the target and achieved share — here the eight six-sample speaker groups can't tile 60/20/20 exactly, so the shares drift. That drift is the cost of never leaking a speaker, and the preview is where you weigh it.

The same flow is on the CLI, driven by the --spec file shown earlier — preview takes the same file as create, minus the writes:

datavo split-set preview voice_train_v1 --spec voice_split.yaml
datavo split-set create  voice_train_v1 --spec voice_split.yaml
datavo split-set splits  voice_train_v1

On a large parent, an exact preview over millions of rows would be too slow, so preview returns an approximate result computed over a bounded sample (is_approximate=True, with source_candidate_count set) rather than erroring. Treat the shares as estimates; create the split set and read the persisted diagnostics off the ready detail for the exact numbers. Persisted diagnostics are bounded too — a large representation failure stores counts plus capped examples, not every unsatisfied value.

Pitfalls

• Splitting a workset. Rejected by design — convert to a dataset first (chapter 11).
• Expecting the seed to reshuffle a ruled split. With exclusivity or stratification the planner is sorted, not seeded; the seed only moves a no-rules (random) split.
• Ratios that leak. Without an exclusivity rule the same speaker can appear in train and test, flattering the test score. Name every leak vector.
• Not recording assignment_seed. It is the reproducibility of a random split; a lost seed is a lost partition.
• Trusting an approximate preview's exact counts. On a large parent, create and read the ready diagnostics.

The full assignment contract — phases, tie-breaking, diagnostics — is the split engine internals.

---

Next: 13. Evaluation — score and compare training runs on a held-out split.