12. Splits and evaluation¶

Runnable companion: examples/09_train_eval_compare.py

A split set partitions a dataset into train / val / test. It needs a member set that cannot move underneath it, which is why splits are a dataset feature: asking for a split over a live workset is rejected, with a pointer to convert first.

Cut a split set¶

from datavo_sdk import (
    DatasetSplitSetCreateRequest, SplitOutputSpec, SplitRatioSelector, ShardPlan,
)

client.create_dataset_split_set(
    "voice_train_v1",
    DatasetSplitSetCreateRequest(
        name="voice_v1",
        assignment_seed=7,
        splits=[
            SplitOutputSpec(name="train", selector=SplitRatioSelector(kind="ratio", ratio=0.6)),
            SplitOutputSpec(name="val",   selector=SplitRatioSelector(kind="ratio", ratio=0.2)),
            SplitOutputSpec(name="test",  selector=SplitRatioSelector(kind="ratio", ratio=0.2)),
        ],
        shard_plan=ShardPlan(shard_size=256),
    ),
)

Creation is worker-deferred. When it settles, the children are ordinary datasets named <split_set>/<split>:

voice_v1/train
voice_v1/val
voice_v1/test

Anything a dataset can do, they can do: pull to disk (chapter 3), open and iterate (chapter 11), seed a store with an add (chapter 5), split again. Poll the split set until it is ready before using them (client.get_dataset_split_set(dataset, split_set); the examples wrap that in a wait_for_split_set helper).

A split's name is a dataset name, so pull every tar for one split exactly as you would any dataset:

datavo dataset pull voice_v1/train --output-dir ./data

Assignment is deterministic from assignment_seed: the same dataset, seed and spec yield the same partition, on any machine, forever. Change the seed and you get a different partition — so record it.

Rules¶

Ratios alone leak. Rules constrain the assignment:

from datavo_sdk import ExclusivityRule, StratificationRule

rules=[
    ExclusivityRule(kind="exclusivity", keys=["speaker.txt"]),      # hard
    StratificationRule(kind="stratification", keys=["language.txt"]),  # best effort
]

Exclusivity (hard) — samples sharing a key value never land in two splits. This is the anti-leakage rule: one speaker, one split.
Stratification (best effort) — keep a key's value distribution similar across splits.
Representation (best effort or strict) — require given values to appear in given splits. Works on any scalar key: a numeric identifier such as session_id.cls is matched by its value like a categorical key, so each session can be required across the splits you name. A key with very many distinct values relative to its samples (a continuous field) draws an advisory warning, since per-value coverage is then rarely meaningful.

Preview before you commit — it reports candidate counts, per-split assignment and rule diagnostics without writing anything:

datavo split-set preview voice_train_v1 --spec voice_split.yaml
datavo split-set create  voice_train_v1 --spec voice_split.yaml
datavo split-set splits  voice_train_v1

client.get_dataset_split_fields(dataset) lists which keys are usable as rule fields. The full assignment contract — phases, tie-breaking, diagnostics — is the split engine.

Score a run against a split¶

The shape that makes runs comparable is: keep the test split fixed, and give each run its own store.

labeled store ─workset─▶ convert ─▶ dataset ─split set─▶ train / val / TEST
                                                              │
                        ┌─────────────────────────────────────┴──────────────────┐
                    run A: add(test) ─▶ attach pred, correct        run B: add(test) ─▶ attach …
                        └──────────── mean(correct.cls) ── compare ──┘

Per run:

Seed a store from the same test split — add_to_store(..., from_dataset="voice_v1/test") (chapter 5). Both runs are scored on an identical held-out set, and each run's outputs are isolated in its own store.
Attach the run's outputs under one producer id per run: pred.cls for the prediction, correct.cls for the per-sample metric against the label (chapter 6). Declaring both output keys up front is worth doing — declared_output_keys is fixed at registration.
Convert a pinned workset into a small eval dataset — producer_pins={"pred.cls": run_id, "correct.cls": run_id}, label unpinned (chapters 9, 11).

Aggregate server-side¶

A numeric scalar key (.cls, .count, a decoded number) lands as a per-sample attribute, so aggregate metrics need no streaming at all:

from datavo_sdk import DatasetScalarStatsRequest

stats = client.get_dataset_scalar_stats(
    "voice_eval_run_a", DatasetScalarStatsRequest(stats_keys=["correct.cls"])
)
stat = next(s for s in stats.scalar_stats if s.key == "correct.cls")
stat.mean_value, stat.present_count     # accuracy, n

Comparing runs is comparing two means, computed in the database. Which is why metrics are worth attaching as scalar keys rather than as blobs: a .npy metric can only be read by downloading it.

One consequence to plan around: a scalar key is one attribute per sample, latest writer wins. Two producers writing the same scalar key in the same store cannot be separated by stats — pin each and read it (chapter 9), which is what gives each run its own store here.

Tearing it down¶

Dependencies point upward, so teardown runs downward: eval datasets → eval stores → split set (archive, then delete, which cascades its children) → parent dataset → base store. A store seeded from a split pins that split child, so it goes first.

Pitfalls¶

Splitting a workset. Rejected by design — convert first (chapter 11).
Not recording assignment_seed. It is the whole reproducibility story of the partition.
Ratios that leak. Without an exclusivity rule, the same speaker can appear in train and test, which flatters the test score.
Using the children before ready. Cutting is worker-deferred.
Comparing runs across different test sets. Seed both from the same frozen split child, or the numbers are not comparable.

Next: 13. Staged pipelines — chain producers end to end.