Skip to content

17. Manual split sets

Runnable companion: examples/14_manual_split.py

Advanced — a branch off chapter 12. Read that first; this chapter assumes you know what a split set is.

Sometimes the partition is already decided, outside datavo. A published benchmark ships its own train / test lists. A colleague hand-curated an eval set. You are reproducing an external split exactly, sample for sample. In all three the split is not something to compute — it is something you already have, and want datavo to materialize faithfully.

That is a manual split set: every split carries an id_list selector, and you supply each split's membership as an explicit list of sample_ids. It produces the same {split-set}/{split} child datasets as any other split set — the difference is only in how membership is decided.

DatasetSplitSetCreateRequest(
    name="benchmark_v1",
    membership_token=token,          # from the upload, below
    splits=[
        SplitOutputSpec(name="train", selector=SplitIdListSelector()),
        SplitOutputSpec(name="test",  selector=SplitIdListSelector()),
    ],
)

It bypasses the engine

A manual split set does not run the assignment engine from chapter 12. There is no ratio to balance, no attribute predicate, no exclusivity or stratification or representation rule — a set is either all id_list (manual) or all computed, never a mix. There is also no draft preview: a computed split has a preview because the assignment is uncertain until the engine runs; a manual split's membership is exactly what you uploaded, so there is nothing to predict.

You own the partition, and datavo enforces exactly one thing: a split draws from its parent. Each child's members are the ids you uploaded for it that are present in the parent dataset's frozen membership. Two consequences follow, both deliberate:

  • Coverage may be partial. The splits need not cover the whole parent. Hold out a 50-sample eval set from a million-sample dataset and leave the other 999,950 in no split at all — that is a valid manual split set.
  • Unknown ids are dropped, not rejected. An uploaded id that is not in the parent — a stale entry from an older version of the corpus — is simply left out. The dropped count is reported in the split set's diagnostics, never raised as an error, so a lightly-stale external list still lands cleanly.

Cross-split overlap — the same sample uploaded to two splits — is likewise reported, not gated: planning succeeds and both children keep the sample. A split set is conceptually disjoint, but you own correctness here, so datavo surfaces the overlap rather than deciding for you.

Uploading the membership

The lists can be large — up to the parent's whole membership, millions of ids — so they do not ride in the create request. They upload through a staging area, in parts, each part a plain newline-delimited sample_id text file tagged with the split it feeds. One upload covers the whole set, and a single split's ids may be spread across as many parts as you like, so no one file has to be huge.

From the SDK, one call does the whole dance — reserve, upload each part, commit — and returns the membership_token you pass on the create request:

token = client.upload_split_membership(
    "my_dataset",
    {
        "train": train_ids,                 # a flat list → one part
        "test":  [test_ids_a, test_ids_b],  # a list of lists → one part each
    },
)

From the CLI, split-set create uploads local files for you. The spec names the splits and marks each id_list; each --membership-file SPLIT=PATH uploads one file as membership for that split, and repeating it for a split spreads that split across files:

datavo split-set create my_dataset --spec manual.yaml \
  --membership-file train=train_ids.txt \
  --membership-file test=test_part_a.txt \
  --membership-file test=test_part_b.txt

# manual.yaml
name: benchmark_v1
splits:
  - name: train
    selector: {kind: id_list}
  - name: test
    selector: {kind: id_list}

Every id-list split named in the spec must have at least one uploaded part, and every uploaded part must name a split in the spec — a mismatch is a clear error at create time, before any planning work.

Reading the result

Creation leaves the split set planning; a worker materializes it, exactly as for a computed set, and moves it to ready. The materialized children are ordinary datasets — you read them, split them again, share them, and score runs on them like any other. Each carries the parent's order.

The plan-time outcome lands in the split set's manual diagnostics: per split, how many ids were staged, how many survived into the child, and how many were dropped, plus the cross-split overlap. Read them to confirm the partition materialized the way you expected — a large drop count usually means the list was built against a different version of the parent.

train  staged=800  materialized=800  dropped=0
test   staged=205  materialized=200  dropped=5     # 5 ids not in this parent
overlap: 0

When to reach for it

Reach for a manual split set when the membership is given: a benchmark you must reproduce bit-for-bit, an eval set someone curated by hand, a partition exported from another system. Reach for the computed engine when you want datavo to decide the partition under constraints — a ratio, a temporal hold-out, no speaker leaking across splits. The two never mix in one set; pick the one that matches where the decision lives.

Everything here materializes server-side and scales to the parent's whole membership — the uploaded ids are intersected with the parent in the database, never pulled into memory — so a manual split of a million-sample dataset is as routine as one of fifty. The full mechanism is in the split engine internals.

Back to 12. Split sets, or on to 13. Evaluation to score runs on a split.