13. Evaluation

Runnable companion: examples/10_train_eval_compare.py

Evaluation is the end-to-end ML shape, and where a dataset earns its keep: you score two (or more) candidate runs on the same held-out test split and compare their aggregate metrics. The split itself comes from the previous chapter — here its test child is the fixed yardstick both runs are measured against.

Score a run against a split

The shape that makes runs comparable is: keep the test split fixed, and give each run its own store.

labeled store ─workset─▶ convert ─▶ dataset ─split set─▶ train / val / TEST
                                                              │
                        ┌─────────────────────────────────────┴──────────────────┐
                    run A: add(test) ─▶ attach pred, correct        run B: add(test) ─▶ attach …
                        └──────────── mean(correct.cls) ── compare ──┘

Per run:

1. Seed a store from the same test split — add_to_store(..., from_dataset="voice_v1/test") (chapter 5). Both runs are scored on an identical held-out set, and each run's outputs are isolated in its own store.
2. Attach the run's outputs under one producer id per run: pred.cls for the prediction, correct.cls for the per-sample metric against the label (chapter 6). Declaring both output keys up front is worth doing — declared_output_keys is fixed at registration.
3. Convert a pinned workset into a small eval dataset — producer_pins={"pred.cls": run_id, "correct.cls": run_id}, label unpinned (chapters 9, 11).

Aggregate server-side

A numeric scalar key (.cls, .count, a decoded number) lands as a per-sample attribute, so aggregate metrics need no streaming at all:

from datavo_sdk import DatasetScalarStatsRequest

stats = client.get_dataset_scalar_stats(
    "voice_eval_run_a", DatasetScalarStatsRequest(stats_keys=["correct.cls"])
)
stat = next(s for s in stats.scalar_stats if s.key == "correct.cls")
stat.mean_value, stat.present_count     # accuracy, n

Comparing runs is comparing two means, computed in the database. Which is why metrics are worth attaching as scalar keys rather than as blobs: a .npy metric can only be read by downloading it.

One consequence to plan around: a scalar key is one attribute per sample, latest writer wins. Two producers writing the same scalar key in the same store cannot be separated by stats — pin each and read it (chapter 9), which is what gives each run its own store here.

Tearing it down

Dependencies point upward, so teardown runs downward: eval datasets → eval stores → split set (archive, then delete, which cascades its children) → parent dataset → base store. A store seeded from a split pins that split child, so it goes first.

Pitfalls

• Comparing runs across different test sets. Seed both from the same frozen split child, or the numbers are not comparable — this is the whole reason the split is cut once, up front.
• Two producers, one scalar key, one store. Latest writer wins, and stats can't separate them; give each run its own store and pin its producer (chapter 9).
• Scoring before the split is ready. Cutting is worker-deferred (chapter 12); wait for the split set before seeding from its children.

---

Next: 14. Staged pipelines — chain producers end to end.