Local shards (compute and train on one machine)

When a producer and a trainer run on the same worker — compute features, finetune, export, all in one GPU container — the round trip through Datavo buys nothing: the write-back would be immediately followed by a server-resolved re-download of bytes that never left the machine.

LocalShardWriter and SampleStream.from_local_shards are the server-free pair for that case. The tar layout and the decode rules are identical to the server-mediated path, so the trainer code does not change.

This is an explicit trade-off, covered at the end of the page.

Write

LocalShardWriter is the local counterpart of attach_keys: it writes derived key-group tars keyed by canonical sample_id and emits a manifest describing them.

from datavo_sdk import LocalShardWriter

with LocalShardWriter("/scratch/increment") as writer:
    for shard_index, samples in enumerate(input_shards):
        # reference the raw tar already on disk (raw shard or cache tier)
        writer.add_existing_tar(
            shard_index=shard_index,
            key_group="audio",
            keys=["audio.wav"],
            tar_path=raw_tar_path_for(shard_index),
            sample_count=len(samples),
        )
        # write the derived group, in the SAME sample order
        writer.write_group(
            shard_index=shard_index,
            key_group="f0",
            keys=["f0.npy"],
            samples=(
                {"__key__": sample_id, "f0.npy": f0}
                for sample_id, f0 in compute_f0(samples)
            ),
        )
# /scratch/increment/manifest.json is now readable

add_existing_tar folds in tars you already have — raw shards, or inference outputs written with altavo_inference --output-write-policy cache_only.

The ordering contract: every key group written for one shard_index must cover the same sample_ids in the same order. The writer enforces matching per-group sample counts, and the reader verifies sample_id ordering at merge time and raises on a mismatch.

Read

from datavo_sdk import SampleStream

ds = SampleStream.from_local_shards(
    "/scratch/increment/manifest.json",
    keys=["audio.wav", "f0.npy"],
)
for sample in ds:
    sample["__key__"], sample["audio.wav"], sample["f0.npy"]

No client, no get_dataset, no workset, no download. The manifest maps each shard index to its per-key-group tars — the local stand-in for the server plan — and iteration merges those tars by sample_id and auto-decodes exactly as the server-mediated path does. sample_count, shard_count and len(ds) come from the manifest, and the whole thing works with no Datavo server reachable.

For a DataLoader, the torch subclass partitions shard indices across DDP ranks and workers as usual:

from datavo_sdk.torch_dataset import DatavoTorchDataset
from torch.utils.data import DataLoader

ds = DatavoTorchDataset.from_local_shards(
    "/scratch/increment/manifest.json", keys=["audio.wav", "f0.npy"]
)
loader = DataLoader(ds, num_workers=4, batch_size=8, multiprocessing_context="spawn")

The trade-off

Locally written shards are not published to Datavo. They therefore have no cross-run feature caching, no lineage, no producer identity, and no idempotency tracking: the intermediates exist only on that worker's disk, and a second run recomputes them.

Use this for single-worker increments. Anything read by another worker, another run, or another machine belongs in Datavo — attach it (attach_keys) and read it with SampleStream.from_workset / from_dataset.