Retries¶

A backend is briefly not there more often than you would think. A managed Postgres takes a maintenance restart. A Container Apps replica recycles. A streaming service sheds load while it scales out. None of these is an error in your code, and none of them should cost you a four-hour training run.

So the SDK retries — and the question that matters is how long it is willing to wait, because a client that gives up at fifteen seconds turns every routine restart into a failed job.

The budget is a deadline¶

Retries are bounded by a wall-clock deadline, not by a number of attempts. The number of attempts is whatever fits: delays grow exponentially from one second, capped at thirty, so a five-minute budget is roughly fifteen tries.

There are two deadlines, because there are two different things to wait for.

	Applies to	Default
Error budget	a transport failure, `500`, `502`, `503`, `504`, `408`	300 s
Flow-control budget	`429` — the server is busy and said when to return	600 s

The split is between broken and busy, which the status code states outright. A 429 is flow control: nothing has failed, and what it waits for is a service scaling out rather than a process restarting, so it gets the longer allowance.

Everything else is terminal on the first response. A 404, a 422, a 403 is an answer, not an outage — it is raised immediately and spends none of the budget.

What the deadline measures is time without progress¶

For a control-plane call the distinction does not arise: the whole request is one round trip. For a download it decides whether a slow transfer survives. A large shard on a shared link can spend minutes transferring successfully, and a clock that started at the first attempt would abandon it for being slow rather than for being stuck.

So bytes that survive a failure — ones a resumed request will not fetch again — restart the deadline. A download that keeps advancing keeps going, however long it takes. One that is starved for the whole deadline still fails, which is the signal the budget exists to give. An attempt whose bytes had to be thrown away refunds nothing.

Downloads have their own read timeout¶

The read timeout measures the gap between chunks of a response body, so on a contended link it is what fires when a connection is starved outright: no bytes at all for the whole window. A bulk shard body and a catalog listing should not share that number, so downloads get their own:

export DATAVO_DOWNLOAD_READ_TIMEOUT_SECONDS=120

The default is 60 s, against 30 s for everything else. It is deliberately not minutes. The stream service holds one of its bounded per-replica request slots for the whole response, so a client that waits out a starved flow holds capacity other clients are being shed with 429 for — and an interrupted download now resumes rather than restarting, so a read timeout costs nothing already transferred. It releases the slot and the link, and the retry picks up where it left off.

Only the read timeout moves. A download that cannot connect still fails on the client-wide connect timeout.

A download continues where it stopped¶

An interrupted shard download keeps its partial file and asks the server for the rest, rather than re-fetching from byte 0 over exactly the link whose scarcity interrupted it.

Nothing needs configuring, and nothing breaks against a server that cannot do it: the client keeps a partial only when the response it was reading advertised Accept-Ranges: bytes, and any answer that does not confirm the resume — a plain 200, a Content-Range starting elsewhere, a 416 — makes it start over. A completed download is checked against the size the server reported for the shard before it is handed back, so a short file is retried rather than used.

A partial belongs to one download. It is not resumed by a later process.

Tuning it¶

client = DatavoClient(retry_deadline_seconds=600)

or, for a whole process:

export DATAVO_RETRY_DEADLINE_SECONDS=600

The argument wins over the environment variable, which wins over the default. The flow-control budget scales with whatever you set, keeping its proportion.

The datavo CLI defaults to 30 s instead. An operator at a terminal would rather see the failure than be held for five minutes by a broken endpoint; DATAVO_RETRY_DEADLINE_SECONDS overrides that too.

What you will see¶

Each attempt logs a WARNING naming the endpoint, the elapsed and remaining budget, and the delay before the next try. A download's line also carries the bytes that attempt transferred and the offset the next one resumes from, which is what tells a starved download apart from a hung one:

retrying datavo file download attempt=2 elapsed=31.0s remaining=269.0s delay_seconds=1.4 bytes_transferred=41943040 resume_offset=41943040 url=... error=The read operation timed out

When the budget runs out, the raised DatavoApiError says what it spent:

GET https://prod.datavo.io/api/datasets failed with 500 (after 14 attempts over 301.2s): Internal Server Error

That sentence is the whole point of the summary: "the backend was unavailable for five minutes" and "this request was rejected" are the same status code otherwise, and only one of them is worth trying again by hand.

Waiting is not free¶

A retry holds the calling thread. Inside a PyTorch DataLoader worker, a shard read that waits out a three-minute restart is a three-minute stall in your training loop — better than the run dying, but it is not nothing.

If you set DataLoader(timeout=...), set it above the retry deadline you expect to use, or the loader will kill the worker while the SDK is still legitimately waiting. Lowering DATAVO_RETRY_DEADLINE_SECONDS is the other half of that trade.

Retries and replay¶

Retrying re-sends a request; replay is what makes a re-send safe. A request is only re-sent when either the failure proves it never reached the server, or the operation converges on a second execution — through a caller-owned operation_id, a path that already names the resource, a state transition that is idempotent by nature, or a create whose key is derived from the request. A DELETE has none of those, so it is re-sent only when the request provably never left, and otherwise settles by checking whether the resource is gone.