How to Spec a Custom
Robot Training Dataset

The Spec Is the Product

Disappointing datasets are almost never caused by sloppy collection. They are caused by underspecified collection: an action space nobody confirmed, success criteria that lived in someone's head, diversity that defaulted to whatever was convenient on the studio floor. Every ambiguity in the spec gets resolved by chance instead of by you. The checklist below closes those gaps in the order we work through them.

1. Task Definition

Write the task as a behavior, not a headline. "Laundry" is a research program; "grasp a single garment from a mixed pile and hang it on a rack" is a collectable task. Define the start condition, the end condition, and what the demonstrator is allowed to do in between. If the task decomposes into stages, name the stages now, because stage labels are cheap during collection and expensive afterward.

2. Embodiment and Action Space

State the target robot, the end effector, and, most critically, the action space your training code consumes: joint positions, end-effector deltas, absolute poses, with control frequency. This single line determines the capture method. On-robot action streams point to teleoperation; end-effector trajectories at volume point to UMI-style handheld capture. If the embodiment is undecided, say so, and spec end-effector space with a feasibility filter for the candidate platforms.

3. Observation Streams

List every stream: which cameras, where mounted, at what resolution and frame rate, plus proprioception and force sensing if the rig provides it. Then state the invariants: which camera placements are fixed across all episodes, and which vary deliberately. Cameras are the piece buyers most often discover too late, so pilot episodes exist partly to let you look through the actual lenses before committing the run.

4. Diversity Axes

For each axis, decide varied or controlled, and write it down: object instances and categories, initial states, layouts, lighting, distractors, backgrounds. Uncontrolled variation is not diversity, it is noise you cannot audit. A spec line like "each episode starts from one of five defined crumple classes, uniformly sampled" is worth more than a thousand accidental episodes.

5. Episode Count and Length

Episode counts are a budget decision informed by task difficulty, policy class, and whether you co-train on open corpora. Two rules hold regardless. Short episodes beat long ones, because one bad segment voids an entire episode. And the pilot batch, trained into your actual pipeline, is the only honest way to estimate the count, which is why our engagements put pilot data in your hands inside two weeks. For the open-data side of the budget question, see custom versus open datasets.

6. Success Criteria and Labels

Write success criteria a stranger could apply: measurable end states, tolerances, and reference images where words run out. Decide now whether you want recovery demonstrations, operator recoveries from staged error states, and whether pure failures are worth keeping for your training method. Label taxonomy belongs in the spec too: per-episode success, per-stage boundaries, and any event tags your evaluation depends on.

7. Format and Delivery

Name the delivery format, LeRobot, RLDS, or GR00T compatible, and require calibration files, a data card, QA results, and a loader script with the delivery. The trade-offs are covered in the format guide. If two ecosystems matter to you, ask for both exports up front.

8. Acceptance Criteria

Agree the test before collection starts: random-sample review of delivered episodes against the written success criteria, automated checks for timestamp monotonicity, stream completeness, and alignment, and the re-collection terms when an episode fails. Acceptance written into the agreement protects both sides and removes the awkward conversation at delivery.

A Spec in Miniature

Common Questions

How long should demonstration episodes be?

As short as the task allows. Long episodes concentrate risk: one bad segment can void the whole episode. If the task decomposes into stages, spec them as separate episode types and stitch at training time.

Should the dataset include failed attempts?

Include labeled recovery demonstrations, where the operator recovers from a realistic error state. Pure failures without recovery are worth keeping only if your training method explicitly uses negatives. Say which you want in the spec.

What is a reasonable acceptance test?

Random-sample review of delivered episodes against written success criteria, plus automated checks: timestamp monotonicity, stream completeness, calibration presence, and action-observation alignment. Agree the sampling rate and re-collection terms before full collection starts.