What it does
A VR rhythm game records controller positions in three dimensions against time. The world stores that as score data. The pipeline reads it as movement: a large library of two-handed trajectories performed under real time pressure by players the game itself selected through difficulty gating.
The run covers roughly 20,000 map files and yields about 17.7 million extracted patterns, at around 23 milliseconds of processing per map, which is a complete pass in about a minute when parallelized.
In the code
@dataclass(frozen=True)
class Pattern:
hand: str # left | right
positions: list # (x, y, t) samples for one stroke
difficulty: str # the tier the source map was rated at
bpm: float # tempo, which sets the time pressure
# Difficulty is carried through because it is the label. The game
# already sorted competent players from incompetent ones; discarding
# that field would throw away the only free supervision in the set.Illustrative shape. The dataset is valuable because of the two fields most people would drop as game metadata.
How this differs from the ordinary version
The labels come free, and they are strict
Robot training data for two-handed manipulation is expensive: arms, demonstrators, motion capture, a controlled space. This dataset is self-labeled twice over, by the game mechanics and by human performance standards, and the difficulty system that selected the expert players is more rigorous than most collection protocols.
The method generalizes past this game
Any activity that records spatial movement against time is a candidate. The contribution is the reclassification, not the specific title.
In the field
Why a manufacturing lab cares
Two-handed assembly is the motion that dominates a panel shop floor. A cheap source of validated bimanual trajectories is directly relevant to any attempt to model or assist that work.
Questions
- How long does a full extraction take?
- About 23 milliseconds per map, and roughly a minute for the full set when run in parallel.