Gameplay video and action alignment: a format guide

Three files describe a recording

• clip.mp4 contains H.264 video with a constant encoded frame rate.
• clip.jsonl contains one JSON object per line, intended to correspond to each encoded video frame.
• clip.meta.json describes the game, frame rate, dimensions, frame count, duration, alignment convention, and control bindings.

This is the current collector format, not a promise that a requested dataset is already in stock. Delivery requirements should be confirmed with the project brief. The format does not include controller state, game-engine state, rewards, or task-success annotations.

Read one action row

This illustrative row uses the current field names. At 30 frames per second, encoded frame 1042 is approximately 34.733 seconds into the saved video.

{
  "frame": 1042,
  "t_s": 34.733,
  "keys": ["W", "SPACE"],
  "keys_down": ["SPACE"],
  "keys_up": [],
  "mouse": {
    "dx": -14,
    "dy": 3,
    "buttons": ["LBUTTON"],
    "buttons_down": ["LBUTTON"],
    "buttons_up": []
  }
}

The documented convention assigns action[t] to the interval between frame t and frame t+1. The held-key and held-button lists describe the state at the end of that interval. The down and up lists record transitions observed within it; mouse deltas are summed over the interval.

A quick press and release can therefore appear in both transition lists while being absent from the final held state. Repeated transitions for the same key within one interval are deduplicated in each list. The row is an interval summary, not a complete event timeline with individual event timestamps.

Key names need a control map

W identifies a recorded key; it does not inherently mean “forward.” Metadata can map a key to a control name, for example "W": "forward". The collector’s configured bindings supply that map. Validate it against the player’s actual in-game settings, especially when combining titles or custom key layouts.

The collector records key names and mouse buttons while the game process is foreground, including inputs without a configured binding. The binding map labels inputs; it does not restrict collection to gameplay controls. Recording instructions should account for in-game chat and other personal information visible or typed inside the game.

Mouse deltas are captured input movement, not calibrated camera rotation in degrees. Sensitivity, input settings, menus, and game behavior affect their interpretation. Mouse-wheel actions and controllers are not represented by this schema. Agree the device and settings you need before collection.

Encoded time is not a capture-quality guarantee

The collector calculates t_s from the encoded frame index divided by the configured frame rate. The current encoder assigns a fixed duration to each saved frame. A constant encoded frame rate does not, by itself, prove that the source frames arrived evenly in wall-clock time.

The recorder can pause after the game loses focus and resume into the same recording. The current sidecar does not include explicit pause-boundary markers. Saved-video time continues across the retained frames, so do not assume every neighboring pair spans an uninterrupted interval of play. For tasks requiring continuous trajectories, agree how interruptions will be identified and excluded.

The metadata names QueryPerformanceCounter as the collector clock. It does not provide a measured input-to-image latency or an individually timestamped raw-event stream. Treat the alignment convention as a documented association to validate, rather than proof of the exact simulation step in which an input took effect.

Validate structure, then inspect behavior

Count decoded frames and parsed action rows independently. Check that indexes start at zero and increase by one, that each row has the expected types, and that the metadata agrees with the video. The collector logs a frame/row mismatch; downstream validation must still reject or investigate it.

For a behavioral spot check, render the action state next to the video. Compare held movement, short taps, mouse turns, menus, and the start and end of recording. Check the first frame after any suspected interruption. Do not create a next-frame training target from the final row when no subsequent observation exists.

Validate timing again after preprocessing. Dropping frames, changing frame rate, trimming clips, or resizing observations requires an explicit policy for the associated actions. A mouse delta accumulated across an interval cannot simply be relabeled as an instantaneous action.

Document the transformation your model uses

Keep the original files and version your loader. Record how raw keys become your action vocabulary, how you handle unbound inputs, and how you construct observation/action pairs. Test these decisions on a representative sample before scaling.

MP4 plus JSONL is also used in the public VPT contractor demonstrations, but shared file extensions do not imply compatible fields or action semantics. Use a deliberate adapter for each source.

Send us your required action space and sample validation checks so we can discuss the collection and format your experiment needs.