2026-09-17 · post #8 · Doom + Pong
Can a small model learn to play Doom and Pong?
We trained a model to choose the next button to press. It learned to copy most of the training answers—and still lost. Two simple controllers show what it was missing.
In Doom, a useful decision is “turn left” or “fire.” In Pong, it might be “move the paddle up.” Neither needs a paragraph of explanation. That makes games a good place to try an idea inspired by TypeSafe’s Jev: give a model the current situation and a menu of actions, then let code use its choice.
Our experiment cost $1.62 in GPU runs. The small language model struggled in both games. Simple rules, fitted to the same teaching examples, did much better: 9 kills per episode in Doom and 5–3 in Pong.
Watch Doom first, then Pong. Each video starts with the Qwen model and its action probabilities, then switches to the fitted rule for comparison. The highlighted action is the one actually sent to the game.
Doom: turn toward the target, then fire
In ViZDoom’s Defend the Center scenario, monsters approach from around an arena. The player stays in the middle and has three actions: turn left, turn right, or attack. The engine can report targets outside the camera view, so the controller knows more than the screenshot shows.
The simple controller has one setting: how closely the player must aim before firing. We tried six angles against the teaching examples and recovered the teacher’s 8-degree tolerance. It averaged 9 kills across eight episodes; random actions averaged 1.13.
The Qwen sample agreed with about 66% of the offline labels, but chose “turn right” every time during gameplay. It never fired and scored no kills in its three evaluation episodes.
The decision: target more than 8° left → turn left. More than 8° right → turn right. Within 8° → fire.
In the video, watch how the model distributes probability across left, right, and fire. Then compare its choices with the fitted rule’s response to the target bearing.
Pong: knowing the answer is not enough
We used Atari Pong in the Arcade Learning Environment, playing first to five points. The controller can serve or hold, move up, or move down.
Qwen agreed with the teacher on 98.3% of the offline examples in the recorded successful run. Put it in charge of the paddle, though, and it lost 0–5. The run’s action mix was roughly 43% hold, 29% up, and 29% down: it moved the paddle, but never scored in the evaluation. A high average score on individual questions had hidden a failure at the moments that mattered.
For comparison, we fitted a tiny controller with just two settings: how far ahead to aim, and how close is close enough. Searching 12 combinations recovered the teacher’s settings: look ahead by 1.6 times the ball’s vertical movement, and tolerate a six-pixel gap. That controller won 5–3.
The decision: estimate the ball’s target height → compare it with the paddle → move up, move down, or hold.
In the video, watch the model’s probabilities for hold, up, and down. The fitted-controller segment then shows how the target height determines each move.
What Jev changes about training
Jev takes a situation and returns structured decisions with probabilities. For a game, that could mean a probability for each allowed action. TypeSafe says it computes its outputs in parallel, without writing an answer one token at a time. Its own Doom demo also uses a text description of game state, rather than images.
TypeSafe calls its training method Reinforcement Learning for Calibrated Decisions (RLCD). The goal is useful uncertainty: across many predictions assigned an 80% probability, the predicted outcome should happen about 80% of the time. That lets software decide when to act and when to ask for help. It is a different objective from rewarding answers people prefer (RLHF), or rewarding answers that pass a correctness check (RLVR). TypeSafe’s training primer explains the distinction.
The launch material describes that goal, but does not give a training recipe we can reproduce. Our experiment borrows the interface—state in, action out. We train by copying a teacher’s choices, and we do not establish that our model’s probabilities are calibrated.
How we taught our model
We used Qwen2.5-0.5B-Instruct from Alibaba’s Qwen team, a language model with roughly half a billion parameters. A small scoring layer gives each candidate action a number. A softmax turns those scores into probabilities that sum to 100%; the highest-probability action becomes the next button press. An 80% preference for “fire” does not mean an 80% chance of a kill—we have not measured calibration.
- Describe the situation. In Doom, read the nearest object’s direction and distance from the engine. In Pong, extract the ball and paddle positions from the game image. The model receives text, not the screenshot.
- Collect teaching examples. A hand-written controller picks an action for each situation. Doom’s teacher turns toward the nearest target and fires when it is lined up. Pong’s teacher follows where the ball is heading.
- Learn to score the choices. First train only the scoring layer. Then use LoRA, which updates a small set of additional weights, to adapt the language model too.
- Practice after making mistakes. The training script includes a DAgger round: let the learner visit new situations, ask the teacher what it should have done, and train on those examples. Finally, measure how it actually plays.
This is supervised imitation learning. The game score evaluates the policy; it is not the reward used to train Qwen. The Doom full run was interrupted before it finished this pipeline, so its Qwen result below comes from the smaller sample run.
The training details
Each input pairs the observation with one candidate action. A shared scalar head scores the candidates, and a softmax converts their scores into a distribution. Training minimizes its difference from the teacher’s labels. The scripts shuffle candidate order and group full-run splits by episode. The tiny Doom sample reused its 80-state pool across splits, so its label agreement is a training sanity check, not a held-out accuracy estimate. Stage A freezes the backbone; Stage B uses rank-16 LoRA. The Pong revision reduced the LoRA learning rate to 0.00002 after an earlier run became worse.
What actually worked
| Controller | Doom | Pong |
|---|---|---|
| Random actions | 1.13 · 8 games | 0–5 · 8 games |
| Qwen scorer | 0 · 3 games | 0–5 · 8 games |
| Fitted rule | 9.00 · 8 games | 5–3 · 6 games |
| Hand-written teacher | 9.00 · 8 games | 5–3 · 8 games |
The rules recovered the settings of the teachers that supplied their labels. They did not discover a new strategy. These are small evaluations on structured observations, with limited variation in Pong’s starting states.
The useful lesson is how to judge an action model: test the sequence of decisions, not just isolated answers. A missed move changes the next situation. Repeated mistakes compound. Jev’s emphasis on calibrated decisions is interesting, but this experiment tests a much smaller question: can our imitation learner turn good-looking training numbers into useful control? So far, it cannot.
What the experiment cost
The Pong revision used $1.57 in GPU runs, including failed and cancelled attempts. Doom added $0.05: a $0.03 sample and a $0.02 run lost when its rented machine disappeared. That is $1.62 for these two experiments, or $2.30 including an earlier $0.68 prototype. Local controller evaluation and replay rendering are outside those GPU charges.
The Doom result table comes from local evaluation of the fitted rule, not a completed full Qwen training run. The per-run records are available in the Pong receipt and Doom receipt.
Models and projects behind this experiment
The model we trained: Alibaba’s Qwen2.5-0.5B-Instruct, with our action-scoring head and a separate LoRA adapter for each game. TypeSafe’s Jev inspired the decision interface and the Doom experiment.
The research in the original ticket also drew on these open-source projects:
- vinnylarouge/jevlike — the reference for learning to score a menu of candidate actions.
- harshatheg/Qwen-2.5-1B-RLCD — reviewed for its parallel constrained-inference approach.
- AlexWortega/OpenJev — reviewed as a larger, separately trained decision-model reference.
Those projects informed the research; their checkpoints were not used in the reported game runs. Our game environments come from ViZDoom, with Freedoom assets, and the Arcade Learning Environment with Gymnasium. Hugging Face Transformers and PEFT provide the model and adapter tooling.
Try the experiment
Each game has a standalone training script. Start with the sample run, then check gameplay as well as label agreement before spending more. An agent can use the Compute setup guide with the Pong instructions or Doom instructions.
Run Doom
curl -fsSL https://raw.githubusercontent.com/theoriclabs/letsusecompute/main/posts/jev-doom/train.py -o train.py
compute run train.py::train --gpu cheap --dry-run
compute gpu list
compute run train.py::train --gpu L4 --timeout 2400 --wait --args '{"sample": true}'
# Full training:
compute run train.py::train --gpu L4 --timeout 5400 --waitThe pinned ViZDoom package includes Freedoom. If the requested GPU is unavailable, choose an available SKU from compute gpu list and check its quote.
Run Pong
curl -fsSL https://raw.githubusercontent.com/theoriclabs/letsusecompute/main/posts/jev-games/train.py -o train.py
compute run train.py::train --gpu cheap --dry-run
compute run train.py::train --gpu cheap --timeout 2400 --wait --args '{"sample": true}'
# Full training:
compute run train.py::train --gpu cheap --timeout 5400 --waitThis guide combines the original Doom and Pong posts. Both training scripts remain available. Replays are recordings, and do not measure live inference speed.