LLMs Push TIS-100 Optimization Frontier

Sixteen large language models spent days improving programs in Zachtronics' TIS-100, and the effort produced 133 new points on the Pareto frontier, according to the TIS-100 Bench write-up. The result matters because it shifts AI evaluation toward iterative optimization, where a model must refine code under constraints rather than simply answer correctly once.

What Happened in the TIS-100 LLM Benchmark

The source described a benchmark in which 16 LLMs, covering leading frontier and open-weight models, attempted all 47 TIS-100 puzzles plus eight community tournament puzzles. Each model made one scored attempt, pass@1, and spent 10 to 45 minutes per puzzle depending on difficulty. The team froze the community leaderboard on October 15, 2026, before submitting anything, and asked the models to keep improving. The effort produced 133 new frontier points, with 18 of 55 puzzles gaining new points and three outright cycle records. Within days, players picked up the models' programs and improved on most of them. The launch video is at the launch video.

The source said twelve nodes, two registers per node and fifteen lines of code are enough to make even multiplying two numbers a small engineering project. The team said players had spent a decade optimizing these programs on a public community leaderboard. A new frontier point was defined as a solution that no community solution tied or beat on all three scores at once. It could use more code to run faster, for example.

How the Benchmark Tested Optimization

According to the source, the harness design gave models access to the manual, visible test inputs, registers and game feedback through text tools instead of a mouse and screenshots. They did not get community code or exact records, only the game's coarse score histograms. Models could read, edit and run. Diagnostics returned mismatches, registers and blocked ports. Saved progress let a model recover a passing program with get_best. Reminders asked it to keep testing. A visible clock showed time used against limits. Run limits stopped execution after roughly one second without state changes, five seconds without output, or 60 seconds overall.

The headline count required a fresh-game replay and 100,000 passing tests on the independent TIS-100-CXX simulator, the source said. The game checks a small set of cases per run, so the stronger count was meant to provide more evidence that a program generalized.

The source reported 880 episodes, 95.5 million output tokens and 62,400 game-tool calls. It said models get what a player can inspect, through text tools instead of a mouse and screenshots. Solving and optimizing are different skills, the source noted.

What the Results Mean for AI Coding and Game AI

The source said Opus 5.5 and GPT-6 Astra led. Both solved all 55 puzzles and both found new frontier points on 11. Opus 5.5 found more points, 72 to 32, while GPT-6 Astra had the higher record proximity, 78.0 to 77.2. The proximity gap's 95% bootstrap interval included zero, from -0.7 to +2.4. GPT-6 Astra also led in all six tested hypervolume settings, which measure the space covered by each model's tradeoffs.

Solving and optimizing are different. The source said cost did not track quality. GPT-6.1 Sol reached 75.6 record proximity at an estimated 0.75 per puzzle. Opus 5.5 reached 77.2 at 5.08, about seven times the cost for two more points. Fable 5.1 was the most expensive at 7.19 and scored 70.2. New frontier points concentrated on hard puzzles, the source said. They appeared on 14 hard puzzles, four medium puzzles and no easy ones. On puzzles with a cycle record of 200 or less, 88% of episodes passed and 4% found a new point. Above 500 cycles, 76% passed and 8% found one.

The source said the data did not separate Opus 5.5 and GPT-6 Astra completely. The closeness in record proximity was within uncertainty. The source also said model activity did not explain quality. GPT-6.1 Sol averaged 136 game calls per puzzle at 75.6 record proximity, while Gemini 4 Argon averaged 41 at 76.2. Five models got under the old record in-game. The other two cycle records were GPT-6.1 Sol's 3,460 on Intersection Identifier SPEC64160260, against 3,872, or 11% faster, and Gemini 4 Argon's 188 on Error: 8 SPEC79324720, against 198, or 5% faster. All three were on tournament puzzles, none in the main game. Of the 133 new frontier points, 18 beat a community record outright. The other 115 were new tradeoffs.

How Creators and Developers Could Use This

The overlap with Grafy is genuine. A creator exploring optimization puzzles could use Game Studio to turn a brief into a playable game. A developer who wants to test iterative AI coding loops could use Grafy Code. The story does not mean Grafy has a TIS-100 mode. It means the benchmark fits the broader questions about how AI tools handle constraints, feedback and tradeoffs.

Frequently asked questions

Does Opus 5.5 teach a smaller model TIS-100 optimization techniques?

The TIS-100 Bench did not show that. According to the source, Opus 5.5 was tested directly, and the write-up did not describe a teaching or transfer setup to a smaller model. The claim that it teaches is therefore not supported by this story. A smaller model might learn from its output in a future experiment, but that would require separate tests.

What counts as a new Pareto frontier point in the TIS-100 benchmark?

A new frontier point is one that no community solution ties or beats on all three scores at once. It can use more code to run faster, for example. The source said a point counts only if it also passes 100,000 tests and has a fresh-game replay.

Why did some TIS-100 LLMs fail to find new frontier points?

The source did not call the misses failures. It reported that 16 models were compared and that some solved all 55 puzzles while others solved fewer. Some found no new frontier points. The write-up suggested cost and activity did not cleanly explain quality, so context and puzzle difficulty matter.

---

Source: Hacker News — front page — LLMs Play Zachtronics' TIS-100