Hangar Harness / Model Tests Hangar Harness / Model Tests I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in /goal mode. Prompt: Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript. Model ↕ Harness ↕ File ↕ Duration ↕ TTFT ↕ Input tokens ↕ Output tokens ↕ Reasoning tokens ↕ Total tokens ↕ Cached input % ↕ Tool calls ↕ Tool errors ↕ Opened in browser ↕ Checked screenshots ↕ GLM 5.3 Flash Max Codex Open 9m 0.232s 9.344s 457,685 17,458 7,694 475,143 85.26% 14 1 Blocked No Luna 5.6 Max Codex Open 9m 13.098s 6.199s 1,146,755 25,512 6,879 1,172,267 92.76% 32 2 Yes Yes SOL 5.6 Max Codex Open 10m 48.765s 8.460s 1,069,163 28,278 7,515 1,097,441 94.60% 21 5 No No Astra 6.0 Max Codex Open 37m 29.705s 3.589s 1,292,366 43,129 15,271 1,335,495 94.93% 20 5 Yes No GLM 5.3 Flash Max OMP Open 30m 14.979s 6.596s 1,678,509 63,405 — 1,741,914 78.54% 57 0 Yes Yes Qwen 3.8 27B x-high OMP Open 41m 25.836s 15.275s 3,407,451 71,935 49,131 3,479,386 86.92% 89 0 Yes Yes GLM 5.3 Flash Max OpenCode Open 20m 28.948s 6.284s 4,305,447 50,468 33,414 4,355,915 96.89% 67 0 Yes Yes Qwen 3.8 27B x-high OpenCode Open 8m 48.470s 10.182s 665,490 41,817 28,788 707,307 95.64% 13 0 Yes Yes Qwen 3.8 27B x-high DSH / PTC Open 24m 32.929s 7.724s 1,012,499 89,894 — 1,102,393 91.69% 23 5 Yes Yes Qwen 3.8 27B x-high DSH Open 18m 15.239s 6.755s 2,654,457 78,232 — 2,732,689 95.48% 42 2 No No All GLM runs are labelled GLM 5.3 Flash Max ; input tokens include cached input. Output tokens are the generated total, including reasoning; when a harness reports reasoning separately, the reasoning column shows that subset. The DSH adapter does not report a separate reasoning count. DSH durations sum active turn time across both turns, excluding the pause between turns. Tool errors are recorded failed tool events. A dash means unavailable or not reported.
The Astra version seems to have used three.js r170, which is from October 2024. Sol used an even earlier version. GLM's code used the latest version, but I think it's just getting three.js@latest from jsdelivr so it's unlikely to be writing code against that version. Qwen on OpenCode also fetches from jsdelivr, but using a pinned version at r160.
I don't think any of these examples are using things like tone mapping so they're stuck in sRGB (AgX or ACES look much better), they're not using the node materials (good for programmatic texture implementation), and they're not doing anything cool like baking shadow environments or using post-processing effect.
They're nice, but I think they're showing how far behind AI models are on this sort of project rather than how good they are.
It’s really interesting how much little choices make the result better or worse. Astra and one of the GLMs added bright lights, and thus looked so much better to my eye.
Genuinely happy with some of the Qwen 3.8 results (especially since I can run that model at Q8).
Interesting to see how much better (at this task) Pi (OMP) is over Opencode as a harness.
I’d love to see a few more with outcomes that are as easy to judge but less subjective.
I’ve got a toy project going to make a fun to watch battle simulator where an LLM (or two if playing vs) has to write programs that control multiple bots (each with their own line of sight and limited battle context) that have to coordinate and fight alongside each other. Goal is to have the LLM update the code based on current situations maybe 5-10 times in a 5 min simulated battle. Exploring even allow the bots to request new programming and score based on number of reprogram steps.
I wish there was another column with the estimated cost for each, with a specific date.
Ideally also finding somehow (not sure what would be the right away) what is publicly available before running the test. It's quite a different outcome if there are competitions, e.g. js13k, live code examples from books, even templates, on specific that topic. Visually here the results looks very very similar to the point that I can't help but wonder if it's the result from the short yet relatively descriptive prompt or because some template was always found and relied on.
Nice. But these tests raise a question what results are reproducible, the final visual, time, tool calling or it's mostly noise. Like, have you tried same combination or model and harness multiple times?
Btw in my quest for a good desktop codex/claude like app - https://github.com/openchamber/openchamber - this seems to be front-runner. I pair it with OMP via https://github.com/alvins82/omp-openchamber-server/ .
I wanted a powerful GUI+harness setup for open models so I could use/test as they came out.
Should I be able to see the actual results? Because I don’t see a hangar anywhere.
Checkout https://threejseval.com
I have been building something like this with human voting, elo and ranking on multiple scenes and many models.
I love everything about OpenCode except for the stuff it outputs. On paper it has everything I want in a harness and more, but it tends to struggle to deliver the desired outcome. Last I used it the context was massive and tool calls were reliably unreliable. Is it worth revisiting as a daily driver?
I saw examples on X of people building amazing things with Three.js and models, and I figured out how to build them myself. But the results are mostly low-poly materials and crappy animations (pretty sure because I am not the best at these things). I thought it was my prompt (it was detailed), but it seems to be mostly a limitation of the models.
From all the examples I've seen, Astra does it really well, and I suspect it's because they wanted to attract game designers, so they trained the model more on 3D, animation libraries, etc.
Would be good to include Fable and Opus. Based on the results with Astra vs GLM (not even in the same ballpark), these are probably the realistic alternatives to consider.