Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
Research Track · 20 representative scenes · 14 coding-agent configurations.
Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
Quality against the cost of one case · Research Track, 20 scenes
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
Research Track, 20 scenes. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical. Detailed and Overview are means of the per-case scores.
| # | Agent configuration | Score | Detailed | Overview | Physical | Cost / case |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 (max)ParetoAnthropic | 0.725 | 0.772 | 0.898 | $18.16 | |
| 2 | GPT-6 Astra (max)OpenAI | 0.707 | 0.733 | 0.712 | $22.47 | |
| 3 | Claude Opus 5 (max)Anthropic | 0.631 | 0.716 | 0.811 | $34.50 | |
| 4 | GPT-5.6 Sol (high)ParetoOpenAI | 0.632 | 0.710 | 0.772 | $4.07 | |
| 5 | Gemini 3.8 Flash (high)ParetoGoogle | 0.637 | 0.664 | 0.655 | $2.49 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.598 | 0.661 | 0.651 | $0.10 | |
| 7 | Grok 4.6 (high)xAI | 0.522 | 0.576 | 0.585 | $4.33 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.507 | 0.530 | 0.688 | $0.49 | |
| 9 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.505 | 0.460 | 0.694 | $0.33 | |
| 10 | GLM-5.3 Flash (max)open weightsZ.ai | 0.403 | 0.537 | 0.533 | $0.71 | |
| 11 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.412 | 0.416 | 0.694 | $0.03 | |
| 12 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.438 | 0.370 | 0.725 | $0.02 | |
| 13 | Inkling (high)Thinking Machines | 0.393 | 0.337 | 0.722 | $0.20 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.217 | 0.161 | 0.513 | $0.19 |
Click a column to sort. Pareto: on the score-against-cost frontier.
Drag any scene to inspect the saved 3D output, or open a case to explore every agent.
What the evaluation shows across the 14 agents. Every chart below is drawn from the paper's numbers; hover for values.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
@misc{ye2026code4scene,
title = {Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes},
author = {Xiaokang Ye and Siddhant Hitesh Mantri and Zimeng Chen and Edward Zhang and Zhaoxu Zheng and Yuanheng Li and Yizhao Chen and Tianyang Huang and Lianhui Qin},
year = {2026},
eprint = {2609.36777},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2609.36777},
}