Can an AI agent look at a single photograph and rebuild it as an editable 3D scene? A new study by a team from the University of Maryland and Amazon Web Services says it can, up to a point. Their framework, LEGO-Anything, has a coding agent write and refine a Blender program until the scene matches the photo, and their benchmark shows where the approach still breaks down: geometric accuracy and the agent’s ability to judge its own work. The Decoder reported on the paper, which was posted to arXiv on 28 September 2026.
The researchers call the method Image-to-Code. Rather than producing the scene in a single step, the agent loops: write Blender code, execute it, look at the render, fix what is wrong. Because the output is a program, objects, geometry, layout and camera position are all explicit, so the scene can be edited, queried and rerun like any other code.
A benchmark built on simulator scenes
Real photographs have no exact 3D answer key, so the team built LEGO-Bench from professionally made simulator scenes. The set holds 208 images taken from 104 scenes, both interiors and exteriors, built with 443 registered assets. Each submission is scored on three counts: whether a usable scene was delivered at all, how accurate the visible geometry is, and how closely a re-render matches the original image.
All six GPT configurations tested delivered a working scene almost every time, but accuracy varied widely. GPT-6 Astra came out on top with 53.4 percent for interiors and 39.6 percent for exteriors, while weaker configurations scored around 15 percent. Accuracy fell as scenes became more complex. Giving the models a larger reasoning budget helped: on one office subset, Astra’s score rose from 32.3 to 61.8 percent.
Agents cannot tell if they are improving
The most telling result concerns self-assessment. Asked to pick the closer of two versions of a scene, the models’ geometric judgements were at or below chance. In one case Astra degraded its own scene late in the process, from 33.9 to 4.4 percent. The authors conclude that refinement should rely on measurements rather than the agent’s own opinion, and built LEGO-Plugin, a training-free add-on that anchors the starting scene, replaces self-judgement with concrete checks and protects correct progress. It improved all six models, with relative gains of up to 62.7 percent for weaker agents and about two points for the strongest.
For architects using Blender for massing and visualisation, the work points to a near future in which a site photo could become an editable base scene. The researchers are clear that this is not there yet: used for standard vision tasks, the reconstructed scenes reached roughly half the performance of a specialised detection model, and the gap was larger for segmentation and depth.
Leave a comment