Researchers Unveil LegoGPT AI Model That Can Build Physically Stable Design of Lego Structures

LegoGPT, a new artificial intelligence (AI) model that can generate three-dimensional (3D) Lego structure designs, was unveiled by researchers recently. The new AI model is an open-source project to determine whether AI models can generate structures that are consistent with real-world physics and are consistently stable. The researchers have shared details of how the model was built, as well as have made the dataset available to the public domain. The AI-generated Lego structures were also tested by humans and robots to confirm the stability of the structures.

LegoGPT AI Model Was Built on LLaMA-3.2-Instruct

In a post, researchers from Carnegie Mellon University detailed the LegoGPT AI model. The large language model (LLM) can generate a Lego structure from a text prompt, ensuring it is physically stable and buildable. The open-source model is available to download and use on GitHub with a permissive MIT licence.

Users can prompt the model to design a “streamline elongated vessel” or a “backless bench with armrest,” and it can generate a design that not only matches the description but also can be placed upright without the structure collapsing.

This is possible due to two components that make LegoGPT — the base AI model and a stability analysis system. For the base model, the researchers used a fine-tuned version of the Llama-3.2-Instruct with one billion parameters. This was paired with Gurobi, a mathematical optimisation solver, that runs stability analysis for each generated structure.

Alongside building the refined architecture, the researchers also created a dataset to train the model on Lego structures. Dubbed StableText2Lego, it is a dataset containing more than 47,000 Lego structures of over 28,000 unique 3D objects. Each structure is accompanied by detailed captions, design code, and models.

To verify that the generated structures are indeed stable, the researchers also tested them with a dual robot assembly. The assembly was tasked with recreating the designs and testing whether they could stand upright. Some of the designs were also recreated by humans to see the impact on stability if less dextrous hands were involved. The research paper claims that 99.8 percent of all structures passed the stability test.