Build materials and UMG widgets with an AI agent in Unreal Engine

How to ask an editor agent for material variants, material graphs and UMG widget layouts, what it checks after each change, and prompts that get usable results.

6 min read

What the agent does here

Kismet’s agent works on materials and UMG Widget Blueprints inside the editor, with the same conversation it uses for gameplay logic. It edits the assets in place, so the result is an ordinary material or widget you keep editing by hand. It does not generate textures, images or meshes: it works with the assets already in your project.

Material variants: ask for an instance first

Most material requests are variants: a red version, a rougher one, a different texture. The right tool for those is a material instance. An instance reuses its parent’s compiled shader, so it is nearly free, and a project stays manageable with a few parent materials and many instances.

The agent reads the parent’s parameters, creates the instance and overrides scalar, vector, texture and switch parameters by name. It edits the parent’s graph only when no parameter exposes the change you need.

Prompt to try
Make three instances of M_Crate in /Game/Props/Materials: worn, clean and burnt. Use the existing Roughness and Tint parameters, and tell me the values you chose.

Editing a material graph

When the change does need the graph, the agent reads the material first: every expression node, what feeds each input, and which nodes connect to the root properties such as Base Color, Roughness and Normal. It can descend into material functions the same way. Then it adds expression nodes, sets their properties and connects them.

It can also change the material’s shading settings: blend mode, shading model, domain and whether it is two-sided.

After editing, the agent recompiles the material. The compile reports shader errors along with vertex and pixel instruction counts and the sampler count, so you see the cost of a change next to the change. Unreal’s limit is 16 samplers.

Prompt to try
Add a scalar parameter EmissiveStrength to M_Sign, multiply the existing emissive color by it, and compile. Report the pixel instruction count before and after.

UMG widgets: layout and events

A Widget Blueprint has two sides: the Designer tree of panels and controls, and an event graph. The agent works on both.

  • On the Designer side it reads the full hierarchy, including each widget’s slot layout: position, size and anchors in a Canvas Panel, padding and fill in a box, row and column in a grid.
  • It adds, moves and configures widgets in batches, then recompiles and reads the resulting tree back.
  • On the graph side it wires widget events, such as a button’s OnClicked, with the same tools it uses on any Blueprint.
  • A widget the graph refers to must be exposed as a variable. The agent can set that when it adds a widget, and the tree it reads shows which existing widgets are exposed.
Prompt to try
In WBP_PauseMenu, add a vertical box with Resume, Settings and Quit buttons under the title, 12 pixels apart. Wire Resume to remove the widget and unpause the game. Leave Settings and Quit unwired.

Getting better results

  • Name the asset and its folder. The agent can search the asset registry, but an exact path removes a guess.
  • Describe the result you want to see, with numbers where they matter: spacing, sizes, parameter ranges.
  • For anything that touches several assets, ask for a plan first. Plan mode explores read-only and presents the plan for your approval before any edit.
  • Work in steps. Ask for the layout, look at it, then ask for the events.
  • Put standing conventions, such as naming prefixes and where UI assets live, in an AGENTS.md file in the project root so you do not repeat them.

Limits to know

  • Material instances inherit their parent’s shaders and cannot be compiled on their own. Compile results come from the parent material.
  • From the Kismet panel, edits and new assets under /Game are allowed by default. Anything else is asked in the panel first.
  • A successful compile says the asset is valid, not that it looks right. Check the material on a mesh and the widget at your target resolutions.
  • Niagara effects are on the roadmap, not in the current plugin.

In the docs

Try it in your own project.

Kismet is free and open source. Install the plugin, connect a model, and ask for one small change.

Related guides