Agent or operator: twelve unlabelled variants and a rule written in advance

Same brief, same six reference images, two ways to make the screens: an AI agent on its own, or the design tool's own model with me at the controls. I scored twelve unlabelled variants before I knew which was which.

Twelve screen variants, each marked only with a letter, each scored on its own. The two I rated highest both came from the agent. I found out only after I had handed in my scores and opened the key, and the first thing I felt was relief.

I'm building a system that helps generate interface screens in a product's own style. I had designed it around one assumption: a person sits at an AI design tool, pastes in a package the system prepared (a written direction, reference images, a brief) and starts the generation. Then I read the tool's instructions closely. They describe a supported mode in which an outside AI agent writes the screens itself, as code, following the same instructions the tool gives its own model.

Relief, because if that worked, the last manual step in the pipeline would go away. If it didn't, I'd be automating my way to worse screens. So I wanted evidence before switching, and a rule for reading it.

The rule came before the screens

Before generating anything, I wrote down when the agent would take over. Its average across both scores could be at most half a point below the operator path's on a five-point scale. Each screen needed at least one variant I would take further. No variant could copy a reference image. And it had to cost me clearly less time. If any condition failed, the operator path would stay the default and the agent would be an option.

The package was fictional: a neobank feature for couples saving towards a shared goal, in a warm, homely mood with a bank's clarity. It came with six reference images and two screens to design, an invitation for a partner and a savings goal, at least three variants each. In the agent path, the agent wrote the screens and the tool only stored and displayed them. In the operator path, the tool's own model generated them and I operated it, pasting the same brief and images into its interface.

How the two paths worked

The generator is Claude Design. Agent path: Claude Code (Opus) running headless on my Mac, connected to the tool through its MCP server. It fetched the tool's own system prompt and its hi-fi design skill, wrote the screens as HTML and saved them to the project with write_files. The tool's model generated nothing on this path.
Agent run: 6.2 minutes, 33 turns, $2.04, one self-initiated round of fixes for non-text contrast (WCAG 1.4.11).
Operator path: the same prompt pasted into the Claude Design UI with the six images attached; about seven minutes, no corrections.
The inputs matched but travelled differently: the agent read all six references locally before generating and did not upload them, while in the UI they went into the project and to the tool's model.
Output on each side: the partner invitation in three variants, each showing both the inviter's and the invitee's screen, and the savings goal in three variants.

Both paths took about seven minutes. The agent ran one round of fixes on its own, raising the contrast of graphic elements for accessibility; the operator path got none. The agent's run cost $2.04; the operator path was covered by the subscription.

4.17 against 3.50

Separate agents, not the one under test, cropped the frames, stripped anything that gave the path away, shuffled them and later added up the averages. I gave each variant two scores from 1 to 5, for holding the written direction and for craft, marked whether I'd take it further, and decided what the numbers meant for the system.

On direction the agent averaged 4.17 against the operator path's 3.50. On craft it trailed slightly, 3.50 against 3.67. It produced variants worth taking further on both screens, four in total; the operator path produced three. Neither path copied a reference. My time: zero on the agent path, a few minutes of pasting and uploading on the operator path. The rule was met on every count, so I made the agent the default.

Blinding and all twelve scores

Frames were rendered and cropped automatically by a headless browser, so no path label or interface chrome was visible. Letters K–Z were assigned at random, and the key stayed outside the repository until scoring was done.
Why only half-blind: I had seen the operator path's output in the UI, and the paths differ in small signatures. The goal amounts were 8,000 and 9,000 zloty (both invented, since the brief gave none), and one path drew a phone frame with the camera island, the other without.
Direction / craft, agent path. Invitation: 3/3, 4/3, 5/4. Goal: 5/4, 4/3, 4/4.
Direction / craft, operator path. Invitation: 3/3, 3/3, 4/4. Goal: 4/4, 4/4, 3/4.
Averages: direction 4.17 vs 3.50, craft 3.50 vs 3.67, combined 3.83 vs 3.58 (the rule's floor for the agent: 3.08). Variants worth taking further: agent 1 invitation + 3 goal, operator 1 + 2. Copies of references: 0 and 0.

I had expected it to win on speed and lose on quality. Speed was a draw, and the craft gap was under a fifth of a point. It pulled ahead on what this system is for: holding a mood described in words. And it did that blind to its own work. It wrote code and never looked at a single rendered screen.

What the scores can't carry

The rater was me, and I scored half-blind: I had seen the operator path's results, and the two paths left small fingerprints, such as invented savings targets a thousand złoty apart. One brief, six variants per side. I treat it as a lead to follow up: the next round uses a second brief and a rater who has seen neither path.

The part I'd reuse doesn't depend on the scores. Had I seen 4.17 against 3.50 before writing the rule, I would have read it whichever way I already leaned. Next time an agent is up for a step a person does today, I'll again write down first how much worse it may be and what I expect in return.

For now, default means new packages go to the agent first, and the interface stays for when I want to steer by hand.

One moment from the trace stayed with me longer than any score. Twice, the agent called a tool my standing rules forbid. It hadn't broken anything open: the block list I had set up for that run was incomplete, and the agent used what it was given. That was a cold shower: an agent will do whatever you haven't explicitly ruled out, so the list has to be declared before every run and checked after it, and that check comes out of the time I counted as zero. Default, yes. Alone, not yet. Not until it can see its own screens and the list holds.

What has to change before it runs unattended

Renders: the preview tool returns only a tokenised URL that must not be printed, and a local headless browser needed a permission the agent's session didn't have. The fix is a screenshot script allowed in that session.
Block list: the agent called create_support_js twice. My standing rules block it, but the list configured for this run left it out. Blocks need declaring before a run and checking after it.
Paper trail: the decision is recorded as D53; the protocol, the rating sheet and the screenshots sit with the pilot package.

Drafted with AI agents, edited and fact-checked by me.