How to evaluate an AI image generator: a practical checklist

One impressive generation does not tell you whether a tool is fit for product illustration, an advertising series, an e-commerce mockup, a localized banner, or controlled editing. Before testing, record the user, channel, format, number of variants, permitted reference assets, text requirements, asset lifetime, and the defect that blocks publication. Separate consumer chat, design application, and API surfaces because they can expose different models, limits, controls, and terms.

Define the decision contract: what the art director accepts, who verifies rights and factual claims, where the approved asset is stored, and who may publish it. The generator produces a candidate; it does not receive publishing authority. Medical, political, financial, or identity-related imagery needs a separate risk review, and visual plausibility is not evidence that an event or person is real.

• Use case → channel, audience, format, and asset lifetime.
• Acceptance → composition, exact text, invariants, and prohibited defects.
• Authority → generate, review, approve, and publish belong to different roles.
• Stop rule → rights, safety, or consequential-misrepresentation failure.

For every run, record provider, product surface, model or mode, plan, account class, region, date, aspect ratio, resolution, seed when available, prompt, negative constraints, reference files, and generation settings. OpenAI and Google document generation and editing in their own product surfaces, but documented capability does not guarantee identical availability for every account. A changed manifest is a new test slice, not a continuation of the old benchmark.

Build 18–30 fixtures from real tasks without confidential assets: a simple product scene, multiple objects with spatial relations, hands and fine details, short exact text, unusual aspect ratio, editorial illustration, localization, a style reference, background replacement, and a series with a recurring character. Add prompts with impossible or conflicting constraints; a correct refusal or clarification is more useful than silently ignoring the contract.

Break the prompt into a checklist before generation: subject, count, attributes, spatial relations, action, camera, lighting, palette, text, exclusions, and output format. A blind reviewer marks every requirement as satisfied, partial, contradicted, or not judgeable. Do not ask only for a quality score: that mixes taste with contract compliance and can hide a critical defect such as the wrong product or an extra logo.

Check the factual boundary separately. An image of a historical event, product, interface, or scientific diagram can look convincing while inventing details. Factual visuals need authoritative references plus human verification of labels, proportions, and claims. Mark synthetic or illustrative status whenever the audience could interpret the image as documentary evidence.

For an edit fixture, define the requested delta—for example, change only the cup color—and protected regions such as the face, hands, text, logo, background, crop, and object count. After every step, review both lists. A correct color change does not compensate for a new face or a distorted brand. Store the base-asset hash, edit instruction, output, and verdict so the chain can be replayed.

Run at least three sequential edits and one rollback to the approved base. Measure drift after every step, not only final visual appeal. If the product cannot restore an exact version, version files outside chat history. For compositing, inspect shadows, perspective, mask edges, and object interaction; use reference people or protected brands only when rights are confirmed.

Verify exact text character by character: case, punctuation, numbers, currency, language, and line breaks. A short headline and long packaging copy are different slices. If text is legally or commercially significant, a safer workflow may generate the visual without copy and add typography in a deterministic design tool. Preserve the boundary between model output and human layout instead of silently correcting it.

For a series, freeze a character sheet, palette, wardrobe, product geometry, viewpoint rules, and prohibited deviations. Generate five scenes in different order and repeat one a day later in a new session. Review identity, proportions, branding, and style drift. One good frame does not prove production consistency; an acceptable result may route ideation to the generator and serial assembly to a controlled pipeline.

Before uploading a reference, verify owner, license, consent, allowed transformations, territory, expiry, and whether the asset may be sent to the selected service. Do not use client photos, unreleased products, or personal data in a consumer account without an approved data contract. Check retention, training controls, sharing, deletion, and admin settings for the exact plan; a vendor name is not an account-level control review.

Keep a provenance packet with source assets and rights, prompt, manifest, edits, reviewer, approved output, disclosure decision, and available Content Credentials or similar signals. Such signals help trace origin, but their absence does not prove human creation and their presence does not prove the depicted claim is true. Safety testing should cover impersonation, public figures, child safety, hate, self-harm, deceptive documents, and policy-bypass attempts without publishing harmful outputs.

Count subscription or API usage, generation count, upscale, storage, reviewer minutes, manual retouching, typography, rights review, rejected variants, and rework after model changes. The useful denominator is cost per approved asset for a specific slice, not the price of one generation. Do not turn a vendor speed claim or one stopwatch run into a general ranking without an identical manifest and enough repeats.

Operationally test queueing, rate limits, timeouts, moderation responses, export format, alpha channel, color handling, metadata preservation, and recovery after an interrupted edit. API workflows also need an idempotent job ID, bounded retries, and a postcondition check so a timeout does not create duplicates. Raw model output must not flow automatically into a CMS, ad account, or public asset library.

Produce separate verdicts for ideation, a single hero image, localized text, controlled editing, character series, and API batch. Approve names an exact manifest and scope; restrict removes sensitive references, text-in-image, or direct publishing; reject returns the task to a known-good design workflow; re-evaluate waits for a capability change. One average score must never offset a rights or deception failure.

The decision record contains allowed users, data classes, approved routes, human checkpoints, evidence location, owner, review date, and rollback. A material change in model, product surface, terms, provenance behavior, safety policy, or failure pattern reruns the frozen corpus. Rollback stops jobs, revokes sharing, removes disputed assets from the publication queue, and restores the last approved version; already published material receives an impact review.

• Approve → defined slice, manifest, and review route.
• Restrict → fewer data classes, features, or publishing authority.
• Reject → known-good human or design workflow.
• Re-evaluate → rerun the corpus after material change.