Design + Build

Gold Set Tool

I conceived and built an internal tool for sorting, labeling, and QA-checking product photos used to train an AI image model.

This case study describes work done at Block, Inc. (Square). It covers process and design thinking only. Specific metrics, screenshots, and proprietary details are omitted per NDA. More information available upon request.

Role Design + Build
Year 2026
Type Internal Tooling
Internal Tooling AI Vision QA Workflow Cost Optimization

The Problem

The team maintains curated sets of product photos used to fine-tune an internal generative AI model. Each product line needs photos covering specific angles, finishes, and environments. When photos are missing from the set, the model produces worse results for those cases.

Photos arrive in bulk as unsorted folders. When I was asked to assemble a test set myself, I felt the pain firsthand: manually sorting hundreds of images, checking coverage by hand, no way to tell what was missing at a glance. So I built a tool to fix it. The first iteration used AI vision for sorting, but it was slow, expensive, and inconsistent. I iterated from there.

The Approach

The tool needs to do three things:

01

Sort incoming photos

Automatically place each uploaded photo into the correct product-line category (e.g., by finish color or material type).

02

Show coverage gaps

Display a matrix showing which categories have enough photos and which ones are missing coverage, without requiring the user to click through hundreds of images.

03

Enable human QA

Let a reviewer click through sorted images and approve or reject each one. Every image needs a human sign-off before it enters the final reference set.

How I Worked with AI

1
Optimize the AI pipeline

I rewrote the sorting pipeline: swapped to a smaller, cheaper vision model, reduced the image resolution sent to the API, and switched to parallel batch processing with structured JSON output. This cut costs significantly while improving consistency.

2
Fix inconsistent labels

The AI model would label the same finish differently across images. I added three layers of normalization: deterministic key sorting handles the obvious cases, a curated synonym table catches known variants, and a drag-to-merge gesture in the UI lets reviewers handle the rest.

3
Use AI only when needed

Most files already had the category in the filename (e.g., "PRD_BLK_01.png"). A regex extracts those first. The AI vision model only runs on genuinely ambiguous files. Reviewers only see low-confidence results. This hybrid approach handles about 60% of files without touching the AI at all.

  INCOMING FILES
        |
        v
  +-----------------------+
  | Filename Extractor    |    has finish in name?
  +-----------------------+
        |            \
       no           yes ---> DONE (~60% of files)
        |
        v
  +-----------------------+
  | Vision Model          |    confidence >= 0.7?
  | gpt-4o-mini, low-res  |
  +-----------------------+
        |            \
       low          yes ---> DONE (with badge)
        |
        v
  +-----------------------+
  | FLAGGED "?"           |    surfaced for human review
  +-----------------------+
4
Match the existing UI

This tool lives inside a larger asset management system. I fed screenshots of the existing UI to Claude alongside my design and asked it to identify styling differences. It found 14 small mismatches in spacing, color, and type. None were noticeable on their own, but fixing all of them made the tool feel native instead of bolted-on.

Design Decisions

Show gaps, don't fill them

The tool identifies what's missing from the photo set. Actually creating those photos is a creative decision that belongs to the design team.

Coverage matrix over per-image badges

Instead of putting status badges on every image, I put coverage information in a sidebar matrix. This keeps the photo grid clean and lets you see overall status at a glance.

Show where every label came from

Every label shows its source: extracted from filename, assigned by AI, or manually overridden by a reviewer. This transparency is what makes people trust the tool's output.

01Upload
02Sort + Review
03Export
↑
Drop your zip file here
01 Upload a zip of product photos
02 AI sorts by color and type
03 Review, fix, then export
01Upload
02Sort + Review
03Export
Categories
Black 24
White 18
Glow 12
Matte 9
Silver 6
Unsorted 3
Black (24 images)
[img]
PRD_BLK_01.png
ok
[img]
PRD_BLK_02.png
ok
[img]
PRD_BLK_03.png
fixed
[img]
PRD_BLK_04.png
ok
[img]
PRD_WHT_09.png
fixed
[img]
PRD_BLK_06.png
ok
[img]
IMG_4821.jpg
unsorted
[img]
PRD_BLK_08.png
ok
01Upload
02Sort + Review
03Export
Coverage
Black 23 of 25
92%
White 18 of 23
78%
Glow 9 of 14
64%
Matte 5 of 10
51%
Silver 3 of 9
35%
Summary 72 sorted / 3 unsorted / 5 categories
Copy CSV
Export ZIP

Reflection

Limit what the AI decides. A human makes the final call.

The AI labels photos, deterministic rules deduplicate categories, and the coverage matrix highlights gaps. But a person always approves the final result. I designed each layer to be cheaper and easier to verify than the one above it: filename extraction costs nothing, AI sorting costs a little, and human review is reserved for the cases that actually need judgment.

Related Project

I built a related open-source project that uses a similar pattern of automated analysis with human oversight, applied to code review: RoastMyRepo. Full source code available.

github.com/naeema-scopes/roastmyrepo