Pass, fail, unknown: designing honest results in a scanner

When you build a tool that checks things, the instinct is to produce two outcomes: pass or fail. Green tick or red cross. It makes for a clean interface.

It also makes for lying software.

Where it breaks

My scanner checks whether Shopify product images meet Google's minimum size. To do that, it reads each image's dimensions from the Shopify API.

Sometimes those dimensions simply aren't there. An image may still be processing after an upload. Processing may have failed. In some cases the image isn't hosted by Shopify at all, and the values come back empty.

If I forced those into pass/fail, I'd have two bad options:

• Mark them as pass. The merchant gets a clean report and a nasty surprise later.
• Mark them as fail. The merchant is told to fix something that may be perfectly fine, and loses trust in every other result.

The third state

So there is a third outcome: unknown, with a reason attached. Still processing. Processing failed. Not hosted by Shopify.

That one decision changes how the whole tool feels:

• Pass means pass. If it says an image is fine, I actually measured it.
• Fail means fail. Every red item is a real, measured problem.
• Unknown tells you what to do next. Wait a minute and rescan, or look at the failed upload.

A few related rules I follow

Store raw values, derive verdicts later. I save the measured width and height, and work out the status when displaying. Google's rule changes on a date, so the same data needs to produce different verdicts before and after. Storing the verdict would bake in a rule that expires.

Rank by consequence. A failing main product image is likely to get a product disapproved. A failing extra image is less urgent. Sorting issues by severity means merchants fix the expensive ones first.

Explain every flag. A red mark with no explanation is homework. Each result in the scanner says why it was flagged and what to do about it.

Say what you checked. The scanner reports on the images in your Shopify catalogue. It does not claim to know exactly what a particular Google feed app will send, because that depends on the setup. Narrow, accurate claims beat broad, shaky ones.

The general lesson

Diagnostic tools earn trust by being precise about their own limits. An honest "I can't tell" is worth more than a confident wrong answer, because it keeps every other answer believable.

I take the same approach in client work. When a store has a performance problem, I'd rather report "this is measured, this is an estimate, this I haven't tested yet" than hand over a tidy story that falls apart on contact.

This is part of the thinking behind Storko Product Health Scanner.