Explainable & Auditable Alloy Defect Detection

A working model that explains itself

Upload a micrograph (or pick an example below). The model predicts the most likely defect class; the heatmap shows which pixels drove that prediction (Grad-CAM, implemented from first principles with PyTorch hooks, following the course's own notebook).

The classifier itself is deliberately ordinary — a compact 164k-parameter CNN, trained by the same recipe as the notebook: a stratified 70/10/10/10 split where the test images are never used to fit weights or pick a checkpoint. The notebook's run scores 94.1% on that test split; the separately trained checkpoint served here scores 95.6% on the same split — same recipe and same split, different training run. That is the point: the question is not whether a CNN can score well on NEU-DET (it can), but whether the explanation sitting on top of it means anything. Tabs 2 and 3 test exactly that.

We planted a shortcut. The obvious test missed it.

Grad-CAM tells you where the model looked, not whether what it found was real. So we planted a fake: a second model was trained from scratch on data where every image of one defect class (scratches) carries a crude synthetic "batch ID stamp" in the corner. Nothing about that stamp is metallurgical — it is exactly the kind of acquisition artifact (line ID, scale bar, microscope watermark) Problem #10 warns about.

The strategy is the course's: this is exercise 5 of the course Grad-CAM notebook — "create a challenge set and test for shortcut learning" — proposed there but left unbuilt. Building it out is our contribution.

Below: four images that are not scratches, with the stamp painted on at test time. Top row is what the model sees; bottom row is where it looked. (Each row's captions sit above its own images, so the Grad-CAM labels belong to the row beneath them.) The p= values are the model's scratches probability for images containing no scratch defect at all. The stamp pulls attention in every panel — which is what the third test below quantifies.

Strip the stamp off and recall stays at 100%. The model never became dependent on the artifact — it learned the real defect signal and the stamp in parallel. That is what makes this dangerous: the test almost anyone would run first (remove the suspicious variable, check whether performance drops) returns a completely clean bill of health.

What the artifact did do is become an independently sufficient cue. Painting it onto a different defect flips the model to scratches 15.6% of the time against a 0% baseline on identical unstamped images — and it captures 21× more Grad-CAM attention than its area warrants, firing on the stamp even in cases where the prediction did not flip.

So the honest statement to a plant manager isn't "the model is broken." It's: this model attends to something with no metallurgical meaning, it will act on it in about one case in six, and neither accuracy, per-class metrics, the heatmap, nor the obvious hold-out test can see it. Latent until a line re-labels, a microscope is swapped, or sample prep changes — which is why this belongs in periodic revalidation, not a one-time sign-off.

First: can we even tell whether the heatmap is faithful?

Tab 2 showed the explanation can be captured by an artifact we planted ourselves. Fair objection: real data has no planted stamp. So we tested the heatmap directly on clean images, using the course's own deletion test — delete the pixels Grad-CAM ranks most important and measure how much the prediction suffers, versus deleting the same number of random pixels.

It passes cleanly. At 30% of pixels replaced, Grad-CAM-guided deletion has cost 0.898 of predicted probability; random deletion has cost 0.016. The pixels the heatmap points to are the pixels the prediction actually depends on.

A note on method. An earlier version of this project used an 85/15 split and got the opposite verdict on this same test (0.546 versus 0.718). We therefore ran it properly — two split designs, three seeds each, six independent runs — and the separation was perfect: 3/3 unfavourable under 85/15, 3/3 favourable under the 70/10/10/10 split used here. Notably it is the random baseline that moves, not the attribution. The full analysis is in the notebook. We report the clean-split result because that design never lets test images influence weights or checkpoint selection — but the sensitivity is why we treat this curve as supporting evidence rather than proof, and why the guarantee below is what we would actually put in front of an auditor.

So what survives? Conformal Risk Control

Not covered anywhere in the course — taken from Shen & Liu, arXiv:2504.17721 (2025). It wraps around any existing detector and needs no retraining.

Why not just calibrate the confidence scores? Calibration says "when I output 0.9, I'm right about 90% of the time" — a description of average past behaviour, true only while conditions hold, and guaranteed by nothing. Conformal risk control says something categorically different: "the expected error rate on future batches is at most α" — a bound that is derived, not fitted. To an auditor those are not the same kind of evidence. (It carries preconditions, and we examine ours in the limits below rather than waving them through.)

How it works, in one paragraph. Hold back a set of images the model never trained on. Sweep a confidence bar across them and find the least strict bar that still keeps the error rate under your chosen budget α. Provided future images come from the same process as those held-back ones (exchangeability), a future image is just another draw from the same pool — so the bar that worked on them must work in expectation on it too. The guarantee is arithmetic, not optimism.

It was validated, not just asserted: empirical error stayed at or below target at six of the nine risk levels tested (α = 0.1 through 0.9). The three that came in over — α = 0.10, 0.20 and 0.90 — miss by 1.12, 0.55 and 0.83 standard errors on 180 test images. The bound is on expected error, so realisations scatter either side of the line; near-misses in both directions are what a correctly behaving bound looks like. An earlier version of this project reported nine out of nine, which looked stronger and was weaker — that model had been checkpoint-selected on the very images it was scored against.

What the plant actually gets

For each micrograph the method returns a prediction set — the defect classes clearing the calibrated bar. Its size is the instruction:

The plant picks α. That single dial trades throughput against escalation volume. Pick it at the strict end: α = 0.10 abstains on 12.8% of images, α = 0.90 on 91.7% — within budget and operationally useless. This, not the accuracy number and not the heatmap, is what we would put in front of an auditor.

Four honest limits

• The bound is an average, not a per-image promise. It constrains the error rate across future batches. It does not say this particular micrograph has a ≤α chance of being wrong, and must not be sold that way.
• Exchangeability is the whole vulnerability. A new production line, a replaced microscope, or changed sample prep breaks it silently — no error is raised, the number just quietly stops meaning what you think. Recalibrate on data from the new source.
• At lenient α the metric is dominated by abstentions. We adapted a method built for pixel-level segmentation, where a prediction set spans thousands of pixels and is essentially never empty. With six classes, empty sets are common at strict bars — 91.7% of images at the most lenient α — and an empty set is scored as a full error. So at that end the reported "false discovery rate" is very nearly a synonym for "abstention rate." The useful operating range here is the low-α end, where empty sets are far rarer (12.8%).
• A precondition of the theorem is not strictly met. Conformal Risk Control requires the loss to be monotone non-increasing — as you flag more, the risk must never climb back. False-negative rate satisfies that; false discovery rate does not. Ours is U-shaped: it falls from 0.949 to a minimum of 0.029 around λ ≈ 0.461, then climbs back to 0.755 as the flagged set fills with false positives — rising at 80 of 399 grid steps. Every threshold we actually selected (0.004–0.219) sits on the descending branch, where the condition effectively holds; the first rise is not until λ = 0.486. Capping the search grid at λ ≤ 0.48 would satisfy the precondition outright without changing a single reported number. But that is a property of this data, not a proof. It is why the chart above is load-bearing rather than decorative: with a precondition in question, checking the bound empirically on held-out data is doing real work.