AI / Data / SoftwareMedium (months)Detectability: Moderate
Anomaly detection trained on “normal” that no longer existed
A manufacturing anomaly detector was trained on pre-upgrade process data and deployed after equipment changes.
Decision summary
- Year
- 2021
- Failure mode
- Concept drift hidden by superficial equivalence: “same process, better sensors” was not the same process.
- Silent failure window
- 3–5 months: false negatives increased while operators became desensitized by intermittent false positives.
The original logic
The model reduced false alarms in the old regime, and the upgrade was considered “equivalent with better sensors,” so the detector was reused to avoid re-validation delays.
Key assumptions
- The upgraded process remained statistically comparable to the training regime.Confidence at decision: MediumExpected lifetime: 1–3 months
- Operators would escalate edge-case alarms for review.Confidence at decision: LowExpected lifetime: Weeks
What changed
New sensors changed measurement resolution and noise characteristics. The process mean shifted slightly, but the bigger change was variance and correlation structure—making the old “normal” definition obsolete.
Outcome
Quality escapes increased, with a late discovery that the detector was systematically under-reporting anomalies in specific product variants.
Early warning signals (missed)
- Feature distribution drift (variance/correlation) post-upgrade
- Alarm acknowledgement patterns trending toward “auto-dismiss” behavior
- Variant-level defect rates rising while global KPIs stayed flat
How AssureAI would have helped
- Assumption registry: “training regime remains valid” expires after equipment change unless re-validated.
- Operational signals: acknowledgement friction and dismissal rates tracked as leading indicators.
- Decision record: equipment change triggers “review due” for all dependent models.
Non-obvious lessons
- Sensors are part of the system; changing them changes the system.
- When alarms become noise, silence becomes dangerous.
- Variant-level monitoring often reveals drift first.