The Time My AI Model Looked 90% Accurate and Was Actually 12%

I built a six-stage machine learning pipeline on 400,000+ consumer complaints from the CFPB (the federal agency that handles complaints about banks, credit bureaus, and lenders)— cleaning raw data, discovering hidden topics, extracting structured facts from free text, predicting outcomes, retrieving similar past cases, and finally, building an AI assistant that triages new complaints using everything the earlier stages learned.

The part of this project I'm proudest of isn't any of that. It's the moment I caught myself being wrong.

The setup

The last stage of the pipeline is a triage tool. Feed it a new complaint, and it pulls up similar historical complaints, then generates a predicted category, a suggested next step, and a confidence level. To know whether an automated prompt-optimization step actually improved this tool, I needed a way to score its predictions against the true category.

I wrote what looked like a reasonable check: does a meaningful word from the correct category show up in the model's prediction? Simple, fast, easy to defend.

I ran it. Baseline: 90% accurate. After optimization: 100% accurate.

That's the kind of result that goes straight into a slide deck. It's also, it turned out, almost entirely fake.

Where it fell apart

Before trusting that number, I looked at what the metric was actually rewarding. Half the complaint categories in this dataset contain the word "report" — "Incorrect information on your report," "Improper use of your report," and so on. My keyword check would pass any prediction that used the word "report," regardless of whether it named the right issue at all.

I checked directly: 9 of 10 baseline predictions and all 10 optimized predictions contained the word "report." My "accuracy" metric was mostly measuring whether the model could say a common English word, not whether it understood the complaint.

Fixing it, and fixing it again

My first fix was blunt — strip out generic words and only count meaningful overlap. The number swung hard: 0%, then 40%. That swing was too big to trust either. Digging into individual predictions showed something more specific: after optimization, the model had started copying the government's exact category names verbatim, pulled straight from its training examples. That's a real behavior change, but it's closer to memorization than understanding — the model was also confusing the two largest categories with each other in both directions.

So I built something better: a metric that checks for actual meaning per category, not just any shared word, and I scaled up the test set from 10 examples to 25 so a single lucky guess couldn't swing the whole result.

The real number: baseline 12% accurate, optimized 36% accurate.

Not the headline-grabbing 90%. But it's a number I can defend, and digging into why the optimized version did better turned up something genuinely interesting: the unoptimized model kept guessing "identity theft" for complaints that had nothing to do with identity theft — a bias that likely came from fraud-related language showing up often in the similar past cases it was shown. The optimized version showed that bias less. That's a real, useful finding. The 90% number would never have surfaced it, because the 90% number wasn't measuring anything real to begin with.

The part that generalizes

I don't think the specific lesson here is "keyword-matching metrics are bad." The lesson is bigger than that: any measurement can look reasonable at first glance and still not measure what you think it measures. And the only way to find out is to stop trusting the aggregate number and go read the actual failures.

That's true whether you're grading an AI model, reading a quarterly report, or deciding whether a new process is "working" at your job. A good-looking summary statistic is a starting point for questions, not an ending point for conclusions.

What the whole pipeline does, briefly

For context on the rest of the project: it starts with cleaning a massive, messy government dataset (finding, along the way, that nearly half the complaint narratives were exact duplicates of a handful of copy-pasted templates). It moves through unsupervised topic discovery, structured fact extraction from free text, and a comparison of five different modeling approaches — from simple structured features all the way up to a fine-tuned transformer — to see how much predictive lift each one actually earns. It builds a retrieval system that finds genuinely similar past complaints (validated at 86% accuracy on held-out test cases). And it ends with the triage tool described above.

Full code for all six stages is on GitHub, built with an AI coding assistant doing most of the implementation — my role throughout was deciding what to check, catching what didn't hold up, and making sure the final numbers were ones I could actually stand behind. That last part is the whole point.