Organisations run bias evaluations, publish the results, and users continue reporting biased behaviour the evaluation did not find. The gap is methodological rather than dishonest.
What an audit actually measures
A bias audit compares outcomes across defined groups on a fixed set of test cases, checking whether the system treats them differently.
That requires the groups to be specified in advance and the test cases to be constructed, which means the audit can only find the disparities someone anticipated.
Categories nobody listed produce no measurement, and the audit reports no disparity where it simply did not look.
Why aggregates hide individual harm
An audit reports averages across many cases, and averages tolerate a substantial number of bad outcomes without moving.
A system can be balanced overall and still fail consistently on a specific combination of attributes that is rare in the test set and common in someone's life.
The person experiencing that failure encounters it every time, which is a different reality from the one the summary statistic describes.
How test sets diverge from use
Evaluation cases are constructed to isolate the variable being tested, so they are clean, short and unambiguous.
Real requests arrive with context, informal phrasing, mixed languages and details that interact with each other in ways a controlled case removes deliberately.
Bias frequently emerges from those interactions rather than from any single attribute, which is exactly the structure a clean test set is designed to eliminate.
Why the reported metric shapes the fix
Once a specific fairness metric becomes the target, engineering effort concentrates on moving it, and there are several ways to move a metric.
Some of those genuinely improve behaviour. Others adjust surface responses on cases resembling the test set while leaving the underlying tendency untouched.
The second kind is difficult to distinguish from the first using the same metric that motivated the change, which is a structural weakness of any published target.
What complaint data supplies that audits cannot
User reports find the cases nobody thought to construct, and they arrive with the context that made the failure matter.
They are unrepresentative, since only some users report and reporting requires effort, so they cannot replace systematic testing.
The arrangement that works treats audits as coverage of known risks and complaints as discovery of unknown ones, with a standing process for turning a recurring complaint into a new permanent test case.