Facial Recognition: 34% Error for Dark-Skinned Women, 0.8% for Others
Mortgage algorithms deny Black applicants 80% more than white applicants with identical finances. AI systems trained on biased data bake discrimination into infrastructure at scale.
34%Face recognition error rate — dark-skinned women (MIT)
0.8%Error rate — light-skinned men
80%Higher mortgage denial rate for Black applicants
42×Disparity ratio in facial recognition accuracy
Choose your depth. The data doesn't change — just the explanation.
Computers are now making decisions that used to be made by people — like whether to give someone a home loan, or recognizing faces on camera. But these computer programs learned from old data that had discrimination baked in. So now the computers discriminate too, but faster and at larger scale. A face recognition program is wrong 34% of the time for dark-skinned women, but only wrong 0.8% of the time for light-skinned men. That's 42 times worse.
Algorithmic discrimination occurs when AI systems trained on historically biased data perpetuate or amplify those biases. The landmark MIT Media Lab study (Buolamwini & Gebru, 2018) tested three commercial facial recognition systems on a dataset balanced by gender and skin tone. Error rates for dark-skinned women averaged 34.7% vs 0.8% for light-skinned men — a 43× disparity. Mortgage algorithms: Bartlett et al. (2022) study of 9.5 million mortgage applications found Black applicants denied 80% more than white applicants with identical debt-to-income ratios, controlling for all financial variables.
Buolamwini & Gebru (2018) "Gender Shades" tested Microsoft, IBM, and Face++ APIs on Fitzpatrick scale skin tone × gender cells. Worst performer: 46.5% error on dark-skinned women. NIST FRVT evaluation (2019): most algorithms show 10-100× higher FMR for Black Americans than white Americans, independent of gender. Mortgage bias: Bartlett et al. (2022, NBER WP 25943) uses HMDA data, fintech vs. bank comparison — fintech shows smaller (but still significant) disparities, suggesting data-driven models reduce but don't eliminate bias. Criminal justice: COMPAS recidivism risk scores show 2× false positive rate for Black defendants (Angwin et al. ProPublica 2016), disputed by Northpointe (Dieterich et al. 2016). This launched the "fairness impossibility" literature showing multiple fairness criteria cannot simultaneously be satisfied (Chouldechova 2017, Kleinberg et al. 2017).
Buolamwini & Gebru (2018) FAccT: gendershades.org. NIST FRVT 2019: nvlabs.nist.gov/frvt/reports/facerec/performance-by-demographics.pdf. Bartlett et al. NBER WP 25943: nber.org/papers/w25943. ProPublica COMPAS: propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. HMDA mortgage data: ffiec.gov/hmda. EU AI Act prohibited use cases (biometric remote ID, social scoring): eur-lex.europa.eu/AI-Act. CFPB guidance on algorithmic lending fairness: consumerfinance.gov/data-research/research-reports/cfpb-report-algorithmic-fairness.
Bias by the Numbers
Facial Recognition Error Rate by Demographics
MIT Gender Shades Study — average error across 3 commercial APIs
Mortgage Denial Rate — Black vs. White Applicants
Controlling for income, debt-to-income ratio, and loan amount
Algorithmic Discrimination: Domains and Disparity Ratios
How much worse AI performs for minority groups vs. reference group
⚖️ The Fairness Impossibility Problem
Mathematicians proved something disturbing in 2016-2017: you cannot simultaneously satisfy all intuitive definitions of algorithmic fairness. If an algorithm is calibrated (equally accurate predictions for all groups), it cannot also have equal false positive/negative rates across groups with different base rates. This "fairness impossibility" means that fixing bias in one dimension often creates it in another. The implication: algorithmic bias isn't just a technical bug that can be patched — it requires explicit policy choices about which type of fairness to prioritize, made transparently by humans, not buried in code. Currently, almost none of these decisions are made transparently.