What Is AI Bias and How Do Companies Reduce It?

An AI tool that screens resumes, approves loans or flags fraud can be accurate on average and still treat some groups of people unfairly. That gap between “works well overall” and “works fairly for everyone” is what people mean by AI bias, and it is now a legal, reputational and product-quality issue for any company that deploys AI.

This guide explains what AI bias is, where it creeps into a model, a few well-documented examples, and the concrete testing steps teams can run before launch.

What AI bias means

AI bias is a systematic error that produces unfair outcomes for particular people or groups. “Systematic” is the key word. Every model makes mistakes; biased models make more of them, or different kinds of them, for some groups than for others.

The National Institute of Standards and Technology (NIST) describes AI bias in 3 broad categories in its Special Publication 1270 (March 2022):

• Systemic bias: Patterns built into institutions and society that show up in historical data, such as past hiring or lending decisions.
• Statistical and computational bias: Errors from unrepresentative samples, missing data, poor proxies or modeling choices.
• Human bias: How people design, label, interpret and act on AI outputs, including automation bias (trusting the machine too much).

NIST’s point is that bias is not only a data problem. It is socio-technical: it comes from the data, the model, the people around it and the context it is used in. NIST also notes that zero risk of bias is not a realistic goal; the aim is to identify, measure and manage it.

Where bias enters an AI system

Bias can slip in at every stage of the AI lifecycle:

Generative AI adds its own forms: image generators that default to stereotypes for certain jobs, or chatbots that respond differently depending on the name or dialect in a prompt. These outputs can be subtle, so they need deliberate testing too. Related reliability issues are covered in our guide to AI hallucinations.

Real examples of AI bias

A few well-documented cases show how bias plays out:

• Face recognition. NIST’s 2019 study of face recognition algorithms (NISTIR 8280) tested 189 algorithms from 99 developers and found demographic differences in error rates for most of them, with false positive rates varying by factors of 10 to 100 across groups for many algorithms. It also found wide variation between vendors: the most accurate algorithms showed much smaller differences.
• Hiring tools. Resume screening models trained on a company’s past hires can learn to favor the profiles of people hired before, reproducing historical imbalances. This is why New York City’s Local Law 144 has required bias audits of automated employment decision tools used for NYC jobs since July 5, 2023.
• Lending and pricing. Models can produce disparate outcomes when they rely on variables correlated with protected characteristics, which is why fair lending laws still apply to algorithmic credit decisions. See our explainer on how AI is used in loan decisions.

Key takeaway: A model can be accurate overall and still unfair. You only find bias by measuring results separately for the groups your system affects.

How to measure fairness

There is no single fairness metric, and some metrics conflict with each other. Teams typically choose a few based on the use case:

• Selection rate or impact ratio. Compares the share of each group receiving a positive outcome (hired, approved). In US employment, the “four-fifths rule” from the federal Uniform Guidelines on Employee Selection Procedures treats a selection rate for one group below 80% of the highest group’s rate as general evidence of adverse impact, though it is a rule of thumb rather than a legal bright line. NYC’s bias audits report impact ratios by sex and race or ethnicity.
• Error rate parity. Compares false positive and false negative rates across groups. Useful for fraud, security and medical screening tools.
• Calibration. Checks whether a score of 0.7 means the same probability of the outcome for every group.
• Performance by subgroup. Accuracy, precision and recall broken out by group, including intersections (for example, older women).

The right choice depends on the harm you are trying to prevent. Document why you chose each metric; regulators and auditors will ask.

Testing steps to run before launch

Here is a practical pre-launch process that works for most business AI systems, whether you built the model or bought it:

1. Define the decision and the people affected. Write down what the system decides or recommends, who is affected, and what a harmful error looks like for them.
2. Set fairness goals and metrics. Pick 2 or 3 metrics from the list above and agree on acceptable thresholds before you see the results.
3. Audit the training and test data. Check representation by relevant group, missing data, label quality and obvious proxies. Note where demographic data is unavailable and how you will test anyway (for example, with a carefully governed test set).
4. Test performance by subgroup. Run the metrics for each group and key intersections. Look at real example outputs, not just numbers.
5. Probe for proxies and edge cases. Remove or shuffle suspected proxy variables to see how results change. For generative AI, run paired prompts that differ only in name, gender, age or dialect and compare responses.
6. Mitigate and retest. Options include rebalancing or augmenting data, removing problematic features, adjusting decision thresholds, constraining the model during training, or narrowing the use case. Retest after every change.
7. Design human oversight. Decide when a person reviews outputs, give them the authority to override, and train them on automation bias.
8. Document and approve. Record the data, tests, results, limitations and sign-off in a model card or impact assessment.
9. Monitor after launch. Track the same fairness metrics on live data, watch for drift, and give affected people a way to appeal or report problems.

If you buy AI rather than build it, ask vendors for their bias testing results, the data used, subgroup performance and audit reports, and run your own tests on your population.

Laws and frameworks that address AI bias

There is no single US AI bias law, but several rules apply. Existing anti-discrimination laws in employment, lending, housing and insurance cover decisions made with AI. New York City’s Local Law 144 requires annual bias audits for automated hiring tools. Several states have passed AI laws covering consequential decisions, summarized in our guide to state AI laws. Companies selling into Europe should read what the EU AI Act means for US companies, since high-risk systems there face data governance and bias requirements.

The voluntary NIST AI Risk Management Framework is the most widely used US reference for managing bias as part of broader AI risk. This is not legal advice.

Where to start

If your company is just beginning, take 3 steps this quarter:

1. Inventory AI that affects people. List every tool that influences hiring, pay, credit, pricing, access to services or customer treatment. Our AI governance checklist shows how to build that inventory.
2. Rank by risk. Start testing the tools that make or shape consequential decisions about individuals.
3. Assign owners. Every high-risk system needs a named owner responsible for fairness testing and monitoring.

Good data governance makes all of this easier, because you cannot test what you cannot trace.

Conclusion

AI bias is not a rare glitch. It is a predictable result of training systems on imperfect data and deploying them in an unequal world. The fix is not a single tool but a habit: define what fair means for your use case, measure results group by group before launch, keep people in the loop and keep monitoring. Companies that build that habit reduce legal risk and ship better products.

Sources

• NIST SP 1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (March 2022) — systemic, statistical and human bias categories
• NISTIR 8280: Face Recognition Vendor Test Part 3, Demographic Effects (December 2019) — demographic differences in face recognition error rates
• NYC Department of Consumer and Worker Protection: Automated Employment Decision Tools FAQ — Local Law 144 bias audit requirements
• eCFR: 29 CFR 1607.4, Uniform Guidelines on Employee Selection Procedures — the four-fifths rule of thumb for adverse impact
• NIST AI Risk Management Framework — voluntary framework for managing AI risks, including bias