AI bias audits for hiring tools: how to read one and what to ask

Impact ratios, sample sizes, intersectional groups and the gaps an audit can't cover, explained for recruiting leaders.

The short answer

An AI bias audit is an independent check of whether a hiring tool selects or scores some demographic groups at lower rates than others. Under New York City's Local Law 144, the minimum version compares selection or scoring rates across sex, race/ethnicity and intersectional groups and reports an "impact ratio" for each. A good audit tells you whether a disparity showed up in real use. It does not tell you whether the tool measures job skills well, and it can't make a biased process fair on its own. Read the date, the data source, the sample sizes and the lowest impact ratio before you trust the headline.

This is not legal advice. Laws on AI in hiring differ by city, state and country, and the ones we cite are linked at the end.

What an AI bias audit actually measures

Most hiring-tool audits follow the same arithmetic, which goes back to the federal Uniform Guidelines on Employee Selection Procedures. New York City's rules for Local Law 144 say their required calculations are consistent with that framework.

Split candidates into groups by sex, by race/ethnicity, and by the combination of the two (for example, Black women or Asian men).
Work out each group's rate. For a pass/fail tool, that's the selection rate: the share of the group the tool advanced. For a tool that outputs a score, the city's rules use a scoring rate: the share of the group scoring above the median of the whole sample.
Divide by the top group's rate. The impact ratio is each group's rate divided by the rate of the highest group. The top group always scores 1.00.

The EEOC's long-standing Q&A on the Uniform Guidelines describes a selection rate below four-fifths (80%) of the top group's rate as generally regarded as evidence of adverse impact. Its own example: 48 of 80 white applicants hired (60%) and 12 of 40 Black applicants hired (30%) gives a ratio of 0.50, well under 0.80.

How to read an AI bias audit summary

Here is an illustrative summary, with made-up numbers, laid out the way a Local Law 144 audit usually reports a scored tool:

Category (illustrative)	Candidates	Scoring rate	Impact ratio
Male	1,200	51%	1.00
Female	1,000	48%	0.94
Hispanic or Latino female	140	41%	0.80
Two or more races female	18	33%	0.65
Unknown	310	Not calculated	Not calculated

What to notice:

The sex-level numbers look fine and hide the detail. Women overall sit at 0.94, but intersectional rows can be much lower. Always scroll to the lowest ratio.
Small groups swing wildly. The 0.65 row rests on 18 people. Three more candidates scoring above the median would push it past 0.80. The EEOC Q&A itself warns that the four-fifths rule may not support a finding of adverse impact when the number of people selected is very small. Under the city's rules, a category under 2% of the data can be excluded from the impact ratio math, but its counts must still be reported.
Look at the unknown row. Hundreds of candidates without demographic data means the ratios describe only part of your pool. The city's rules require the summary to report this number.
Check the data source. An audit on historical data from real hiring tells you more than one on synthetic test data. The city's rules allow test data only when historical data is insufficient, and the summary must explain it.

What a bias audit doesn't cover

An impact ratio is an outcome measure. A tool can pass every ratio and still be a poor tool, and a tool can fail one for reasons that sit outside the software (for example, a job requirement that screens out a group before the AI ever sees them). Ask separately about:

Job-relatedness. Do the questions and rubric come from the actual job? Structured, job-based questions and scoring are covered in our guide to how AI interviews are scored.
Accessibility and accommodations. Can candidates with a disability, a speech difference or poor bandwidth take an equivalent path? NYC's notice rules already require you to tell candidates how to request an accommodation.
What drives the score. If a score depends on accent, background noise or camera quality, a disparity can appear without anyone intending it.
Drift. A model, prompt or rubric that changes after the audit isn't the tool that was audited. Ask how the vendor handles material changes between annual audits.

Where bias audits are required, and where they are not

Jurisdiction	What the law says about bias testing	Status (checked October 2026)
New York City (Local Law 144)	Independent bias audit within one year before use, public summary, candidate notice	Enforced since July 5, 2023
Illinois (AI Video Interview Act, 820 ILCS 42)	No audit, but employers that rely solely on AI analysis of video interviews to pick who gets an in-person interview must report race and ethnicity data to the state each year by December 31	In effect (reporting section effective January 1, 2022)
Illinois (Human Rights Act, 775 ILCS 5/2-102(L))	Civil rights violation to use AI that has the effect of discriminating, or to use zip codes as a proxy; employers must notify employees when they use AI this way	Effective January 1, 2026 (P.A. 103-804)
Colorado (SB24-205, then SB26-189)	SB24-205 asked deployers to run impact assessments and annual reviews for algorithmic discrimination. SB25B-004 delayed it to June 30, 2026. SB26-189, signed May 14, 2026, repealed and reenacted those provisions, with new duties built around notice, explaining adverse outcomes and human review	SB26-189 enacted; developer documentation duties start January 1, 2027, and the Attorney General must adopt rules by the same date
European Union (AI Act)	Providers of high-risk systems handle data governance and bias checks. A 2026 amendment added Article 4a, allowing exceptional processing of sensitive data to detect and correct bias	High-risk duties for recruitment AI apply from December 2, 2027

Federal anti-discrimination law applies everywhere in the US whether or not a city requires an audit. That is why many employers run the same impact ratio math internally, even where no published audit is required.

How vendors describe their audits

Vendors publish very different levels of detail. From each company's own site, as recorded in our research in October 2026:

HireVue says its fairness audits cover NYC Local Law 144, and a HireVue press release says DCI Consulting Group ran external bias audits of its algorithms.
Eightfold says its AI Interviewer has been independently bias-audited under Local Law 144, with results published.
Ribbon says it has a Local Law 144 bias audit, last audited in 2025, and links the report.
Vervoe says its NYC bias audit was verified by Holistic AI, with an annual audit.
Classet says it runs monthly bias audits with Warden AI.
HackerRank's AI interviewer page mentions bias audits in line with Local Law 144.
Sapia.ai links an "independent bias audit" from its homepage, and BrightHire says its Screen product has undergone an independent AI bias audit, without naming the city's law.

For our own product, NTRVSTA publishes an independent NYC Local Law 144 bias audit by Asenion, finalized October 20, 2026, on our compliance page. For a side-by-side list, see AI hiring tools with published Local Law 144 audits.

What to ask a vendor before you sign

When was the most recent audit finalized, and when is the next one due?
Who ran it, and are they independent of you under NYC's definition (no employment, no involvement in building the tool, no material financial interest)?
Was it historical data or test data? If pooled across customers, can we contribute our data so we can rely on it?
What is the lowest impact ratio in the report, for which group, and on how many candidates?
How many candidates fell into the unknown category?
Has the model, prompt or rubric changed since the audit?
Does the score use facial expressions, voice tone or anything that could act as a proxy for a protected trait?

Then read our full NYC Local Law 144 guide for the notice and posting steps, and our guide to AI resume screening, since resume filters need the same scrutiny.

Questions

What is an AI bias audit in hiring?

It is an independent statistical review of whether a hiring tool advances or scores some demographic groups at lower rates than others. NYC's version requires selection or scoring rates and impact ratios across sex, race/ethnicity and intersectional groups, published in a summary.

Is an impact ratio below 0.80 illegal?

Not by itself. The EEOC's guidance treats a ratio under four-fifths as a general sign of adverse impact, not a legal definition, and warns that it may not hold with very small numbers. NYC's law requires the audit but sets no pass mark. A low ratio is a reason to investigate with counsel.

How often should a hiring tool be audited?

In New York City, at least once a year: an employer can only rely on an audit for one year from the date it was conducted. It also makes sense to re-check after any material change to the model, the questions or the rubric.

Who pays for the bias audit, the vendor or the employer?

Either can. Many vendors commission an independent audit of their tool, and NYC allows that. But the city's FAQ puts the legal duty on the employer or employment agency, so make sure the vendor's audit covers your use and is current.

Curious what our audit covers and how our interview scores are built? Book a 20-minute demo and we'll walk through it, or try a sample interview first.

Sources

NTRVSTA