AI CV screening software: what it must do before you buy

You are pricing AI CV screening software, and your third AI resume screening demo ends like the 2 before it. You ask why the top CV ranks first, and the rep shows its match score again.

In Greenhouse's vendor-run November 2025 AI in Hiring survey, only 21% of US recruiters were very confident their AI rejected no qualified applicant.

That is not your fault: vendors rarely show the working. But the risk lands on you, because a tool you cannot check can bury good people with your approval on it.

This guide shows what the software should do, how to test it on 20 past CVs first, and 12 points to hold vendors to.

Key takeaways

• What does AI CV screening software do with 200 CVs? It scores each CV against your criteria, quotes the evidence and ranks the pile. You read the top and decide.
• Should I only interview CVs that score 8 out of 10? No. Of 105,648 CVs Flowmingo scored 10 Jul to 24 Sep 2026, only 4.0% reached 8. Rank everyone and read the band under your cut.
• Is a person approving the AI's picks enough to stop bias? No. In an experiment with 528 people, reviewers drifted toward whichever group a biased AI favoured.
• How do I test a tool before I buy it? Run 20 past CVs whose outcome you know, re-run them shuffled, then change only the name on 3 copies.
• Can I use it on New York City jobs without a bias audit? Not if it substantially helps decide who moves on. You, the employer, need an audit from the past year and must notify NYC residents 10 business days before use.

1. We get 200+ CVs per role. What does AI CV screening software do with them?

It reads every CV against criteria you set, scores each one with a written reason and ranks the pile.

Among projects that sent our platform 10 or more CVs to score, the busiest 1 in 10 got 228 or more. Greenhouse's 2025 North American benchmark counts every application and puts the average job at 244, so 200+ is an ordinary busy role.

It runs the first pass of CV screening in 5 steps:

1. You paste the job brief.
2. The software drafts weighted criteria, and you edit them.
3. It scores each CV per criterion, quoting evidence.
4. It ranks everyone, and you read the top and the band below: rank, never reject.
5. You export the shortlist with the reasons.

How it works: job brief, weighted criteria you edit, each CV scored with evidence quoted, then rank, never reject. The shortlist goes out with the reasons.

An automated resume screening tool scores what the CV says, and many CVs now copy the job description (a foundational 2024 Reddit thread).

Some candidates go further. In Greenhouse's 2025 survey of 1,200 US job seekers, 41% admitted hiding text aimed at AI resume screening filters. That's why the score is a first pass, and screening questions for job applicants should test the person next.

Flowmingo data · 2,294 hiring projects with 10 or more CVs, at 1,004 companies, CVs received 3 Apr to 20 Oct 2026

• 33 CVs was the median for projects with 10 or more CVs.
• 228 or more CVs landed in the busiest 1 in 10 of those projects.

Counts only CVs sent to Flowmingo's CV evaluation, not every application a role received elsewhere.

CVs per role: across 2,294 projects with 10 or more CVs, the median got 33. The busiest 1 in 4 got 83 or more, and the busiest 1 in 10 got 228 or more.

2. How do I tell the software what 'qualified' means for this role, beyond keywords?

Write 4 to 6 criteria from the job itself, give each a weight, and say what CV evidence earns a high score. Leave out anything the job does not need, such as school name, postcode or a gap year.

Keep pass/fail for 2 or 3 facts you ask in the application, such as right to work or a licence (candidate shortlisting). Weight everything read off the CV, because a hard filter on a CV proxy removes qualified people unnoticed.

A foundational 2021 Harvard Business School and Accenture study gives 1 example: a filter that ranks down anyone out of work for 6 months.

Proxies need the same care. Newer AI models show less explicit gender and race bias, but bias tied to education remains significant (Iso and colleagues). Illinois bans zip codes as a proxy from 1 January 2026, and Singapore's fairness law will protect language ability, nationality and age.

Here is a worksheet for a Customer Service Team Lead (illustrative example, not customer data):

Must-haves the job title hides, such as years per level or working on site, go in the worksheet or in 1 or 2 screening questions. Agree it with the hiring manager before the first CV gets a score, and calibrate it on 5 CVs as our shortlisting guide shows.

How Flowmingo helps

Paste your job and Flowmingo drafts weighted criteria from it, and your team sets each one's importance. A Must Have counts 4 times as much as a Good To Have (How we score).

3. My hiring manager will ask why. Can AI CV screening software explain each score?

Good AI CV screening software shows a score for every criterion and quotes the line on the CV that earned it. If you cannot see the evidence, you cannot defend the shortlist to your hiring manager.

The November 2025 Greenhouse survey from the intro shows the cost of not seeing it. Beyond the US recruiters who were very confident, 25% were not confident in their AI at all, and 8% had no idea what it prioritises.

US recruiter confidence: 21% very confident, 25% not confident in their AI at all, and 8% with no idea what it prioritises.

Here is 1 CV scored line by line against the section 2 worksheet (illustrative example, not customer data):

The weighted score beats the plain average, because the strongest evidence sits on the must-have. The silent retail line scores 3, the flagged default in our shortlisting guide. A zero would punish someone who simply left it off the CV.

Then run 3 checks in about a minute. Each quoted line should prove its criterion, no score should rest on missing evidence, and a decent total should not hide a weak must-have.

In a demo, a single match percentage with no breakdown is the red flag, and so are reasons that copy the job ad.

How Flowmingo helps

Flowmingo scores every CV against your weighted criteria out of 10 and writes out the evidence behind each score (FAQ). The overall score is a weighted average, as in the table above (How we score).

4. What happens when the AI scores a strong candidate low, and can I override it?

Nothing should happen automatically: a low score should move a CV down the list, not out of it. You read the band under your cut and move people up.

Flowmingo data · 105,648 CVs · 50,546 candidates · 3,815 hiring projects · 1,761 companies · current scorer, 10 Jul to 24 Sep 2026

• 5.4 out of 10 was the median CV score against each job's own weighted criteria.
• 4.0% of CVs scored 8 or more, and 19.0% scored 7 or more.

AI CV scores, not candidate quality or hiring outcomes, not comparable across jobs. Strict criteria or bulk uploads pull them down; unscorable CVs are left out. The scoring method changed on 9-10 Jul 2026 and earlier scores are not comparable.

CV score spread, current scorer: of 105,648 CVs, 56.4% scored 5 or more and 19.0% scored 7 or more. Only 4.0% scored 8 or more, and 0.1% reached 9.

So "only interview the 8s" leaves many roles with a tiny list, or none. In half of projects with 20 or more scored CVs, not one reached 8 (candidate shortlisting).

In a 2025 experiment with 528 people, reviewers followed a simulated biased AI's picks up to 90% of the time (Wilson and colleagues).

So read the evidence before you accept a score, our advice, not the study's finding. Keep 3 habits:

• Read the top slice plus the next band down, where the foundational 2024 Reddit thread saw career changers, bootcamp graduates and 2-column CVs land.
• When you move someone up, write 1 sentence on what the score missed.
• After the role closes, check whether your overrides were right.

Say your cut is 7 and a career changer scores 6.1, because her CV says "shift supervisor", not "team lead". She ran 6 people at a retailer, so move her up and note "led 6, title differs".

From 2 December 2027, the EU needs a trained person with authority to overrule the tool (Article 26).

How Flowmingo helps

Nobody is rejected automatically: Flowmingo returns a score with its reasons, and every decision stays with your team (FAQ).

5. We hire across markets. Does AI CV screening software read other languages and odd formats?

Sometimes, and the gap can be large, so test it on the languages and layouts you receive. Ask the vendor for results by language, then check 2 or 3 of your own non-English and scanned CVs.

The model behind the tool matters more than a "multilingual" label. In the JobResQA benchmark (a 2026 preprint), Llama 3.3 70B scored 0.73 on English CV questions but 0.47 in German and 0.48 in Chinese.

The drop is uneven, though. Italian held at 0.70, and Mistral Large stayed closer, at 0.69 in English and 0.61 in Chinese.

Same model, other languages: Llama 3.3 70B scored 0.73 in English, 0.70 in Italian, 0.48 in Chinese and 0.47 in German. These are answer-quality scores, checked by an AI judge against human answers, not hiring accuracy.

Odd formats trip tools too. In the same foundational 2024 thread, recruiters named 2-column CVs, tables and misread dates, and a scanned CV needs text recognition first.

Then ask what happens to files the tool cannot read: you want them flagged, never silently dropped.

Flowmingo data · 210,364 CVs sent to CV evaluation at 2,223 companies, 3 Apr to 20 Oct 2026

• 99.7% of CVs sent to Flowmingo's CV evaluation came back with a score.
• 345 CVs could not be scored because the file was corrupt, password-protected or unreadable.

This measures how often scoring finished, not how accurately each CV was read. Some unscored CVs may still have been in the queue.

For example, hiring in Singapore, you will see CVs in English with Chinese, Malay or Tamil names and overseas degrees. Use 2 such CVs in the name-swap test in section 9, and keep the odd-file slots for a real non-English CV and a scanned one.

6. After Amazon and Workday, how do I check AI screening for bias against women or over-40s?

Ask the vendor for audit numbers, then check your own shortlist rates by group. The risk is real, because tools learn from past hires, read names and favour the CV they read first.

The foundational case is Amazon's experimental CV tool (Reuters, 2018). It learned from 10 years of CVs, penalised the word "women's" and downgraded graduates of 2 all-women's colleges.

Names sway models. In a foundational October 2024 audit of 3 CV-matching models, White-associated names won 85.1% of race tests (Wilson and Caliskan). Black male names lost in up to 100%.

Order sways them too. Across 22 AI models and 30,690 decisions on pairs of equally qualified CVs, the CV shown first won 63.5% of the time (Rozado).

First CV wins: across 22 AI models, the CV shown first won 63.5% of the time and the second 36.5%. Fair odds are 50%.

In Bloomberg's foundational 2024 test, GPT-3.5 ranked 8 near-identical CVs, where a fair top rate is 12.5%. Asian women's names came top 17.2% of the time and Black men 7.6% (figures). Yet interviewing.io disputes the method, and a chatbot is not a tested screening tool.

In the Mobley case against Workday, a court let applicants aged 40 and over proceed as a nationwide group in May 2025 (Holland & Knight). These are allegations, not findings.

Divide each group's shortlist rate by the highest group's rate (illustrative example, not customer data):

Group Y's 6.0% divided by Group X's 10.0% is 60%, under the 80% line. So find out why before the role closes.

The 80% line comes from the foundational 1978 Uniform Guidelines, a rule of thumb for race, sex and ethnic groups. Using it for over-40s is our suggestion. With small groups, use a significance test.

As of 2 November 2026 the rule is still in the eCFR. On 9 June 2026, the Justice Department's Office of Legal Counsel called the guidelines behind it an unconstitutional reading of Title VII.

The EEOC plans to rescind them in November 2026 (Ogletree). Title VII disparate-impact claims remain, so keep the check as your own monitor.

7. We hire in New York, the EU and Singapore. Which AI screening laws apply?

It depends where the job is and where your candidates live. EU and Singapore rules start late 2027, but New York City's apply now.

New York City bars a covered tool without a bias audit from the prior year, a public summary and notice. This is general information, not legal advice.

Laws by place: New York City bars covered tools without a bias audit and 10 business days' notice. Illinois requires notice and no zip-code proxies, California 4-year records, the EU applies from 2 December 2027 and Singapore from end-2027.

The city's DCWP has enforced the foundational Local Law 144 since 5 July 2023, with fines of $500 to $1,500 a day. Yet the State Comptroller found at least 17 potential violations among 32 companies, where the city found 1. A foundational 2024 study found only 18 of 391 employers had posted an audit (Null Compliance).

The EU lists filtering job applications as high-risk (Annex III), from 2 December 2027 under the Digital Omnibus (Lewis Silkin). Illinois rules (a foundational 2024 Morgan Lewis note) and California's 2025 rules (Mayer Brown) already apply.

The EEOC removed its foundational 2023 AI guidance in January 2025 and stopped disparate-impact investigations (Seyfarth). Private claims such as Mobley continue.

Singapore's Act will bar decisions on 11 protected traits and exempt employers under 25 staff, apart from fair consideration (Mondaq, DLA Piper). It does not name AI, so applying it to automated filters is our reading.

7.1 Should we tell candidates that AI screens their CV, and let them opt out?

Yes, because New York City and Illinois both require notice. An opt-out, where a person reads the CV instead, is good practice, though NYC's rules do not require one.

Put the notice on your careers page, in the job ad or by email at least 10 business days before scoring NYC candidates (DCWP rules). Name 1 person to answer opt-out replies within 2 business days, our suggestion, not a legal deadline.

For example, a Singapore and NYC role advertised on Monday the 1st cannot score NYC CVs until Monday the 15th. Singapore CVs can be scored from day 1, and NYC CVs too if your careers page already carried the notice for 10 business days.

Sample wording, adapt with your counsel:

We use AI software to score CVs against the criteria for this role: [list]. A member of our team reviews the results and makes every decision. For New York City roles, our latest bias audit, from the past year, is summarised here: [link]. To have a person review your application without the software, or to ask for an adjustment, reply to this email and [name] will answer within 2 business days.

8. Can I export the scored shortlist and send it to the hiring manager?

Yes, and the export should carry the reasons, not just names and numbers. Send each score with its evidence lines, so you argue about evidence instead of gut feel.

A useful export holds 6 things: name, overall score, score per criterion, the evidence for each must-have, your override note and the date. With those, a hiring manager can dispute 1 line instead of the whole list, and you keep a record of why.

The export is also your record. California's rules need ADS records kept for 4 years, and the EU AI Act needs logs for at least 6 months from 2 December 2027.

So ask each vendor how long it keeps scores, and whether you can export them all. Check that override notes export too, since they show a person made each decision. For example, an agency may want a PDF per candidate for a client.

An in-house team may want a CSV for its ATS, or an API so scores land there without copy and paste. Either way, the hiring manager gets the list with evidence, never a bare ranking.

How Flowmingo helps

On the Free plan you can view everything in the app, share reports by link and download 3 PDF or DOCX reports in total. For unlimited downloads, upgrade to Flowmingo Pro (Starter, Growth or Scale), and CSV export starts on Growth (pricing). There is no native sync with Greenhouse, Lever or Workday, so teams connect through the REST API, webhooks and MCP, free on every plan (FAQ).

9. How do I test AI CV screening software on 20 of my own past CVs?

Pick 20 past CVs whose outcome you know, run them through the trial and compare the ranking with what happened. Then shuffle and re-run them, and change only the name on 3 copies.

Mix the 20: 5 people you shortlisted and were glad you did, 5 near-misses, 8 clear rejections and 2 non-English or scanned CVs. Remove personal data you do not need, and check your trial terms and privacy notice allow reusing past CVs this way.

Then run 5 checks:

• Agreement: at least 4 of 5 known-good CVs should land in the tool's top 8.
• Shuffle: re-upload the 20 in reverse order. No CV should move more than 2 places.
• Name swap: copy 3 CVs and change only the name on each copy. The score gap should stay at 0.5 or less out of 10.
• Reasons: every must-have score on 5 random CVs shows its evidence, checkable in under 1 minute.
• Odd files: non-English and scanned CVs get read, not left blank.

20-CV test: agreement, shuffle, name swap, reasons and odd files, each with a suggested pass mark you can check in 1 sitting.

The shuffle matters because of Rozado's first-CV effect in section 6. These pass marks are our rule of thumb, not a standard.

9.1 How accurate is it compared with my own team reading the CVs?

No independent benchmark compares tools with human screeners on your roles. Your 20-CV test is the honest answer, so compare it with the real outcome.

If it puts 4 of 5 known-good CVs in its top 8 and 0 of 8 rejections in its top 5, it agrees with you. If only 2 of 5 make the top 8, read its reasons for the other 3 before you trust it.

How Flowmingo helps

You can run this test free: CV evaluation is unlimited on every plan, including Free, with no per-CV charge (FAQ).

10. After 3 vendor demos, which checks must AI CV screening software pass before I buy?

It must score against your criteria with visible evidence, never reject on its own, and pass your 20-CV test. It must also keep your candidates out of its training.

Ask to see each of these 12 points live on your own CVs, because a vendor's sample data proves nothing about your roles. For example, change 1 weight and watch the ranking move.

Points 1 to 10 come from sections 2 to 9 of this guide, and points 11 and 12 cover data handling.

Points 11 and 12 cover data handling, which most resume screening software pages skip. The ICO's November 2024 audit found most providers it checked had reused candidate data to train their tools.

It also found tools that let recruiters filter out protected characteristics, or guessed ethnicity from names. So get 4 answers from every vendor in writing, before you sign:

• Where they store your CVs, and for how long.
• Whether they train any model on your candidates.
• How you delete a candidate's CV and scores.
• Whether they support your DPIA, the data protection impact assessment, before you start.

How Flowmingo helps

Flowmingo is a free AI hiring platform for recruiters. Its CV Evaluation covers points 1 to 3: criteria you edit, evidence behind each score and no automatic rejection. You run point 9, section 6's group-ratio check, on your own shortlist. Run your 20-CV test on Flowmingo before you buy any AI CV screening software.

11. Sources

Every study, law and survey cited above, in APA style.

• Bloomberg Graphics. (2024). 2024-openai-gpt-hiring-racial-discrimination [Data and code]. GitHub. Percentages as reported by Business Insider via Yahoo.
• Carrino, C. P., Estrella, P., Zbib, R., Escolano, C., & Fonollosa, J. A. R. (2026). JobResQA: A benchmark for LLM machine reading comprehension on multilingual resumes and JDs (arXiv:2601.23183). arXiv.
• Cornell Law School, Legal Information Institute. (n.d., read 2 November 2026). 42 U.S. Code 2000e-2, Unlawful employment practices. Cornell LII.
• Dastin, J. (2018, October 10). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters.
• DLA Piper. (2024, December). First Workplace Fairness Bill introduced in Singapore to protect employees against discrimination. DLA Piper.
• Electronic Code of Federal Regulations. (current text, read 2 November 2026). 29 CFR Part 1607: Uniform guidelines on employee selection procedures. eCFR.
• European Parliament and Council. (2026). Regulation (EU) 2026/1744, the Digital Omnibus on AI. Official Journal of the European Union.
• European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III. artificialintelligenceact.eu.
• European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 26. artificialintelligenceact.eu.
• Fuller, J. (2021, November 5). How to tap the talent automated HR platforms miss. Harvard Business School Working Knowledge.
• Greenhouse. (2025, November 19). An AI trust crisis (2025 AI in Hiring Report). Greenhouse press release.
• Greenhouse. (2026, March). The Hire Standard: Greenhouse benchmark report, North America. Greenhouse.
• Herbert Smith Freehills Kramer. (2025, December 10). Workplace Fairness Act to take effect end of 2027. Mondaq.
• Holland & Knight. (2025, May 27). Federal court allows collective action lawsuit over alleged AI hiring bias. Holland & Knight.
• Information Commissioner's Office. (2024, November). AI tools in recruitment: Audit outcomes report. ICO.
• interviewing.io. (n.d., read 2026). Refuting Bloomberg's analysis: ChatGPT isn't racist. interviewing.io blog.
• Iso, H., Pezeshkpour, P., Bhutani, N., & Hruschka, E. (2025). Evaluating bias in LLMs for job-resume matching: Gender, race, and education. Proceedings of NAACL 2025 Industry Track.
• Lewis Silkin. (2026, July 27). The Digital Omnibus on AI enters into force today. Lewis Silkin.
• Mayer Brown. (2025, September 2). California adopts new employment AI regulations effective October 1, 2025. Mayer Brown.
• Morgan Lewis. (2024, September 12). Illinois passes new law to address AI in the workplace. Morgan Lewis.
• New York City Department of Consumer and Worker Protection. (n.d., read 2026). Automated employment decision tools (AEDT). City of New York.
• New York City Department of Consumer and Worker Protection. (2023, April). Notice of adoption of final rule: Use of automated employment decision tools (6 RCNY 5-304, notice to candidates). City of New York.
• Office of Legal Counsel, US Department of Justice. (2026, June 9). Constitutionality of disparate-impact liability under Title VII. DOJ.
• Office of the New York State Comptroller. (2025, December 2). Enforcement of Local Law 144: Automated employment decision tools. Office of the New York State Comptroller.
• Ogletree Deakins. (2026, July 7). The EEOC's regulatory agenda: 10 signs of intent. Ogletree Deakins.
• Reddit r/Recruitment. (2024, May 6). Is anyone using AI for resume screening [Online forum thread]. Reddit.
• Rozado, D. (2026). Gender and positional biases in LLM-based hiring decisions: Evidence from comparative CV/resume evaluations. PeerJ Computer Science.
• Seyfarth Shaw. (2026). EEOC-initiated litigation: 2026 edition. Seyfarth Shaw.
• U.S. Code of Federal Regulations (current text). 29 CFR 1607.4, Uniform Guidelines on Employee Selection Procedures, information on impact. Legal Information Institute.
• Wilson, K., & Caliskan, A. (2024). Gender, race, and intersectional bias in resume screening via language model retrieval. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1). Erratum: arXiv v3, 2026.
• Wilson, K., Sim, M., Gueorguieva, A.-M., & Caliskan, A. (2025). No thoughts just AI: Biased LLM hiring recommendations alter human decision making and limit human autonomy. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(3).
• Wright, L., Muenster, R. M., Vecchione, B., Qu, T., Cai, P., Metcalf, J., Matias, J. N., et al. (2024). Null compliance: NYC Local Law 144 and the challenges of algorithm accountability. Proceedings of FAccT 2024.