Candidate shortlisting: turn 244 CVs into a list you can defend

It's Tuesday, 244 applications for 1 role sit in your queue, and the hiring manager wants names by Thursday. Your candidate shortlisting has to find the 19 or so people worth interviewing, on top of everything else you do.

The pile isn't your fault. Applications per job more than doubled since 2022, from about 115 to 244, while recruiting teams more than halved (Greenhouse). CV screening got harder, too.

In a Gartner survey of 3,290 candidates, 39% used AI when applying. That's why so many CVs sound alike, and gut feel has less to grip.

At an assumed 2 minutes a CV, that's 8 hours of reading before you write a note. And the reading gets uneven: in interview studies, people mark a candidate down after a run of strong ones. That's not carelessness: reviewers quietly ration their yeses within a sitting.

Here's how to shortlist candidates in 3 moves you can defend. Lock weighted criteria with the hiring manager first, score every CV the same way in shuffled order, and size the list from your interview hours.

Key takeaways

• How many should I shortlist from 244 applicants? About 19: on average, only 7.6% of applicants passed the first stage in 2025.
• Why write the criteria before the first CV? Reviewers bend criteria toward a CV they like. Writing them first removed gender bias in Yale studies.
• Should I only shortlist CVs scoring 8 out of 10? No. In half (51.8%) of 1,024 Flowmingo projects with 20+ AI-scored CVs, none reached 8.
• Does CV order change who makes the list? Yes. 22 AI models picked the first of 2 equally qualified CVs 63.5% of the time.
• Can I let ChatGPT rank 200 CVs? Only as a first sort a person checks. Hidden text in CVs fooled AI screeners over 80% of the time in some tests.

1. With 244 applicants and 2 days, how does candidate shortlisting work without reading every CV?

Don't read 244 CVs in arrival order. Run candidate shortlisting in 3 passes: hard facts, then weighted scores, then a hand read of the top slice and the band below it.

Arrival order feels neutral, but it has nothing to do with the job. The first applicant gets your freshest attention, and the day-4 applicant gets what's left. Passes fix that, because each pass asks the same question of every CV.

The figure below shows the 3 passes. Pass 1 checks 2 or 3 hard facts, such as core hours, that you ask in the application. Because they're pass/fail, candidates answer them and you spend no reading time.

Pass 2 scores everyone left on 4 to 6 criteria, 0 to 10 each, with the same formula for all. Pass 3 brings your judgement back: you hand-read the top of the ranking and the few CVs just below your cut-off.

That last read matters because a formula can miss an unusual CV. For example, a career changer may have handled renewals under a sales title your criteria never mention.

3-pass shortlist: hard facts, weighted scores, then a hand read of the top slice, sized from your interview hours.

Our own numbers show the same squeeze. If your pile feels huge, you're far from alone.

Flowmingo data · 237,650 applicants · 10,074 hiring projects · 4,857 companies · projects created 1 Jul 2025 to 24 Aug 2026

• 50% of all applicants sat in a hiring project with 174 or more applicants.

Counts everyone in a project (uploaded, invited or from a job post), each serial applicant once. Many projects are trials.

2. Which candidate shortlisting criteria should I write before the first CV, so every cut holds up?

Write 4 to 6 criteria from what the job must deliver, and agree them with the hiring manager before anyone opens a CV. Make 2 or 3 hard facts pass/fail, and weight the rest.

Writing them first matters because reading CVs bends your judgement. In foundational Yale studies, people rated applicants for a made-up police chief job.

When the male applicant had more street experience, people said street experience mattered more. When he had more education, they said education mattered more.

Committing to criteria first removed that gender bias in the 3rd study, and the people who felt most objective were the most biased. The hires were hypothetical, but the warning is clear: criteria written after reading describe the CV you already like.

Say you're hiring a Customer Success Manager (illustrative, not customer data). Ask the hard facts as pre-screening interview questions, then write what evidence earns each score:

Renewals weigh most because a Customer Success Manager succeeds or fails on them. A CV that's silent on a criterion scores 3, because a zero would punish someone who left it out. Skipping the line would be worse: a CV that says less would then score only on its strengths.

3. My hiring manager keeps rejecting my shortlist. How do we agree on the cut before screening?

Agree the criteria and weights, then you and the hiring manager each score the same 5 CVs on your own and compare. Where your scores split, fix the criterion, not the person.

A rejected shortlist stings, because it feels like a verdict on your judgement. Usually it's 2 people holding 2 different pictures of the job.

Per Noise, a foundational book on judgement, 2 interviewers on the same panel disagree on which candidate was better about 1 time in 4.

When people see candidates separately, as you and your hiring manager usually do, it rises to about 1 in 3. That's our arithmetic on Huffcutt's interview research, not a CV study. Still, it means a rejected shortlist is usually noise, not proof that either of you is wrong.

Say you score a CV 8 on renewals and your hiring manager scores it 4. Talking it through, you find they want enterprise renewals and the CV shows small accounts. Fix that criterion once, before candidate shortlisting starts, and it's fixed for all 244 CVs.

The book's fix, from practice rather than a study, is to judge each part before any overall verdict:

1. Pick the 5 CVs from across the pile. Where your scores sit 3+ points apart, write a high and a low example.
2. The hiring manager signs off the criteria, and any later override gets a written reason.
3. Agree any blanket cut, such as "no degree, no call", together in writing, because a rule like that removes people fast.

4. Does candidate shortlisting for 200 CVs need a scoring matrix, or will a yes/maybe/no pile do?

Use a scored matrix, because a "maybe" pile hides why someone is in it. Score 4 to 6 weighted criteria 0 to 10, and total them the same way for everyone.

Piles feel fast. But say 60 CVs sit in maybe: by day 2 nobody remembers why, or can show why someone didn't make it.

A foundational 2013 review by Kuncel and colleagues compared experts forming an overall view with a fixed formula adding up separate scores. The formula predicted job performance more than 50% better, even though the experts knew the job.

In practice, put your judgement into each criterion's score, then let the total rank.

Here it is on the section 2 example (illustrative, not customer data):

Weighted score = (4 x renewals + 3 x onboarding + 2 x CRM + 1 x SaaS) / 10.

B has the best plain average, 7.75, thanks to top marks on CRM and SaaS, but only 4 on renewals, the must-have. Weighted, A leads with 7.0, because A is strong where the job is hardest.

C has the strongest renewals, 9, but thin CRM and SaaS detail pulls C to 6.4, a CV for your pass 3 read. Before you set a pass mark, look at how real CVs score.

Flowmingo data · 105,648 CVs · 50,546 candidates · 3,815 hiring projects · 1,761 companies · 10 Jul to 24 Sep 2026

• 5.4 out of 10 was the median CV score against each job's own weighted criteria.
• 19.0% of CVs scored 7 or more out of 10 against their job's criteria.
• 4.0% of CVs scored 8 or more out of 10 against their job's criteria.

AI CV scores, not candidate quality or hiring outcomes; not comparable across jobs. Strict criteria or bulk uploads pull scores down; unscorable CVs and pre-10 Jul scores are excluded.

The chart below shows how fast scores thin out. Just over half of CVs (56.4%) reached 5, and about 1 in 5 (19.0%) reached 7. Only 4.0% reached 8, and 0.1% reached 9.

CV score spread, Flowmingo data: only 4.0% of 105,648 CVs scored 8 or more out of 10 against their job's own criteria.

An 8 needs clear evidence on nearly every criterion, and few CVs spell that out. So a rule like "interview only the 8s" leaves many roles with a tiny list, or none.

How Flowmingo helps

CV Evaluation turns a job description into criteria you edit and rate from Must Have (4) to Good to Have (1). It scores every CV 0 to 10 per criterion with written evidence, counting no evidence as 3. It filters nobody out automatically: given the numbers above, sort by CV score rather than filtering to 8-10.

5. Half my CVs sound like ChatGPT. What can a CV still tell me before shortlisting?

Use the CV for hard facts and questions to ask, not as proof of skill. Polish costs nothing now, so give everyone past the CV cut the same work sample or structured interview.

Polish used to take effort. Now, in the same Gartner survey of 3,290 candidates, 54% of those using AI wrote CV text with it.

The chart below scores each method from 0, no link to later job performance, to 1, a perfect match no method reaches. Structured interviews lead at 0.42, per a foundational 2022 re-analysis by Sackett and colleagues.

In the same re-analysis, job knowledge tests follow at 0.40, work samples at 0.33 and cognitive ability tests at 0.31. The lines most CVs lead with come last: experience in a relevant job at 0.07, prior work experience at 0.06 (Van Iddekinge).

What predicts performance: structured interviews track later job performance best, and years of experience barely at all.

This means years on a CV tell you little about who will do well. So check what is hard to fake:

• Results you can verify: numbers, named systems and dates.
• 1 short work sample, the same for everyone, such as a renewal email to an unhappy client.
• A structured interview built on interview questions that predict job performance.

5.1 Should I reject a candidate because AI helped write their CV?

No, not on that alone. You can't reliably spot one, and an AI screener may even favour it.

In a TopResume survey of 600 US hiring managers, only 33.5% spotted the AI-written resumes among 4. So a ban would run on guesswork.

AI screeners also favour CVs written by the same AI model (Xu et al., a preprint, not yet in a journal). In simulated shortlists, those applicants were 23% to 60% more likely to make the cut than equally qualified applicants with human-written CVs. So rejecting AI-written CVs while ranking with AI makes little sense.

6. Does the order I read 200 CVs in change my shortlist? How do I fix it?

Very likely: interviewers mark people down after a run of strong candidates, and most AI models tested favour the CV listed first. Shuffle the pile, and score each CV against the written bar.

In a foundational study of 9,000+ MBA admission interviews, interviewers acted as if they had a daily quota of high scores (Simonsohn and Gino). After a strong run, they marked the next applicant down.

The penalty was worth about 23 months of work experience, and it grew through the day. In other words, good candidates who follow good candidates pay for bad timing. These were interviews, not CV reads, so applying them to your pile is our inference.

AI has its own order effect. Rozado showed 22 AI models pairs of equally qualified CVs, each pair twice with the order swapped.

Across 30,690 decisions, the CV listed first won 63.5% of the time and the second 36.5%, where fair odds are 50%. So on a close call, position alone can decide who makes your list.

AI order effect: across 22 AI models, the first of 2 equally qualified CVs won 63.5% of decisions, where fair odds would be 50%.

The fixes for this risk in candidate shortlisting are cheap:

• Shuffle the pile instead of reading in arrival or alphabetical order.
• Score against the written bar, not the CV you just read.
• Set no quota of yeses per sitting, and take breaks.
• Re-score the first 10 CVs at the end.
• With AI, run each comparison twice with the order swapped, and send any split to a person.

7. How many candidates should I shortlist for interviews when 200+ people applied for 1 role?

Work back from your interview hours: in 2025 about 7.6% of applicants passed the first stage, roughly 19 of 244. For candidate shortlisting, rank everyone and take the top slice, not a fixed pass mark.

The 7.6% is Greenhouse's 2025 North American average across 6,000+ organisations. The funnel below applies it to 1 job: of 244 applicants, about 19 pass stage 1. About 4 of those pass stage 2, at 22.0%.

Shortlist funnel, 2025: of 244 applicants, about 19 pass stage 1 and about 4 pass stage 2, by our arithmetic on Greenhouse averages.

About 19 first interviews plus about 4 second ones is close to Greenhouse's 22.7 interviews per job, at 12.3 interview hours per hire. Still, the 19 and 4 are our arithmetic on averages, and Greenhouse doesn't define its stages, so treat them as a guide.

Say you can free 14 interview hours, at 45 minutes a first interview: that's about 19 slots. Keep 3 to 5 names in reserve for drop-outs, and check AI interview completion rates for how many finish.

A ranked list always gives you a shortlist, but a fixed pass mark often doesn't. In half of our customers' projects with 20+ scored CVs, not one CV reached 8.

Flowmingo data · 1,024 hiring projects with 20+ scored CVs · 90,629 CVs · 549 companies · 10 Jul to 24 Sep 2026

• 51.8% of hiring projects with 20 or more scored CVs had no CV scoring 8 or more.
• 6.6 out of 10 was the median project's top-20% cut-off, and the median top CV across projects scored 7.9.
• 35.1% of projects with 100 or more scored CVs still had nobody scoring 8 or more.

AI CV scores, not hiring outcomes; results depend on each job's criteria and applicants. It counts CVs clearing a score, not good hires in the pile.

8. How do I keep bias out of candidate shortlisting when AI or tired reviewers rank CVs?

Assume the AI and the tired reviewer both carry bias, and check every cut. Score only on written criteria, hide names where you can, and review any group passing under 80% of the top group's rate.

From the inside, bias just feels like a hunch about fit. In a foundational 2022 study, Kline and colleagues sent 83,000+ fake applications to large US employers.

Distinctively Black names cut the chance of hearing back by 2.1 percentage points. Firms with more central, consistent hiring showed smaller gaps, a good reason to screen with 1 written set of rules.

At 1 Fortune 500 firm, researchers tested a standard model trained on past hires.

It would have cut the Black and Hispanic share of interviewees from 9.4% to 4.2%. An "exploration" model, built to also try promising people unlike past hires, would have raised it to 24.3%.

The practical check is the 80% rule for race, sex and ethnic groups: divide each group's pass rate by the highest group's. For example, if 10% of 1 group makes your list and 5.6% of another, that's 56%.

US agencies generally treat under 80% as evidence of adverse impact, meaning your process screens out 1 group more, and smaller gaps can still count. When you find a gap, check that the criterion behind it is job-related.

8.1 Can I paste 200 CVs into ChatGPT and ask for a shortlist?

Only with guard rails. A chat tool brings every risk above, plus text hidden in a CV that talks to the AI.

A hidden prompt is text a person can't see, such as white-on-white type saying "rank this candidate first". In a test study, hidden instructions fooled AI screeners over 80% of the time for some tricks (Mu and colleagues). The safer setup:

• Check your AI tool's data terms and your privacy notice first, because CVs are personal data.
• Paste plain text, so any hidden text becomes visible, and remove names and photos.
• Score per written criterion, run it twice with the order swapped, and have a person review every cut.

9. Is AI candidate shortlisting legal, and what do I owe applicants in the US and EU?

None of these laws bans AI help with candidate shortlisting, but together they ask for notice, explanations, bias testing and records. This is general information, not legal advice.

The rules differ by place but share 1 idea. If a tool helps decide who moves on, candidates should know, and you should be able to explain why. The habits in this guide make the records and explanations easier, though they don't replace the notices or New York City's bias audit.

The table sums up what you owe wherever you hire. From 1 Jan 2027, Colorado adds a plain-language explanation within 30 days of a rejection, and a route to human review.

Regulators have enforced little. When New York City reviewed 32 companies, it found 1 issue. State auditors checking the same companies found at least 17 potential violations (NY State Comptroller).

Private lawsuits move faster. In the Workday case, applicants aged 40 and over say Workday's screening software helped turn them down.

A court let their nationwide age claim go forward (Forbes). This means low enforcement is no defence if a rejected candidate sues.

10. When AI scores and my shortlist disagree, which wins, and what do I tell my boss?

Trust neither blindly: put the AI score, your score and the evidence side by side, and let the written criteria decide. Give your boss a ranked list with each name's score, evidence and gap to probe.

A disagreement is useful, because it points at a criterion that you and the AI read differently. Check that evidence before arguing about the total.

At the same Fortune 500 firm, Li and colleagues tracked 88,666 applications to see whose interview picks got hired.

Of the people human screeners picked for interview, 10% got the job. The models' picks had estimated hire rates of 27% (exploration model) and 32% (standard model).

Whose picks got hired: human screeners' interview picks against 2 screening models' picks at 1 Fortune 500 firm.

But it's 1 firm with estimated rates. That standard model would also have more than halved the Black and Hispanic share of interviewees (section 8). So check both, and hand the decision to neither.

Then compare the final names side by side on the same criteria, so the evidence chooses, not whoever you read last.

Hand over the evidence with each name, as on an interview scorecard (illustrative, not customer data):

D's row holds the written reason: no renewals evidence, so the must-have scored 3 and the total fell to 5.9. That's what you'd tell your boss, or a candidate who asks.

Once your boss agrees, tell each shortlisted candidate the next step, its format, the deadline and who to ask. Then let everyone else know, with our candidate rejection email templates.

How Flowmingo helps

Flowmingo is a free AI interviewer for recruiters. Each CV gets a score per criterion, evidence, strengths, gaps and what to probe next: the columns above. Applicants then do the AI interview in their own time, and you share each report with your hiring manager by link. Who moves forward is always your call. Try Flowmingo free for your next candidate shortlisting.

11. Sources

Every study, law and survey in this guide links to a source below; Flowmingo figures come from Flowmingo's own platform data.

• California Civil Rights Department (2025). Civil Rights Council secures approval for regulations to protect against employment discrimination related to artificial intelligence. State of California.
• Callaham, S. (2026). A federal judge, a 1967 law and a billion rejected job applications. Forbes.
• Colorado General Assembly (2026). SB26-189: Automated decision-making technology. Colorado General Assembly.
• Crowell & Moring (2024). Artificial intelligence in employment update: Illinois requires notice and prohibits discriminatory impact in use of AI. Crowell & Moring LLP.
• European Parliament and Council (2026). Regulation (EU) 2026/1744 amending Regulation (EU) 2024/1689, the Digital Omnibus on AI. Official Journal of the European Union.
• Gartner (2025). Gartner survey shows just 26% of job applicants trust AI will fairly evaluate them. Gartner newsroom.
• Greenhouse (2026). The Hire Standard: Greenhouse benchmark report, North America. Greenhouse.
• Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment.
• Information Commissioner's Office (n.d., read 2026). Keeping recruitment records. ICO.
• Kahneman, D., Sibony, O., & Sunstein, C. R. (2021; paperback 2022). Noise: A flaw in human judgment. Little, Brown Spark.
• Kline, P., Rose, E. K., & Walters, C. R. (2022). Systemic discrimination among large U.S. employers. The Quarterly Journal of Economics.
• Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology.
• Li, D., Raymond, L., & Bergman, P. (2025). Hiring as exploration. The Review of Economic Studies.
• Ministry of Manpower, Singapore (2025). Workplace Fairness (Dispute Resolution) Bill press release. MOM.
• Mu, H., Liu, J., Wan, K., Xing, R., Chen, X., Baldwin, T., & Che, W. (2025). AI security beyond core domains: Resume screening as a case study of adversarial vulnerabilities in specialized LLM applications. arXiv preprint.
• NYC Department of Consumer and Worker Protection (n.d., read 2026). Automated employment decision tools (AEDT). City of New York.
• Office of the New York State Comptroller (2025). Enforcement of Local Law 144: Automated employment decision tools. Report 2024-N-6.
• Rozado, D. (2026). Gender and positional biases in LLM-based hiring decisions: Evidence from comparative CV/resume evaluations. PeerJ Computer Science.
• Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology.
• Simonsohn, U., & Gino, F. (2013). Daily horizons: Evidence of narrow bracketing in judgment from 10 years of MBA admissions interviews. Psychological Science, via EurekAlert.
• TopResume (2025). AI in hiring survey. TopResume.
• U.S. Code of Federal Regulations (current text). 29 CFR 1602.14, preservation of records. Legal Information Institute.
• U.S. Code of Federal Regulations (current text). 29 CFR 1607.4, information on impact. Legal Information Institute.
• Uhlmann, E. L., & Cohen, G. L. (2005). Constructed criteria: Redefining merit to justify discrimination. Psychological Science, via Yale News.
• Van Iddekinge, C. H., Arnold, J. D., Frieder, R. E., & Roth, P. L. (2019). A meta-analysis of the criterion-related validity of prehire work experience. Personnel Psychology.
• Xu, J., Li, G., & Jiang, J. Y. (2026). AI self-preferencing in algorithmic hiring: Empirical evidence and insights. arXiv preprint.