A Filter Is Not the Same as a Finding
Ask a hiring manager why steady career progression, no employment gaps, or a specific degree predicts good performance, and the explanation tends to get thin fast. Most of these criteria became "standard" simply because they were repeated long enough to feel load-bearing. That is worth sitting with, because it describes a failure mode that shows up far beyond recruiting: a signal survives not because someone checked that it correlates with the outcome you care about, but because it is easy to measure, easy to defend, and uncomfortable to challenge once it is embedded in a process.
Resume structure, job titles, and career history are conventions — they tell you how someone packaged their experience, not what they can actually do. One study of resume photos found that candidates who included one landed jobs at a higher rate, even in markets like the U.S. and Canada where the practice is actively discouraged, especially in senior, client-facing roles. That is not a finding about competence. It is a finding about reviewer comfort, dressed up as a hiring standard.
The Evidence Hiding in the Reject Pile
Here is the part most teams skip: if you want to know whether a screening criterion actually predicts performance, you cannot just look at the people it let through. You have to look at the people it filtered out and ask what happened to candidates like them elsewhere. That rejected pool is a kind of hidden counterfactual — and in hiring, researchers occasionally get to check it.
Research from the Burning Glass Institute and Harvard Business School did exactly that. At firms that genuinely committed to skills-based hiring, employees hired without a degree into roles that had previously required one stayed in their jobs at a two-year retention rate 10 percentage points higher than their degree-holding peers, and saw roughly 25% higher average pay gains. That does not prove degree requirements are bad policy everywhere — the researchers were looking at firms that implemented the change seriously, and other differences between companies could be contributing to the gap. But it is a genuine test: a comparison between who was screened out and how they actually performed once given the chance. Most companies never run that comparison on their own filters. They just keep the filter.
When the Policy Changes but the Habit Doesn’t
This is where the hiring story becomes a sharper lesson about validation in general: changing a document is not the same as changing a decision. The Burning Glass Institute and Harvard Business School tracked more than 11,000 roles at large firms that had formally dropped degree requirements, following roughly 65 million career histories to see whether actual hiring behavior shifted. It mostly didn’t. Fewer than one in 700 new hires in 2023 benefited from the reform. Only about 37% of the firms that announced the change actually altered who they hired in a measurable way; nearly half changed the job posting and little else.
Meanwhile, the policy language keeps spreading. NACE’s Job Outlook 2026 survey found 70% of employers now use some form of skills-based hiring for entry-level roles, up from 65% the year before, while GPA screening has fallen from 73% of employers in 2019 to 42% today. That is real movement in stated practice. But job postings tell a more complicated story: the share of U.S. postings requiring a bachelor’s degree actually climbed back to 19.3% in November 2025 after bottoming out the year before. Policy and job-market behavior are not moving in lockstep, which is exactly the kind of gap a team should notice before declaring victory on an assumption.
| Screening criterion | Why it feels like a safe bet | What the evidence actually shows | What to do with it |
|---|---|---|---|
| Degree requirement | Signals effort, credibility, a known filter | At firms that dropped it seriously, non-degree hires retained better, but most firms never changed actual hiring | Test against your own performance data before keeping |
| Resume photo | Feels like a personal, trustworthy touch | Correlates with reviewer comfort, not demonstrated ability | Retire as a signal |
| No employment gaps / linear career | Reads as reliability | Not consistently linked to on-the-job performance in the available research | Test; don’t assume |
| GPA cutoff | Easy, quantifiable, defensible in a meeting | Adoption has fallen from 73% to 42% of employers as other signals replace it | Deprioritize unless tested for the specific role |
| Structured work-sample or skills assessment | Feels slower, more subjective to set up | Organizations that can assess skills directly are more likely to meet financial targets and report lower turnover | Keep, but verify it predicts your outcome |
The Automation Problem: Bias at Scale, Trust by Default
Adding AI to a screening process does not neutralize an untested assumption — it can simply execute it faster and at greater scale. A University of Washington study had 528 people select candidates with help from simulated hiring AI across 16 job types. When the AI gave no recommendation, people picked candidates across races at roughly equal rates. When the AI showed even moderate bias toward one race, human reviewers followed its lead — and in cases of severe, obvious bias, people still followed the AI’s picks around 90% of the time, even when they personally rated those recommendations as low quality. The lead researcher put it plainly: "Unless bias is obvious, people were perfectly willing to accept the AI’s biases". That is not evidence that any particular AI hiring tool is neutral or malicious by design — it is evidence that a tool’s assumptions get inherited by the humans using it unless someone actively checks.
From Announcement to Actual Behavior
flowchart LR
A[Policy announced] --> B[Job posting edited]
B --> C{Rubric and interviews also rewritten?}
C -->|No| D[Managers default to old habits]
C -->|Yes| E[Criteria tested against performance]
D --> F[AI tools reinforce existing patterns]
F --> G[Outcome: hiring barely changes]
E --> H[Outcome: measurable shift in who gets hired]
The gap in that diagram — between "policy announced" and "outcome changes" — is where most validation efforts quietly die, in hiring and in product work alike. A founder adds a feature because users "asked for it" in a survey, then never checks whether the people who received it actually behaved differently. A team "validates" a pricing model because a handful of interviews sounded enthusiastic, without comparing that enthusiasm to who actually paid.
A Simple Rule Before You Defend Any Filter
The fix Fuller and his co-researchers describe for hiring managers applies almost word-for-word to anyone testing an assumption: look at who your filter rejects, follow enough of them to see what would have happened, and only keep the criterion if it demonstrably predicts the outcome you care about. If a rule survives only because it is familiar, easy to write into a form, or hard to argue with in a meeting, that is not evidence — that is habit wearing the costume of evidence.
The organizations quietly gaining ground are not the ones with the boldest policy language. They are the ones treating every screening rule, hiring or otherwise, as a hypothesis that has to keep earning its place — and willing to retire it the moment it stops predicting anything real.
