CV Matching · 31 August 2026
Does a name change which jobs you see?
An audit of Teamme.link’s AI job matching: 980 tests checking whether identical skills produce identical job suggestions when the candidate’s name, gender, ethnicity or age is changed.
- Model
- gpt-4o-mini-2024-07-18
- Temperature
- 0
- Calls
- 980 completed, 0 failed
- Corpus
- 25 CVs × 12 identities
- Prompt SHA-256
- 5e2fa289c28c6e75
- Redaction SHA-256
- 68bb26f4106ffa72
Summary
We could not detect any effect of name, gender, ethnicity or age on which roles a candidate is shown.
The honest caveat sits alongside it. The model’s own run-to-run variability is large: an identical CV resubmitted returns a different set of suggestions 61% of the time. That variability — not bias — is the most significant thing this audit found, and it limits how small an effect the audit could have detected.
Findings
- Personal information removed before analysis
- 100% · 0 leaks in 505
- Redaction verified on real candidate CVs
- 0 leaks in 70
- Effect of name, gender or ethnicity on matches
- None detectable
- Effect of age on matches
- None detectable
- Repeat consistency — high-confidence matches
- 86.0%
- Repeat consistency — lower-confidence matches
- 35.5%
- Indirect signals surviving redaction
- Inconclusive
It reorders a list. It does not screen anyone.
A visitor to a company’s careers site may upload a CV. The roles already shown on that page are then filtered and reordered to those that fit their background.
The CV is never stored, never scored, and never seen by the employer. It changes only what the candidate themselves is looking at, and every role stays open to apply to regardless of what the suggestions say.
How we tested it
25 base CVs spanning junior to senior across engineering, data, product, design, sales, marketing, finance and support. Each was rendered under 12 identities — 6 ethnicity signals crossed with two gender signals — changing only the name, email handle and filename. Skills, employers, achievements and years of experience were identical across all 12.
This is a paired design. Each CV is its own control, which is far more sensitive than sampling hundreds of unrelated CVs and comparing group averages.
Age was isolated separately: the same CV under three graduation-year cohorts, with years of experience held constant. Varying experience alongside age would have made the audit flag legitimate seniority judgement as age discrimination.
The order of the job list was shuffled on every call, because language models are sensitive to list position and that would otherwise be mistaken for identity bias. Every CV was run twice — once through the redaction step, once raw — to measure what redaction buys.
The control that makes the numbers mean anything
A fourth arm submitted an identical CV five times. Language models are not deterministic, so some share of results differ for reasons having nothing to do with identity. Everything below is measured against that noise floor.
The audit calls the production code directly — the same redaction and matching functions that serve real candidates, imported from the deployed source rather than reimplemented. An audit of a reimplementation would prove nothing about what ships.
Tested against real CVs, not only synthetic ones
The bias measurements use synthetic CVs, because they require swapping one candidate’s identity for another. Redaction — the step that removes identity before anything reaches the model — was separately validated against 70 real candidate CVs, supplied for testing and processed entirely offline: no model, no network, nothing retained.
It found three defects the synthetic corpus could not, because those CVs had been written to the shape the redaction step already assumed:
- Letter-spaced extraction defeated redaction entirely. Some PDFs position every character individually and extract as “D A N A C O H E N”. No pattern matched, so nothing was removed and the whole document would have reached the model spelled out — which a language model reads perfectly well. It also looked like a clean pass: with nothing matchable, there is nothing for a leak check to find either.
- National identity numbers were not detected at all. Several CVs carried a nine-digit identity number through redaction — the most sensitive field in the set.
- Unlabelled dates of birth survived, because the rule recognised only the labelled form and real CVs often just print the date.
All three are fixed and re-verified, and this report was re-run against the corrected version. That is why the redaction fingerprint above differs from the one published first: the earlier fingerprint described a step that would not have held up against real documents.
Across 70 real CVs, no email address, phone number, link, identity number, date of birth or name now survives redaction, and no CV loses more than 40% of its text. 2 of 72 files could not be read at all — scanned images rather than text — and are rejected rather than processed.
Result: no detectable effect
If identity had no effect, swapping it should disturb results no more than resubmitting the same file does. Each bar is the 95% confidence interval on that excess. Every interval crosses zero.
Notably the raw condition shows no detectable effect either. With these CVs and this model, redaction is defence in depth rather than the thing holding bias at bay. One incidental result: redaction slightly improved consistency, from 0.305 to 0.390.
Why we do not headline a parity score
Standard demographic parity ratios were computed with Fairlearn. At face value they look alarming — a minimum of 0.400 for ethnicityagainst the four-fifths rule’s 0.80 threshold.
Those numbers are not evidence of bias.We tested this directly by shuffling the identity labels within each CV — where they cannot possibly matter, because every variant is the same document — and recomputing 2,000 times.
| Condition | Attribute | Observed | Median under noise | p | |
|---|---|---|---|---|---|
| Redacted | Gender | 0.806 | 0.757 | 0.913 | noise |
| Redacted | Ethnicity | 0.400 | 0.444 | 0.389 | noise |
| Redacted | Age | 0.500 | 0.500 | 0.786 | noise |
| Raw | Gender | 0.842 | 0.769 | 0.908 | noise |
| Raw | Ethnicity | 0.500 | 0.500 | 0.724 | noise |
| Raw | Age | 0.500 | 0.429 | 0.993 | noise |
Random labels produce ratios just as low as the real ones, and not one comparison is significant. At this sample size, against a model this variable, the parity ratio fails the four-fifths rule even when there is provably nothing to find. It cannot separate bias from noise here. The paired comparison in §6 is the informative measurement; this table is reported because parity is the metric most often asked for, not because it carries weight in this setting.
This is also why no “perfect parity” target is quoted. It is not attainable and never was. A perfect score against a non-deterministic model would indicate a broken measurement, not a fair system.
The finding that actually matters
An identical CV resubmitted returns a different set of suggestions 61% of the time. The movement is small — a median of one position — and almost entirely confined to the lower-confidence tier.
High-confidence matches are stable. Lower-confidence matches are close to coin-flips at the margin. This is a product-quality issue before it is a fairness issue — a candidate who re-uploads sees a visibly different list — and it caps the audit’s sensitivity, since an effect smaller than the noise would not be visible.
Indirect signals: inconclusive, and reported anyway
Redaction removes names and dates. It does not remove signals that correlate with identity, so we probed those directly with skills held constant.
| Comparison | Identical result | Overlap | n |
|---|---|---|---|
| "Women in Tech" vs neutral mentoring | 33% | 0.631 | 6 |
| 14-month career break vs none | 0% | 0.581 | 6 |
The affinity-organisation result sits within the noise floor. The career-break result — the set changed in all 6 cases — is suggestive but not established: n = 6 is far too small, and unlike the name test it is not a clean comparison, because the career-break CV contains extra text the other does not. It warrants a properly powered follow-up rather than a conclusion. We report it because it is the test most likely to hide real bias, and publishing only the clean results would misrepresent the audit.
How the feature is built to be fair
First · carries the most weight
No employer ever sees the output. The CV is not stored, not scored, not ranked, and not transmitted to the hiring team. It changes only what the candidate is shown. This, not the model's behaviour, is the primary safeguard.
Second
Identity is removed before analysis. Names, contact details, addresses, dates of birth, nationality, gender and graduation years are stripped on our servers before anything reaches the model. Verified at 100% recall on the synthetic corpus and against 70 real CVs.
Third · weakest
The prompt constrains the model to competencies and seniority. A prompt instruction is not a guarantee, and we present it as defence in depth rather than as the control.
What this audit does not establish
- The bias measurements use synthetic CVs. 25 templates, professionally realistic but not real candidate documents. Redaction is separately validated against 70 real CVs; the bias arms are not.
- Real-CV validation covers redaction only. Deliberately so. Running the bias arms on real CVs would mean sending real candidates' details to the model — the exact thing this feature exists not to do.
- Names as identity proxies. The standard device in audit studies, and an imperfect one: a name signals perceived ethnicity, not actual ethnicity.
- English only, Latin script only for the bias arms, one synthetic employer with 22 roles.
- Underpowered proxy arm. 6 comparisons per probe. Directional at best.
- Limited detectable effect size. The model's own variability sets the floor; effects smaller than that band would not have surfaced. The finding is that no effect was detectable, which is not the same as establishing that none exists.
- Not tested: writing style and non-native phrasing, disability disclosure, parental status beyond a career gap, religious affiliation, socioeconomic signals, or CV formatting quality.
- Point-in-time. Valid for the exact model, prompt and redaction versions recorded above. Any change to those invalidates it — as the redaction fixes in §5 in fact did, which is why it was re-run.
What we are changing
- Show high-confidence matches only, or visually separate the lower tier. 86.0% consistent versus 35.5%. The highest-value change this audit identified, improving product quality and fairness headroom at once.
- Pass a fixed seed to the model so a candidate re-uploading the same CV sees the same list.
- Re-run the proxy probe properly— career gap and affinity signals across 25+ CVs rather than 6.
- Re-audit whenever the model, prompt or redaction logic changes. The fingerprints recorded at the head of this report define its scope.
Regulatory posture
Because no employer receives the output and the tool neither screens nor scores applicants, we do not consider it an Automated Employment Decision Tool under NYC Local Law 144, nor a high-risk employment system under Annex III of the EU AI Act. We consider GDPR Article 22 not to be engaged, as no decision with legal or similarly significant effect is made. Data minimisation is addressed by redaction and by retaining nothing.
This section states our reasoning, not a legal opinion.
Questions we are asked
- Is my CV stored anywhere?
- No. It is read in memory, stripped of personal details, used to reorder the list you are looking at, and discarded when the request ends. It is not written to disk, to a database, or to a log, and no candidate profile is created.
- Does the employer see my CV, or the suggestions?
- No. Nothing from the upload reaches the hiring team. This is the single most important fact about the feature and the reason it is not a screening tool — see §10.
- Can I still apply to roles that were not suggested?
- Yes, all of them. The upload only reorders and filters the list already on the page. Nothing is closed off, and the suggestions are not an assessment of whether you are qualified.
- I uploaded twice and got different suggestions. Why?
- Because the model is not deterministic. We measured this rather than hiding it: an identical CV returns a different set 61% of the time, almost entirely among the lower-confidence matches. §8 has the numbers and what we are doing about it.
- Does this fall under NYC Local Law 144?
- We do not consider it to. Local Law 144 governs Automated Employment Decision Tools — systems that screen, score or rank candidates to substantially assist an employment decision. No employer receives this output, nothing is scored or ranked for a hiring team, and no decision is made. It changes only what the candidate themselves sees. §13 sets out the reasoning, which is ours rather than counsel’s.
- Is it high-risk under the EU AI Act?
- We do not consider it to be. Annex III treats employment systems as high-risk where they are used to filter applications or evaluate candidates for an employer. This one is candidate-facing navigation and reaches no employer. As with Local Law 144, this is our reasoning and should be reviewed by your own counsel.
- What about GDPR Article 22?
- Article 22 covers decisions producing legal or similarly significant effects, made solely by automated means. No decision is made here at all. Data minimisation is addressed by removing personal details before analysis and by retaining nothing.
- What actually leaves your servers?
- The redacted text only, plus the job titles already public on the careers page. Name, contact details, address, date of birth, nationality, graduation year and identity numbers are removed first — verified at 100% recall on the synthetic corpus and against 70 real CVs. The uploaded file itself never leaves our servers.
- How often is this re-run?
- Whenever the model, the prompt or the redaction step changes — those three fingerprints are recorded at the head of this report and define its scope. That is not an aspiration: a test in our build pipeline compares the published redaction fingerprint against the code actually deployed and fails if they diverge.
- Can we see the underlying results?
- Yes. The harness, the corpus generator, the per-call raw results and the analysis scripts are retained, and every figure on this page is generated from them rather than typed in. Ask and we will share them.
Harness, corpus generator, per-call raw results and analysis scripts are retained and reproducible.