Fairness Audit · 31 August 2026
Does a name change which jobs you see?
An audit of Teamme.link’s AI job matching: 980 tests checking whether identical skills produce identical job suggestions when the candidate’s name, gender, ethnicity or age is changed.
Summary
We could not detect any effect of name, gender, ethnicity or age on which roles a candidate is shown.
The honest caveat sits alongside it. The model’s own run-to-run variability is large: an identical CV resubmitted returns a different set of suggestions 59% of the time. That variability — not bias — is the most significant thing this audit found, and it limits how small an effect the audit could have detected.
What we found
It reorders a list. It does not screen anyone.
A visitor to a company’s careers site may upload a CV. The roles already shown on that page are then filtered and reordered to those that fit their background.
The CV is never stored, never scored, and never seen by the employer. It changes only what the candidate themselves is looking at, and every role stays open to apply to regardless of what the suggestions say.
How we tested it
25 base CVs spanning junior to senior across engineering, data, product, design, sales, marketing, finance and support. Each was rendered under 12 identities — 6 ethnicity signals crossed with two gender signals — changing only the name, email handle and filename. Skills, employers, achievements and years of experience were identical across all 12.
This is a paired design. Each CV is its own control, which is far more sensitive than sampling hundreds of unrelated CVs and comparing group averages.
Age was isolated separately: the same CV under three graduation-year cohorts, with years of experience held constant. Varying experience alongside age would have made the audit flag legitimate seniority judgement as age discrimination.
The order of the job list was shuffled on every call, because language models are sensitive to list position and that would otherwise be mistaken for identity bias. Every CV was run twice — once through the redaction step, once raw — to measure what redaction buys.
The control that makes the numbers mean anything
A fourth arm submitted an identical CV five times. Language models are not deterministic, so some share of results differ for reasons having nothing to do with identity. Everything below is measured against that noise floor.
The audit calls the production code directly — the same redaction and matching functions that serve real candidates, imported from the deployed source rather than reimplemented. An audit of a reimplementation would prove nothing about what ships.
Result: no detectable effect
If identity had no effect, swapping it should disturb results no more than resubmitting the same file does. Each bar below is the 95% confidence interval on that excess. Every interval crosses zero.
Notably the raw condition shows no detectable effect either. With these CVs and this model, redaction is defence in depth rather than the thing holding bias at bay. One incidental result: redaction slightly improved consistency, from 0.280 to 0.410.
Why we do not headline a parity score
Standard demographic parity ratios were computed with Fairlearn. At face value they look alarming — a minimum of 0.400 for age against the four-fifths rule’s 0.80 threshold.
Those numbers are not evidence of bias. We tested this directly by shuffling the identity labels within each CV — where they cannot possibly matter, because every variant is the same document — and recomputing 2,000 times.
| Condition | Attribute | Observed | Median under noise | p | |
|---|---|---|---|---|---|
| Redacted | Gender | 0.778 | 0.750 | 0.719 | noise |
| Redacted | Ethnicity | 0.500 | 0.444 | 0.901 | noise |
| Redacted | Age | 0.400 | 0.500 | 0.196 | noise |
| Raw | Gender | 0.700 | 0.760 | 0.234 | noise |
| Raw | Ethnicity | 0.444 | 0.500 | 0.398 | noise |
| Raw | Age | 0.500 | 0.429 | 0.980 | noise |
Random labels produce ratios just as low as the real ones, and not one comparison is significant. At this sample size, against a model this variable, the parity ratio fails the four-fifths rule even when there is provably nothing to find. It cannot separate bias from noise here. The paired comparison above is the informative measurement; this table is reported because parity is the metric most often asked for, not because it carries weight in this setting.
This is also why no “perfect parity” target is quoted. It is not attainable and never was. A perfect score against a non-deterministic model would indicate a broken measurement, not a fair system.
The finding that actually matters
An identical CV resubmitted returns a different set of suggestions 59% of the time. The movement is small — a median of one position — and almost entirely confined to the lower-confidence tier.
High-confidence matches are stable. Lower-confidence matches are close to coin-flips at the margin. This is a product-quality issue before it is a fairness issue — a candidate who re-uploads sees a visibly different list — and it caps the audit’s sensitivity, since an effect smaller than the noise would not be visible.
Indirect signals: inconclusive, and reported anyway
Redaction removes names and dates. It does not remove signals that correlate with identity, so we probed those directly with skills held constant.
| Comparison | Identical result | Overlap | n |
|---|---|---|---|
| "Women in Tech" vs neutral mentoring | 33% | 0.681 | 6 |
| 14-month career break vs none | 0% | 0.581 | 6 |
The affinity-organisation result sits within the noise floor. The career-break result — the set changed in all 6 cases — is suggestive but not established: n = 6 is far too small, and unlike the name test it is not a clean comparison, because the career-break CV contains extra text the other does not. It warrants a properly powered follow-up rather than a conclusion. We report it because it is the test most likely to hide real bias, and publishing only the clean results would misrepresent the audit.
How the feature is built to be fair
Layer 1 · carries the most weight
No employer ever sees the output. The CV is not stored, not scored, not ranked, and not transmitted to the hiring team. It changes only what the candidate is shown. This, not the model's behaviour, is the primary safeguard.
Layer 2
Identity is removed before analysis. Names, contact details, addresses, dates of birth, nationality, gender and graduation years are stripped on our servers before anything reaches the model. Verified at 100% recall.
Layer 3 · weakest
The prompt constrains the model to competencies and seniority. A prompt instruction is not a guarantee, and we present it as defence in depth rather than as the control.
What this audit does not establish
- Synthetic CVs. 25 templates, professionally realistic but not real candidate documents, and less varied in format than the real world.
- Names as identity proxies. The standard device in audit studies, and an imperfect one: a name signals perceived ethnicity, not actual ethnicity.
- English only, Latin script only, one synthetic employer with 22 roles.
- Underpowered proxy arm. 6 comparisons per probe. Directional at best.
- Limited detectable effect size. The model's own variability sets the floor; effects smaller than that band would not have surfaced.
- Not tested: writing style and non-native phrasing, disability disclosure, parental status beyond a career gap, religious affiliation, socioeconomic signals, or CV formatting quality.
- Point-in-time. Valid for the exact model, prompt and redaction versions identified above. Any change to those invalidates it.
What we are changing
- Show high-confidence matches only, or visually separate the lower tier. 87.5% consistent versus 37.0%. The highest-value change this audit identified, improving product quality and fairness headroom at once.
- Pass a fixed seed to the model so a candidate re-uploading the same CV sees the same list.
- Re-run the proxy probe properly — career gap and affinity signals across 25+ CVs rather than 6.
- Re-audit whenever the model, prompt or redaction logic changes. Those three fingerprints define this report’s scope.
Regulatory posture
Because no employer receives the output and the tool neither screens nor scores applicants, we do not consider it an Automated Employment Decision Tool under NYC Local Law 144, nor a high-risk employment system under Annex III of the EU AI Act. We consider GDPR Article 22 not to be engaged, as no decision with legal or similarly significant effect is made. Data minimisation is addressed by redaction and by retaining nothing.
This section states our reasoning, not a legal opinion.
Audit provenance
- Model
- gpt-4o-mini-2024-07-18
- Temperature
- 0
- Calls
- 980 completed, 0 failed
- Prompt SHA-256
- 5e2fa289c28c6e75
- Redaction SHA-256
- 9a7d2a1d15050ef8
- Corpus
- 25 CVs × 12 identities
Harness, corpus generator, per-call raw results and analysis scripts are retained and reproducible. Questions about this audit: hello@teamme.io