Experiment 3: LLM-as-judge, same job scored 20 times (run 2026-10-22, gpt-4o-mini)
Why: my job digest only sends me a role if an LLM scores it ≥ 4.0 (~/projects/jobsearch/jobsearch/calibrate.py: “Aplicar” if nota ≥ 4.0). The digest uses another model (Sonnet + a ranker); gpt-4o-mini here is a cheap stand-in, so this says nothing direct about the digest’s accuracy.
Scripts: judge.py (10 postings = corpus[::3], 10 runs at temperature 0 + 10 at 1.0 = 200 calls), judge_why.py (6 calls, asks for a reason). Raw: judge_raw.json, judge_why_raw.json. Cost: cumulative US0.0822 after both (see `cost_log.jsonl`).
Prompt: candidate profile (senior full-stack/product, TS/React/Node/Python/AI agents, lives in Brazil, wants fully remote from a foreign company ≥ US100k) + “score 1.0-5.0, one decimal, consider whether someone in Brazil can be hired, reply with only the number”.
Results
| posting (location) | temp 0: min-max, ≥4.0 | temp 1: min-max, ≥4.0 |
|---|---|---|
| Founding Senior SWE (Europe – Remote) | 4.2-4.5, 10/10 | 4.0-4.5, 10/10 |
| Senior Fullstack Engineer (BR based only), US YC startup, Node/TS, “ship with coding agents” | 1.0-1.0, 0/10 | 1.0-1.5, 0/10 |
| Senior Backend, Foundations (LATAM / US / UK remote) | 4.0-4.0, 10/10 | 3.5-4.3, 8/10 (flips) |
| Senior Backend (Canada + EU countries) | 2.5, 0/10 | 1.5-2.5, 0/10 |
| Software Engineer Python (Brazil/Argentina) | 3.5, 0/10 | 3.5-3.8, 0/10 |
| Ramp - Software Engineer (Remote / Argentina) | 3.5, 0/10 | 3.5-4.0, 2/10 (flips) |
| Senior SWE Ledger (Global Anywhere) | 4.5, 10/10 | 4.2-4.5, 10/10 |
| Senior Agentic AI Engineer (Latin America) | 4.5, 10/10 | 4.0-4.5, 10/10 |
| Senior SWE Product/AI (Americas Remote) | 4.5, 10/10 | 4.5-5.0, 10/10 |
| Technical Solutions Engineer (US-Remote) | 3.5, 0/10 | 3.5-3.9, 0/10 |
Summary: temperature 0 → 0/10 postings flip across the 4.0 line, mean spread 0.03, max 0.3. Temperature 1.0 → 2/10 flip, mean spread 0.53, max 1.0.
Asking why (judge_why.py, temp 0, 3 runs each)
- Europe-only role: “Score: 4.0” x3, reason says strong fit “although the location preference for Europe may pose a challenge for hiring from Brazil”.
- BR-based-only role: “2.0” x3 (it was 1.0 without the reason request), reason x3: “explicitly states that it is only for candidates based in Brazil, while the candidate is seeking a fully remote role from a foreign company.” The model read “Brazil-based” as the opposite of “foreign company”.
Takeaways (honest version)
- Temperature 0 made the judge consistent, not correct: the Europe-only role passed my 4.0 line 20/20 times, and the posting that matched my profile best on paper (senior full-stack, Node/TS, agents, Brazil-based) scored 1.0 20/20 times.
- At temperature 1.0, 2 of 10 postings land on both sides of 4.0 depending on the run.
- Asking for a one-sentence reason moved the scores (1.0 → 2.0; 4.2 → 4.0). The reason is the useful part: it exposed the misreading in one line.
- Caveats: 10 postings, 1 cheap model, 1 untuned prompt. A better prompt (“Brazil-based roles at foreign companies are a fit”) would probably fix the 1.0 case; the point is that nothing in the score told me it was broken.