Experiment 2: structured output, how often “reply with JSON” breaks (run 2026-10-22, gpt-4o-mini)
Scripts: corpus.py (picks 30 real postings from teste-a/data/ats_detail_cache.json, 1 per company, saved to corpus.json), json_extract.py (run 1), json_extract2.py (run 2b). Raw outputs: json_extract_raw.json, json_extract2_raw.json. Cost: see cost_log.jsonl (run 1 + 2b = US$0.0557 cumulative at the end of 2b).
Setup: 20 real postings (corpus[0:20]), same extraction prompt (7 fields incl. accepts_brazil = yes/no/unclear), API default temperature 1.0, 3 runs per posting per mode. Modes: plain (prompt says “reply with JSON only”), object (response_format: json_object), schema (json_schema, strict).
Run 1 (180 calls)
| mode | n | json.loads on raw text OK | needed ``` fence strip | schema-valid after parse | postings where 3 runs disagree on accepts_brazil |
|---|---|---|---|---|---|
| plain | 60 | 0 | 60 | 60 | 3/20 |
| object | 60 | 60 | 0 | 59 (1x accepts_brazil="unknown", outside the enum) | 3/20 |
| schema | 60 | 60 | 0 | 60 | 8/20 |
Other disagreement (postings with ≥1 differing run of 3): seniority plain 3, object 2, schema 2; main_languages plain 6, object 3, schema 8; salary_min_usd 0/1/1; min_years 0/0/0.
Run 2b (120 calls): the 4 postings whose location excludes Brazil
Postings: Europe-only founding role; a list of EU countries + Canada; San Francisco + other US cities (“Location: United States, Remote”); “Remote United States”. 10 runs per mode each, counting accepts_brazil.
| mode | wrong “yes” (of 40) | other |
|---|---|---|
| plain | 2 | 2x “unknown” (outside the enum), rest “unclear”, 2 “no” |
| object | 2 | rest “unclear”, 1 “no” |
| schema | 12 | ”Remote United States” → “yes” in 8/10 |
Combined run 1 + 2b, those 4 postings (13 runs each per mode = 52): wrong “yes” plain 2/52, object 2/52, schema 16/52 (31%).
Takeaways (honest version)
- Asking nicely for JSON: 0 of 60 replies parsed as-is. All 60 were wrapped in ```json fences. Strip the fence and 60/60 parse. So “it worked” depends on whether your parser strips fences.
json_objectfixes syntax, not the contract: 1 of 60 used a value outside the allowed enum (and plain mode did it twice in 2b).- Strict schema gave 100% valid shape (60/60 + 40/40) and the most wrong answers on the Brazil field: 16/52 “yes” on postings that exclude Brazil, vs 2/52 for the other two modes.
- Caveats: 1 model, 1 prompt, 20 postings, 3-10 runs. “unclear” for a Europe-only role is arguably fine; I only count “yes” as wrong. I don’t know why schema mode answers “yes” more; my guess is constrained decoding shifts which token wins, but I didn’t test that.