job-ad-extractor
The reliability layer around an LLM call.
Messy job ad in, strict validated JSON out. Prompt versioning, schema validation and an eval harness turn "it seems to work" into a measured pass rate.
"Were hiring!! Sr. Support Eng (Sydney/hybrid) - 90-110k + super, must know SQL & APIs, 3+ yrs..."
{
"title": "Senior Support Engineer",
"location": "Sydney",
"employment_type": "full_time",
"seniority": "senior",
"salary_min": 90000,
"salary_max": 110000,
"skills": ["SQL", "APIs"]
}
LLM steps fail quietly: prose instead of JSON, a dropped field, an invented enum, a salary band where max is below min. Every output here passes a pydantic gate or gets rejected and logged. An eval runner scores each prompt version against a hand-labelled golden set, and a regression check flags any case that passed before and fails now.
- prompt versioning
- structured-output validation
- eval harness
- pydantic
- provider-agnostic client
weekend hack
What it actually took. Small on purpose. I built it to internalise one idea: the difference between an LLM call you wrote and an LLM call you can prove behaves. Writing the golden set by hand was tedious and it is exactly where the value lives, because "quality" became a number I can watch move between prompt versions instead of a vibe.