Building an agentic pipeline to evaluate jobs in minutes, not hours

Two hours a day deciding which jobs were worth applying to

As a product manager in an active job search, I was losing up to two hours a day evaluating roles. The slow part was never reading the posting. It was the judgment around each one: is the location actually compatible or is that buried three paragraphs down, how would a recruiter read my non-obvious background against this specific role, and how much work would the application really take before I committed to it.

The two decisions that cost the most time were the ones I cared about most. Assessing recruiter-perceived fit meant digging through old resume versions to decide what to actually change. Estimating application effort meant guessing at custom questions and cover-letter work with no fast way to plan before committing. I wanted a single command that took a posting and returned a clear go or no-go plus an honest effort estimate, in the time it takes to read a few paragraphs.

A sequential funnel that fails fast

I designed the system as a funnel rather than running every check at once. Each stage can stop the pipeline before the next one runs, so a role that fails a hard rule, the wrong location or not a product role, never spends tokens on the expensive reasoning stages downstream. Ordering the stages this way is a deliberate cost decision, not just a logical one.

The criteria live in plain-text config files, not in the code. I can tighten a disqualifier rule and rerun without touching Python, and every change is versioned in Git for free. Every failed role also appends one row to a market-intelligence log, capturing the company and reason, because postings expire but companies do not. When my constraints change, for example if I relocate, I can filter that log for companies that would now be in scope.

The payoff runs in both directions. When a role clears the funnel, I arrive at the recruiter conversation already knowing where my background needs translation and where it maps cleanly, which makes those first calls better for both of us.

Where the real work was

The pipeline itself was fast to build. The decisions that mattered were about cost, reliability, and how to make the thing tunable without rewriting code each time.

Model selection as a cost decision. Not every stage needs the same horsepower. The hard disqualifier check has to reason about ambiguous location language, "globally distributed team," "European time zone preferred," so it runs on a stronger model. Pulling a company name out of a posting or scoring against a fixed rubric is mechanical, so those run on a cheaper, faster one.

Graceful degradation over hard failures. A personal tool that crashes is a tool you stop using. Company research is optional: if the Perplexity key is missing or the call fails, the run notes the gap and continues instead of dying. URL extraction falls back to paste. A lightweight content check catches bad fetches, login walls, privacy pages, full-page dumps, before they waste a real API call.

Prompts and criteria as config, not code. Every prompt and every rule lives in its own versioned file. This turned out to be the decision that mattered most, for a reason I did not fully appreciate until I started tuning.

I tested it like a product, not a script

Before writing any pipeline code, I built a golden set of eight real job postings I had already evaluated by hand, each labeled with the outcome I expected: hard fail, soft fail, or pass, at different effort tiers, plus one deliberately awkward edge case. That set became a regression suite. Every time I changed a prompt, I reran the whole set and checked nothing regressed.

Because LLM outputs are probabilistic, I also ran the same postings several times to confirm the signal was stable. Variation in wording is fine. Variation in the go/no-go result or the effort tier is not. And I weighted the testing toward false positives on purpose: a strong role wrongly flagged as disqualified is a real opportunity lost, which is worse than a missed disqualifier I would have caught by eye anyway.

Did it work?

The validation run came back clean: 8 of 8 golden-set postings landed on the expected outcome, stable across reruns. In day-to-day use, a full run takes about 30 seconds, and reviewing the report takes me under five minutes.

Before the pipeline, my daily review meant scanning job alerts, then opening each posting and reading it end to end to decide whether I was qualified. Now I scan the alerts, run the postings I’m unsure about through the pipeline, and decide from the one-page report whether a role goes on my apply list. Counting the runs and reading the reports, that review went from up to two hours a day to roughly ten minutes, by my own estimate from daily use. That time didn’t disappear, it moved. I spend it on the applications themselves, the part of the search that actually needed me. The measurement still in progress is estimated versus actual application effort, which I’m logging as I dogfood the tool through my own search.

What building it actually taught me

The biggest surprise was where the time went. I expected the engineering to be the hard part. Instead the build was quick and the tuning was slow, and almost all of the tuning was prompt design validated against the golden set. The effort estimator over-flagged resume work until I told it to default to "none" unless the posting demanded otherwise. These were judgment calls encoded in language, not logic in code, which is a different muscle than I expected to use.

It is honest about its limits. Paste mode loses some application-effort signals that are only visible in the live page, so the estimate can read one tier high. A few job boards still need the paste fallback. The next step is the most useful one: dogfooding it through my own search to log estimated versus actual application effort, then tuning the rubric against real data.

Built with: Python, Anthropic Claude API (Sonnet and Haiku), Perplexity Sonar API, Jina AI Reader. Config-driven criteria, versioned in Git.

Part of my side projects: a set of builds from my 2026 sabbatical, each exploring a different AI paradigm hands-on. A walkthrough of the code is available on request.