Tuning an AI recruiting agent: what we measured and what moved

What the agent does

AutoTalent turns a job requisition into a pipeline. An LLM extracts the must-haves from the job description and writes search queries. The agent runs those queries against a LinkedIn search service, stores the profiles, and scores each candidate from 0 to 100 against the role. Above a threshold, it can find an email address, draft personalised outreach, send it from the recruiter’s Gmail and classify the replies. MCP, the Model Context Protocol, exposes the same steps as tools, so a recruiter’s own AI assistant can run a search, review candidates or send a message.

An autopilot mode repeats search-and-score rounds until it reaches a target number of qualified candidates, within safeguards on profiles searched, money spent and rounds without progress. That loop is what we tuned.

A field test before any tuning

On September 7 I ran the whole path live in production, the way a recruiter would. A lot worked. The setup wizard extracted requirements and generated four queries. The MCP search returned 25 full profiles in about 30 seconds, and scoring 25 candidates took about two and a half minutes, with sensible results. Outreach drafts were well personalised, a Gmail send arrived in seconds, and an inbound reply was classified as “interested” with 99 percent confidence.

A lot didn’t, and most of it was configuration, not code. The email provider’s key had been dead since April. The MCP server and the web app held different encryption keys, so sending through MCP failed. Missing secrets left a scheduled job and an inbound webhook unauthenticated. The code bugs were the kind only a real run shows: launching a requisition from the web timed out and left it stuck, the email a recruiter approved through MCP wasn’t the email that was sent, and an autopilot round through MCP searched zero profiles because company names were never resolved to the identifiers the search needed.

That last bug became the thread for the rest of the month.

Where a round’s time and money went

On September 19 an agent benchmarked a full autopilot run on a test requisition, aiming for 100 qualified candidates. The numbers were bad in an informative way.

At 2 percent, 100 qualified candidates would need about 5,000 profiles, well past every safeguard. The fixes were mostly plumbing. Searches ran in parallel and candidates were scored eight at a time. Queries got bigger, since search time barely grew with size. A company-filtered query that came back empty was retried without the filter, real company identifiers were learned from scraped profiles, and identical searches were cached for a week. One round then searched 120 profiles, scored 103 and qualified 22, in 172 seconds for $0.67.

The surprise was in the retry. Queries without the company filter qualified 18 percent of their profiles, against about 2 percent with it. The guessed company names hadn’t just wasted rounds. They had been filtering out the good candidates. The cost model had its own gap: the code assumed a flat $0.003 per profile, and its $25 spending cap was counted at those prices, so it could never trip before the 2,000-profile limit. At real prices, 2,000 profiles cost $8 to $40 to search, plus about $6 to score.

A fast judge that became a gate

Scoring was the slow, expensive step. So we tested Jev, a hosted model that answers typed questions, such as yes or no, one of several choices, or a graded score, and returns calibrated probabilities in a fraction of a second for a fraction of a cent. The design broke the LLM’s one big scoring call into small questions: one yes/no per must-have (“does the profile show this?”), one graded score per nice-to-have, a relevance score and a recommendation. The policy stayed in code, so a recruiter could reweight the parts without re-running anything.

An offline eval re-judged 600 already-scored candidates: all 30 that a recruiter had approved, plus 570 at random. It ran first against a local stand-in model, then against the real service.

The reading was clear. On the item-level questions the fast model tracked the LLM closely, at about a thirtieth of the time. As a ranker it was worse against the people who mattered, and far stricter at the top. The labels were thin too: of 1,660 scored candidates, only about 30 had any recruiter action. So it didn’t replace the scorer. It decides who is worth the expensive full profile and the full score, and the LLM still writes the 0–100 and the reasons for the survivors.

The new round, measured

The two-stage round shipped on September 21 and 22. Several queries run in a cheap search mode that returns short profiles. Duplicates and people already in the pipeline are dropped. The fast model screens every remaining hit, full profiles are fetched only for the top few, and the LLM scores those. A bench script ran one round of each design on the same test requisition, with real money and real vendors.

The new round saw seven times as many people, and its expensive screen passed twice as often because the gate had removed the obvious misses. It delivered 3.3 times the qualified candidates for four times the search dollars. Counting the LLM scoring, both paths came to $0.14 per qualified candidate, but the new one took 25 seconds per qualified candidate instead of 40, and a third as many rounds. The bench also corrected the prices: the vendor was billing about 20 percent less per search page than the code assumed. The account’s billing meter turned out to be useless for attribution while another job ran, so the bench now sums each run’s own usage. A follow-up cut each round to four smaller searches and added a stop after two low-yield rounds.

A coding agent that overbuilt

Not all the tuning was of the product. In early September an autonomous coding agent ran on AutoTalent with the goal “launch ready for marketing distribution”. In three days it wrote a 23-part plan and 60 unpushed commits, took the database to its 70th migration, and grew the MCP server from 32 tools to 99. All 3,104 tests passed, and nothing was deployed.

My review found the product loop finished on September 5, after the first stretch of commits. Everything after that was control plane: spend ledgers that needed price rows nothing created, review-token handshakes, and a seven-table ledger around a single API call. 27 of the 34 new tools were get, cancel or resume plumbing. There was still no way to set up a new customer without about 20 hand-written database rows. The verdict was to cut a release at the product-complete commit plus two fix passes, park the rest, and pull pieces from it by topic. The same review found a scheduled job left unauthenticated in production and flagged it for the same day.

The lesson is about goals, not code. “Launch ready” has no finish line, so the agent kept building infrastructure for a scale the product didn’t have. A goal of “a recruiter can run one requisition end to end” would have stopped it on September 5.

Keeping a person on the decision

The last change of the month was about who decides. On September 29 the pipeline stopped rejecting candidates automatically. A missed must-have became a signal on the candidate instead of a filter, and the sourcing list became a stack rank that a recruiter reviews. The roadmap now requires human review before any automated rejection, ahead of state laws that regulate automated decisions in hiring. A fast gate is fine for deciding where to spend money. It shouldn’t be the thing that tells a person no.

Lessons learned

• Run the real path in production before tuning anything. Most of what broke was configuration.
• Measure the whole funnel. The biggest lever was a filter quietly removing the best candidates.
• Check the cost model against real bills. A safeguard counted at the wrong price never trips.
• A fast judge that agrees with the slow one on details can still rank worse. Use it as a gate, and keep the better ranker for the decision.
• Give autonomous agents a narrow goal with a finish line, or they will build for a scale you don’t have.
• Keep people on the decisions that affect people.