A second AI judge for my leads: 600 posts, 3 cents

TL;DR — SignalScout reads Reddit looking for people who are asking for a tool to solve their own problem. A classifier decides which posts count. I wanted to know whether a newer model would decide better, so I put it behind the existing one in shadow mode: it reads the same posts the real classifier reads, writes down its own verdict, and is not allowed to change anything. Three days, 600 judgments, zero failures, about 240 ms each, and roughly three cents in total. The two models disagree a lot. I don't yet know which one is right, and this post is mostly about why that's the honest place to be.

Update, Oct 13: the CPU model described below has since been switched off on my server — it was starving video rendering of CPU — so SignalScout's main classifier now runs on a cloud model. The numbers here are from Oct 9–11, before that change. Any comparison from Oct 13 on is against the cloud model, and the follow-up will keep the two periods separate.

The setup

SignalScout's main intent classifier is a small open model running on CPU on my own server, with a cloud model as a fallback. It's free to run, but slow, and it hedges. For each Reddit post it records a post type (request, promotion, advice, discussion, unknown), whether the author is looking for a solution, and a 0–10 score. Anything at 6 or above becomes a lead a customer might see.

The challenger is Jev, a hosted judging model from TypeSafe. I asked it three narrow questions about each post:

1. What kind of post is this? Same five types.
2. Is the author looking for a solution to their own need? Yes or no, with a probability.
3. How strong or urgent is that need? One of four levels.

The wiring is deliberately boring:

• The judge only runs after the main classifier has finished and its result is saved. If the judge errors, times out or runs out of budget, the main pipeline never notices.
• It stores its verdict next to the original classifier's, in its own table. Lead scores, rankings and what customers see are untouched.
• The model and the rubric are pinned to exact versions, so the numbers stay comparable across days.

That's what shadow mode means here. You get real production traffic without real production risk.

Three days of numbers

Operationally there's nothing to complain about. It's fast, it didn't fail once, and a day of judging costs about a penny.

Where the two models disagree

The current classifier hedges. It labelled 57% of posts "unknown" (341 of 600). The challenger did that for 9%. One model shrugs most of the time; the other commits.

The challenger is much stricter about buyers.

• Of the 117 posts the current classifier called a request, the challenger agreed on 33. It called 44 of the rest discussion and 34 advice.
• Of the 24 posts that cleared the lead threshold, the challenger would have flagged 7 as genuine buyers.
• Across all 600 it flagged 27 buyers.

It's also often unsure. By the rule I set in advance (unknown type, low type confidence, or a buyer probability between 0.2 and 0.8), 52% of its answers count as uncertain.

The tempting headline is "new model is 3x stricter, therefore 3x better." I can't write that, because agreement isn't accuracy. If the challenger rejects 17 of 24 leads, either it's catching 17 false positives or it's throwing away 17 real buyers. Those are opposite conclusions, and nothing in these 600 rows tells me which one is true. The only thing that can is a human reading the posts.

Three things I'd tell anyone doing this

1. Charge the budget before every attempt, not after. Each HTTP request, retries included, claims one unit from a daily counter in Postgres before it's sent, with an atomic update shared across every worker. Failures don't get a refund. If the counter can't be read, the judge simply doesn't run. A per-day number that's checked after the fact is a hope, not a cap.

2. A daily cap is also a sampling decision. I set the cap at 200 calls a day. Every day it was used up roughly 70 minutes after midnight UTC, so all 600 judgments come from the same slice of the clock: evening in the US, and probably a narrower mix of subreddits than the full day. The rest of each day got no second opinion at all. At about a penny a day, the cap was protecting a budget that didn't need protecting while quietly skewing the data. The fix is to raise it, or to sample one post in N evenly through the day.

3. Expect the provider to round. During a test run, one response reported a score of 0.01 alongside probabilities whose weighted sum was 0.02. Both numbers had been rounded to two decimals independently, and my strict consistency check rejected a valid response I'd already paid for. The check now allows exactly the drift that two-decimal rounding can produce, and still rejects anything bigger. I keep the provider's numbers as reported rather than renormalizing them into fake precision.

What happens next

The evaluation tool is already built. What's missing is the slow part: hand-labelling 100–200 real posts, deciding for each one whether the author is actually asking for a solution, before looking at either model's answer. Then both models get scored against those labels, with precision, recall and how often each one abstains, broken down by source.

If the challenger holds up, it could replace the CPU model as the main classifier: answers in a quarter of a second instead of the seconds a CPU model takes, for pennies a day. If it doesn't, shadow mode will have cost me three cents to find out. That follow-up post will have the real numbers.