Building Jerry: The Model Is Not the Product

Most AI demos can summarize a stack of links. The harder job is deciding what matters to one person, reconciling sources that disagree, keeping every claim fresh and attributable, publishing on a schedule, and knowing when there isn't enough evidence to publish at all.

Jerry is the daily briefing in the Spot Sports app. Every morning it turns the teams and sports a fan follows into one ranked, source-backed read that ends. Underneath is a production agent that watches scores, schedules, reporting, video, and podcasts, decides what changed for each fan, writes, checks its own work, and publishes only when the whole package is ready.

I built it solo, after running analytics for ESPN+ and building the experimentation platform at The New York Times. That constraint shaped everything: a media-company-sized problem with a one-person operating footprint. It also settled an argument early. The model is not the product. The product is the system that decides what the model may see, say, and ship.

Start with the fan's morning

Those three screens are the whole product brief. The job is concrete: help a fan understand what happened, what matters next, and where to go deeper, without making them work through another endless feed. A late Mets result, an Arsenal injury, a race starting this afternoon, and a new podcast may all matter differently to the same person.

That gave me four rules before I chose any model. The briefing had to be personal, finite, source-backed, and ready on time. Those product decisions became the system's boundaries.

One of them is a bet against the industry's favorite metric. Feeds optimize for time in app. A briefing that ends optimizes for a fan trusting it enough to open it again tomorrow. The bar I set was simple and personal: keep iterating until Jerry was something I couldn't wait to open. A daily check-in that leaves you caught up instead of drained. Jerry is measured on whether fans come back and finish, not on how long they stay.

One morning inside Jerry

Imagine the Mets played after midnight. Five outlets recap the same final score, the schedule feed already knows the next game, and a new podcast lands before sunrise. Jerry collapses the five recaps into one story, raises it because the fan follows the Mets and the result is fresh, writes the summary from that evidence only, attaches the most useful media, and checks the score against the event record before publishing.

That is the loop: observe, decide, create, check, publish. Jerry is a workflow, not a free-running agent, and that is on purpose. The stages run in a fixed order. Observing and deciding are deterministic code: normalize, deduplicate, group reporting into source-neutral stories, and score each one on follow priority, freshness, live or upcoming events, impact, and whether yesterday's lead has actually changed. Creating and checking are where the model works, and each story block costs a handful of model calls: one to draft, and separate ones to grade every claim against the evidence the draft was given. No stage can widen its own scope. The model writes from the evidence Jerry selected. It does not decide what the evidence set should be.

Three decisions that shaped the system

1. Decide what Jerry is allowed to decide

Jerry may decide what matters. It may not decide what is true.

The hardest product decision was not which model to choose. It was how much freedom Jerry should have, and what proof it needed to earn more.

• Scores, schedules, and stats come from official feeds. Story claims come from linked reporting, not model memory.
• Each stage sees only the context it needs. Article text is treated as content, never instruction.
• Every run keeps its sources, model version, cost, checks, and failures so a decision can be replayed.
• The briefing publishes as one complete package. If a run is not trustworthy, the last valid briefing stays in place.

These boundaries make Jerry both safer and more capable, because a bounded decision does not need a human to review it. Jerry can make thousands of small decisions a morning without any one of them being an unreviewable leap of faith.

2. Shipping nothing beats shipping wrong

A bad demo is embarrassing. A bad recurring product erodes trust one morning at a time. So a run has to earn publication. Structured output is parsed against a schema, factual claims are checked against stored event and stat data, and the narrative is checked for unsupported claims, repetition, and policy. A story block can pass, be repaired, or be held. Archiving the old briefing and publishing the new one commit together, and an invalid draft never replaces the last valid one.

The boundaries earned their keep in August. One morning, Jerry drafted a Giants story that called quarterback Jaxson Dart a rookie. He isn't. He was a rookie the season before. Nothing in that day's sources used the word; the model reached into its training memory and filled it in. That is exactly the mistake a sports product cannot ship, because it sounds authoritative and is a year stale. The fact check caught the unsupported claim and held the story, and Giants fans saw yesterday's briefing instead of a wrong one.

3. The eval is the spec

A production agent needs a clear way to see its work, not just logs. Two internal reports do that job. The trace answers, "Why did this run behave that way?" The shadow report answers, "Did the change make Jerry better?"

The shadow report scores every replayed briefing on accuracy, relevance, sourcing, freshness, cadence, media, missing signals, and tone, across multiple days rather than a single sample, because the question is whether one briefing improves the next. Together, the two reports turn a model change into a product change I can review before releasing it.

That apparatus paid for itself the first time a cheaper model showed up. A newer variant priced about 75 percent lower looked like an obvious swap. I replayed 36 story blocks across 12 teams with the same evidence and the same checks, and had three different judge models score the results pairwise with positions shuffled. The candidate was equally grounded and about a second faster per block, but wrote roughly 25 percent shorter briefings and dropped storylines. A one-line prompt change closed the coverage gap, and it then won the comparison two to one. The swap is still on hold, for a reason covered below, but the decision now rests on a number I trust rather than a demo I liked.

What the first versions got wrong

The checker was too blunt. The same August check that caught the Dart claim also held three other stories that morning, and on replay all three were supported. The evidence was in a neighboring sentence the checker had not been shown. One true catch, three false alarms, four team briefings held for no reason. Failing closed was the right default. Silently losing one or two percent of briefings was not an acceptable steady state. So I added a narrow repair step: when the only problem is a risky label like "rookie," rewrite the story without the label and check it again. Every other factual failure still holds the story. Trust runs in both directions. Block what you cannot prove, and keep what you can.

On time can fail quietly. The morning run normally finishes in about three minutes. One day in August it took thirteen. Nothing errored. A cost guard I had added to cap spend was quietly serializing every model call to one at a time. The fix took a morning. The lesson took longer to sink in: "ready before the alarm" has to be a metric with an alert, not something you notice when the briefing feels late.

A cheaper model is only cheaper where it is billed. The model swap above stalled on the invoice, not the quality. The price depended on which endpoint the gateway routed to, and the production pin was landing on the expensive one. The real saving was half the headline. Cost has to be measured at the bill, per story block, or the eval is grading the wrong thing.

Shipping without betting the briefing

Before fans saw Jerry, it ran silently beside the deterministic baseline it grew out of. Same cases, same quality bar, old and new paths side by side. I shifted traffic only after Jerry held up across repeated mornings.

Every release was gradual and reversible. Feature flags and allowlists controlled who saw what, retries were bounded, and failures had a known recovery path. The Giants briefing held in August was republished by a recovery run once the repair step landed. No fan saw a blank. The product never had to bet a fan's morning on a good demo.

What good means in production

I judge Jerry from three directions. Fans should open, finish, explore, and respond. The briefing should cover the right stories without repetition and keep its sources attached. The platform should publish on time while staying fresh, fast, and affordable. A model upgrade is better only if it improves that whole picture against the same evidence.

Spot is early, so I won't dress up fan numbers that aren't meaningful yet. The editorial and platform numbers are already real. Over the last thirty days of scheduled morning runs:

The full run takes about three minutes at the median, from the morning trigger to one atomic publish, and every briefing is ready before a 7 a.m. Eastern alarm goes off. The one thirteen-minute morning is the p95, and it is the reason on-time now has an alert.

That cost line is the business story, not just a platform stat. A newsroom-shaped product that a media company would staff runs for about $80 a month, and the marginal cost of one more fan is close to zero. That is what lets one person operate it.

The result is a product, not a demo

Fans never need to see the stages, the checks, or the recovery paths. They see the right stories waiting, the sources close at hand, and a briefing that knows when to end. The complexity belongs to the platform, not the person using it.

Building Jerry hardened four rules I would now apply to any product that has to turn fast-moving information into something one person can trust:

• The model decides what matters. Data decides what is true. Salience is a judgment call the model is good at. Facts are not.
• Shipping nothing beats shipping wrong. The last valid version is always there, and a held story is a feature, not an outage.
• Every run is replayable. If you cannot see why it did that, you cannot ship the next change.
• Finite is a feature. A product that ends is one people can trust with a daily habit.

The next useful AI products won't be chat boxes. They will be systems that do a recurring job end to end: grounded in evidence, checked before they publish, cheap enough to run every morning, and honest enough to ship nothing when the evidence isn't there. The model is the smallest part of that build. The work is the contract, the ranking, the checks, the scorecard, and the rollout, and most of it is code you can read.

Jerry does that for sports fans. The same pattern fits a portfolio, a customer base, a league, or a beat. I built Spot to prove it, and Jerry publishes every morning.