Scoping an AI Feature So It Doesn't Balloon

AI features tend to balloon. The initial scope says "add an AI summary button." Three months later the team is arguing about eval harnesses, streaming architecture, and whether the summary should support 12 languages. Meanwhile the button still isn't in production.

This post is the working version of how to scope an AI feature so it ships in weeks instead of quarters. Not a template. The actual questions to answer before writing any code.

Why AI features balloon

Three common reasons. First, teams underestimate the plumbing — evals, monitoring, fallbacks, cost tracking, user feedback. The model call is 10% of the work; the surrounding system is the rest.

Second, "AI" tempts teams to over-scope quality. "Should be perfect" replaces "should be useful enough to ship." That gap swallows months.

Third, boundaries between AI features and adjacent features blur. An AI search inside a document editor turns into a general chatbot, which turns into a full agent, which turns into a rewrite of the app. Every AI project we've seen ship on time had aggressive scope discipline.

The one-page scope document

Before writing code, write a page. Not a spec, not a PRD — a scope. If you can't fit it on a page, you don't understand the feature yet.

Who is this for

Which user, doing what job, in which moment. "Users" is not an answer. "Support agents drafting a first-response reply within 60 seconds of a ticket being assigned" is an answer.

The narrower the user + moment, the smaller the feature. A support-agent drafting tool is smaller than "an AI assistant." A summary of the last 5 emails in a thread is smaller than "an inbox AI."

What does success look like at v1

What does the user do that they weren't doing before, and how do you know? Concrete: "Agent uses the draft as-is or after light editing on X% of tickets in the first two weeks."

Not "improved productivity." Not "10x faster." A specific, measurable action. If you can't articulate this, ship a smaller feature first.

What is explicitly out of scope

The most valuable section. "This feature does not support translation. Does not handle attachments. Does not learn per-user preferences. Does not remember prior conversations." Write down what the feature doesn't do so nobody adds it mid-build.

A pattern we see often: the "out of scope" list is longer than the "in scope" list. That's a healthy sign.

What quality bar ships

What accuracy, hallucination rate, or user-signal threshold does v1 need to clear? Concrete numbers. "The draft is accepted or lightly edited 60% of the time on the first 100 tickets." "Fewer than 5% of drafts introduce a factual error caught in review."

The point of writing the bar down is that "good enough" is not a shared understanding until it's a number. Teams that skip this end up polishing forever.

The plumbing to plan for

Every AI feature needs, at a minimum:

An eval set. 30–100 examples with expected outputs. Built before the feature ships, run on every prompt change. Our AI evals post is a starting point.

User feedback capture. Thumbs up/down, edit tracking, or acceptance rate. Something that tells you if the feature is working after launch.

Cost tracking. Tokens per user per day. If you can't see this, cost surprises will find you.

An outage plan. What happens when the provider is down. Even a "try again in a minute" message counts, as long as it's designed.

A prompt-versioning story. When the prompt changes, you need to know when and why. A git-tracked file is enough at v1.

These are not premature optimization. They're the difference between an AI feature that ships once and an AI feature that keeps working.

Where scope actually creeps

"What if the user wants to…"

Every AI feature invites use cases that weren't in scope. "What if they ask a follow-up?" "What if they want to see the source?" "What if they want to edit the prompt?" Each is reasonable. Each is out of scope for v1.

Write those in a follow-up backlog. Ship v1. See if anyone actually wants them.

Quality goalposts

A demo that works 70% of the time can trigger a push to make it work 95% of the time before shipping. If 70% is good enough to be useful, ship at 70%, learn, then improve. If 70% isn't good enough, you probably need a different feature, not more polish.

Model shopping

Teams start on GPT-4, wonder if Claude would be better, try both, argue, wonder if fine-tuning would help. Pick a reasonable default, ship, iterate. Model choice can be revisited in a week; feature launch delay compounds.

Building the framework instead of the feature

A generic prompt manager. A pluggable eval framework. A multi-provider abstraction. None of these are the feature. Build the feature first. Extract the framework later, if ever.

The scope test

Read the scope back and ask three questions:

Can I build v1 in four weeks? If not, the scope is too big. Cut until it fits.

Will the user experience an obvious improvement, not a subtle one? Small AI features can be great, but "subtle improvement" is often invisible and users don't retain them.

If v1 gets zero adoption, do I know why? If not, add measurement.

Mistakes teams make

Skipping the out-of-scope list. Every scope needs a "does not" section. Skipping it invites creep.

Writing goals as adjectives. "Fast." "Accurate." "Helpful." Rewrite as numbers or behaviors.

Treating AI features like frontend features. Frontend features don't need evals. AI features do. Plan for the differences.

Over-scoping fallback. A working "try again later" message is a fine fallback for v1. Don't design a multi-provider failover before you have users.

Trade-offs

Small scopes ship. Big scopes learn slower. Cutting features cuts optionality. Skipping plumbing (evals, monitoring) makes v1 faster and every version after slower.

The right stopping point is where v1 is useful enough to test the core hypothesis, and no bigger.

Common misconceptions

"AI features take longer than normal features because AI." Sometimes. Often the actual reason is scope. A tightly scoped AI feature can ship as fast as a UI feature.

"We need to build for future flexibility." Rarely. Build the feature. If the future arrives, refactor then.

"Fine-tuning is worth exploring at v1." Almost never. Ship on off-the-shelf models. Fine-tune only when you have production data and a specific quality gap.

Frequently asked questions

How long should v1 of an AI feature take?

Four to six weeks is realistic for a scoped v1 with basic evals and monitoring. Longer usually means the scope was too big.

Should I ship an AI feature without evals?

You can, but you'll regret it. Build a small eval set as part of v1. Skipping this is the difference between shipping once and shipping continuously.

How do I say no to scope additions?

Point at the out-of-scope list. If it's not there, add it. If it is, add the addition to the follow-up backlog and ship.

What if the feature isn't good enough at v1?

Ship to a small user group behind a flag. Real usage will tell you what to fix better than internal debate.

How do I know when to stop iterating?

When the acceptance metric hits your v1 target. Not when the demo feels perfect.

Conclusion

Scoping an AI feature is unglamorous but decisive work. A one-page scope with a clear user, a specific quality bar, and an aggressive out-of-scope list is worth more than a 40-page PRD. Ship v1 in four to six weeks. Learn. Iterate.

The teams that ship AI features that stick treat scope as a design constraint, not a wish list. They ship earlier and improve faster.

If you're staring at a scope that's grown to a quarter and want a second opinion on what to keep and what to cut, the AI Audit can help you separate signal from ambition.