How to Run Long Jobs on Serverless Without Losing Your Mind

Serverless is great until a job takes longer than the platform's timeout. Then it isn't. Long-running jobs in serverless environments — LLM calls, PDF generation, video processing, batch imports — are where most early-stage teams first hit the wall, and where the wrong choice locks you into painful rewrites.

Here's how we think about it when we're building.

The wall you're about to hit

Cloudflare Workers gives you 30 seconds of CPU on the paid plan. Vercel functions cap at 60 seconds on Pro, longer on higher tiers. AWS Lambda tops out at 15 minutes. These aren't quirks — they exist because serverless is priced on the assumption that requests are short.

The first LLM call that streams a 90-second response, or the first CSV import with 50k rows, breaks the model. And retrying the whole request doesn't work: half of it already ran.

The mistake most teams make

The instinct is to "just make the function longer." That works for a week. Then a real user uploads a bigger file, and you're back to square one — this time with a partial write in the database and no way to resume.

The right question isn't "how long can this run?" It's "how do I make this restartable?"

The pattern we default to

Split every long job into three phases: accept, process, notify.

The HTTP request accepts the work, writes a job row with status queued, and returns immediately with a job ID. A worker picks up the job asynchronously and updates status as it goes. When done, the worker fires a webhook, sends an email, or the UI polls.

This maps cleanly to whatever queue primitive your platform gives you — Cloudflare Queues, AWS SQS, Supabase pg_boss, Inngest, Trigger.dev.

Where to actually run the worker

Cloudflare Queues + Workers

Good when you're already on Workers. Cheap, fast, integrates with the same code. Downside: each message still has a per-invocation CPU limit, so chunk carefully.

Durable execution (Inngest, Trigger.dev, Temporal)

The right answer for anything with more than three steps or retries that must be idempotent. You write the job as a normal function, the runtime handles checkpoints, retries, and state. Costs more, saves weeks of debugging.

A dedicated worker on a VM

Sounds old-fashioned. Still the correct answer for CPU-heavy jobs, video encoding, or anything that needs local files and a real filesystem. A single Fly.io or Railway VM beats fighting serverless limits for weeks.

Idempotency is not optional

Every job gets an idempotency key — usually the job row's UUID. Every side effect the worker performs checks whether that side effect has already happened for that key. Sending an email? Look up whether it's already been sent. Charging a card? Same check.

Retries will happen. Duplicate sends and double charges are what tank customer trust.

Chunking long work

When a job is too big for a single worker invocation, split it. A 50k-row CSV becomes 50 chunks of 1k rows. Each chunk is its own message. The parent job tracks progress by counting completed chunks.

This is not fancy. It's the difference between "our import broke and we don't know where" and "we retried chunk 37 and moved on."

Streaming vs. background

Not every long job needs background execution. Streaming an LLM response is long in wall-clock time but short in CPU. Server-sent events over a normal HTTP handler works — the client stays connected while the worker generates tokens.

The tell: does the user need to sit and wait for the result to be useful? Stream it. Do they want to close the tab and come back? Queue it.

Progress and observability

Every job writes progress to a row the frontend can poll or subscribe to. Percentage, current step, last error. This is what turns a "spinner that runs for 4 minutes" into a UI that feels responsive.

Log every state transition. When something goes wrong at 2am, you want a timeline, not a stack trace.

Common misconceptions

"Queues are overkill for early stage." Adding one takes an afternoon. Retrofitting one after a data-loss incident takes weeks and destroys trust.

"Durable execution is only for big companies." Inngest and Trigger.dev have generous free tiers and cut orchestration code by 80%. Small teams benefit most.

"Just use cron." Cron is fine for scheduled work. It's the wrong tool for user-triggered jobs — you get unpredictable latency and no per-job visibility.

A concrete decision tree

• Under 5 seconds and streamable → normal handler, stream the response.
• Under 30 seconds and needs a result → normal handler, block.
• Longer than the timeout, single step → queue + worker, poll for status.
• Longer than the timeout, multiple steps or retries → durable execution.
• CPU-bound (encoding, ML inference on CPU) → dedicated VM.

FAQ

Can I just use setTimeout in a Worker?

No. The Worker instance can be recycled at any time. Anything scheduled that hasn't fired will be dropped.

What about waitUntil?

Useful for fire-and-forget work that finishes within the CPU budget. Not a substitute for a queue when the job is genuinely long.

Do I need Redis for this?

Usually not. Your primary database plus a queue is enough for most SaaS workloads. Reach for Redis when you have real hot-path caching needs.

Where to go from here

If your app is starting to accumulate handlers that "usually finish in time but sometimes fail," it's time to move that work to a queue. Our SaaS and web apps engagements often start with exactly this refactor — same features, calmer nights. Related reading: Background jobs for modern SaaS.