Faster Uploads with Background Processing

Uploading a document sounds simple, but the work behind it can be surprisingly expensive.

A typical document workflow may involve storing the file, extracting text, splitting content into chunks, generating embeddings, and updating a search index. Depending on the file size and processing requirements, those steps can take several seconds or longer.

If all of that work runs inside the upload request, users are left waiting for operations they don't need to wait for.

In one document-processing workflow I worked on, separating upload acknowledgment from processing brought the average acknowledgment time to approximately 200 milliseconds. We also supported multiple file uploads and provided status updates so users could see when their documents were ready.

That measurement reflects the acknowledgment stage of the workflow, not the total time required to transfer, process, or index a document. File size, network conditions, and system load still affect the overall experience.

The key improvement was straightforward: accept the file, process it in the background, and make the current state visible to the user.

This article walks through how I would structure that approach using FastAPI, Redis, and Celery. The architecture is illustrative rather than a reproduction of the original application's internal implementation.

1. Keep the upload request focused

The first design decision is deciding what the upload endpoint is responsible for.

Its job is to validate the request, store the uploaded file, and reliably record that processing needs to happen. It shouldn't need to extract text or generate embeddings before responding.

A typical flow looks like this:

1. The client uploads a file.
2. The API validates the user's access, file size, and supported format.
3. The application stores the file in persistent storage.
4. A document record and processing request are saved.
5. The API returns 202 Accepted with a document ID.
6. A background worker processes the document independently.

The important distinction is between accepted and ready.

Accepted means the system has safely recorded the work. Ready means the document has finished processing and is available for its intended use.

Returning a fast response before the file or job is safely recorded would only move the failure somewhere less visible.

FastAPI provides BackgroundTasks, which can be useful for small tasks that run after a response. For heavier processing, I generally prefer separate workers. They can be scaled independently and don't tie expensive document operations to the API server.

FastAPI background-task documentation

2. Use Redis and Celery to separate the workload

A practical setup is to use Redis as the message broker and Celery as the worker system.

The API accepts uploads, while Celery workers handle expensive operations such as text extraction, chunking, and embedding generation.

The processing flow can be represented as:

Upload: Browser → FastAPI → Persistent File Storage → Database

Dispatch: Processing Request → Redis → Celery Worker

Processing: Extract Text → Create Chunks → Generate Embeddings → Update Search Index

Status: Worker → Database → Status API → Browser

Files should remain in persistent storage rather than being passed through the message queue. Celery tasks only need a document identifier and enough information to locate the work.

The application database remains the source of truth for document ownership and processing status. Redis transports tasks; it should not be the only place where the application remembers that a document exists.

There is one subtle failure case worth addressing.

Suppose the database successfully saves a document record, but the application crashes before publishing the processing task to Redis. The document now exists, but no worker knows it needs processing.

For systems that require reliable recovery, a transactional outbox is one solution. The application saves the document record and an outbox entry in the same database transaction. A separate dispatcher publishes pending entries to Redis.

This adds complexity, so it isn't always necessary for a small application. A simpler design may use direct task publishing with a recovery process that periodically finds jobs stuck in the accepted state.

The important part is having a way to recover work when the database-to-queue handoff fails.

Celery Redis documentation

3. Design workers for retries and failures

Background processing makes the API more responsive, but it introduces another set of problems.

Workers can crash. External AI services can time out. A document can be malformed. A task may also be delivered more than once.

A reliable worker should be safe to run again without creating duplicate output.

For document processing, I would use a stable document ID and processing version to identify the work. Before writing results, the worker checks whether the same version has already completed.

This is especially important for RAG applications. Retrying a failed embedding job should not leave duplicate chunks in the vector index or overwrite results from a newer document version.

Retries also need limits.

Temporary failures, such as an unavailable external API, can use bounded retries with backoff. Permanent failures, such as an unsupported or unreadable file, should stop processing and return a clear error.

Celery supports task retries and configurable acknowledgment behavior, but those settings alone don't guarantee that every failure will recover cleanly. Application-level idempotency and state handling still matter.

Celery task documentation

4. Make processing status useful to users

Moving work into a queue is only half the solution. Users also need to know what's happening.

I prefer a small set of clear states rather than an estimated progress percentage that doesn't reflect real work.

For multi-file uploads, each document should have its own status.

If a user uploads five PDFs and one fails, the other four should still be available when their processing finishes.

Polling a status endpoint is often enough to get started. WebSockets or server-sent events can improve the experience later, but they aren't necessary for every application.

The status must come from durable application state so that a page refresh doesn't reset the apparent progress.

Security matters here as well. Status and download endpoints must check document ownership. Knowing a document ID should never be enough to access another user's information.

5. Measure more than API response time

An average acknowledgment time of approximately 200 milliseconds is a useful result, but it doesn't tell the whole story.

I would also track:

• Upload transfer time: How long it takes to receive the file.
• API acknowledgment latency: How quickly the application confirms acceptance.
• Queue wait time: How long a job waits before a worker starts.
• Processing duration: How long extraction, embedding, and indexing take.
• Time to ready: How long users wait before they can actually use the document.
• Failure and retry rates: How often processing fails or requires another attempt.

It is important to define the start and end points of each measurement. Otherwise, upload latency and processing latency can easily be confused.

Average latency is useful, but I also look at the 95th percentile.

A system can have a good average response time while still giving some users a poor experience.

Queue depth and job age are equally important. If jobs arrive faster than workers can process them, the API may continue responding quickly while the processing backlog grows.

At that point, adding workers, limiting concurrent jobs, or applying backpressure may be more useful than further optimizing the upload endpoint.

6. Test what happens when things go wrong

Happy-path tests aren't enough for an asynchronous workflow.

I want to know what happens if a worker crashes after generating embeddings but before marking the document ready. I also want to know whether a duplicate task creates duplicate search results.

Other important scenarios include a temporary Redis outage, an interrupted upload, invalid documents in a batch, and a user refreshing the page during processing.

These cases reveal problems that ordinary unit tests may not catch.

It is also worth verifying that one user cannot view another user's upload status or retrieve their files.

In production, reliability often depends on what happens between components, not just whether each component works on its own.

Final thoughts

Background processing doesn't make document extraction or embedding generation inherently faster. It changes how that work affects the application.

By separating upload acceptance from processing, the API can respond without waiting for expensive operations. Workers can scale independently, and users can continue working while their documents are prepared.

The approximately 200-millisecond average acknowledgment in the measured workflow was one useful outcome. The more important design goal is to make processing reliable, recoverable, and visible.

For document-heavy applications, especially those using RAG, that's a better foundation than keeping the user waiting on a long-running HTTP request.

A fast upload experience isn't just about returning a response quickly. It's about accepting work reliably and making sure users know when that work is truly complete.