Design a file upload service that handles large files with resumable uploads and processing

Architecture

Client → Upload API → S3 (multipart) → SQS → Transcoder → CDN
              ↓
         Progress Tracking (Redis)

Chunked/Resumable Upload

// Step 1: Initialize upload
POST /api/uploads/initiate
{ fileName: "video.mp4", fileSize: 5_000_000_000 }
→ { uploadId: "up_123", chunkSize: 5_000_000, presignedUrls: [...] }

// Step 2: Upload chunks (parallel, resumable)
PUT { presignedUrls[0] } → chunk data
PUT { presignedUrls[1] } → chunk data
// Failed chunk? Just retry that chunk, not the whole file

// Step 3: Complete upload
POST /api/uploads/complete
{ uploadId: "up_123", parts: [{ partNumber: 1, etag: "..." }, ...] }
→ Triggers transcoding pipeline

Server Implementation

// Initiate upload
export async function POST(req: Request) {
  const { fileName, fileSize } = await req.json();
  const uploadId = crypto.randomUUID();
  const chunkSize = 5 * 1024 * 1024; // 5MB chunks
  const totalParts = Math.ceil(fileSize / chunkSize);

  // Generate presigned URLs for each chunk
  const presignedUrls = await Promise.all(
    Array.from({ length: totalParts }, (_, i) =>
      s3.getPresignedUrl(`uploads/${uploadId}/${fileName}`, i + 1)
    )
  );

  // Track progress
  await redis.set(`upload:${uploadId}`, JSON.stringify({
    fileName, fileSize, chunkSize, totalParts, uploadedParts: 0,
  }));

  return Response.json({ uploadId, chunkSize, presignedUrls });
}

Processing Pipeline

// After upload completes → SQS message → Transcoder
async function processVideo(uploadId: string) {
  // 1. Download from S3
  // 2. Transcode to multiple resolutions (720p, 1080p, 4K)
  // 3. Generate thumbnails
  // 4. Upload processed versions to S3 + CDN
  // 5. Update database with video metadata
  // 6. Notify user (WebSocket or push)
}

Key Design Choices

Decision	Approach
Upload	Presigned S3 URLs (client uploads directly to S3)
Resumable	S3 multipart upload (retry individual parts)
Progress	Redis counter, client polls or WebSocket
Processing	Async job queue (SQS/RabbitMQ)
Storage	S3 + CloudFront CDN
Max file size	5TB (S3 multipart limit)