Serverless image pipeline on AWS

The problem

Users upload images. The application needs several versions of each one — a thumbnail for lists, a mid-size for detail pages, a WebP variant for browsers that support it — and has to serve them quickly to viewers in different places.

This is a write-once, read-many problem. Uploads are sparse and bursty; reads are dense and should be cached. The architecture below separates those two halves completely.

How it flows

1. The client PUTs the original into the upload bucket (n1).
2. S3 emits an event notification and invokes the resize function (n2). S3 can invoke Lambda directly — no intermediate component is needed.
3. The function reads the original, produces the derivatives, and PutObjects them into the derivatives bucket (n3) — a different bucket. The reason is below.
4. CloudFront (n4) sits in front of the derivatives bucket, caches at the edge, and signs origin requests using origin access control (OAC). The bucket is never exposed to the internet.
5. If the function fails after Lambda has exhausted its retries, the invocation record is pushed to an SQS queue (n5) configured as the on-failure destination.
6. Function logs flow to CloudWatch Logs (n6) with no configuration at all.

Why two buckets

This is the most important trap in the architecture, and AWS states it plainly in the documentation:

If your notification writes to the same bucket that triggers the notification, it could cause an execution loop.

If the resize function writes derivatives back into the bucket that triggered it, each write produces another event, and that event invokes the function again. The function indirectly triggers itself, with no natural stopping point. The bill grows exponentially, and because every invocation succeeds, no error alarm fires.

AWS documents two remedies: use two buckets, or restrict the trigger to a prefix reserved for incoming objects. This architecture takes two buckets because the boundary is easier to audit — a misconfigured prefix can still look almost right, whereas a bucket cannot.

Trade-offs

This is the section worth reading. The architecture above is not the only correct answer.

Asynchronous processing: fast for the uploader, but "not there yet"

The user gets a response as soon as the original lands, without waiting for the resize. In exchange, the derivatives do not exist for anywhere from a few hundred milliseconds to a few seconds afterwards. The interface has to handle that state — a placeholder, or a fallback to the original — rather than assuming the thumbnail is already there.

If the business requires the thumbnail to exist the moment the upload request returns, this architecture is wrong: resize synchronously inside the request and accept that the user waits.

Lambda's 15-minute ceiling

Lambda's timeout is capped at 900 seconds (15 minutes) and cannot be raised. For images that is ample. But if the pipeline later grows to handle video, that number becomes a hard wall, and the work must be split into orchestrated steps.

This is exactly where Google Cloud's Cloud Run differs sharply: its request timeout reaches 60 minutes (3,600 seconds), four times Lambda's. The same transcode job that Lambda forces you to break apart usually fits in a single Cloud Run request.

Lambda memory: more is not reliably more expensive

Lambda bills per GB-second, and CPU is allocated in proportion to memory — at 1,769 MB a function has the equivalent of one vCPU. Raising memory therefore makes each millisecond cost more, but usually shortens the run.

Cost is consequently not monotonic in memory. A 1,024 MB configuration can be cheaper than 512 MB for the same resize work. This has to be measured, not guessed.

A failure queue: four lines of config for recoverability

Without an on-failure destination, a failed resize simply vanishes — one log line, then it scrolls away. With SQS, the failure becomes a message you can inspect and reprocess.

One technical constraint: only standard queues work; FIFO queues are not supported as on-failure destinations. The real cost is close to nothing — the first million requests each month are free, and a healthy pipeline sends almost nothing here.

CloudWatch Logs: the quiet cost

Log groups never expire by default. On a pipeline running for years this is a steadily growing charge that nobody notices, because it never appears as a prominent line on the bill. Set a retention policy from the start.

OAC instead of a public bucket

Making the derivatives bucket public is simpler, but costs two things: no control over who reads what, and every request billed as S3 egress instead of being cached at the CDN. With OAC the bucket stays closed and CloudFront is the only way in.

Three things to know when using OAC with S3:

• The bucket's Object Ownership must be set to Bucket owner enforced.
• HTTPS between CloudFront and S3 is only guaranteed when the signing behaviour is always sign requests.
• A bucket configured as a website endpoint cannot use OAC (or OAI) at all — it must be declared as a custom origin.

OAI is the legacy mechanism and AWS recommends against it: it does not support SSE-KMS, does not support dynamic requests (PUT/POST/DELETE), and does not support Regions launched after January 2023.

Event notifications: at least once, no ordering guarantee

S3 event notifications are designed to be delivered at least once, and are not guaranteed to arrive in the order the events occurred. On rare occasions S3's retry mechanism can produce duplicate events for the same object.

The practical consequence: the resize function must be idempotent. Processing the same image twice must produce the same result as processing it once. Writing to a deterministic key (rather than appending, or naming by timestamp) is the simplest way to get there.

Cost

Assuming 10,000 images per month at 2 MB each, three derivatives per image, Lambda at 1,024 MB for about 2 seconds, and 50 GB of CDN egress at a 90% cache hit rate, the total comes to roughly 6.20 USD per month.

That figure is computed by hand from published unit prices on the AWS pricing pages, not taken from a saved AWS Pricing Calculator estimate, and it excludes the Free Tier. Read it as an order of magnitude.

More interesting than the number: egress dominates. Lambda and S3 are almost negligible at this scale. If the cache hit rate drops from 90% to 60%, the total moves more than any other change would — including doubling the number of images.

When not to use this architecture

• Derivatives needed immediately when the upload request returns → resize synchronously.
• Work exceeding 15 minutes (long video) → Cloud Run, ECS, or split the steps.
• Very many variants, hard to predict in advance → consider resizing on-the-fly at the edge instead of pre-generating everything, so you do not store variants nobody ever reads.