Image Recognition as a Service

Instances step up behind a request burst and back down, illustrative

What I built9

Developed an elastic cloud infrastructure SaaS using AWS EC2, AWS SQS, and Lambda.
Enabled automatic linear scaling based on demand, serving 100 concurrent requests in 5 seconds.
Split the queue in two, requests in and results out, so a web tier can return immediately with a request id and a slow inference never holds an HTTP connection open.
Scaled on queue depth rather than CPU, because an instance waiting on a GPU-bound model looks idle to a CPU metric right when more capacity is needed.
Made the scaler decide from one number, messages in flight divided by per-instance throughput, and capped it, so a traffic spike cannot spin up an unbounded bill.
Scaled in conservatively and out aggressively, with a cooldown, after an early version thrashed instances up and down on a sawtooth load.
Used SQS visibility timeouts as the retry mechanism: a worker that dies mid-image releases the message back rather than losing it, so at-least-once delivery does the failure handling for free.
Made the workers idempotent on an input hash, since at-least-once means the same image will occasionally be classified twice and the second result must not conflict with the first.
Measured the cold path honestly: end-to-end latency for the first request after a scale-out is dominated by model load, not inference, which is what the instance warm pool exists for.