TL;DR
|
Rate limiting is a rule that limits how often someone can do something. A login page might allow five attempts a minute. An API might allow 1,000 requests an hour per key. Go over the limit, and further requests are refused until enough time passes.
It used to be something mainly large services worried about. Now that many apps call paid AI models behind the scenes, it matters for everyone: a feature that costs a fraction of a cent per use can cost thousands if someone scripts it. This guide explains how rate limiting works, why AI-built apps especially need it, and where to add it.
Key takeaway: the server counts requests from each user or key, and refuses them once the count passes the limit for the current time window.
A good 429 response tells the client when to try again, using a Retry-After header. Many APIs also send headers showing the limit and how many requests are left, so well-behaved clients can slow down before they hit it.
Key takeaway: the methods differ in how they handle bursts. Token bucket is a popular middle ground.
| Method | How it works | Trade-off |
|---|---|---|
| Fixed window | Counts requests in fixed blocks of time, such as each minute | Simple, but allows bursts at the edges, such as 100 requests at 12:00:59 and 100 more at 12:01:00 |
| Sliding window | Counts requests in the last rolling period, such as the past 60 seconds | Smoother, slightly more work to track |
| Token bucket | Each user has a bucket of tokens that refills at a steady rate. Each request uses one | Allows short bursts while holding the average steady |
| Leaky bucket | Requests are processed at a fixed rate, and extras wait or are dropped | Very steady output, less tolerant of bursts |
Key takeaway: if every request to your app costs you money, anyone who can send lots of requests can run up your bill.
Apps built quickly, often with AI tools, frequently skip rate limiting, because everything works fine in testing. The problems appear once the app is public:
| Risk | What happens |
|---|---|
| Surprise AI bills | Your app calls a paid AI model for each chat message or generation. Someone scripts thousands of requests, and you pay for every one. This is sometimes called a denial-of-wallet attack |
| Free accounts abused | Bots create accounts in bulk to use free credits or free features |
| Password guessing | Attackers try thousands of passwords against your login page |
| Scraping | Bots copy your content or data, using up resources and bandwidth |
| Runaway loops in your own code | A bug calls a paid API in a loop, and nobody notices until the invoice arrives |
| One user slowing everyone down | A single heavy user, or a misbehaving script, takes capacity from everyone else |
The expensive endpoints matter most. Put the strictest limits on anything that costs money or is attractive to attackers: AI features, sign-up, login, password reset, and checkout. Our guide What is an API key? covers the related risk of a leaked key being used to run up your bill.
Key takeaway: limits work best in layers, from blocking floods at the edge to precise per-user limits in your app.
| Layer | What it's good for | Examples |
|---|---|---|
| CDN or edge | Stopping floods before they reach your servers, by IP address or path | Rate limiting rules in a CDN or web application firewall |
| Reverse proxy | Limits per IP address in front of your app | Nginx, HAProxy, Varnish |
| Your app | Precise limits per logged-in user, per API key, or per feature | Framework tools and libraries |
| The services you pay for | Capping your total spending, whatever happens | Spending limits and usage alerts in your AI or API provider's dashboard |
Many web frameworks have rate limiting built in or available as a package:
| Framework | Rate limiting option |
|---|---|
| Express (Node.js) | The express-rate-limit package |
| Django (Python) | The django-ratelimit package |
| Laravel (PHP) | Built-in rate limiting and the throttle middleware |
| Symfony (PHP) | The RateLimiter component |
| Rails (Ruby) | Built-in rate_limit in controllers, from Rails 7.2 |
Spending limits deserve special mention. Whatever else you do, set a monthly spending cap and usage alerts with every AI and API provider you pay. It's the safety net if every other layer fails.
Key takeaway: if a service returns 429 errors, slow down and retry later, rather than retrying immediately.
Rate limits work in both directions. If your app calls an API that returns 429 errors:
Retry-After header, and wait that long before retrying.Key takeaway: many rate limiting bugs come from counting the wrong thing, or counting in the wrong place.
| Mistake | What happens | How to avoid it |
|---|---|---|
| Limiting by the proxy's address | Behind a CDN or proxy, every request seems to come from the same address, so one limit blocks everyone | Use the visitor's real address, which the proxy passes in a header |
| Counting separately on each server | With three app instances, each keeps its own count, so the real limit is three times higher | Keep counts in a shared store, such as Redis |
| No limits on expensive features | A free AI feature gets scripted, and the bill arrives | Limit costly endpoints strictly, per user and per address |
| Limits so strict they block real users | Customers on a shared office or mobile network get blocked together | Prefer limits per logged-in user where you can, and tune limits using real traffic |
| No spending caps with providers | Rate limits have a gap, and costs run unchecked | Set spending limits and alerts with every paid service |
| No monitoring | Nobody notices abuse until something breaks | Log 429 responses and alert on unusual spikes |
Key takeaway: on Upsun Cloud, you can rate limit in your app, with Varnish, or at the edge with a CDN add-on.
Upsun Cloud gives you several places to apply limits:
| Option | What it does | Details from the Upsun docs |
|---|---|---|
| In your app | Precise limits per user, per API key, or per feature, using your framework's tools | The router passes each visitor's real IP address to your app in the X-Client-IP header, so limits by address work correctly |
| Varnish | Limits per IP address in front of your app, using Varnish's throttling module | Add Varnish as a service, and use the vsthrottle module in your Varnish configuration |
| Edge-level rate limiting | Limits applied at the CDN, before requests reach your app, scoped by path, request type, or IP address | Available as an add-on at the CDN layer, arranged through Sales and configured with Upsun Support. No rules are applied by default |
The Upsun docs on Varnish include an example that blocks any IP address making more than 20 requests in 10 seconds, for two minutes:
import vsthrottle;
sub vcl_recv {
# The Upsun Cloud router provides the real client IP as X-Client-IP
if (vsthrottle.is_denied(req.http.X-Client-IP, 20, 10s, 120s)) {
return (synth(429, "Too Many Requests"));
}
set req.backend_hint = application.backend();
}
The edge-level rate limiting add-on is rule-based. According to the docs, it doesn't identify bots automatically or present challenges such as CAPTCHAs, and Upsun Cloud offers separate integrations for advanced bot protection.
If your app runs several instances, keep app-level rate limit counts in a shared store. A Redis service on Upsun Cloud works well for this. Our guide What is caching? Redis explained for beginners covers adding one.
For hands-on examples, read the Dev Center posts Varnish 102: Protecting your application with rate limiting on Upsun and Rate limiting Rails apps when you can't put Varnish in front.
You can deploy on Upsun Cloud free for 15 days, with no card needed, or see how security and compliance work on Upsun Cloud.
A limit on how many requests someone can make in a set time, such as 100 a minute. Requests over the limit are refused until the time window resets.
That you've sent more requests than the server allows in a period of time. Wait, ideally for as long as the Retry-After header says, and try again.
To stop abuse such as bots, scraping, and password guessing, to keep one user from slowing down everyone else, and to stop people running up your bill if your app calls paid services.
An attack that doesn't try to take your app down, but to make it cost you money, for example by sending thousands of requests to a feature that calls a paid AI model.
Both, where possible. Limit by IP address for anonymous traffic and login pages, and by user or API key for logged-in features, so people sharing a network aren't blocked together.
Not on its own. It limits how fast anyone can send requests, but it doesn't tell bots from people. Combine it with bot protection, spending limits, and monitoring.