A langchain 429 error rate limits gateway failure shows up when your chain sends more requests than the upstream model provider allows, and the gateway in front of it returns HTTP 429 before any fallback kicks in. This guide reproduces the error with a minimal script, inspects the raw response, and walks through concrete fixes: client-side throttling, retry backoff, and gateway-aware routing.
Step 1: Reproduce the 429 with a minimal LangChain script
Point LangChain’s ChatOpenAI at your gateway endpoint. If you run a local mock or a hosted OpenAI-compatible gateway, set base_url and a dummy key. The snippet below fires 50 concurrent calls with no delay.
import asyncio
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="gpt-3.5-turbo",
base_url="https://gateway.example.com/v1",
api_key="sk-test",
max_retries=0, # disable built-in retries to see raw 429s
)
async def call_once(i: int):
try:
await llm.ainvoke(f"Say hello {i}")
except Exception as e:
print(f"Request {i} failed: {type(e).__name__}: {e}")
async def main():
await asyncio.gather(*[call_once(i) for i in range(50)])
asyncio.run(main())Run this against a provider with a low quota (or a gateway that enforces per-key limits) and you will see OpenAIError wrapping a 429. That is the langchain 429 error rate limits gateway pattern: the client is unaware of the server’s rate window.
For synchronous chains, the same burst happens with ThreadPoolExecutor:
from concurrent.futures import ThreadPoolExecutor
def call_sync(i):
try:
llm.invoke(f"Say hello {i}")
except Exception as e:
print(f"Request {i} failed: {e}")
with ThreadPoolExecutor(max_workers=50) as ex:
list(ex.map(call_sync, range(50)))The error rate scales with concurrency, not just total volume.
Step 2: Capture and inspect the raw HTTP error
LangChain swallows the response body unless you dig into the exception. Wrap the call and log the response object from the underlying openai library.
from openai import APIStatusError
async def call_and_log(i: int):
try:
await llm.ainvoke(f"Say hello {i}")
except APIStatusError as e:
body = e.response.json()
print(f"Status {e.response.status_code}: {body}")A typical 429 body from a gateway looks like:
{
"error": {
"type": "rate_limit_error",
"message": "Rate limit reached for requests per minute: 20",
"code": 429
},
"x-ratelimit-limit": "20",
"x-ratelimit-remaining": "0",
"retry-after": "2.5"
}The retry-after header is your signal. If the gateway provides per-token metering, the body may also include usage details. Capture these headers; they drive the backoff in later steps.
You can also attach a LangChain callback to record every LLM call without modifying business logic:
from langchain_core.callbacks import BaseCallbackHandler
class RateLimitSniffer(BaseCallbackHandler):
def on_llm_error(self, error, **kwargs):
if "429" in str(error):
print("Captured 429 in callback:", error)
llm = ChatOpenAI(..., callbacks=[RateLimitSniffer()])Step 3: Configure LangChain retry with exponential backoff
LangChain’s ChatOpenAI delegates to the OpenAI SDK, which respects max_retries. The default is two retries with a short fixed delay. That bursts retries immediately after a 429 and can extend the saturation window. Use tenacity to wrap your own logic instead.
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from openai import RateLimitError
@retry(
retry=retry_if_exception_type(RateLimitError),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(5),
)
async def safe_invoke(prompt: str):
return await llm.ainvoke(prompt)For synchronous code, the same decorator works on llm.invoke. The key change is the exponential wait: first retry waits ~2s, then 4s, then 8s, giving the gateway’s limit window time to reset.
This reduces immediate retries but does not solve concurrency: 50 tasks still hit the gateway simultaneously. You need throttling before the network call.
Step 4: Add client-side rate limiting
Use an asyncio.Semaphore to cap concurrent requests, and a simple token bucket for per-minute rates. Below is a minimal bucket that reads the limit from the first 429.
import time, asyncio
class TokenBucket:
def __init__(self, rate: int, per: float):
self.rate = rate
self.per = per
self.tokens = rate
self.last = time.monotonic()
self.lock = asyncio.Lock()
async def acquire(self):
async with self.lock:
while True:
now = time.monotonic()
elapsed = now - self.last
self.tokens = min(self.rate, self.tokens + elapsed * (self.rate / self.per))
if self.tokens >= 1:
self.tokens -= 1
self.last = now
return
await asyncio.sleep(0.1)
bucket = TokenBucket(rate=20, per=60) # match gateway limit
async def throttled_call(i: int):
await bucket.acquire()
return await safe_invoke(f"Say hello {i}")Now run 50 calls; they will be spaced to 20 per minute. The langchain 429 error rate limits gateway loop disappears because the client respects the window.
In a threaded sync context, use threading.Semaphore plus a blocking token bucket:
import threading, time
class SyncTokenBucket:
def __init__(self, rate, per):
self.rate = rate; self.per = per
self.tokens = rate; self.last = time.monotonic()
self.lock = threading.Lock()
def acquire(self):
with self.lock:
while True:
now = time.monotonic()
self.tokens = min(self.rate, self.tokens + (now-self.last)*(self.rate/self.per))
if self.tokens >= 1:
self.tokens -= 1; self.last = now; return
time.sleep(0.05)Step 5: Leverage gateway-level fallback and routing
A gateway that fronts multiple providers can shield you from single-provider limits. For example, n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and automatically fails over when a provider is rate-limited or degraded. Even with that, the first request may still get a 429 if all backing providers are saturated.
You can send client routing directives via extra_headers so the gateway prefers a less busy provider:
llm = ChatOpenAI(
model="auto",
base_url="https://gateway.example.com/v1",
api_key="sk-test",
extra_headers={"x-prefer-provider": "anthropic"}, # gateway honors directive
)If the gateway forwards provider cache-control hints, add them to cut token cost on repeated prompts:
extra_body = {"cache_control": {"type": "ephemeral"}}
await llm.ainvoke("Long system context...", extra_body=extra_body)The gateway passes the hint to the provider, reducing repeated input tokens and indirectly lowering rate-limit pressure. Do not assume fallback is instant; design your client to tolerate a 429 even when a gateway promises automatic failover.
Step 6: Track per-token usage to anticipate limits
Rate limits are often token-based, not request-based. Read usage from the LangChain result and maintain a rolling sum.
response = await llm.ainvoke("Explain rate limits")
usage = response.usage_metadata # {'input_tokens': 10, 'output_tokens': 20}If your gateway provides per-token metering, log these to detect when you approach the token ceiling. Switch models or batch prompts when the sum exceeds 80% of the known quota. A simple rolling window:
from collections import deque
import time
class TokenWindow:
def __init__(self, limit, seconds):
self.limit = limit; self.seconds = seconds
self.q = deque()
def add(self, tokens):
now = time.time()
self.q.append((now, tokens))
while self.q and now - self.q[0][0] > self.seconds:
self.q.popleft()
@property
def used(self):
return sum(t for _, t in self.q)Check used < limit*0.8 before sending a large batch.
Step 7: Verify success with a controlled load test
Replace the dummy gateway with your real endpoint and run the throttled client under expected peak load. Success criteria:
- Zero
RateLimitErrorexceptions over a 5-minute window. x-ratelimit-remainingstays above zero in logged headers.- End-to-end latency per call stays under your SLA despite backoff.
A quick verification script:
async def verify():
tasks = [throttled_call(i) for i in range(100)]
results = await asyncio.gather(*tasks, return_exceptions=True)
errors = [r for r in results if isinstance(r, Exception)]
print(f"Errors: {len(errors)} / 100")
asyncio.run(verify())If Errors: 0 / 100 prints, your langchain 429 error rate limits gateway handling is correct.
Common pitfalls
- Setting
max_retriestoo high: The OpenAI SDK will block the event loop or thread pool, masking the real throughput problem. - Ignoring
retry-after: A static exponential backoff may wait longer or shorter than the gateway needs; parse the header when present. - Routing headers without gateway support: If the gateway does not honor the directive, the header is silent dead weight.
- Mixing sync and async throttles: An
asyncio.Semaphoredoes nothing in a thread pool; use the right primitive for the execution model.
Following these steps moves the rate-limit concern from reactive exception handling to proactive client/gateway coordination.