Data APIs: the pagination and volume traps that break systems at scale
Data APIs: the pagination and volume traps that break systems at scale
Your integration works perfectly in staging. The first query returns in 180 ms. The data looks clean. The team signs off. Three weeks into production, the pipeline stalls at 2 a.m. on a Tuesday, and nobody knows why.
This is not a rare story. It is the default outcome when teams skip one specific audit step: stress-testing how a data API behaves not when it works, but when it reaches its natural limits. Pagination logic and volume ceilings are where most integrations silently degrade — not crash loudly, but drift into incompleteness that nobody catches until the analysis is already wrong.
Why pagination is the first thing to stress-test
Most data APIs return results in pages. That is expected. What is not always documented clearly is what happens when you try to reach deep pages — page 50, page 200, page 1,000.
Several patterns emerge at depth:
Hard page caps. The API silently stops returning results after a fixed offset, often somewhere between 1,000 and 10,000 records. You keep paginating, get 200 OK responses, but the result set is empty or repeated. If your code does not detect this, you assume you have everything.
Cursor drift. Cursor-based pagination sounds more reliable than offset-based — and it is, until the cursor expires. Cursors tied to a session or a time window expire if your pipeline is slow or if a retry loop adds latency. The next call returns an error or, worse, resets to page one silently.
Rate-limited pagination. Some APIs throttle not per request but per paginated sequence. You can pull 100 requests per minute globally, but only 10 consecutive paginated calls before a backoff is enforced. This is almost never in the main documentation; it surfaces in the HTTP 429 response headers if you read them carefully.
The audit: before production, run a full pagination sweep on a known large result set. Count the actual records returned. Compare against the total count the API reports in the first response. If they do not match, you have a structural problem to solve before shipping.
Volume ceilings: the three types your SLA probably ignores
Volume limits in data APIs come in three distinct shapes. Most teams only account for one.
Request volume (RPM/RPS). This is the visible limit. Easy to find in the docs, easy to handle with a token bucket or a simple sleep. Most teams get this right.
Data volume per response. This is subtler. APIs often cap the payload size per response — 1 MB, 5 MB, 10 MB. When a single record is large (long text bodies, embedded metadata, nested arrays), you may hit the payload cap before you hit the record count limit. The result: truncated records that look complete. Your parser does not error; it just processes less data than it should.
Cumulative daily volume. Pay-as-you-go APIs, including those structured around actual consumption rather than flat tiers, tend to enforce daily or monthly caps that reset on a calendar boundary, not a rolling window. If your pipeline runs a backfill job on the last day of the month and the next morning hits zero quota, you have a 24-hour blind spot in your data continuity. Map the reset boundary before you design your scheduler.
The completeness problem nobody measures
Here is the real issue: most monitoring stacks track latency and error rates, not data completeness. An API that silently truncates results at 5,000 records will look healthy in your dashboards if you are only watching response codes and p99 latency.
The fix is instrumentation, not trust.
Add a completeness check at the pipeline layer:
def fetch_with_completeness_check(api_client, query, expected_total_field="total"):
results = []
page = 1
reported_total = None
while True:
response = api_client.get(query, page=page)
if reported_total is None:
reported_total = response.get(expected_total_field, 0)
batch = response.get("results", [])
if not batch:
break
results.extend(batch)
page += 1
fetched = len(results)
if reported_total and fetched < reported_total:
raise DataCompletenessError(
f"Expected {reported_total} records, fetched {fetched}. "
f"Check pagination ceiling or payload cap."
)
return results
This pattern is simple. It is not in enough codebases. The reported_total field is available in most data API responses — use it as a contract, not a curiosity.
Burst behavior under concurrent workers
If your architecture uses parallel workers to speed up ingestion — multiple threads or async coroutines hitting the same API — you will encounter a class of problems that sequential testing never reveals.
Quota contention. Two workers consuming quota simultaneously can exhaust a per-minute limit in 20 seconds. If your retry logic uses fixed backoffs instead of jitter, all workers retry at the same moment and repeat the exhaustion cycle.
Deduplication gaps. Concurrent workers operating on overlapping time windows will pull the same records twice. Without a deduplication layer downstream, your dataset has phantom volume — it looks larger than it is, and aggregations are inflated.
State collisions. If workers share a cursor or a checkpoint store without proper locking, one worker can overwrite another's progress marker. On restart, the pipeline replays records or skips a window entirely.
The standard fix for burst deduplication is a fast key-value store at the ingestion boundary — Redis with a short TTL keyed on the record identifier works for most pipelines. For quota contention, exponential backoff with full jitter (not fixed jitter) is the pattern that actually distributes retries across the rate limit window.
What to document before the first production deploy
Three things that belong in your runbook, not just your head:
The actual pagination ceiling. Not what the docs say — what you measured in a sweep test. If the documented limit is 10,000 records and your sweep hits a wall at 8,500, write 8,500.
The quota reset boundary. UTC timestamp, not "daily." APIs operated globally reset at midnight UTC. If your team is in a different timezone, "daily reset" means something different at 11 p.m. your time than the API expects.
The completeness check alert threshold. Do not alert at 0% completeness (pipeline is already broken). Alert at 95% — a 5% gap in a large dataset is a symptom, not noise.
APIs like FeedScale — structured around pay-as-you-go consumption of public internet signals — expose the full quota state in each response. That makes this instrumentation straightforward. Not every API does. When one doesn't, you build the tracking layer yourself, or you fly blind.
Data API failures at scale are rarely spectacular. They are quiet, incremental, and invisible until someone asks why the numbers from last Tuesday don't match. The systems that stay reliable are not the ones with the best APIs — they are the ones whose teams measured the limits before depending on them.