Data APIs and Pagination: Why Your Retrieval Strategy Breaks at Scale
Data APIs and Pagination: Why Your Retrieval Strategy Breaks at Scale
Most integrations fail not because the API is bad, but because the team chose the wrong pagination model for the wrong workload. By the time the problem surfaces, you are already in production, missing data silently, and the logs are not telling you anything useful.
Pagination is not a detail. It is the structural backbone of any data retrieval loop. Get it wrong and you get incomplete datasets, duplicated records, runaway memory consumption, or requests that simply time out at 50,000 results when your use case demands 500,000.
This post is about what actually happens to pagination strategies under real load — and how to choose, implement, and harden the one that fits your workload.
The Three Models You Will Actually Encounter
When you call a data API, the provider controls the pagination contract. You work within it. Understanding the tradeoffs of each model is what lets you design the correct client-side logic.
Offset/limit pagination is the most common and the most dangerous at scale. You request page 1 with offset=0&limit=100, then offset=100&limit=100, and so on. The problem: if a new record is inserted between your first and second request, every subsequent page shifts by one. You miss records silently. The API itself has no way to warn you. At low volumes this is acceptable. At tens of thousands of results per hour, it is a data integrity issue.
Cursor-based pagination hands you an opaque token representing your position in the result set. The API guarantees that the next request using that cursor returns the correct next batch, regardless of writes happening in parallel. This is the correct model for high-throughput, append-heavy data streams — the kind common in public-source signal pipelines. The tradeoff: you cannot jump to an arbitrary page. Traversal is strictly sequential.
Keyset (seek) pagination uses a known, indexed column — typically a timestamp or an ID — as the anchor for the next page. You request created_at > last_seen_timestamp ORDER BY created_at ASC LIMIT 100. It is stable, fast, and stateless from the client perspective. It is also the model that most developers implement incorrectly, because they forget to handle ties (two records with identical timestamps) or fail to account for DST gaps in timestamp ranges.
What "Losing Data Silently" Looks Like in Practice
Consider a pipeline consuming signals from public sources at a rate of 3,000 records per hour. The team uses offset pagination with a fixed 200-record page size. Every six minutes, a scheduled job runs the full retrieval loop.
At normal load, this works. At peak — a breaking topic generating 800 new records per minute — the offset drift means the job skips entire segments of the result set. The pipeline reports success. The downstream model trains on incomplete data. No alert fires. The problem only surfaces three weeks later during a quality audit.
This is not a hypothetical. It is a class of incident that any team running high-frequency API polling against a write-heavy index will eventually encounter. The fix is architectural: switch to keyset pagination anchored on an indexed timestamp, persist the last-seen value between job runs, and add a reconciliation window (re-query last_seen_timestamp - 5 minutes) to catch any late-arriving records.
Parallelism and Pagination: Where Developers Overcorrect
When retrieval is too slow, the natural instinct is to parallelize: spawn N workers, each requesting a different page range simultaneously. This works under offset pagination — until it doesn't. Parallel offset reads amplify the drift problem proportionally. You are not just missing records from one sequential pass; you are missing them across N concurrent passes, each independently shifting.
The correct approach to parallelism in data API retrieval is shard by a stable dimension, not by page offset. If the API supports time-range filtering, split the workload by time windows (00:00–06:00, 06:00–12:00, etc.) and run each shard sequentially within its own cursor or keyset anchor. Each worker operates on a stable, non-overlapping slice of the index. Writes outside a shard's window cannot affect it.
Some APIs — including FeedScale's endpoints — expose explicit from/to timestamp parameters precisely to support this pattern. Use them. Do not paginate across the entire index in a single sequential loop if you can partition it first.
Handling Rate Limits Without Losing Your Position
Pagination and rate limiting interact in a way that many integrations do not handle correctly. When the API returns a 429 Too Many Requests, the typical retry logic pauses and resumes — but with what cursor or offset?
If you did not persist the pagination state before hitting the rate limit, you restart from the beginning or from an arbitrary checkpoint. With offset pagination this is an invisible data gap. With cursor pagination it is an explicit failure — the cursor may have expired.
Design your retrieval loop to persist the pagination state atomically after every successful page, not at the end of the full run. Use a lightweight state store (Redis, a small Postgres table, even a local file for low-volume jobs) that records the last-seen cursor or timestamp. Your retry logic then reads this state and resumes from the exact position, not from the start.
Treat pagination state as you would a transaction log: durable, write-ahead, never ephemeral.
Before You Write the Client Code
The single most valuable step most teams skip is reading the API's pagination documentation as a contract, not a tutorial. Identify:
- What happens if the cursor expires? Some APIs invalidate cursors after a TTL. Know this before you build retry logic that stores cursors for hours.
- Is the total count reliable? Many APIs return a
totalfield that reflects the index state at query time, not at retrieval time. Building pagination loops around total counts is fragile. - Does the API support partial page detection? If a page returns fewer records than the limit, is that always EOF, or can the API return short pages mid-stream?
These are operational questions, not onboarding questions. If the documentation does not answer them, test them explicitly in a staging environment at the volumes you expect in production — not at the toy scale you use for demos.
The APIs that power real analytical workflows are only as reliable as the retrieval logic built on top of them. Pagination is where that reliability either gets engineered in from the start or gets discovered missing under pressure. The models exist. The failure modes are well-documented. The only variable is whether your team treats this as a design decision or an afterthought.
If you are evaluating how a data API handles pagination before committing to an integration, FeedScale's documentation exposes the specific pagination parameters and rate-limit behavior for each endpoint — test against them directly, at the scale you intend to run.