Blog

Pagination strategies for data APIs: what breaks at scale and how to fix it

18 de agosto de 2026 · FeedScale Team

Pagination strategies for data APIs: what breaks at scale and how to fix it

Most developers don't think about pagination until it causes an incident. You build the integration, it works fine in staging with a few hundred records, and then production hits a window with 40,000 signals and the pipeline stalls, duplicates, or silently drops data. At that point, the architecture decision you made in week one becomes the most expensive line of code in the codebase.

This is not a theoretical problem. Data APIs returning signals from the public internet — mentions, trends, derived analytics — are inherently bursty. Volume is tied to external events: a brand crisis, a regulatory announcement, a viral story. The pagination strategy you chose during calm periods will be tested at the worst possible moment.

Here is a practical breakdown of the three dominant approaches, where each one fails, and what you should actually do depending on your pipeline's requirements.

Offset pagination: fast to implement, dangerous at scale

Offset-based pagination (?page=3&limit=100 or ?offset=300&limit=100) is the default choice because it requires almost no cognitive overhead. The API documentation shows it, you implement it in ten minutes, and it works.

The problem emerges when the underlying dataset is live. If new signals are ingested between your first and second request, the entire offset shifts. Records that were on page 2 are now on page 3. You skip them. Worse: records on page 1 are still on page 1, so you re-process the same data. In a pipeline processing public internet signals in near-real time, offset drift happens constantly — not occasionally.

There is also a performance cliff. For most database backends, offset queries require scanning all preceding rows before returning results. At ?offset=50000, you are paying for the cost of 50,000 discarded rows on every single request. The API provider pays for it too, which is why many cap offsets or add latency beyond a threshold you never see in the docs until you hit it.

When offset is acceptable: small, static datasets with infrequent updates; internal tools where correctness is auditable manually; rapid prototyping before committing to an architecture.

Cursor-based pagination: the right default for live data

Cursor pagination replaces numeric offsets with an opaque token — typically a base64-encoded internal pointer (a timestamp, a UUID, or a composite key) returned by the API in each response. You pass the token back in the next request, and the API resumes exactly where it left off.

// Response fragment
{
  "data": [...],
  "pagination": {
    "next_cursor": "eyJpZCI6IjE4Mzk5MiIsInRzIjoiMjAyNi0wOC0xNlQxNDozMjowMFoifQ==",
    "has_more": true
  }
}

# Python: walking a cursor-paginated endpoint
import requests

url = "https://api.example.com/v2/signals"
params = {"q": "brand_name", "limit": 200}
headers = {"Authorization": "Bearer YOUR_TOKEN"}

while True:
    response = requests.get(url, params=params, headers=headers)
    payload = response.json()

    process(payload["data"])

    if not payload["pagination"]["has_more"]:
        break

    params["cursor"] = payload["pagination"]["next_cursor"]
    # Remove page param if present — cursor replaces it
    params.pop("page", None)

Cursor pagination survives live dataset mutations because the pointer is anchored to a record position, not a numeric offset. New signals ingested after your first request don't shift the cursor window — you stay synchronized with the state at the time the first request was made, or you explicitly move forward with a time-anchored cursor.

The failure mode here is cursor expiration. APIs commonly expire cursors after a time window (15 minutes, 1 hour, 24 hours). If your pipeline pauses — for a retry, a downstream queue backup, a deployment window — the cursor is gone and you have to decide how to resume. This must be a first-class concern in your error-handling logic, not an afterthought.

Persist the cursor in a durable store (Redis with persistence, a database, a config file you can inspect). Never keep it only in memory.

Keyset pagination: when you own the sort and need resumability

Keyset pagination (sometimes called "seek" pagination) is cursor-based at the conceptual level but uses explicit, readable fields instead of opaque tokens. Instead of passing a cursor, you pass the last seen value of the sort key:

GET /signals?published_after=2026-08-16T14:32:00Z&limit=200&sort=asc

This is more transparent, easier to debug, and trivially resumable: if the pipeline crashes, you reload the last processed timestamp from your state store and resume from there. No cursor expiry, no opaque token to decode.

The trade-off: it only works reliably when your sort field is unique or combined with a tiebreaker (published_at + id). If two records share the exact same timestamp and your page boundary falls between them, you will miss one silently. Design the query accordingly — always include a secondary unique field in the keyset.

This approach maps well to pipelines that use FeedScale or similar services where you are pulling signals over time windows and need deterministic resumption. The pagination state is just a timestamp and an ID — human-readable, storable as a plain string, and independent of API session state.

The hybrid reality: what most production pipelines actually do

Stable pipelines rarely use a single strategy in isolation. A practical pattern:

Initial historical backfill: keyset pagination over a time range, resumable from a checkpoint file.
Ongoing near-real-time ingestion: cursor-based, with cursors persisted to Redis and a fallback to a "last 5 minutes" time window if the cursor expires.
Ad-hoc analytical queries: offset, with explicit awareness that results may overlap, followed by deduplication at the storage layer.

The deduplication layer is not optional regardless of which strategy you use. Idempotent inserts — using the signal's unique ID as the primary key — are the simplest form of protection. Do not treat your pagination strategy as the sole guarantee of exactly-once delivery. It isn't.

What to ask before you choose

Before committing to a pagination strategy with any data API, get answers to these four questions — ideally from the documentation, but test them empirically if the docs are vague:

Does the API support cursor expiry, and what is the window? This determines whether cursors are safe for pipelines with variable throughput.
Is there a maximum offset or page depth? Many APIs silently degrade or error beyond a threshold.
Does the sort order change between requests on the same query? Unstable sort breaks both offset and keyset approaches.
Are there deduplication guarantees on the API side, or does it emit the same signal via multiple sources? This is common in public internet monitoring and must be handled downstream regardless.

Choosing the right pagination strategy is not about picking the most sophisticated option. It is about matching the mechanism to the guarantees your pipeline actually needs — and being honest about what breaks when volume spikes unexpectedly.

If you are building pipelines against APIs like FeedScale that return signals from the public internet, cursor or keyset approaches with durable state checkpointing are the baseline. Offset is a convenience that costs you correctness the moment the real world starts generating events faster than your pipeline expects.

← Volver al blog