Data APIs: why your pagination strategy decides whether the pipeline survives or not
Data APIs: why your pagination strategy decides whether the pipeline survives or not
Most teams pick a pagination strategy the way they pick a font: once, early, without much thought. Then six months later the pipeline starts dropping records silently, the queue backs up every Tuesday morning, and nobody can explain why. The root cause is almost always the same — the pagination model doesn't match the data's temporal behavior.
This is not a theoretical problem. It shows up the moment your upstream API starts returning high-velocity, continuously updated datasets — exactly what happens with public internet signals, media monitoring feeds, or any source that backfills historical data while simultaneously pushing real-time mentions. Get the pagination wrong and you either re-ingest duplicates at scale or miss an unpredictable slice of the window.
The three models and what they actually cost you
Offset-based pagination (?page=2&size=100) is the default in most legacy APIs and the worst fit for live data. The problem is mathematical: offset is calculated at query time. If 40 new records arrive between your first and second request, every subsequent page shifts by those 40 rows. You silently skip records. No error is raised. The pipeline confirms success. The data is gone.
Use offset-based pagination only when the dataset is static or append-only with guaranteed ordering — batch exports, historical archives, anything that won't mutate between your requests.
Cursor-based pagination solves the drift problem by anchoring position to a server-side pointer, not a numeric count. The API returns an opaque cursor (next_cursor: "eyJ0..."), and your next request continues from that exact position regardless of inserts or deletions upstream. This is the correct model for any real-time feed.
The engineering cost: cursors are stateful. Your pipeline must persist the last-known cursor reliably. If your worker crashes mid-window and you haven't committed the cursor to a durable store (Redis, Postgres, a managed state backend), you restart from an arbitrary position and either re-ingest or lose coverage. Treat cursor state with the same discipline you'd apply to Kafka consumer offsets.
Keyset pagination (also called "seek pagination") is the strongest option when the upstream API exposes a monotonic field you control — typically a timestamp or an auto-incremented ID. Instead of a cursor managed by the server, you filter directly: ?published_after=2026-08-22T09:00:00Z&limit=200. You own the state. Restarts are deterministic. You can parallelise windows across workers without coordination overhead.
The catch: keyset requires that the API's ordering is truly monotonic and stable. If the provider can backfill older records into a timestamp range you've already consumed, you'll miss them unless you build a lookback window into every request.
Matching the model to the API's temporal contract
Before choosing a strategy, ask three questions about the upstream API:
- Can records be inserted into a range I've already paginated? If yes, offset and naive keyset both fail. You need cursor-based pagination with a configurable lookback, or a change-data-capture equivalent.
- Does the API guarantee monotonic ordering on the field I intend to use as a key? If the timestamp resolution is only to the second, two records can share the same key. Your keyset query will skip one of them or loop indefinitely.
- What happens when the cursor expires? Many APIs expire cursors after 5–15 minutes. If your worker is slow or the queue is saturated, the cursor becomes invalid and the API returns an error — or silently restarts from the beginning. Know the TTL. Design your worker throughput around it.
These questions are rarely answered in the API docs. You find out by running load tests at production volume, not by reading the OpenAPI spec.
Practical patterns for high-velocity feeds
Sliding window with overlap. For keyset pagination on feeds where backfill is possible, always request published_after = last_checkpoint - Δt where Δt is a configurable overlap (5–15 minutes is typical). You will ingest duplicates. Handle them downstream with a deduplication layer keyed on a stable document identifier, not on ingestion timestamp.
Cursor checkpointing with write-ahead semantics. Before processing a batch, persist the cursor to durable storage. After processing is confirmed, update a "committed" pointer. On restart, read the committed pointer, not the latest cursor. This mirrors the write-ahead log pattern in databases and prevents the "processed but not committed" gap that causes data loss under failure.
Parallel keyset workers with partition keys. If the API supports filtering by source, geography, or topic category, you can shard the ingestion across multiple workers, each owning a keyset window over its partition. This reduces per-worker throughput pressure and makes cursor state simpler — each worker manages one independent position. The coordination cost shifts to the merge layer downstream.
Adaptive page size. Don't hardcode limit=500. If the API returns variable-size payloads (media signals, enriched text records), large pages under high load will hit timeout thresholds before the response is fully serialised. Start with a conservative page size, instrument median response times per window, and increase only when p95 latency stays below half your timeout threshold.
The operational blind spot: what you can't see without instrumentation
The most dangerous property of pagination bugs is that they look like healthy pipelines. Throughput is positive. No HTTP 4xx or 5xx errors. The queue is draining. But coverage is silently degraded because records are being skipped between pages.
Add two metrics your monitoring stack probably doesn't have yet:
- Inter-page gap count. If the API returns a sequence ID or timestamp on each record, track the delta between the last record of page N and the first record of page N+1. A gap larger than expected page density is a signal of drift.
- Expected vs actual record count per window. If you know the upstream volume profile (even roughly), compare ingested count against a rolling baseline. A consistent 8–12% shortfall that doesn't trigger errors is pagination drift, not a network issue.
Tools like FeedScale expose windowed query semantics that make keyset-style access natural — but the instrumentation layer is still your responsibility. The API can't tell you that you're missing records if you never ask for the count.
Before you ship the next pagination refactor
Pagination is infrastructure, not a feature. It defines the reliability contract between your pipeline and every downstream consumer that depends on completeness guarantees. Teams that treat it as a configuration detail end up rebuilding the ingestion layer under pressure, after an incident, with incomplete data already in the warehouse.
Map your upstream API's temporal contract first. Choose the model that matches it. Instrument the gaps. Then make the pagination strategy explicit in your internal architecture docs — not buried in a worker config file nobody reads.
The records you don't know you're missing are the ones that matter most when the analysis lands in front of a decision-maker.