Blog

Data APIs: how to handle pagination when the dataset never really ends

15 de agosto de 2026 · FeedScale Team

Data APIs: how to handle pagination when the dataset never really ends

Most pagination bugs don't crash the pipeline. They just make it quietly wrong.

You get records 1–100, then 101–200, then you miss a batch because the dataset shifted under you mid-traversal. Offset-based pagination against a live feed is the canonical example: new signals arrive while you're paginating, the index shifts, and you skip rows without a single error thrown. The pipeline finishes, the job reports success, and you've silently lost coverage.

This isn't a niche problem. Any team consuming a high-frequency data API — signals from public sources, mention streams, social data — hits this wall within weeks of going to production. The fix isn't heroic engineering. It's choosing the right pagination model for the data shape you're actually consuming.

Why offset pagination breaks on live data

Offset pagination is the default because it's trivial to implement: ?page=2&limit=100, ?offset=200&limit=100. The problem is structural. Offset is a positional instruction to the database. If a new record is inserted at position 50 while you're fetching page 3, every subsequent page shifts by one. You either duplicate records or drop them — and unless you have a deduplication layer downstream, you won't know which.

For static datasets — a historical archive you export once — offset is fine. For any feed where new data arrives continuously, offset pagination gives you the illusion of completeness. The more frequently the source updates, the faster the illusion collapses.

The inflection point where offset breaks is roughly correlated with dataset volatility. If the underlying source receives more than a handful of new records per minute, you're already in dangerous territory.

Cursor-based pagination: what it solves and what it doesn't

Cursor pagination replaces the positional offset with an opaque pointer to the last record seen. The API returns a next_cursor token; you pass it on the next request. The server resolves position relative to that anchor, not relative to the full dataset.

This solves the shifting-index problem. New inserts don't affect the traversal because you're not navigating by position — you're navigating by identity. The traversal is stable even if the dataset grows during consumption.

The catch: cursor pagination is strictly sequential. You can't jump to page 47 without traversing pages 1–46. For use cases that require random access or parallelized ingestion across segments, cursors force you into a linear bottleneck.

There's also the question of cursor expiration. Many APIs invalidate cursors after a TTL (30 minutes is common). If your pipeline stalls — network issue, rate limit backoff, downstream failure — you may return to an expired cursor and need to restart the traversal from a checkpoint. Designing that checkpoint logic is non-trivial if the pipeline doesn't own it natively.

Keyset pagination: the model that actually scales

Keyset pagination (also called seek-based or cursor-forward pagination using indexed columns) uses a real field value — typically a timestamp or an auto-incrementing ID — as the anchor. Instead of ?offset=200, you send ?after_id=8472910 or ?published_after=2026-08-14T10:30:00Z. The server applies a WHERE id > 8472910 ORDER BY id ASC LIMIT 100 query against an indexed column.

This is the model that holds up at scale. It's deterministic, it's index-efficient, and it's parallelizable when the keyspace allows segmentation (e.g., splitting traversal by time window across multiple workers). It also survives pipeline restarts cleanly: you checkpoint the last ID or timestamp seen, and you resume from there. No cursor expiry, no positional drift.

The constraint is that keyset pagination requires a monotonic, indexed field on the API side. Well-designed data APIs expose this. APIs that don't — or that expose it inconsistently across endpoints — create real operational pain. Before you commit to an API integration in production, checking the pagination model offered is as important as checking the rate limit policy.

Building the checkpoint layer your pipeline actually needs

Regardless of pagination model, your pipeline needs an explicit state layer. Not an implicit assumption that the last successful run ended cleanly.

A minimal checkpoint implementation looks like this:

import time
import requests

CHECKPOINT_FILE = "last_seen_id.txt"
API_ENDPOINT = "https://feedscale.trawlingweb.app/api/v1/signals"
API_TOKEN = "your_token_here"

def load_checkpoint():
    try:
        with open(CHECKPOINT_FILE, "r") as f:
            return f.read().strip()
    except FileNotFoundError:
        return None

def save_checkpoint(last_id):
    with open(CHECKPOINT_FILE, "w") as f:
        f.write(str(last_id))

def fetch_page(after_id=None):
    params = {"limit": 100, "token": API_TOKEN}
    if after_id:
        params["after_id"] = after_id
    response = requests.get(API_ENDPOINT, params=params)
    response.raise_for_status()
    return response.json()

def run_pipeline():
    last_id = load_checkpoint()

    while True:
        data = fetch_page(after_id=last_id)
        records = data.get("results", [])

        if not records:
            print("No new records. Sleeping.")
            time.sleep(60)
            continue

        for record in records:
            process(record)  # your processing logic

        last_id = records[-1]["id"]
        save_checkpoint(last_id)
        print(f"Processed {len(records)} records. Last ID: {last_id}")

def process(record):
    # placeholder — route to your storage or transformation layer
    pass

run_pipeline()

This is deliberately minimal. In production you'd replace the flat file with a distributed store (Redis, DynamoDB, Postgres), add retry logic with exponential backoff, and instrument the checkpoint state for observability. But the structural logic — load state, fetch, process, save state — is the pattern. Any pipeline that skips the explicit checkpoint step is betting on perfect execution every time.

What to verify before you integrate a data API in production

The pagination model is one signal. Before committing to an integration, validate:

These aren't theoretical checklist items. They're the questions that surface as production incidents at 2am if you don't ask them before go-live.

Pagination feels like a solved problem until it isn't. The teams that get it right treat pagination model selection as a first-class architectural decision — not a detail to sort out after the pipeline is already running. The dataset will keep growing. The model you choose at the start will determine whether you can keep up with it.

If you're evaluating data APIs for a continuous ingestion pipeline, FeedScale exposes keyset-compatible endpoints designed for exactly this pattern — incremental, resumable, and built for teams that can't afford silent data loss.

← Volver al blog