Data APIs in Production: What Nobody Tells You About Pagination, Rate Limits and Reliability
Data APIs in Production: What Nobody Tells You About Pagination, Rate Limits and Reliability
Most data API documentation looks clean. Neat endpoints, tidy JSON responses, an example in curl that works on the first try. Then you move to production, ingest 40 million records over a weekend, and the pipeline starts failing in ways the docs never mentioned.
This post is about the gap between the sandbox and real workloads. Specifically: the mechanics of consuming data APIs at scale — pagination patterns, rate limit arithmetic, error classification, and the architectural decisions that determine whether your integration holds up or collapses under pressure. If you are building a data pipeline that depends on external REST APIs for media signals, public web data, or large-scale Text and Data Mining (TDM), this is for you.
Pagination Is Not a Detail — It Is the Core Problem
Offset-based pagination (?page=2&size=100) feels intuitive until the dataset is large and live. The core failure mode: if the underlying dataset changes between requests — new records indexed, old ones removed — your offset drifts. You end up with gaps or duplicates you will not notice until much later.
Cursor-based pagination solves the drift problem. Each response returns an opaque cursor pointing to the next slice. The cursor is stable regardless of what happens to the dataset between calls. If an API offers both modes, prefer cursor-based for any dataset that updates frequently.
There is a third pattern worth knowing: keyset pagination, where you paginate by an indexed field (typically a timestamp or a monotonic ID). For time-series data — which is the dominant shape in media intelligence and public signal analysis — this is often the most efficient approach. It maps cleanly to published_after / published_before parameters, makes retries idempotent, and lets you resume mid-job without re-fetching everything from the start.
Practical rule: always log the last successful cursor or timestamp before your pipeline exits. Do not assume the API will give you a clean recovery point if you did not save yours.
Rate Limits: The Arithmetic You Need to Do Before You Start
Rate limits are usually documented as a ceiling (1000 requests/minute, 10000 requests/day). Most engineering teams read that number and move on. The number you actually need to calculate is throughput per record, not per request.
Example: an endpoint returns 100 records per call. At 1000 requests/minute, that is 100,000 records/minute, or roughly 144 million records/day. Sounds fine — until you factor in retries, pagination overhead, fan-out queries across multiple topics or date ranges, and the burst spikes your pipeline produces at startup.
A pattern that works at scale: adaptive backoff with a token-bucket model on your side. Do not simply catch a 429 and sleep for a fixed interval. Implement a client-side rate limiter that tracks your own request rate before the API starts refusing you. This reduces 429s, improves effective throughput, and makes the pipeline predictable.
Also account for quota windows. An API with a daily quota that resets at UTC midnight behaves very differently from one with a rolling 24-hour window. If your pipeline runs overnight and hits the reset boundary mid-job, you need logic to pause and resume cleanly, not just retry blindly.
Classifying Errors: Not All 5xx Are the Same
A 503 during a transient overload is safe to retry. A 500 caused by a malformed query parameter will fail every time regardless of how many retries you throw at it. Treating all errors as retriable is a common mistake that burns quota, inflates costs, and masks real bugs.
A working error taxonomy for data API consumers:
- Retriable / transient: 429, 503, 502, connection timeouts. Retry with exponential backoff and jitter.
- Retriable / idempotent failure: 500 with a server-side trace ID. Log it, retry once or twice, escalate if it persists.
- Non-retriable / client error: 400, 401, 403, 422. These require human intervention or a code fix. Retrying wastes quota.
- Non-retriable / data error: the request succeeds (200) but the response is structurally invalid or empty when it should not be. This is the hardest to catch and the most dangerous for downstream data quality.
That last category deserves a dedicated validation step in your ingestion layer. Schema validation on every response — not just on failure — catches silent data degradation before it poisons your analysis.
Reliability Patterns That Actually Matter
Idempotent writes at the destination. Your ingestion pipeline will re-process records. Accept it. Design your storage layer so that writing the same record twice is harmless — use upserts keyed on a stable record ID rather than blind inserts.
Dead-letter queues for failed batches. When a batch fails after exhausting retries, do not discard it. Route it to a dead-letter store with enough metadata to replay it: endpoint, parameters, timestamp, error code. This turns an operational incident into a recoverable state.
Health probes that test real endpoints. A synthetic ping to /status or /health tells you the API is reachable. It does not tell you that the endpoint you actually use is returning valid data. Add a lightweight canary query — a narrow, well-known request — and assert on the response shape, not just the status code.
Version pinning in your integration layer. APIs evolve. When a new version ships, run both in parallel on a sample of queries before cutting over. The cost of a parallel run for a few days is orders of magnitude lower than debugging a broken pipeline in production.
Pay-as-You-Go APIs Change the Cost Calculus
The shift toward consumption-based pricing in data APIs — where you pay per record returned or per API call rather than a flat monthly seat — changes how you architect your queries. Wide, open-ended queries that would have been fine on a flat subscription become expensive at scale. You need to push filtering as far upstream as possible: use every available server-side filter (date range, source type, language, relevance score) to reduce the payload before it hits your quota.
This is not just a cost issue. It is a data quality issue. Pulling fewer, more relevant signals and processing them well beats pulling everything and hoping your downstream model figures it out.
Platforms like FeedScale are built on this model — you consume what you need, when you need it, and the API is designed to support precise, filtered access to public web signals rather than bulk dumps.
Where to Focus First
If you are starting a new data API integration or hardening an existing one, prioritize in this order:
- Pagination strategy — cursor or keyset, always with checkpointing.
- Client-side rate limiting — proactive, not reactive.
- Error taxonomy — build it once, apply it everywhere.
- Response validation — schema assertions on every call, not just on error.
- Cost-aware query design — server-side filters first, always.
The teams that ship reliable data integrations are not the ones with the most sophisticated infrastructure. They are the ones who treated these five mechanics as first-class engineering problems from day one, rather than operational afterthoughts.