You point eth_getLogs at a contract's full history, fromBlock genesis, toBlock latest, and wait. The request hangs, then errors. So you halve the range. It errors again. You halve it again, something comes back, and you hardcode that number as a constant and ship a backfill loop around it.
Two weeks later the loop breaks, because the contract got popular and the same window now returns more logs than the node will serve. Nobody can say what the correct value is, only that the current one has stopped failing.
That loop is the symptom. You are asking a node a question it was never built to answer, and the fix is not a bigger node or a wider range limit. It is an index.
What Is Blockchain Indexing?
Blockchain indexing is the process of reading raw chain data (blocks, transactions and event logs) and reorganising it into a queryable database. Nodes are optimised for consensus and sequential access, not for questions like "every transfer involving this address since 2021", which is why applications put an indexer in front of an archive node.
The short version: nodes answer "what is the state now". Indexers answer "what happened over time".
A node holds the canonical data and can prove it. What it lacks is a layout that makes arbitrary historical lookups cheap. TrueBlocks puts it bluntly: without an index, a node cannot produce the transactional history of an address.
Why Nodes Are Slow at Historical Queries
Every Ethereum block header carries a 2048-bit logsBloom, a probabilistic filter saying a log might be in this block. That design assumed sparse blocks. The EIP-7745 motivation records that mainnet blocks now emit "over 1000 log addresses and topics in average" against those 2048 bits, that restoring an acceptable false-positive rate would need the filter to grow "about tenfold", and that searching one year of history requires reading "over 6 gigabytes of data" before retrieving a single actual log. At modern gas limits the header bloom has become, in the EIP's words, "practically useless".
Geth's filter path in eth/filters/filter.go runs a bloom pre-screen, then an indexed search through FilterMaps, then re-reads real receipts for every candidate block to discard false positives. Bloom hits are probabilistic, so that receipt read is mandatory. When index data is missing or pruned, Geth falls back to unindexedLogs(), walking blocks one at a time. That O(blocks) path is where large-range queries go to die.
Storage layout compounds it. Geth moves blocks and receipts older than 90,000 blocks behind head into a freezer, flattened raw binary blobs explicitly designed to tolerate slow disks: cheap for long-term storage, poor for random access. Erigon 3 keeps compact receipt metadata in a required domain, while full receipts with logs live in a cache domain that is off by default in every prune mode (--prune.include-receipts). With that cache off, receipts get reconstructed by re-executing transactions. Erigon's PR #23774 reports a 60x eth_getLogs throughput gain from serving cached receipts instead.
This is a storage-layout problem, not a bandwidth problem. Faster NVMe shifts the constant, not the shape of the curve.
The eth_getLogs Range Problem
The JSON-RPC specification sets no limit at all. The filter schema defines fromBlock, toBlock, address, topics and blockHash, and nothing else: no maximum block span, no result count, no response size. The only MUST clause in the method definition is -32602: Invalid params when a bound exceeds head or fromBlock is greater than toBlock. Every cap you have hit is a client setting or a provider policy.
Clients disagree on the defaults. Stock Geth ships --rpc.rangelimit at 0, meaning unlimited, but its --history.logs default of 2,350,000 blocks means a stock node only indexes logs for roughly the last year. Older ranges fall into the unindexed scan. Reth is the only major client shipping non-trivial defaults for both dimensions, at 20,000 logs per response and 100,000 blocks per filter.
Providers are where you actually meet a wall:
| Provider | Block range | Result or size cap |
|---|---|---|
| Alchemy | Free 10 blocks; PAYG and Enterprise unlimited on major chains | 150 MB response cap, all tiers |
| QuickNode | Free Trial 5 blocks; paid 10,000 blocks | None documented |
| Chainstack | Developer 100 blocks; paid 10,000 blocks | None documented |
| Ankr | Public 1,000; Freemium 3,000; Premium 10,000 | None documented |
| Dwellir | Not included on Free or Starter; Developer 500; Growth 10,000; Scale 10,000 with custom on request | 20,000 logs, rejected rather than truncated |
Two ceilings operate at once, which is why one magic constant never holds. The plan block-range cap is toBlock - fromBlock + 1, fixed and knowable before you send the request. The node result cap depends on what is inside the window, so it moves as you scan. On a busy contract the result cap binds first, and filters move the arithmetic further than range tuning does: measured over 10 Ethereum blocks in the Dwellir eth_getLogs limits guide, an unfiltered query returns 8,082 logs while address plus topic returns 710, roughly 11x more blocks per request.
So parse the rejection instead of bisecting toward a constant that goes stale. Dwellir names the window it will serve, in strings like query exceeds max results 20000, retry with the range 25954788-25955001:
import re
RETRY_RANGE = re.compile(r"retry with the range (\d+)-(\d+)")
PLAN_CAP = re.compile(r"(\d+)-block limit for this plan")
def next_window(err, cursor, window):
"""Resize from the node's own rejection rather than halving blindly."""
message = str(err)
plan = PLAN_CAP.search(message)
if plan: # range ceiling: fixed by plan, learn it once
return cursor, int(plan.group(1))
retry = RETRY_RANGE.search(message)
if not retry: # not a limit error, let it surface
raise err
lo, hi = int(retry.group(1)), int(retry.group(2))
return lo, hi - lo + 1 # result ceiling: moves with contract activity
Checkpoint after each window, and stay roughly 64 blocks behind head so a reorg does not rewrite what you just stored. The limits guide has the full scanner.
What an Indexer Actually Does
An indexer is a pipeline with five stages: ingest, decode, handle reorgs, store, serve.
Ingest is where the approaches diverge most. Ponder polls JSON-RPC and exposes an ethGetLogsBlockRange setting, noting that if it is undefined "Ponder will attempt to determine the block range automatically based on error messages", the same trick as the scanner above. Ponder's docs also state that most Ponder apps require a paid RPC provider plan to avoid rate limits, which tells you what a backfill does to your RPC bill.
The alternative is a purpose-built extraction layer. Envio's HyperSync is "a purpose-built data retrieval layer for onchain data, built from the ground up in Rust", built because historical retrieval over JSON-RPC "can take days". SQD runs worker nodes that independently fetch, decode and validate data and cross-check each other before the data lake finalises, with a Portal merging archived history and real-time RPC into one stream. Firehose, built by StreamingFast with The Graph Foundation, goes deepest: a file-based streaming extraction layer requiring an instrumented node, such as a patched Geth, per chain.
Decode turns raw topics into typed events. A subgraph is a manifest plus a GraphQL schema plus AssemblyScript mappings run by Graph Node; Ponder uses TypeScript transforms over the same inputs.
Reorg handling is the stage people underestimate. Ponder's indexing model uses a reorgWindow defaulting to 180 seconds. On a reorg it evicts non-canonical entries from the RPC cache, rolls the database back to the common ancestor using a transaction log, refetches, and re-runs the handlers. Without that machinery, an indexer ingesting at head silently accumulates events that no longer exist on chain.
Storing and serving is the visible part. Ponder writes to Postgres and exposes GraphQL, SQL over HTTP, or direct Postgres access. SQD's Portal streams NDJSON over arbitrary block ranges with internal pagination. Both give you what a node cannot: a query plan tuned for your access pattern rather than for consensus.
Envio published a vendor-run benchmark in late 2023 over Uniswap V3 ETH-USDC, blocks 12,376,729 to 18,342,024, about 5.4 million events with identical schemas and handlers: Envio 9.67 minutes, Subsquid 20.50 minutes, Ponder 780.37 minutes, The Graph hosted around 1,000 minutes, Substreams subgraph 1,529.33 minutes. Treat it as directional. One of the compared vendors ran it, it is almost three years old, and Envio's own caveat is that Envio and Subsquid ran locally while the subgraphs ran hosted, "introducing potential performance variance".
Comparing the Four Approaches
| Approach | Setup effort | Query latency | Historical depth | Cost model | Best for |
|---|---|---|---|---|---|
| Raw RPC calls | None | Seconds to minutes per range | Whatever the node retains; Geth indexes logs for 2,350,000 blocks by default | Per request or per compute unit | One-off analysis, narrow recent ranges |
| Self-hosted indexer (Ponder, SQD) | Days, plus ongoing ops | Milliseconds from Postgres | Full, bounded by your RPC or extraction layer | Your infra plus a paid RPC plan for backfill | Production apps needing custom schemas |
| The Graph subgraph | Hours to write, then deploy | Milliseconds over GraphQL | Full, from the configured start block | 100,000 free queries/month, then $2 per 100,000 | Public APIs other teams query |
| Managed data API | Minutes | Milliseconds | Vendor-defined | Credits, compute units or points | Getting to a working product fast |
The published anchors: The Graph Studio gives 100,000 free queries per month and charges $2 per 100,000 beyond, so 300,000 queries is $4 per month, across 60+ networks. Covalent GoldRush charges $250 per month for 300,000 credits at 50 RPS on Professional. Bitquery sells points from $49 per month for 100,000, but self-serve plans are real-time only and historical access is an add-on from $70 per month per chain. SQD Cloud bills infrastructure directly: processors $0.048 to $0.72 per hour, storage $0.60 per GB per month.
The Archive Node Dependency
"Do I need an archive node?" usually gets a shrug. Reth's pruning documentation answers it precisely, by listing which methods break when each segment is pruned.
Read that as a requirements spec. Log-only indexing, which covers most subgraphs and most Ponder apps, needs receipt retention and nothing else. It does not need historical state. Indexing that calls eth_call at past block heights, or reads traces to capture internal transfers, needs Account History and Storage History, which is what archive means. Reth's full mode keeps current state plus a 10,064-block window, which does not reach back far enough for either.
Geth draws the same line: a path-based full node keeps exactly one full state, 128 blocks in the past, while archive is --history.state=0 with --syncmode=full. Geth's separate history pruning deletes historical block bodies and receipts up to the merge block while preserving the state trie and all headers, which is exactly the wrong trade for a log indexer.
Archive disk cost is client- and mode-dependent, and the quoted figures often measure different things. Geth's docs put path-based archive at roughly 2 TB for flat state only, about 6.5 TB with historical trie data, and over 20 TB for the legacy hash-based archive, which is where most alarming numbers come from. Erigon's hardware requirements, measured on v3.6 on 2026-07-19, list 2.03 TB archive against 419 GB full. Dwellir's hidden cost of archive nodes covers what those disks become as a monthly bill once you add replication and backups.
Which Approach Should You Choose?
- One-off analysis, a few thousand blocks. Raw RPC with a chunked scanner. Do not build a pipeline for a question you will ask once.
- A handful of events on one contract, one chain. Raw RPC plus a checkpointed backfill into your own Postgres.
- Multi-chain production app with custom schemas. A self-hosted indexer such as Ponder or SQD, with a paid RPC plan sized for the backfill.
- A public API other teams query. A subgraph on The Graph. The free 100,000 queries per month absorbs a lot of early traffic.
- Historical state or traces, not just logs. Archive access with Account History and Storage History retained, whichever indexing layer sits on top.
- Sub-second freshness at head. Real-time streaming plus explicit reorg handling, and a decision about how far behind head you trust. A 180-second
reorgWindowis a reasonable starting point.
Where Your RPC Bill Comes From
Indexers are the heaviest archive-RPC consumers there are. A backfill is millions of eth_getLogs calls against historical data, and that is the method compute-unit pricing weights most heavily. Alchemy's published compute unit costs put eth_getLogs at 60 CU against 10 for eth_blockNumber and 26 for eth_call. The weighting is defensible, since a wide log query does cost more to serve. It also means your indexer's bill is set by a 6x multiplier on the one method it calls most, and that multiplier is not on the pricing page.
Indexing teams are a large part of who runs on Dwellir, so the shape of the workload is familiar: a backfill that hammers one method for days, then a long tail of head-following calls that never stops. Teams usually arrive at the same moment in that curve. The pipeline works, the data is right, and the bill has grown past what the queries are worth. Archive access is included on Dwellir's paid plans across 140+ networks with no premium for historical depth, and the model is flat: 1 RPC response is 1 API credit, trace and debug included, with no per-method multiplier to reverse-engineer. Developer is $49 per month for 25 million responses with a 500-block eth_getLogs range; Growth is $299 for 150 million at a 10,000-block range; Scale is $999 for 500 million. RPC providers without compute units compares the billing models in more depth, and self-hosted vs managed RPC nodes works through the build-or-buy side.
The behaviour at the result cap matters just as much to a scanner. Dwellir returns at most 20,000 logs from a single query and refuses anything larger rather than truncating, so a query is either fully answered or rejected and you never silently lose events. The rejection names the range that will work, which is what lets a scanner size its own window instead of bisecting.
FAQ
What is blockchain indexing?
Blockchain indexing is the process of reading raw chain data (blocks, transactions and event logs) and reorganising it into a queryable database. Nodes are optimised for consensus and sequential access, not for questions like "every transfer involving this address since 2021", which is why applications put an indexer in front of an archive node.
What does a blockchain indexer do?
An indexer ingests blocks and receipts from a node or a purpose-built extraction layer, decodes raw topics into typed events, handles chain reorganisations, stores the output in a database such as Postgres, and serves it over GraphQL, SQL or a streaming API.
Do I need an indexer or just an archive node?
An archive node gives you the data; an indexer gives you a query layout. If your queries are narrow and recent, an archive node and chunked eth_getLogs calls are enough. If you need arbitrary historical queries, joins across contracts, or millisecond response times, you need an index in front of the node.
Why is eth_getLogs slow over large block ranges?
The 2048-bit header bloom filter has stopped being selective, because mainnet blocks now emit over 1,000 log addresses and topics on average. Clients pre-screen with the bloom, re-read real receipts to discard false positives, and fall back to scanning blocks one at a time when index data is missing. EIP-7745 estimates that searching one year of history requires reading over 6 gigabytes of data before returning a single log.
What is the difference between The Graph and a self-hosted indexer?
A subgraph on The Graph is a manifest, a GraphQL schema and AssemblyScript mappings executed by Graph Node, billed per query at $2 per 100,000 after a free 100,000 per month. A self-hosted indexer such as Ponder runs your own TypeScript transforms into your own Postgres, so you control the schema and the serving layer, and you pay for infrastructure plus the RPC calls the backfill consumes.
How much does blockchain indexing cost?
It depends on the model. The Graph Studio is free for the first 100,000 queries per month and $2 per 100,000 after. Managed APIs range from $49 per month on Bitquery Personal to $250 per month on Covalent Professional for 300,000 credits. Self-hosting shifts the cost to infrastructure plus RPC, and SQD Cloud publishes that split at $0.048 to $0.72 per hour for processors and $0.60 per GB per month for storage.
Can I index without running an archive node?
Yes, in two ways. If you only index event logs, you need receipt retention rather than historical state, which a pruned node can provide. If you need historical state or traces, you can use a managed archive RPC endpoint instead of operating the node yourself, which is how most teams avoid the multi-terabyte disks.
Wrapping Up
The magic constant in your backfill loop is a guess at two limits at once: a plan block-range cap that never changes, and a node result cap that moves with contract activity. Reading the error message instead of bisecting removes the guess, and an address plus topic filter buys roughly 11x more blocks per request before you reach the result cap at all.
Past that, the decision is about query shape. Logs only, and you need receipt retention and a scanner. Historical eth_call or traces, and you need Account History and Storage History, which means archive. Arbitrary questions over years of data at millisecond latency, and you need an index, because no node layout will give you that.
If you are sizing the RPC side of a backfill, the eth_getLogs limits guide has the window arithmetic and the error strings, and Dwellir's pricing shows what a million historical log queries costs when there is no multiplier on them.
