eth_getLogs Limits: Block Ranges, Pagination, and Backfills

Every EVM backfill starts the same way. You point eth_getLogs at a contract, ask for its whole history, and the call dies. So you halve the range, and it dies again. Somewhere around a few hundred blocks it finally returns, you hardcode that number, and three weeks later the same script falls over on a busier contract.

eth_getLogs is the only way to read historical events over plain JSON-RPC, and it is the method most likely to fail in production. That is not one limit misbehaving. There are two separate ceilings, enforced by different layers, and they fail with different messages. Treating both as "the range is too big" is how indexers end up with a magic constant and a retry loop.

The useful part is that Dwellir tells you what went wrong in enough detail to fix it automatically. A range rejection names your plan's cap. A result rejection names a block range that will work. This guide builds a Python scanner that reads those messages and sizes its own window from them, instead of bisecting toward a number it could have just parsed.

What You Will Learn

• Tell the two eth_getLogs rejection causes apart from the error message alone
• Parse the plan cap and the suggested range out of a rejection, so one failed request teaches the scanner its working width
• Cut response size by about 13x with address and topic filters, measured on Dwellir
• Checkpoint a backfill so a crash resumes at the next block instead of the first
• Decide when chunking has run out and you need a wider range, archive depth, or a subscription

Prerequisites

• Python 3.10 or newer. Run python3 --version to confirm.
• A Dwellir API key. The free tier is enough to work through every step here.
• Your plan's block range cap, from Rate Limits. You will use it as the scanner's starting window.
• One package:

pip install httpx

• Familiarity with EVM event logs: what a topic is, and that topics[0] is the event signature hash.

The two ceilings

A rejected eth_getLogs call has one of two causes, and the fix is different for each.

The block range cap limits toBlock - fromBlock regardless of what is in those blocks. It is plan policy, which makes it fixed and knowable:

The result cap is enforced by the node. Dwellir returns at most 20,000 logs from a single query and rejects anything larger rather than truncating. That distinction matters: a query is either fully answered or refused, so you never silently lose events. Other providers set this cap differently, commonly at 10,000.

The two ceilings behave differently, and that drives the whole design. The range cap depends on your plan, so it is worth learning once and never re-testing. The result cap depends on the data inside the window, so it moves as you scan. On a busy contract it binds first: USDC transfers run around 70 logs per block, so roughly 285 blocks is enough to exceed 20,000 results, long before a 10,000-block plan cap is relevant.

Step 1: A JSON-RPC client that surfaces errors

Most Web3 libraries wrap RPC errors in their own exception types and flatten the message. We want the exact wording, because the wording is what tells us which ceiling we hit and how to respond. So we talk to the endpoint directly.

Create rpc.py:

import os

import httpx

RPC_URL = os.environ.get("RPC_URL", "")

client = httpx.Client(timeout=60.0)


class RpcError(Exception):
    def __init__(self, code, message):
        super().__init__(f"[{code}] {message}")
        self.code = code
        self.message = message


def rpc(method, params):
    if not RPC_URL:
        raise RuntimeError("set RPC_URL to your Dwellir endpoint")
    response = client.post(
        RPC_URL,
        json={"jsonrpc": "2.0", "id": 1, "method": method, "params": params},
    )
    payload = response.json()
    if "error" in payload:
        error = payload["error"]
        raise RpcError(error.get("code"), error.get("message", ""))
    return payload["result"]


def block_number():
    return int(rpc("eth_blockNumber", []), 16)


def get_logs(from_block, to_block, address=None, topics=None):
    log_filter = {"fromBlock": hex(from_block), "toBlock": hex(to_block)}
    if address:
        log_filter["address"] = address
    if topics:
        log_filter["topics"] = topics
    return rpc("eth_getLogs", [log_filter])

RpcError keeps both the code and the message. Reading RPC_URL with .get rather than indexing keeps the module importable without the environment set, which is what lets us unit test the logic later without a network.

Point it at your endpoint and confirm it works:

export RPC_URL="https://api-ethereum-mainnet.n.dwellir.com/YOUR_API_KEY"
python3 -c "import rpc; print(rpc.block_number())"

25959787

Step 2: Read both rejections

Before writing any retry logic, trigger both errors and look at them. Ask for a deliberately wide range of USDC transfers:

from rpc import RpcError, block_number, get_logs

TRANSFER_TOPIC = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
USDC = "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"

head = block_number() - 64
for span in (20000, 5000):
    try:
        get_logs(head - span + 1, head, USDC, [TRANSFER_TOPIC])
        print(f"span {span}: accepted")
    except RpcError as error:
        print(f"span {span}: [{error.code}] {error.message}")

On a Growth key, capped at 10,000 blocks:

span 20000: [-32005] eth_getLogs range of 20000 blocks exceeds the 10000-block limit for this plan; split the request into ranges of at most 10000 blocks
span 5000: [-32602] query exceeds max results 20000, retry with the range 25954788-25955001

Both messages carry the answer. The first names the plan cap, so there is no reason to bisect toward a number already written in the error. The second names a block range the node will actually answer, which is better than any guess a client could make.

Note: The numbers in the first message reflect the plan on the key. A Developer key reports the 500-block limit in the same shape.

Now the classifier. Create errors.py:

import re

# Result/size ceiling: the span was legal, the answer was too big or too slow.
# Checked FIRST because these messages can themselves mention a block range.
TOO_MUCH_DATA = re.compile(
    r"exceeds max results|more than \d+ results|response size|query timeout|too many",
    re.I,
)

# Plan/provider ceiling on the span itself. Fixed, so worth remembering.
RANGE_CAP = re.compile(
    r"range of \d+ blocks"          # eth_getLogs range of 20000 blocks exceeds...
    r"|-?block limit"               # ...exceeds the 10000-block limit for this plan
    r"|blocks? range"               # Block range too large / ...0 - 50 blocks range
    r"|ranges? over \d+ blocks"     # ranges over 10000 blocks are not supported
    r"|range must not exceed"       # log query range must not exceed 25 blocks
    r"|maximum allowed is \d+ blocks"
    r"|limited to .{0,20}blocks"
    r"|exceed\w*\s+maximum block",
    re.I,
)

# Nothing to do with the query. Wait and send the same request again.
# No bare HTTP status digits here: they collide with block numbers in messages.
TRANSIENT = re.compile(
    r"rate limit|overloaded|try again|temporarily unavailable|service unavailable|bad gateway",
    re.I,
)

# Dwellir, and some other clients, name a span they will answer. Use it.
SUGGESTED_RANGE = re.compile(r"retry with the range (\d+)-(\d+)", re.I)

# The plan cap is stated in the message. Parse it instead of bisecting toward it.
STATED_CAP = re.compile(
    r"at most (\d+) blocks|(\d+)-block limit|maximum allowed is (\d+) blocks", re.I
)


def classify(message: str) -> str:
    if TOO_MUCH_DATA.search(message):
        return "too_much_data"
    if RANGE_CAP.search(message):
        return "range_cap"
    if TRANSIENT.search(message):
        return "transient"
    return "fatal"


def suggested_end(message: str):
    match = SUGGESTED_RANGE.search(message)
    return int(match.group(2)) if match else None


def stated_cap(message: str):
    match = STATED_CAP.search(message)
    if not match:
        return None
    return int(next(g for g in match.groups() if g))

Two ordering decisions in there are not obvious and will bite you if you reverse them.

TOO_MUCH_DATA is checked before RANGE_CAP because the result-cap message contains the phrase "retry with the range". Check the range patterns first and every result rejection gets misfiled as a plan cap, which permanently shrinks the scanner's ceiling for no reason.

TRANSIENT contains no bare HTTP status numbers. An earlier version matched 50[234] against the message text, which happily matched the digits inside a block number like 25950211. Match status codes on the response, never on prose.

Anything unrecognised falls through to fatal and stops the scan. For a backfill that is the right default: an unknown error should surface, not disappear into a retry loop.

Check it before trusting it

Every message below was captured from a live endpoint. Put them in test_errors.py and assert the classification rather than eyeballing it:

from errors import classify, stated_cap, suggested_end

# (message, expected_kind, expected_suggested_end, expected_cap)
CASES = [
    ("eth_getLogs range of 20000 blocks exceeds the 10000-block limit for this plan; "
     "split the request into ranges of at most 10000 blocks",
     "range_cap", None, 10000),
    ("query exceeds max results 20000, retry with the range 25949701-25949970",
     "too_much_data", 25949970, None),
    ("ranges over 10000 blocks are not supported on free plan",
     "range_cap", None, None),
    ("Block range too large: maximum allowed is 50 blocks on your current plan.",
     "range_cap", None, 50),
    ("eth_getLogs is limited to 0 - 50 blocks range", "range_cap", None, None),
    ("log query range must not exceed 25 blocks", "range_cap", None, None),
    ("server overloaded, retry later", "transient", None, None),
    ("service temporarily unavailable", "transient", None, None),
    ("execution reverted", "fatal", None, None),
    ("Method not found", "fatal", None, None),
]


def main():
    failures = 0
    for message, kind, end, cap in CASES:
        got = (classify(message), suggested_end(message), stated_cap(message))
        ok = got == (kind, end, cap)
        failures += 0 if ok else 1
        print(f"{'ok  ' if ok else 'FAIL'} {got[0]:14s} {message[:56]}")
    print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())

python3 test_errors.py

ok   range_cap      eth_getLogs range of 20000 blocks exceeds the 10000-bloc
ok   too_much_data  query exceeds max results 20000, retry with the range 25
ok   range_cap      ranges over 10000 blocks are not supported on free plan
ok   range_cap      Block range too large: maximum allowed is 50 blocks on y
ok   range_cap      eth_getLogs is limited to 0 - 50 blocks range
ok   range_cap      log query range must not exceed 25 blocks
ok   transient      server overloaded, retry later
ok   transient      service temporarily unavailable
ok   fatal          execution reverted
ok   fatal          Method not found

10/10 passed

The first two are Dwellir. The rest come from other EVM providers, and they are worth keeping in the suite because the same scanner usually ends up pointed at more than one endpoint.

Tip: Log every fatal message you hit in production and fold the real ones into the patterns. This list is a starting point, not a specification.

Step 3: A window that sizes itself

Now the scanner. It tracks two numbers:

• window is the span we will try next
• ceiling is the widest span the plan allows

A range_cap rejection sets ceiling from the cap named in the message, once. A too_much_data rejection adjusts only window, because the next stretch of blocks might be quiet.

Create scan.py:

import json
import os
import sys
import time

from errors import classify, stated_cap, suggested_end
from rpc import RpcError, block_number, get_logs

TRANSFER_TOPIC = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
USDC = "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"

START_WINDOW = int(os.environ.get("START_WINDOW", "10000"))
MIN_WINDOW = 1
FINALITY_LAG = 64
MAX_RETRIES = 5
TARGET_LOGS = 10000


class Scanner:
    def __init__(self, address, topics, start_window=START_WINDOW):
        self.address = address
        self.topics = topics
        self.ceiling = start_window
        self.window = start_window
        self.requests = 0
        self.rejections = 0

    def _fetch(self, from_block, to_block):
        """One request, with backoff for transient failures only."""
        for attempt in range(MAX_RETRIES):
            try:
                self.requests += 1
                return get_logs(from_block, to_block, self.address, self.topics)
            except RpcError as error:
                if classify(error.message) != "transient":
                    raise
                delay = 2**attempt
                print(f"  transient: {error.message}, retrying in {delay}s")
                time.sleep(delay)
        raise RuntimeError(f"gave up on {from_block}-{to_block}")

    def next_chunk(self, cursor, to_block):
        """Return (end_block, logs), narrowing until the call succeeds."""
        while True:
            end = min(cursor + self.window - 1, to_block)
            try:
                return end, self._fetch(cursor, end)
            except RpcError as error:
                kind = classify(error.message)
                if kind not in ("range_cap", "too_much_data") or self.window <= MIN_WINDOW:
                    raise
                self.rejections += 1
                self.window = self._narrow(error.message, kind, cursor, end)
                print(f"  {kind}: {error.message[:72]} -> window {self.window}")

    def _narrow(self, message, kind, cursor, end):
        """Prefer what the node told us over bisection."""
        cap = stated_cap(message)
        if kind == "range_cap" and cap:
            # The message names the plan cap. Go straight there.
            self.ceiling = cap
            return cap

        hint = suggested_end(message)
        if hint and cursor <= hint < end:
            # The node named a span it will answer.
            return hint - cursor + 1

        narrowed = max(MIN_WINDOW, (end - cursor + 1) // 2)
        if kind == "range_cap":
            self.ceiling = narrowed
        return narrowed

    def resize(self, blocks_scanned, logs_returned):
        """Aim the next window at TARGET_LOGS, based on what this one returned."""
        if logs_returned == 0:
            ideal = blocks_scanned * 2
        else:
            ideal = blocks_scanned * TARGET_LOGS / logs_returned
        self.window = int(max(MIN_WINDOW, min(self.ceiling, blocks_scanned * 2, ideal)))

_narrow is the whole point of reading the messages. Bisection is the fallback, used only when the provider gave us nothing to work with.

resize is clamped three ways: never past ceiling, never more than double in one step, never below MIN_WINDOW. TARGET_LOGS sits at half of Dwellir's 20,000-log cap. That headroom absorbs the variance between adjacent windows, which on a busy contract is routinely 30% or more. Aiming at the cap itself means a rejection every time activity ticks up.

Growth matters as much as narrowing. Doubling after every success is the obvious choice and it is wrong: on a uniformly busy contract it guarantees the next window is rejected, so you pay a failed request at the top of every chunk. Sizing from the last result instead keeps the scan clean once it has settled.

Important: MIN_WINDOW is 1, and next_chunk re-raises once it gets there. A single block that still exceeds the result cap is a dead end for eth_getLogs, not something to retry. Going to Production covers what to do with it.

Step 4: Filter before you paginate

Filters are usually presented as an optimisation. For eth_getLogs they change what is reachable, because the result cap counts logs after filtering. A narrower filter means more blocks fit under the same ceiling.

These four queries cover the same 10 Ethereum blocks on Dwellir:

Address plus topic returns 8.8% of the unfiltered log count and 7.8% of the bytes, and it lets a single request cover roughly 11 times as many blocks.

Pass both whenever you know both:

logs = get_logs(
    from_block,
    to_block,
    address="0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
    topics=["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],
)

topics is positional, and it supports two shapes that are easy to miss. None in a slot matches anything, so [TRANSFER_TOPIC, None, recipient] filters on the third indexed argument while leaving the second open. A nested list is an OR, so [[TRANSFER_TOPIC, APPROVAL_TOPIC]] fetches both events in one request. address also accepts a list of contracts, which is how you scan a whole token set without one pass per address.

Step 5: Checkpoint so a crash costs one window

A backfill that restarts from the beginning after an interruption is not a backfill. Add the loop and the checkpoint to the bottom of scan.py:

def load_checkpoint(path, default_block):
    if not os.path.exists(path):
        return default_block, 0
    with open(path) as fh:
        state = json.load(fh)
    print(f"resuming from block {state['next_block']} ({state['logs_written']} logs already written)")
    return state["next_block"], state["logs_written"]


def scan(from_block, to_block, address, topics, out_path, checkpoint_path):
    scanner = Scanner(address, topics)
    cursor, total = load_checkpoint(checkpoint_path, from_block)
    started = time.monotonic()

    with open(out_path, "a") as out:
        while cursor <= to_block:
            cursor_start = cursor
            end, logs = scanner.next_chunk(cursor, to_block)
            for log in logs:
                out.write(json.dumps(log) + "\n")
            out.flush()
            total += len(logs)
            print(f"  {cursor}-{end} ({end - cursor + 1} blocks) -> {len(logs)} logs")

            cursor = end + 1
            with open(checkpoint_path, "w") as fh:
                json.dump({"next_block": cursor, "logs_written": total}, fh)
            scanner.resize(end - cursor_start + 1, len(logs))

    accepted = max(1, scanner.requests - scanner.rejections)
    print(
        f"done: {total} logs, {scanner.requests} requests, {scanner.rejections} rejections, "
        f"{(to_block - from_block + 1) / accepted:.0f} blocks per accepted request, "
        f"{time.monotonic() - started:.1f}s"
    )
    return total


if __name__ == "__main__":
    span = int(sys.argv[1]) if len(sys.argv) > 1 else 10000
    head = block_number() - FINALITY_LAG
    start = head - span + 1
    print(f"scanning {start}..{head}, start window {START_WINDOW}")
    scan(start, head, USDC, [TRANSFER_TOPIC], "usdc-logs.jsonl", "usdc.checkpoint.json")

The ordering is the whole point: write the logs, flush, then advance the checkpoint. A crash between the flush and the checkpoint write replays one window, which produces duplicate rows. A crash in the other order skips a window, which produces missing rows. Duplicates you deduplicate on (transactionHash, logIndex). Gaps you find out about in production.

FINALITY_LAG = 64 keeps the scan behind the chain head, which is covered below.

Putting it all together

Scan 1,000 blocks of USDC transfers, starting from the plan cap:

export RPC_URL="https://api-ethereum-mainnet.n.dwellir.com/YOUR_API_KEY"
export START_WINDOW=10000
python3 scan.py 1000

scanning 25958788..25959787, start window 10000
  too_much_data: query exceeds max results 20000, retry with the range 25958788-25959132 -> window 345
  25958788-25959132 (345 blocks) -> 20000 logs
  25959133-25959304 (172 blocks) -> 16004 logs
  25959305-25959411 (107 blocks) -> 7635 logs
  25959412-25959551 (140 blocks) -> 9070 logs
  25959552-25959705 (154 blocks) -> 12230 logs
  25959706-25959787 (82 blocks) -> 5009 logs
done: 69948 logs, 7 requests, 1 rejections, 167 blocks per accepted request, 1.9s

One rejection for the whole scan. The node named 25958788-25959132 as a range it would answer, the scanner used it verbatim, and from there resize steered each window from the previous result. No bisection happened at all, and the plan cap never came into it because 1,000 blocks is well inside 10,000.

Confirm the output file matches the reported count:

wc -l usdc-logs.jsonl

   69948 usdc-logs.jsonl

Then check the property that actually matters, that the file covers every block exactly once:

python3 -c "
import json
blocks, keys = set(), set()
for line in open('usdc-logs.jsonl'):
    d = json.loads(line)
    blocks.add(int(d['blockNumber'], 16))
    keys.add((d['transactionHash'], d['logIndex']))
lo, hi = min(blocks), max(blocks)
print(f'span {lo}-{hi} ({hi - lo + 1} blocks), {len(keys)} unique logs')
print('gaps:', sum(1 for b in range(lo, hi + 1) if b not in blocks))
"

span 25958788-25959787 (1000 blocks), 69948 unique logs
gaps: 0

69,948 logs, no duplicate (transactionHash, logIndex) pair, no block missing.

Now test the resume path by running the same scan again:

scanning 25958788..25959787, start window 10000
resuming from block 25959788 (69948 logs already written)
done: 69948 logs, 0 requests, 0 rejections, 1000 blocks per accepted request, 0.0s

Zero requests. Delete usdc.checkpoint.json to start over.

Note: USDC is close to the worst case on Ethereum. A mid-volume contract sits at the plan cap and never sees a result-cap rejection at all.

Going to Production

Test the narrowing logic without a network

The failure modes here are hard to reproduce on demand: you cannot ask a provider to reject on cue. Drive Scanner against a stub that mimics both ceilings, and assert coverage rather than log counts:

def fake_get_logs(from_block, to_block, address=None, topics=None):
    span = to_block - from_block + 1
    if span > RANGE_CAP:
        raise RpcError(-32005, f"eth_getLogs range of {span} blocks exceeds the "
                               f"{RANGE_CAP}-block limit for this plan; split the request "
                               f"into ranges of at most {RANGE_CAP} blocks")
    total = 0
    for block in range(from_block, to_block + 1):
        total += chain.get(block, 0)
        if total > RESULT_CAP:
            raise RpcError(-32602, f"query exceeds max results {RESULT_CAP}, "
                                   f"retry with the range {from_block}-{block - 1}")
    covered.extend(range(from_block, to_block + 1))
    return [...]

Then assert that covered equals the full range in order, with no gaps and no repeats, across a spread of random log densities and starting windows. That check is what catches an off-by-one in _narrow before it silently drops a block in production. It also pins the behaviour that matters most: the plan cap should be re-probed exactly once, never on every window.

Stay behind the chain head

Logs from recent blocks can be un-mined. A chain reorg drops blocks your scanner already wrote, and nothing in the JSON-RPC response tells you it happened. FINALITY_LAG = 64 keeps the backfill roughly two epochs behind head on Ethereum, past the point where reorgs are realistic.

That lag is right for a backfill and wrong for a live feed. If you need events at the tip, run two paths: this scanner for finalised history, and eth_subscribe with a logs filter over WebSocket for the head, reconciling the overlap on (blockHash, logIndex). Finality differs per chain, so check before reusing 64 elsewhere.

Budget requests, not just ranges

Narrower windows mean more requests, and requests are what you are billed and rate limited on. The scan above averaged 167 blocks per accepted request against USDC. At that density a year of Ethereum history is roughly 16,000 requests. A quieter contract that sits at the 10,000-block cap does the same year in about 263.

Dwellir counts every response as one request with no compute unit multipliers, so that number is also your cost. The transient branch in _fetch backs off on a 429, but backoff is damage control. The bigger lever is filter selectivity, then plan range.

Parallelise carefully

The scanner is sequential because the checkpoint is a single cursor. To go faster, split the total range into disjoint segments and give each worker its own checkpoint and output file, then merge. Do not share one cursor across workers: a crash leaves a checkpoint claiming completed work in front of gaps that no worker owns.

Keep the worker count under your plan's responses per second. Four to eight workers saturates most backfills before the rate limiter becomes the bottleneck.

Know when a window of 1 is not enough

If next_chunk re-raises at MIN_WINDOW, a single block produced more than 20,000 matching logs on its own. That happens on airdrop claims and large mints. At that point you are past what eth_getLogs can express, and the options are a more selective filter, eth_getBlockReceipts for that block, or a trace method.

Deep history has a second requirement. Blocks older than the pruning window need archive access, and a scan that runs fine on recent blocks can fail partway back without it. Archive nodes are the expensive part of this, so confirm the depth you need before committing to a backfill plan.

Keep the key out of the URL

The examples put the API key in the endpoint path because it is the shortest thing to paste. In production send it as a header instead, so it stays out of logs and proxy traces:

client = httpx.Client(
    timeout=60.0,
    headers={"X-Api-Key": os.environ["DWELLIR_API_KEY"]},
)

Both forms use the same key and the same quota. See Getting Started for the header form.

Next Steps

The scanner writes raw JSONL. Three things to build on top of it:

• Decode the logs. Non-indexed arguments live in data and need the event ABI. Indexed ones are already split across topics.
• Load into a database. Deduplicate on (transactionHash, logIndex) and the replayed window from a crash resolves itself.
• Add the live tail. Pair the backfill with an eth_subscribe logs stream so history and head land in the same table. The Robinhood Chain whale tracker walks through the subscription side.

The one number to get right before you start is your plan's block range cap, because it sets the ceiling everything else works under. Check it in Rate Limits, or create a free account to get a key and read it straight out of the first rejection.