# StreamRawBookDiffs - Real-time Book Diff Streaming Stream incremental order book changes starting from a position, providing real-time access to every addition, size change, and removal in Hyperliquid's resting book. Clone our [gRPC Code Examples Repository](https://github.com/dwellir-public/gRPC-code-examples) for complete, runnable implementations in Go, Python, and Node.js. ## When to Use This Method `StreamRawBookDiffs` is essential for: - **Order Book Reconstruction** - Maintain a live local book by applying per-block diffs - **Liquidity Analysis** - Track depth, spread, and resting-order churn over time - **Microstructure Research** - Study how the book evolves block by block - **Backtesting Inputs** - Replay historical book changes from a chosen position Method signature ```protobuf rpc StreamRawBookDiffs(StreamRequest) returns (stream Record) {} ``` V3 accepts an inclusive `position.block_number` or Unix-millisecond `position.timestamp`. Explicit zero is invalid. A positioned call replays inclusively and then tails live; an empty request tails newly written diffs. Set `progress` to opt in to [progress frames](#progress-frames). ```protobuf position: { block_number: 1058646974 } filter: { fields: { field: "coins" op: FILTER_OP_IN values: "BTC" values: "ETH" } } ``` Raw book diffs accept `users`, `coins` and `dexes` filters. User matching is case-insensitive, coin matching is case-sensitive and dex matching is exact on the HIP-3 dex prefix; values within a field use OR semantics and separate fields use AND semantics. Records with zero matching events are skipped. Each `Record` contains optional `block_number` and `timestamp` cursors plus the filtered envelope in `data`. A pruned position returns `NOT_FOUND`; malformed filters return `INVALID_ARGUMENT`. Raw diffs are not an initial book state. Seed reconstruction from a snapshot or another known state, then apply ordered diffs. ## V3 reference ### Request Parameters - `position.timestamp` (`int64, optional`): Inclusive V3 Unix-millisecond cursor. Omit the position to tail newly written records. - `position.block_number` (`int64, optional`): Inclusive V3 sequential block cursor. Explicit zero is invalid. - `filter.fields` (`FieldFilter[], optional`): V3 users/coins constraints. Fields AND together; values within one field OR together. - `filter.fields.field` (`string, required`): Feed-advertised filter field name, either users or coins. - `filter.fields.op` (`FilterOp, optional`): FILTER_OP_UNSPECIFIED or FILTER_OP_IN. - `filter.fields.values` (`string[], required`): One or more accepted values; entries within the list use OR semantics. - `progress.interval_ms` (`uint32, optional`): Set progress to opt in to progress frames; leave it unset for none. Longest silence in milliseconds before a progress frame. 0 selects the feed default, currently 20000 within bounds of 10000 to 60000; ListFeeds reports the bounds in FeedInfo.progress. ### Request Example ```bash grpcurl \ -import-path content/docs/hyperliquid/grpc \ -proto v3.proto \ -H "x-api-key: ${DWELLIR_API_KEY}" \ -d '{"position":{"block_number":"1058646974"},"filter":{"fields":[{"field":"coins","op":"FILTER_OP_IN","values":["BTC"]}]}}' \ "${HYPERLIQUID_GRPC_ENDPOINT}" \ hyperliquid_l1_gateway.v3.HyperliquidL1Gateway/StreamRawBookDiffs ``` ### Response Fields - `block_number` (`optional int64, optional`): Sequential Hyperliquid block cursor; never the HyperBFT consensus round. - `timestamp` (`optional int64, optional`): Record time in Unix milliseconds. - `data` (`bytes, required`): Bytes from the node_raw_book_diffs_by_block, with non-matching events removed when a filter is applied. - `progress` (`StreamProgress, optional`): Set only on progress frames, which leave data, block_number, and timestamp unset. Check it before decoding data. - `progress.scanned_through.block_number` (`int64, optional`): Inclusive cursor of the newest block the stream examined, sent or removed by the filter. A valid resume position; unset until the stream has examined a record. - `progress.timestamp` (`optional int64, optional`): Source time at scanned_through in Unix milliseconds. ### Successful Response ```protobuf block_number: 1058646974 timestamp: 1785736814057 data: "{\"block_number\":1058646974,\"events\":[]}" ``` ### Error Responses #### Invalid argument - Code: `INVALID_ARGUMENT` - Description: A cursor is zero or malformed, or a filter has an unsupported field, duplicate field, empty name, or empty value list. #### Invalid progress request - Code: `INVALID_ARGUMENT` - Description: A nonzero progress.interval_ms is outside the bounds advertised in FeedInfo.progress. #### Not found - Code: `NOT_FOUND` - Description: Available replay history varies by feed and endpoint. A cursor outside retained history returns NOT_FOUND. #### Replay capacity exhausted - Code: `RESOURCE_EXHAUSTED` - Description: Replay capacity is temporarily unavailable; retry with backoff. #### Filter operator not implemented - Code: `UNIMPLEMENTED` - Description: The requested filter operator is well formed but is not implemented by this gateway version. #### Stream unavailable - Code: `UNAVAILABLE` - Description: No new records arrived before the stream timeout, or the stream failed mid-stream. Progress frames do not reset the timeout. Reconnect from the last cursor received. #### Records lost - Code: `DATA_LOSS` - Description: The stream lost records, for example because it fell behind the live feed. It ends before sending any cursor past them; reconnect from the last cursor received to re-read the gap. #### Slow consumer - Code: `DEADLINE_EXCEEDED` - Description: The stream could not deliver records in time because the client read too slowly or stopped reading; reconnect from the last processed cursor. #### Unauthenticated - Code: `UNAUTHENTICATED` - Description: The request did not include usable authentication metadata. #### Permission denied - Code: `PERMISSION_DENIED` - Description: The supplied credential is not permitted to call this service. ## V2 reference ### Request Parameters - `timestamp_ms` (`int64, optional`): Inclusive Unix-millisecond cursor. Omit it to start from the latest data and continue streaming. - `block_height` (`int64, optional`): Inclusive block-height cursor. Omit it to start from the latest data and continue streaming. ### Request Example ```bash grpcurl \ -import-path content/docs/hyperliquid/grpc \ -proto v2.proto \ -H "x-api-key: ${DWELLIR_API_KEY}" \ -d '{"block_height":"1058646974"}' \ "${HYPERLIQUID_GRPC_ENDPOINT}" \ hyperliquid_l1_gateway.v2.HyperliquidL1Gateway/StreamRawBookDiffs ``` ### Response Fields - `data` (`bytes, required`): JSON-encoded payload returned in the V2 BlockRawBookDiffs wrapper. ### Successful Response ```protobuf message BlockRawBookDiffs { bytes data = 1; } ``` ```protobuf rpc StreamRawBookDiffs(Position) returns (stream BlockRawBookDiffs) {} ``` V2 accepts the inclusive cursor fields `block_height` and `timestamp_ms`; an empty `Position` tails new diffs. It returns the source JSON in `BlockRawBookDiffs.data`. ## Response Stream V3 carries the JSON-encoded `node_raw_book_diffs_by_block` envelope in `Record.data`. V2 carries the JSON-encoded `node_raw_book_diffs_by_block` envelope in `BlockRawBookDiffs.data`. Decode the `data` field to access: - Block reference (`local_time`, `block_time`, `block_number`) - An `events` array of incremental resting-book changes - Per-change order context (owner, order ID, coin, side, price) ## Full Book Diff Spec `StreamRawBookDiffs` emits a JSON payload matching Hyperliquid's `node_raw_book_diffs_by_block` format. Below is the exact structure. Top-level keys (always present): ```json { "local_time": "2026-04-20T00:00:00.040479928", // ISO-8601 timestamp when node processed the block (nanosecond precision) "block_time": "2026-04-19T23:59:59.576040791", // ISO-8601 timestamp from block consensus "block_number": 12345, // Block height containing these changes "events": [ // Array of incremental book-change objects { "user": "0x1234567890abcdef1234567890abcdef12345678", // Order owner address "oid": 12345, // Order identifier "coin": "BTC", // Coin identifier "side": "B", // "B" (buy) or "A" (sell/ask) "px": "42000.0", // Price (numeric string) "raw_book_diff": "remove" // Change applied to the book (see shapes below) }, { "user": "0x1234567890abcdef1234567890abcdef12345678", "oid": 12346, "coin": "BTC", "side": "A", "px": "42050.0", "raw_book_diff": { "new": { "sz": "0.5" } } // Order added with size }, { "user": "0x1234567890abcdef1234567890abcdef12345678", "oid": 12347, "coin": "BTC", "side": "A", "px": "42100.0", "raw_book_diff": { "update": { "newSz": "0.4", "origSz": "1.0" } } // Resting size changed } ] } ``` Book diff event fields (`events[i]`): | Field | Type | Description | | --------------- | ---------------- | ----------------------------------------------------------------------------------------------- | | `user` | string | Order owner address | | `oid` | number | Order identifier. Not reliably unique across coins. See [Order Identifiers](#order-identifiers) | | `coin` | string | Coin identifier (e.g. `"BTC"`, `"ETH"`) | | `side` | string | `"B"` for buy, `"A"` for sell | | `px` | string | Price (numeric string) | | `raw_book_diff` | string or object | The change applied to the book. One of three shapes (below) | The `raw_book_diff` value takes one of three shapes: | Shape | Meaning | | ----------------------------------------------------- | --------------------------------------------- | | `"remove"` (string literal) | Order removed from the book | | `{ "new": { "sz": string } }` | Order added with the given size | | `{ "update": { "newSz": string, "origSz": string } }` | Resting size changed from `origSz` to `newSz` | Guarantees and alignment: - `events` contains all book changes from the block, across all coins and users. - Messages arrive in strictly increasing `block_number`, one per source block (including blocks with an empty `events` array). - `block_number` is sequential and distinct from the potentially skipping `abci_block.round` consensus value in StreamBlocks; `block_time` aligns with `abci_block.time`. Developer tips: - Apply diffs in order: branch on the `raw_book_diff` shape to add, resize, or remove the resting order. - Key your local book on `(coin, oid)`, never on `oid` alone. See [Order Identifiers](#order-identifiers). - Normalize `px`, `sz`, `newSz`, and `origSz` (strings) to numeric types as appropriate. ## Order Identifiers `oid` is **not reliably unique across coins**. Consumers that index or join per-order state must key on the `(coin, oid)` tuple, never on `oid` alone. Outcome markets (HIP-4) demonstrably reuse oids across the two sides of a single market (for example `#30` and `#31`) within one block, so a diff for an `oid` on one coin is not necessarily the same logical order as a diff for that `oid` on another coin in the same block. Keying on `(coin, oid)` is safe whether or not oids turn out to be globally unique elsewhere. ## Common Use Cases ### 1. Local Book Reconstruction ```python class BookReconstructor: def __init__(self): # (coin, oid) -> {"side", "px", "sz"} self.orders = {} def apply_block(self, block): for event in block.get('events', []): key = (event['coin'], event['oid']) diff = event['raw_book_diff'] if diff == 'remove': self.orders.pop(key, None) elif 'new' in diff: self.orders[key] = { 'side': event['side'], 'px': event['px'], 'sz': diff['new']['sz'], } elif 'update' in diff: order = self.orders.get(key) if order: order['sz'] = diff['update']['newSz'] ``` ### 2. Streaming Diffs in Python Open V3 with an empty `StreamRequest` to tail newly written records, or set `StreamRequest.position.block_number` for inclusive replay. Decode each `Record.data` envelope before applying the diffs with the reconstructor above. ```python import grpc import json import os import hyperliquid_pb2 import hyperliquid_pb2_grpc def stream_book_diffs(): endpoint = os.getenv('HYPERLIQUID_ENDPOINT') api_key = os.getenv('API_KEY') credentials = grpc.ssl_channel_credentials() options = [('grpc.max_receive_message_length', 150 * 1024 * 1024)] # 150MB max metadata = [('x-api-key', api_key)] with grpc.secure_channel(endpoint, credentials, options=options) as channel: client = hyperliquid_pb2_grpc.HyperliquidL1GatewayStub(channel) # Empty Position means latest; set block_height or timestamp_ms to backfill request = hyperliquid_pb2.Position() for response in client.StreamRawBookDiffs(request, metadata=metadata): block = json.loads(response.data) print(f"block {block['block_number']}: {len(block.get('events', []))} diffs") ``` ## Progress Frames Progress frames are an application-level heartbeat. Set `StreamRequest.progress` to receive one after each interval of silence. A `users` or `coins` filter can leave the stream silent for minutes. They keep it open through idle-limited proxies, which HTTP/2 keepalive pings do not, and advance your resume cursor through blocks the filter removed. A progress frame sets `Record.progress.scanned_through.block_number` and leaves the data fields unset, so check `progress` before decoding `data`. Persist `scanned_through` as an inclusive resume cursor. See [progress frames](https://www.dwellir.com/docs/hyperliquid/grpc#progress-frames) for the full contract. V2 streams do not send progress frames. Use the V3 method to opt in. ## Error Handling and Reconnection Reconnect with `StreamRequest.position.block_number` set to the last committed V3 cursor, from a data record or a progress frame's `scanned_through`. Replay is inclusive, so deduplicate that record before applying subsequent diffs. Reconnect by replaying from the last processed block. Pass the next `block_height` in the `Position` so no diffs are skipped or double-applied. ```python def stream_with_resume(client, metadata, start_block): next_block = start_block while True: request = hyperliquid_pb2.Position(block_height=next_block) try: for response in client.StreamRawBookDiffs(request, metadata=metadata): block = json.loads(response.data) process(block) next_block = block['block_number'] + 1 except grpc.RpcError as e: print(f"stream error, resuming from {next_block}: {e}") ``` ## Best Practices 1. **Connection Management**: Implement robust reconnection logic with exponential backoff 2. **Resumable State**: Follow the version-specific recovery guidance above and persist the cursor required by your selected API version 3. **Stable Keys**: Index resting orders on `(coin, oid)` to stay correct across coins 4. **Performance**: Apply diffs asynchronously to avoid blocking the stream 5. **Resource Cleanup**: Properly close streams and connections on shutdown ## Current Limitations - **Replay History**: Available replay history varies by feed and endpoint. A cursor outside retained history returns `NOT_FOUND`. - **Backpressure**: High-volume periods may require careful handling to avoid overwhelming downstream systems ## Resources - **[GitHub: gRPC Code Examples](https://github.com/dwellir-public/gRPC-code-examples)** - Complete working examples - **[GetRawBookDiffs](https://www.dwellir.com/docs/hyperliquid/get_raw_book_diffs)** - Point-in-time book diff retrieval - **[StreamOrderStatuses](https://www.dwellir.com/docs/hyperliquid/stream_order_statuses)** - The sibling block-envelope stream for order lifecycle events - **[Historical Data](https://www.dwellir.com/docs/hyperliquid/historical-data)** - Historical `node_raw_book_diffs_by_block` downloads *Need help? Contact our [support team](mailto:support@dwellir.com) or check the [Hyperliquid gRPC documentation](https://www.dwellir.com/docs/hyperliquid/grpc).*