HTTP API
The HTTP API is the contract of ndb with the world. Every SDK, the CLI and the console are built on it, and anything a client can see on the wire is written on these pages. A behavior that is not written here is a bug in the docs.
The routes have pages by group:
| Page | Routes |
|---|---|
| System | /_health, /_metrics, /_collections, /_stats, /_size, /_backup, /_restore, /_pitr, /_blobs, /_log, /_cluster/settings, /_cluster/key |
| Collections | PUT, GET, DELETE /{coll}, _truncate, _rename, _restore, _pitr, _indexes, _stats, _compact, _quarantine |
| Documents | _doc, _bulk, _update/{id}, _expire, _persist, _ttl, _lock, _unlock, _tx |
| Queries | _search, _count, _aggregate, _update_by_query, _delete_by_query, patterns |
| Watch | _watch, _ack |
| Streams | _streams, append, trim, groups, read, ack, nack, pending, claim (Redis Streams; nack is ndb only) |
| Errors | the error body and every code |
The query language (operators, date math, $elemMatch) is on Query, the pipeline stages are on Aggregate, the events are on Watch, and the stream model is on Streams. These pages link there and do not repeat it.
Rules
_is the system namespace. A path whose first segment starts with_is the daemon (/_health). Any other first segment is a collection name, which never starts with_. Data and system routes never collide.- A document id always comes after an action:
/{coll}/_doc/{id},_update/{id},_expire/{id},_persist/{id},_ttl/{id}. An_idsuch as"stats"or"_search"is a normal id. Escape/in an id as%2F. - Query is what matches; search is the whole request. The filter is always the body key
query: in_search,_count,_watch,_update_by_query,_delete_by_queryand_update/{id}. - The write operators go in
update, apart fromquery. - Options go in the body (
many,upsert,expire_in). Onlyconfirmis in the URL, on purpose, for drop and truncate. - The version of a document is
_v, everywhere: rows, write results, watch events. - No version in the path. A breaking change ships as a major release, with a migration table in the CHANGELOG.
Every route
| Method | Path | Page |
|---|---|---|
| GET | /_health | System |
| GET | /_metrics | System |
| GET | /_collections | System |
| GET | /_stats | System |
| GET | /_size | System |
| POST | /_backup | System |
| POST | /_restore | System |
| GET | /_pitr | System |
| POST | /_blobs | System |
| GET | /_blobs/{ref} | System |
| GET | /_log | System |
| PUT | /_log | System |
| GET | /_cluster/settings | System |
| PUT | /_cluster/settings | System |
| POST | /_cluster/key | System (unix socket only) |
| PUT | /{coll} | Collections |
| GET | /{coll} | Collections |
| DELETE | /{coll}?confirm={coll} | Collections |
| POST | /{coll}/_truncate?confirm={coll} | Collections |
| POST | /{coll}/_rename | Collections |
| POST | /{coll}/_restore | Collections |
| GET | /{coll}/_pitr | Collections |
| PUT | /{coll}/_indexes/{field} | Collections |
| DELETE | /{coll}/_indexes/{field} | Collections |
| GET | /{coll}/_stats | Collections |
| POST | /{coll}/_compact | Collections |
| GET | /{coll}/_quarantine | Collections |
| POST | /{coll}/_doc | Documents |
| POST | /{coll}/_bulk | Documents |
| GET | /{coll}/_doc/{id} | Documents |
| DELETE | /{coll}/_doc/{id} | Documents |
| POST | /{coll}/_update/{id} | Documents |
| POST | /{coll}/_expire/{id} | Documents |
| POST | /{coll}/_persist/{id} | Documents |
| GET | /{coll}/_ttl/{id} | Documents |
| POST | /{coll}/_lock/{id} | Documents |
| POST | /{coll}/_unlock/{id} | Documents |
| POST | /{coll}/_lock | Documents |
| POST | /{coll}/_unlock | Documents |
| POST | /{coll}/_tx | Documents |
| POST | /{coll}/_search | Queries |
| POST | /{coll}/_count | Queries |
| POST | /{coll}/_aggregate | Queries |
| POST | /{coll}/_update_by_query | Queries |
| POST | /{coll}/_delete_by_query | Queries |
| POST | /{coll}/_watch | Watch |
| POST | /{coll}/_ack | Watch |
| GET | /{s}/_streams | Streams |
| POST | /{s}/_streams/append | Streams |
| POST | /{s}/_streams/trim | Streams |
| POST | /{s}/_streams/read | Streams |
| GET | /{s}/_streams/groups | Streams |
| PUT | /{s}/_streams/groups/{g} | Streams |
| DELETE | /{s}/_streams/groups/{g} | Streams |
| POST | /{s}/_streams/groups/{g}/setid | Streams |
| GET | /{s}/_streams/groups/{g}/consumers | Streams |
| PUT | /{s}/_streams/groups/{g}/consumers/{c} | Streams |
| DELETE | /{s}/_streams/groups/{g}/consumers/{c} | Streams |
| POST | /{s}/_streams/groups/{g}/read | Streams |
| POST | /{s}/_streams/groups/{g}/ack | Streams |
| POST | /{s}/_streams/groups/{g}/nack | Streams (ndb only) |
| POST | /{s}/_streams/groups/{g}/pending | Streams |
| POST | /{s}/_streams/groups/{g}/claim | Streams |
Any other path is 404 unknown_route. A known path with another method is 405 method_not_allowed, with an Allow header.
Transport
- HTTP/1.1, on a unix socket (
listen.unix, default/run/ndb/ndb.sock) and, whenlisten.tcpis set, on TCP. Connections are kept alive; the SDKs keep a pool. - On a unix socket the host name in the URL does not matter:
curl --unix-socket /run/ndb/ndb.sock http://ndb/_health. - The daemon has no TLS. Across machines, put it on a private network (WireGuard) or behind a TLS proxy. See Deployment.
- No browsers by default. The daemon is for servers: apps, the console server, the CLI. A request with an
Originheader, or withSec-Fetch-Site: cross-siteorsame-site, is403 forbiddenon every listener (CSRF guard). An origin incors_originsgets the CORS headers (Access-Control-Allow-Origin,Vary: Origin) and a204answer to its preflightOPTIONS; any otherOPTIONSis405. - Host. On TCP with no token configured, the
Hostmust be an IP,localhost, a name with no dot, or a name inallowed_hosts; else403 forbidden(DNS rebinding guard). With a token, and on the unix socket, theHostis not checked. - Limits.
max_connslimits the open TCP connections (a new one waits).max_watcheslimits the open_watchstreams (429 too_many_requests, withRetry-After).
Authentication
- With
-token-fileortokens:, every TCP request needsAuthorization: Bearer <token>, or the answer is401 unauthorized. The scheme is case-insensitive (bearerworks too, RFC 9110 11.1); the token is exact. - A scoped token without the right for the route and collection is
403 forbidden. See Scoped tokens. - The unix socket needs no token: file permissions protect it.
- ndb serves plain HTTP; TLS comes from a proxy in front (Caddy). See TLS.
POST /_backupworks on the unix socket only; over TCP it is403 forbidden.- After
auth_failuresfailed token checks (default 10 in 1 min) one IP gets429 too_many_requestswithRetry-Afterforauth_lockout(default 5 min), also with the right token. The unix socket has no lockout.
See Authentication and Security: the threat model, the browser guards, TLS and the limits.
Requests
- Body: one JSON object, UTF-8,
Content-Type: application/json. A blob upload is raw bytes. - Unknown keys are errors (
invalid_requestorinvalid_query;invalid_pipelineon_aggregate), so a typo never passes silently. - Size: a body is at most the
max_line_sizeof the collection (default 1 MiB), else413 request_too_large. A system route has its own limit:POST /_backuptakes at most 64 KiB,POST /_restoreat most 1 MiB (the engine default),PUT /_logat most 4 KiB. A blob is at mostblob_max_size(default 1 GiB). - Numbers: JSON numbers. Queries and sorts compare them as 64-bit floats; an integer in a document keeps its exact text.
- Times: on the collection
timefield (andexpires_field), RFC 3339 strings compare as instants, and a string that starts withnowis date math (now-7d/d). - Ids:
_idis a string. A document without one gets a UUIDv7 (time ordered). See Document id. - Collection names:
^[a-z0-9][a-z0-9_-]{0,63}$. A name with one*at the end is a collection pattern, for reads.
Answers
Every answer is JSON, except the rows of _search and _aggregate, the _watch stream, /_metrics, /_backup and blob bytes.
Rows (_search, _aggregate) are a JSON array by default. With Accept: application/x-ndjson they are NDJSON, one row per line, for exports and jq. The rows are the same in both formats; only the framing changes:
Accept: application/json (default) [{"_id":"c1","status":"answered"},{"_id":"c2","status":"failed"}]
Accept: application/x-ndjson {"_id":"c1","status":"answered"}
{"_id":"c2","status":"failed"}A row is the document. _v is added with "versions": true, and _coll on a row of a collection pattern.
Page data is in headers, in both formats, sent before the body:
| Header | When | Meaning |
|---|---|---|
X-Ndb-Next | more rows | the cursor of the next page (after). Absent on the last page. |
X-Ndb-Total | "count": true | how many rows match, whatever limit is |
X-Ndb-Plan | always | id, index:<field>, scan, snapshot or pattern |
X-Ndb-Examined | always | candidate rows whose query was evaluated |
X-Ndb-Took-Ms | always | time spent by the daemon, in ms |
X-Ndb-Explain | "explain": true | JSON of the plan counters, see Search |
Decrypt: an answer has the encrypted fields as their stored "ndb:e1:k1:..." value. Add ?decrypt=1 to a read to get the plain values; it works only for a caller with the decrypt right (else the stored value comes back). ?decrypt=0 is the default. A sealed value sent back in a write stays as it is.
Request id: every answer has X-Request-Id: the one of the request when it is 1 to 128 of A-Z a-z 0-9 - _ . :, else a new one. The log lines of the request have it as request_id (see Logs).
Status codes: 200 for every success, the codes of Errors otherwise. A write that matches nothing is 200 with "matched": 0, not an error.
Order and pagination
sortis an object,field: 1or-1, in key order._idis always the last tie-break, in the direction of the last sort key (-1gives_iddescending), so the order is total. With nosort, the order is insertion order.skip+limitgive offset pages; the cost grows withskip.aftertakes theX-Ndb-Nextof the previous page. The cursor is opaque and bound to thequeryandsort: another query getscursor_mismatch. Rows inserted during the walk do not move pages already read."snapshot": truekeeps the rows of the first page for 60 s of idle time, so every page sees the same data; after that,cursor_expired.- A cursor keeps the
nowof its first page, so date math stays the same window across pages.
See Pagination.
Consistency
- A write is acked after the WAL. With
durability: strict(the default) the WAL is synced to disk before the answer; withrelaxedit is synced in the background. See Durability. - Read your writes: a read that starts after a write was acked sees it.
- One write is atomic: a
_doc, a_bulk(all or nothing), an_update/{id}, a_tx(inserts, updates and deletes of one collection, all or nothing). Amanyupdate or delete is atomic per WAL record (about half ofmax_line_sizeof documents), not as a whole. - Update and delete by query match and write in one step, against the newest state: two concurrent updates never interleave. A guard in
query({"_v": 7},{"qty": {"$gte": 1}}) makes a conditional write. - A lock (
_lock/{id}) gives one caller the document: until it is released or expires, only a write with its token changes the document, and the other writes wait. See Locks. - Collection patterns read each collection at its own point in time; there is no snapshot over many collections.
Readiness and retries
- The daemon listens before recovery ends. Until then every route except
/_healthand/_metricsanswers503 not_ready, and/_healthanswers503 {"status":"recovering"}. - On shutdown, new calls get
503 closed; calls in flight finish. - A client that leaves before the answer gets
499 canceledin the logs and metrics. - Safe to retry: every
GET,_search,_count,_aggregate, and a write that certainly did not reach the daemon (not_ready, a refused connection). - Not safe to retry blindly: a write whose connection dropped after it was sent; it may have been applied. Use an explicit
_id(a second insert is409 duplicate_id) or a_vguard. - The SDKs retry
not_readyand refused connections for 10 s by default (readyTimeout), with backoff and jitter, and repeat a read whose connection dropped. They never repeat such a write.
Long-lived requests
Two routes keep the request open. They are not the same feature.
_watch sends the changes of a collection as NDJSON, one event per line. It stays open until the client closes it, limit events were sent, or the daemon ends it.
- After 15 s with no event, the daemon writes a heartbeat line
{"heartbeat":true}. Ignore it. - The response ends on a truncate or a drop of the collection, on shutdown, and when the consumer is too slow (an error line
watch_lagged). Reconnect: the consumer continues after its last_ack. - An error after the response started is a line
{"error":{"code","message"}}. - See Watch routes.
A stream read with block_ms (POST /{s}/_streams/groups/{g}/read, or POST /{s}/_streams/read without a group) is a normal JSON request that waits up to block_ms for new entries. With none, the answer is []. It sends no heartbeat.
- Set the client read timeout above
block_ms. The SDKs do this for you. - See Stream routes.
Limits
| Limit | Value | Error |
|---|---|---|
| Body size | max_line_size of the collection, default 1 MiB | 413 request_too_large |
| Blob size | blob_max_size, default 1 GiB | 413 request_too_large |
| Collection name | 64 characters, ^[a-z0-9][a-z0-9_-]*$ | 400 catalog_invalid |
| Collections of one pattern | 256 | 400 pattern_too_wide |
$regex pattern | 1 KiB | 400 invalid_query |
| Group table of one aggregate | 256 MiB | 422 group_limit_exceeded |
| Snapshot cursor idle time | 60 s | 400 cursor_expired |
| Watch heartbeat | 15 s of silence | — |
Open _watch streams | max_watches, default no limit | 429 too_many_requests |
| Open TCP connections | max_conns, default no limit | a new connection waits |
_quarantine limit | 1000 | — |
Stream read or claim count | 10 000 | 400 invalid_request |
Versioning
There is no version in the path. The API follows the major version of ndb: a route or a key changes only in a major release, and the CHANGELOG has a table from the old form to the new one. Additions (a new route, a new optional key, a new header) can come in a minor release.