TypeScript
The package @thadeu/ndb (the name ndb is taken on npm). Typed collections, a chainable query builder, aggregate, watch, streams, blobs and admin.
The source and the same guide are in sdk/ts.
Install
pnpm add @thadeu/ndb
# or
npm install @thadeu/ndb
bun add @thadeu/ndbIt runs on Node 24 or newer (with undici) and on Bun (with its own fetch), over a unix socket or TCP. You also need a running ndb daemon. See the repository for how to run one.
Connect
import { connect } from '@thadeu/ndb'
// Unix socket.
const ndb = connect('unix:///run/ndb/ndb.sock') // or tcp://host:port, https://host
// TCP with a bearer token.
const remote = connect('tcp://10.0.0.5:7070', { token: 'secret' })
// Address and token from the environment: NDB_URL and NDB_TOKEN.
const fromEnv = connect()
// Retry for up to 5 s while the daemon starts or recovers.
const patient = connect('unix:///run/ndb/ndb.sock', { readyTimeout: 5000 })
await ndb.health()
await ndb.close()connectdoes not connect. The first call does.- The address is
unix:///path,tcp://host:port, orhttps://host(ndb behind a TLS proxy;http://works too). Without an address,connectreadsNDB_URL. With no address and noNDB_URL, it throws aTypeError: there is no default address. - A local daemon that you start with no flags and not as root listens on the unix socket
~/.ndb/ndb.sockand on TCP127.0.0.1:7070. As root (or with/var/lib/ndb), it listens only on/run/ndb/ndb.sock. Give the full path:connect(`unix://${os.homedir()}/.ndb/ndb.sock`)orconnect('tcp://127.0.0.1:7070'). See Configuration. - The token goes only in the
Authorizationheader. A unix socket does not need it. - Every request sends
User-Agent: ndb-ts/<version> (node v24.1.0; linux)(bun v…ordeno v…on those runtimes).connect(addr, { userAgent: 'myapp/1.2' })adds the token of your app at the end, so the daemon can tell the apps apart. A browser cannot set a User-Agent: there the SDK sends none.VERSIONis exported. readyTimeoutis in milliseconds. It sets how long a call retries when the daemon is not ready (not_ready, or the socket is missing or refuses). Default 10000. Zero turns the retry off. Writes are retried only when the daemon certainly did not process them. Reads are also retried after a dropped connection.- The pool has up to 16 connections. There is no setting for it.
new Ndb(addr, opts)does the same asconnect.
Define a collection
const calls = ndb.collection('calls', {
time: 'at',
fields: {
status: { type: 'string', index: true },
caller: { type: 'string', unique: true },
duration: { type: 'number', optional: true },
at: { type: 'date' },
meta: { type: 'json', optional: true },
},
})- Field types:
string,number,boolean,date(aDate),json(any value). - Field options:
index,unique,optional,encrypted: true,blind: 'numeric'(or'exact','lower','upper'). Reads senddecrypt=1;coll.encrypted()is a handle whose reads keep the values sealed. See Encrypted fields: a blind field is found by value withquery({ cpf: '390.533.447-05' }). - A compound index or unique key is a schema option, next to
fields, with the fields in order:indexes: [['caller', 'at']],unique: [['account_id', 'email']]. _idis the primary key of every document. It is not a schema field: leave it out on insert and the daemon makes a UUIDv7 (or usenewId()). For a natural key (email, SKU), keep_idand mark that fieldunique.timeis the collection time field. Used by zone maps,$dateTruncand date math (now-7d).collectionsends nothing. The first call declares the collection (PUT /calls). You can callawait calls.ready()to do it at once. A name with*(a pattern) never declares.- Types come from the schema. Rows have the schema fields,
_idand_v, the document version. Adatefield is aDatein rows. Input takesDatevalues and ISO strings in filters. expiresis thedatefield that holds when a document expires. It turns onexpireIn,expire,persistandttl(below).
Insert, get, update, delete
// One document (POST /calls/_doc): one result.
const one = await calls.insert({ _id: 'c1', status: 'answered', caller: 'a', duration: 30, at: new Date() })
// one: { _id: 'c1', _v: 1 }
// Many in one transaction, all or nothing (POST /calls/_bulk): an array.
const results = await calls.insert([
{ _id: 'c2', status: 'failed', caller: 'b', at: new Date() },
{ status: 'answered', caller: 'c', at: new Date() },
])
// Get by _id (GET /calls/_doc/c1). Returns null when missing.
const row = await calls.get('c1')
// Update by _id, in one atomic step. guard adds conditions on the same
// document: a _v guard is an optimistic lock.
const res = await calls.update('c1', { $set: { status: 'done' }, $unset: { duration: true } }, { guard: { _v: row!._v } })
// res: { matched, modified, doc } (doc is the new row with _v)
// withRetry does the read, the guard and the retry: change the row in the
// function, and the changed fields are written with the _v guard. On a
// conflict it reads again and calls the function again (5 tries,
// { retries }), then throws StaleObjectError. A throw in the function writes
// nothing. The function can run more than once.
await calls.withRetry('c1', (call) => {
if (call.status === 'done') return
call.status = 'done'
})
// lock is the pessimistic lock: one holder at a time, so the function runs
// once. Use it when the function has an effect outside ndb. Another lock
// waits ({ wait }, then LockedError); a plain write waits for the release.
// { ttl } (default 30 s) must cover the function: after it the write throws
// LockExpiredError.
await calls.lock('c1', async (call) => {
if (call.status === 'billed') return
await billing.charge(call._id)
call.status = 'billed'
})
// transaction: the writes of the function go in one step, all or none. The
// ids are locked first (all or none) and given by _id; [] takes no lock. A
// throw writes nothing; throw new Rollback() stops with no error. A stale or
// missing document fails it with TxConflictError.
await calls.transaction(['c1', 'c2'], (tx, docs) => {
tx.update({ ...docs.c1!, status: 'merged' })
tx.delete('c2')
tx.insert({ _id: 'c3', status: 'new', caller: '+5511999990003', at: new Date() })
}, { ttl: 5 })
// Atomic update by query. A guarded decrement: exactly one of two racing calls wins.
await stock.updateBy({ _id: 'a', qty: { $gte: 1 } }, { $inc: { qty: -1 } })
// Every match (the query needs _id or an indexed field), or create it.
await calls.updateBy({ status: 'failed' }, { $set: { status: 'retry' } }, { many: true })
await calls.update('c9', { $inc: { duration: 1 } }, { upsert: true })
// Delete by _id or by query: { deleted }.
await calls.delete('c2')
await calls.deleteBy({ status: 'failed' }, { many: true })- Operators:
$set,$setOnInsert(like$set, but only when an upsert creates the document),$unset,$inc,$push(with{ $each: [...] }),$pull,$min,$max. Field names can be dot paths. - Nothing matched is not an error:
matchedis 0 (for example when the_vguard is stale, so read again and retry).modifiedis 0 when the operators leave the document as it is. update(id, ops, { guard, upsert, expireIn }):upsertcreates the document with that_id.updateBy(query, ops, { many, upsert, expireIn }):manyneeds_idor an indexed field in the query, andupsertcreates the document from the equality fields of the query.- A key of the operators without
$throwsInvalidUpdateErrorbefore anything is sent. The daemon contract errors (unknown_operator,no_update,conflicting_update,invalid_field,unindexed_many,empty_filter) areInvalidUpdateErrortoo.
An insert with a duplicate _id throws DuplicateIdError.
upsert(doc) and upsertAll(docs) create by _id, or merge into the stored document when the _id exists, in one transaction. See Upsert. A duplicate unique value throws UniqueViolationError. An empty or non-string _id fails with code invalid_id.
Truncate and drop
await calls.truncate() // every document goes; the definition and indexes stay
await calls.drop() // the collection, its definition and its watch consumers go- Both take the same time for 10 or 10 million documents. Watch consumers get one event with
op: 'truncate'(_idis''), then the new changes. - After
drop(), the next call on the collection declares the collection again. A collection declared inndb.ymlcomes back, empty, when the daemon restarts.
Rename, restore and PITR
const restored = db.collection('calls_restored', schema)
await calls.restore({ at: new Date(Date.now() - 3_600_000), to: 'calls_restored' })
await restored.rename('calls', { replace: true })
const win = await calls.pitr() // { collection, window, oldest, newest, snapshots, bytes, dropped }rename(to, { replace }): documents, indexes and watch positions move toto. Withoutreplace, an existingtothrowsCollectionExistsError. Withreplace, the existingtogoes aside first. The object keeps the old name: open the collection again underto.restore({ at, to }): makes the collectiontohold this collection as it was atat. The source does not change. Out of the PITR window (or withpitr: off) it throwsPitrError. An existingtothrowsCollectionExistsError.pitr(): the PITR window of the collection, and the disk it holds. See Collections.
Expiry (TTL like Redis)
const sessions = db.collection('sessions', {
expires: 'exp',
fields: { exp: { type: 'date', optional: true } },
})
await sessions.insert({ _id: 's1' }, { expireIn: '1h' }) // or 3600; all docs of the call
await sessions.update('s1', { $set: { user: 'u' } }, { expireIn: '30m' })
const { v, expiresAt } = await sessions.expire('s1', '30m') // EXPIRE
await sessions.expireAt('s1', new Date(Date.now() + 3_600_000))
await sessions.ttl('s1') // seconds left, -1 = never expires, null = missing or expired
await sessions.persist('s1') // PERSIST- A duration is seconds or a string such as
"90m","1h30m","1d".expireInwins over a value of theexpiresfield in the same call. expireandpersistthrowNotFoundErrorfor a missing or expired document.TtlError(ttl_not_enabled,ttl_derived) means the schema has noexpiresfield, or the expiry comes fromttl+time.- Disk space of expired documents returns after compaction, and fully only with
wal_retention; see Freeing disk.
Query
Queries built elsewhere
For a search form or a saved search, take the JSON as is. rawQuery takes an untyped query and chains like query; search takes a whole search body and returns one page.
const form = JSON.parse(input) as Record<string, unknown>
const rows = await calls.rawQuery(form).query({ dnis: '1000' }).sort({ ts: -1 }).all()
const page = await calls.search({ query: { status: 'answered' }, sort: { ts: -1 }, limit: 50, count: true })
page.rows // Row[], decoded like query()
page.next // cursor for { after }; '' on the last page
page.total // with count: trueRows decode like any other query. The daemon validates the JSON (invalid_query), but it takes any field: for a public form, keep an allowlist of fields and operators, cap limit, and ask for an indexed field or a time window, as you would before sending a form to Elasticsearch _search.
Collection patterns
A collection named with a pattern (logs-2026-*) reads every matched collection. It sends no PUT, and its writes reject with code pattern_read_only. Each row has _coll, the name of its collection.
const logs = db.collection('logs-2026-*', logSchema)
const rows = await logs.query({ lvl: 'error' }).sort({ at: -1 }).limit(50).all()
rows[0]._coll // 'logs-2026-11'Count
const answered = await calls.count({ status: 'answered' }) // POST /calls/_count
const total = await calls.count()Cursor pagination
Offset pages get slower as skip grows. Use a cursor for deep pages.
const q = calls.query({ status: 'answered' }).sort({ _id: 1 }).limit(100)
const page = await q
const token = q.nextCursor // '' on the last page
const next = await calls.query({ status: 'answered' }).sort({ _id: 1 }).limit(100).after(token)A token works only with the same query and sort. Another query shape throws CursorError.
To walk every row, use each(). It follows the cursor, with limit as the batch size:
for await (const row of calls.query({ status: 'answered' }).sort({ _id: 1 }).limit(500).each()) {
console.log(row._id)
}Explain
There is no explain method. Send explain: true in a raw request and read the header:
const res = await ndb.stream('POST', calls.path('_search'), { query: { status: 'failed' }, explain: true, limit: 1 })
console.log(res.header('x-ndb-plan')) // 'index:status'
console.log(res.header('x-ndb-explain')) // JSON string with the counters
await res.body.dump()Lookup
lookup is a key of the HTTP search (POST /{coll}/_search): in the same call, the daemon reads the documents of other collections that each row points to, and puts them in the row. The HTTP API is the primitive and has every rule (Lookup); this SDK only builds that JSON.
| Relation | The foreign key lives | Match | The row gets |
|---|---|---|---|
| belongs to | in this collection | row.fk == other.pk | an object, or null |
| has many | in the other collection | other.fk == row.pk | an array (100 unless a limit, at most 1000) |
| has one | in the other collection | other.fk == row.pk | an object, or null |
pk is _id unless you say otherwise. The field that is searched on the other side (its pk for a belongs to, its fk for a has many) must be _id or indexed, else the daemon answers lookup_unindexed.
Basic use
import { belongsTo, hasMany, hasOne } from '@thadeu/ndb'
// belongs to: orders.customer_id -> customers._id
const rows = await orders
.query({ status: 'paid' })
.lookup<'customer', { name: string } | null>('customer', belongsTo('customers', 'customer_id'))
.all()
rows[0].customer?.name // 'Ana' (null when there is no customer)
// has many: order_items.order_id -> orders._id
const withItems = await orders
.query({})
.lookup<'items', Item[]>('items', hasMany('order_items', 'order_id'))
.all()
// has one: invoices.order_id -> orders._id
const withInvoice = await orders
.query({})
.lookup<'invoice', Invoice | null>('invoice', hasOne('invoices', 'order_id'))
.all()The type parameters are optional: the first is the name, the second the type of the field (Record<string, unknown> by default).
Options
hasMany('order_items', 'order_id', {
query: { qty: { $gt: 0 } }, // filter the related documents
sort: { pos: 1 }, // their order
limit: 50, // per row, 1 to 1000
fields: ['sku', 'qty'], // keep these fields
lookup: { // nested, two levels at most
product: belongsTo('products', 'sku', { pk: 'code' }), // products.code instead of _id
},
})belongsTo takes pk, fields and lookup; hasOne takes everything but limit. Each one is the plain JSON object of the HTTP API ({ has_many: 'order_items', fk: 'order_id', ... }), so you can also write it by hand.
Index
A hasMany('order_items', 'order_id') needs an index on order_items.order_id: index: true on that field in the schema of the collection, or items.ensureIndex('order_id').
Errors
LookupError (codes lookup_invalid, lookup_unindexed, lookup_too_large) and NotFoundError for an unknown collection.
Aggregate
The pipeline needs exactly one $group. Before it: $match and one $sort. After it: $match, $sort, $limit, $skip, $project. Accumulators: $count, $sum, $avg, $min, $max, $first, $last.
const res = await calls.aggregate(
[
{ $match: { status: { $in: ['answered', 'failed'] } } },
{
$group: {
_id: { status: '$status', hour: { $dateTrunc: { date: '$at', unit: 'hour' } } },
n: { $count: {} },
avg: { $avg: '$duration' },
},
},
{ $sort: { n: -1 } },
],
{ limit: 100, count: true },
)
res.rows // the group rows
res.next // cursor for the next page ('' on the last one)
res.total // with count: trueOptions: limit, skip, after, count. Pass a type to type the rows: calls.aggregate<{ _id: string; n: number }>(...). $dateTrunc units are minute, hour, day, week and month.
Watch
A watch streams the changes of a collection to a named consumer. First it sends the events after the last ack. Then it sends new events as they happen.
import { HistoryTruncatedError, WatchLaggedError } from '@thadeu/ndb'
const controller = new AbortController()
try {
for await (const ev of calls.watch('relay', { query: { status: 'answered' }, signal: controller.signal })) {
// ev: { consumer, pos, op: 'put' | 'del' | 'truncate', _id, v, doc }
await handle(ev)
await calls.ack(ev)
}
} catch (err) {
if (err instanceof WatchLaggedError) {
// The consumer was too slow. Call watch again. It continues after the last ack.
} else if (err instanceof HistoryTruncatedError) {
// The saved position is too old. Start over.
} else {
throw err
}
}- Options:
from: 'now'(a new consumer starts at the end),limit(stop after n events),query(same filters as a search),signal(anAbortSignalthat stops the stream). - A delete event has
doc: undefined. Withquery, a delete is sent only if the same stream saw an earlier matching put of that id. - Delivery is at least once. After a restart an event can come again, so make the work safe to repeat.
ackstores the position. The nextwatchwith the same consumer name starts after it.HistoryTruncatedErrormeans the WAL files for the position were deleted (wal_retention).
Streams
A stream is an append-only log with consumer groups. It has the behavior of Redis Streams. See Streams for the model and Stream routes for the HTTP API.
db.stream<T>(name) gives a Stream<T>. T is the type of the entry. The entry ID and the time are in Entry, not in T.
import { connect } from '@thadeu/ndb'
const db = connect()
const orders = db.stream<{ order: number; total: number }>('orders')db.stream(name)sends nothing. The first append, or the first group create, makes the stream.await orders.create()makes it at once, or checks that the name is a stream.- The
prefixofconnectgoes before the stream name, as for a collection. db.streamhas two forms. With one argument (a name), it gives aStream<T>. With a method and a path (db.stream('POST', path, body)), it is a raw request: it returns aStreamedanswer that you read as bytes or NDJSON rows (see Explain). The two forms are not related.- An entry is
Entry<T>:{ id, time, doc, deleted }.idis"<ms>-<seq>".timeis the ms part of the ID as aDate.docis the entry without_idand_ts.deletedistruefor a pending entry that a trim or a delete removed; thendocis{}. idTime(id)gives theDateof an ID. A bad ID gives an invalidDate.
append
const id = await orders.append({ order: 42, total: 99.9 }) // "1731164400000-0"
await orders.append({ order: 43, total: 1 }, { trim: { maxLen: 10_000, approx: true } })
await orders.append({ order: 44, total: 2 }, { id: '1731164400500-*' })
await orders.append({ order: 45, total: 3 }, { noCreate: true })Adds one entry and returns its ID (XADD).
id:"<ms>-<seq>"or"<ms>-*". Default"*": the daemon makes the ID. An ID that is not greater than the last one throwsStreamIdError(stream_id_too_small).trim: trims the stream after the append. It takes the trim options.noCreate: when the stream does not exist, throwNotFoundError(unknown_collection) and do not make it (NOMKSTREAM).
appendMany
const ids = await orders.appendMany([
{ order: 46, total: 1 },
{ order: 47, total: 2 },
])- Adds the entries in order, with one fsync for all. Returns their IDs.
- It takes the same options as
append.idapplies to the first entry.
trim
await orders.trim({ maxLen: 1000 }) // keep the newest 1000
await orders.trim({ minId: '1731160000000-0' }) // delete the IDs lower than this
await orders.trim({ maxLen: 1000, approx: true, limit: 5000 })Deletes old entries and returns how many it deleted (XTRIM).
- Give
maxLenorminId. approx: delete only whole blocks of 1 024 entries (~in Redis). It is faster.limit: withapprox, delete at most this many entries.- A trimmed entry that is still pending comes to a group read with
deleted: true. See Trim and retention.
info
const info = await orders.info()
// { length, first_id, last_id, entries_added, max_deleted_id, groups }- The state of the stream (
XINFO STREAM,XLEN).
range and revRange
const first = await orders.range('-', '+', 100) // XRANGE, lowest ID first
const last = await orders.revRange('+', '-', 10) // XREVRANGE, highest ID first
first[0].id
first[0].time
first[0].doc.totalrange(start = '-', end = '+', count?): up tocountentries with an ID fromstarttoend, both included, in ID order.revRange(end = '+', start = '-', count?): the same, fromenddown tostart. Note the order of the arguments:endcomes first, as in Redis.-and+are the lowest and the highest ID. Withoutcount, the daemon default applies.
read without a group
let entries = await orders.read({ after: '0', count: 100 }) // XREAD, from the start
entries = await orders.read({ after: entries.at(-1)?.id, count: 100, blockMs: 5000, signal })
await orders.read({ after: '$', blockMs: 5000 }) // only the entries appended after this call startsorders.read(opts?) gives the entries after an ID, in ID order, without a group (XREAD). It changes no group and no pending list, and nothing is acked. An empty answer is an empty array. The read of a group is group.read.
Fields of StreamReadOptions:
after: give the entries with an ID greater than this one. Default'0', the start of the stream.'$'is the last ID when the call starts: only later entries come.count: the most entries to give. Default 10, at most 10 000.blockMs: when there is no entry, wait up to this long (ms) for one.fields: give only these fields (and_id).signal: anAbortSignalthat ends the read, also a blocked one.
To read on, give the ID of the last entry as after. Do not give '$' again: an entry appended between two calls would be lost.
delete
const n = await orders.delete('1731164400000-0') // 1, or 0 when it is not there- Removes one entry by ID (
XDEL).
groups
const list = await orders.groups()
// [{ name, consumers, pending, last_delivered_id, entries_read, lag }]- The groups of the stream (
XINFO GROUPS).entries_readandlagarenullwhen they cannot be known.
group
import { startAll, startNew } from '@thadeu/ndb'
const g = orders.group('billing') // sends nothing
await g.create(startNew) // XGROUP CREATE ... MKSTREAM
await g.ensure(startAll) // create, or do nothing when it exists
await g.setId('0') // XGROUP SETID
await g.destroy() // XGROUP DESTROYstream.group(name) gives a Group<T>. Each entry goes to one consumer of the group. It stays pending until it is acked.
create(start = startNew): makes the group, and the stream when it does not exist. A group that exists throwsGroupExistsError.ensure(start = startNew): likecreate, but a group that exists is not an error. Use it at the start of a worker.startisstartNew('$': only the entries appended after the create),startAll('0': every entry) or an entry ID.setId(id): moves the last delivered ID of the group.destroy(): deletes the group and its pending entries.- A call on a group that does not exist throws
UnknownGroupError.
consumers
const created = await g.createConsumer('w1') // true when it is new
const list = await g.consumers() // [{ name, pending, idle_ms, inactive_ms }]
const held = await g.deleteConsumer('w1') // how many pending entries it heldcreateConsumer(name): makes a consumer (XGROUP CREATECONSUMER). A read makes it too.consumers(): the consumers of the group (XINFO CONSUMERS).deleteConsumer(name): deletes the consumer and its pending entries (XGROUP DELCONSUMER).
read
const controller = new AbortController()
const entries = await g.read('w1', { count: 100, blockMs: 5000, signal: controller.signal })
const mine = await g.read('w1', { pending: true, after: '0' })Gives entries to the consumer (XREADGROUP). By default it gives new entries, and they become pending for that consumer. An empty answer is an empty array.
count: the most entries to give. Default 10, at most 10 000.blockMs: when there is no new entry, wait up to this long (ms) for one. A blocked read has no request timeout.pending: give the consumer's own pending entries (XREADGROUP ... 0), with an ID greater thanafter, not new ones.noAck: do not add the entries to the pending list (at most once).fields: give only these fields (and_id).signal: anAbortSignalthat ends the read, also a blocked one.
ack
const n = await g.ack(...entries.map((e) => e.id))- Ends the work on the entries (
XACK). Returns how many were pending. An ID that is not pending counts 0. - With no ID, it sends nothing and returns 0.
pending
const sum = await g.pending()
// { count, lowest, highest, consumers: [{ name, count }], waiting }- The pending list in short (
XPENDING).lowestandhighestarenullwhen the list is empty. waiting(ndb only): the nacked entries whose retry time has not come. A nacked entry counts incount, and for no consumer inconsumers.
pendingList
const idle = await g.pendingList({ idleMs: 60_000, count: 100 })
// [{ id, consumer, idleMs, deliveries, retryAt }]- Pending entries, in ID order (
XPENDINGwith a range). - Options:
startandend(default'-'and'+'),count(default 100),consumer(only its entries),idleMs(only the entries idle for at least this long),due(ndb only:truegives only the nacked entries whose retry time passed,falseonly the ones that still wait). retryAt(ndb only): aDatewhen the entry is nacked (it is due again at that time), elsenull. A nacked entry hasconsumer: ''.
claim
const res = await g.claim('w2', 60_000, ['1731164400000-0'])
// { cursor: '', entries, deleted }claim(consumer, minIdleMs, ids): moves the entries idle for at leastminIdleMstoconsumer(XCLAIM). Each one gets one more delivery.entriesare the claimed entries.deletedare the pending IDs whose entry is no longer in the stream. They left the pending list.
autoClaim
let cursor = '0-0'
do {
const res = await g.autoClaim('w2', 60_000, cursor, 100)
await work(res.entries)
cursor = res.cursor
} while (cursor !== '0-0')autoClaim(consumer, minIdleMs, cursor = '0-0', count?): scans the pending list fromcursorand moves up tocountidle entries toconsumer(XAUTOCLAIM).cursorin the answer is where the next call starts. It is'0-0'when the scan is complete.
nack
const n = await g.nack('w1', [entry.id], 60_000) // back in 1 min- ndb only: Redis has no delayed retry.
nack(consumer, ids, delayMs)gives the entries back to the group afterdelayMs, for a retry. Returns how many it nacked. - Only the IDs pending for this consumer count; other IDs count 0, not an error. With no ID, it sends nothing and returns 0.
- The entry stays in the pending list with no owner (
consumer: '') and aretryAt. Its delivery count does not change; the claim that takes it again adds 1. delayMs: 0 makes the entry due at once. At most 30 days: a larger or negative value is a400(NdbErrorwith codeinvalid_request).- While it waits, a nacked entry is in no pending read (
readwithpending: true), so a restarted worker does not take it early, and no claim takes it. When it is due, any consumer of the group takes it withclaimDue(orclaim, at once, whateverminIdleMs). - An
ackof a nacked entry removes it: this cancels the retry.
claimDue
let cursor = '0-0'
do {
const res = await g.claimDue('w2', { cursor, count: 100 })
await work(res.entries)
cursor = res.cursor
} while (cursor !== '0-0')- ndb only.
claimDue(consumer, { cursor, count, justIds }): scans the nacked entries fromcursor(default'0-0') and moves up tocountdue ones (their retry time passed) toconsumer, in ID order. Each one gets one more delivery. The answer is{ cursor, entries, deleted }, asautoClaim. - It looks only at the nacked entries, not at the entries in flight at live workers, so a worker can call it on a short tick.
workdoes it everydueEveryMs. - With
justIds: true, the answer is{ cursor, ids, deleted }and the delivery count does not change.
work
import { PoisonError } from '@thadeu/ndb'
const g = orders.group('billing')
await g.ensure()
const controller = new AbortController()
process.on('SIGTERM', () => controller.abort())
await g.work('worker-1', async (entry, signal) => {
await charge(entry.doc, signal) // resolves: ack; throws: stays pending
}, {
count: 100,
claimIdleMs: 60_000,
maxDeliveries: 5,
deadLetter: 'orders_dead',
onError: (err) => {
if (err instanceof PoisonError) console.warn('dead', err.id, err.deliveries)
else console.error(err)
},
signal: controller.signal,
})group.work(consumer, handler, opts) runs a worker of the group until the signal aborts. Then it resolves. The worker:
- Reads its own pending entries first (the ones it took before a restart and did not ack), and does the work on them.
- Reads new entries, and waits up to
blockMsfor them. - Acks each entry that the handler did without an error, in one call.
- Every
claimEveryMs, claims the entries of any consumer of the group that are idle forclaimIdleMs, and tries them again. - With
retry, nacks each entry that failed, with the wait of its try, and everydueEveryMsclaims the nacked entries that are due (claimDue).
Notes:
- The handler is
(entry, signal) => unknown. When it resolves, the entry is acked. When it throws, the entry stays pending (or is nacked, withretry), andonErrorgets the error. - A throw after the signal aborted is a stop, not a failure: the entry stays pending for the next start.
onErrordoes not get it, and the entry does not go to the dead letter. - Make the group with
ensure(). A read that fails goes toonError, and the worker reads again after a backoff (100 ms, doubled up to 5 s). This is also true for a group or a stream that does not exist (UnknownGroupError,NotFoundError, as thecauseof the reported error): a worker that starts before its group (deploy order) waits for it. Only the abort of the signal ends the worker. - Options and defaults:
count: the most entries per read. Default 10.blockMs: how long a read waits for new entries. Default 5000.claimIdleMs: default 60000. A negative value turns the claim off.claimEveryMs: defaultclaimIdleMs / 2, at least 1000.maxDeliveries: default 0 (no limit).deadLetter: the stream name for poison messages. Default none.retry: the wait (ms) before a failed entry comes back. A function(deliveries) => ms(see tiers), or an array of waits ([15_000, 60_000]: the wait of try n is the n-th one, the last one repeats). Default none: a failed entry stays pending untilclaimIdleMs.maxAgeMs: an entry older than this, by the time of its ID, is not tried again. Default 0 (no limit).dueEveryMs: how often the worker claims the due nacked entries. Default 1000 withretry; withoutretry, the due claim runs only when you set it. A negative value turns it off.onError: gets the errors that do not stop the worker: a failed read (the worker then waits, from 100 ms up to 5 s), a handler error, a failed ack or claim, and aPoisonError. An error thatonErrorthrows is ignored: the worker and its acks go on.signal: anAbortSignalthat stops the worker. Without it, the worker runs forever.
- Stop: abort the signal. The abort also ends a blocked read. The acks of the work that is done are still sent.
- Poison message: an entry that fails on its
maxDeliveries-th delivery. At that failure, whendeadLetteris set, the worker appends aDeadEntryto that stream:{ stream, group, id, deliveries, reason, doc }. Then it acks the entry, andonErrorgets aPoisonError(withid,deliveriesandreason). WithoutdeadLetter, the entry is only acked and reported.reasonis'max_deliveries'here. - Delayed retry: with
retry, an entry that fails is nacked withretry(deliveries)(deliveriesis 1 for the first try), clamped to 0 to 30 days. It has no owner while it waits: a restart of the worker does not take it early, and any worker of the group takes it when it is due, within aboutdueEveryMs. An entry that fails on itsmaxDeliveries-th delivery goes to the dead letter, not to a nack. Aretrythat throws is reported, and the entry stays pending. - Max age: an entry older than
maxAgeMsgoes to the dead letter withreason: 'max_age'and is acked, with no retry. The worker checks it before the handler, and again after a failure (no wait for a try that is too old to run). - An own pending entry at start with more than
maxDeliveriesdeliveries was left by a process that stopped in the handler (a crash, a kill). The worker sends it to the dead letter without the handler, so a poison entry that kills the process does not run forever. - A
deletedentry (trimmed or deleted while pending) is acked without the handler. - Delivery is at least once. Make the work safe to repeat.
- To share the work, run many workers with different consumer names in the same group.
- See The worker in the SDKs.
tiers
import { tiers } from '@thadeu/ndb'
tiers(15_000, 60_000, 600_000) // try 1: 15 s, try 2: 1 min, then 10 mintiers(...steps)gives a function forretry: the wait after a failed try, by its delivery count. A number is the wait of one try. A step{ waitMs, upTo }holds for every try up toupTo, after the steps before it; a step with noupTois the last one. After the last step, its wait repeats.- An array in
retryis the same astiers(...array).
The post-call outbox of fsw-esl: try 1 waits 15 s, tries 2 to 5 wait 1 min, 6 to 11 wait 10 min, 12 to 23 wait 1 h, then 6 h until the entry is 15 days old:
import { PoisonError, tiers } from '@thadeu/ndb'
const minute = 60_000
const hour = 60 * minute
await jobs.group('postcall').work('worker-1', async (entry, signal) => {
await postCall(entry.doc, signal) // resolves: ack; throws: nack with the wait of its try
}, {
retry: tiers(
{ waitMs: 15_000, upTo: 1 },
{ waitMs: minute, upTo: 5 },
{ waitMs: 10 * minute, upTo: 11 },
{ waitMs: hour, upTo: 23 },
{ waitMs: 6 * hour },
),
maxAgeMs: 15 * 24 * hour,
deadLetter: 'postcall_dead',
onError: (err) => {
if (err instanceof PoisonError) console.warn('dead', err.id, err.reason)
else console.error(err)
},
signal: controller.signal,
})With the five waits one try each (retry: [15_000, 60_000, 600_000, 3_600_000, 21_600_000]), try 5 and later wait 6 h.
Stream errors
| Class | Codes |
|---|---|
NotAStreamError | not_a_stream: a stream call on a collection that is not a stream |
StreamIdError | invalid_stream_id, stream_id_too_small |
UnknownGroupError | unknown_group |
GroupExistsError | group_exists (create only; ensure ignores it) |
PoisonError | poison: not from the daemon. work gives it to onError |
All are NdbError subclasses. See the error table below and the stream route errors.
Redis Streams commands
An app that uses Redis Streams can change to ndb and keep its calls. Two subpaths of the package give the stream commands of the two Redis libraries on ndb streams, with the same arguments and the same answers: @thadeu/ndb/ioredis and @thadeu/ndb/node-redis. The app picks its backend in one place, its factory. See Move from Redis Streams for the same change in Ruby and Go.
- The adapters have no dependency. The main entry does not load them, and they do not import
ioredisorredis. Their types are structural. - A type test of the package checks that each adapter is assignable to the stream commands of its library (
Pick<Redis, ...>of ioredis 5,Pick<RedisClientType, ...>of node-redis 5). So the two sides of the?:in the factory have the same methods. - The two libraries have different shapes, so there are two adapters. Use the one of the library that the app has.
ioredis
import { connect } from '@thadeu/ndb'
import { ioredis } from '@thadeu/ndb/ioredis'
import { Redis } from 'ioredis'
const streams = process.env.STREAMS_BACKEND === 'ndb' ? ioredis(connect()) : new Redis()
await streams.xadd('orders', 'MAXLEN', '~', 1000, '*', 'order_id', '42')
await streams.xgroup('CREATE', 'orders', 'billing', '$', 'MKSTREAM')
const got = await streams.xreadgroup('GROUP', 'billing', 'w1', 'COUNT', 10, 'BLOCK', 5000, 'STREAMS', 'orders', '>')
// [['orders', [['1731164400000-0', ['order_id', '42']]]]], or null
await streams.xack('orders', 'billing', '1731164400000-0')ioredis(db) gives an IoredisStreams: the methods of ioredis 5. They have lowercase names, the arguments of the Redis command in the same order (option words in any case), an optional callback as the last argument, and the raw arrays of Redis as the answer.
| Method | Arguments | Answer |
|---|---|---|
xadd | key [NOMKSTREAM] [MAXLEN|MINID [=|~] n [LIMIT n]] id field value ... | the ID; null with NOMKSTREAM and no stream |
xlen | key | a number |
xrange / xrevrange | key start end [COUNT n] ((id leaves the ID out) | [[id, [field, value, ...]], ...] |
xread | [COUNT n] [BLOCK ms] STREAMS key ... id ... | [[key, [[id, [field, value]]]]] or null |
xdel | key id ... | a number |
xtrim | key MAXLEN|MINID [=|~] n [LIMIT n] | a number |
xgroup | CREATE key group id [MKSTREAM] [ENTRIESREAD n], SETID key group id [ENTRIESREAD n], DESTROY key group, CREATECONSUMER key group consumer, DELCONSUMER key group consumer | 'OK', 'OK', 1/0, 1/0, the pending count |
xreadgroup | GROUP group consumer [COUNT n] [BLOCK ms] [NOACK] STREAMS key ... id ... | as xread; with an ID that is not >, [[key, []]] when there is no entry, and [id, null] for a deleted entry |
xack | key group id ... | a number |
xpending | key group | [count, lowestId, highestId, [[consumer, '2']]], or [0, null, null, null] |
xpending | key group [IDLE ms] start end count [consumer] | [[id, consumer, idleMs, deliveries], ...] |
xclaim | key group consumer minIdle id ... [IDLE ms] [RETRYCOUNT n] [FORCE] [JUSTID] [LASTID id] | entries as xrange; IDs with JUSTID |
xautoclaim | key group consumer minIdle start [COUNT n] [JUSTID] | [nextStart, entries or IDs, deletedIds] |
xinfo | STREAM key, GROUPS key, CONSUMERS key group | flat [name, value, ...] arrays, with the field names of Redis |
- The return types are those of ioredis:
unknown[]forxreadgroup,xpending,xclaimandxautoclaim;unknownforxgroupandxinfo. The subpath exportsEntryReply,PendingEntryReply,StreamsReplyandGroupStreamsReplyfor a cast. - The option words go in any order where Redis takes them:
xadd key MAXLEN 10 NOMKSTREAM * f v,xautoclaim ... JUSTID COUNT 5,xgroup CREATE key g $ ENTRIESREAD 0 MKSTREAM. - There are no
Buffervariants (xreadBuffer, ...).
node-redis
import { connect } from '@thadeu/ndb'
import { nodeRedis } from '@thadeu/ndb/node-redis'
import { createClient } from 'redis'
const streams = process.env.STREAMS_BACKEND === 'ndb' ? nodeRedis(connect()) : await createClient().connect()
await streams.xAdd('orders', '*', { order_id: '42' }, { TRIM: { strategy: 'MAXLEN', strategyModifier: '~', threshold: 1000 } })
await streams.xGroupCreate('orders', 'billing', '$', { MKSTREAM: true })
const got = await streams.xReadGroup('billing', 'w1', { key: 'orders', id: '>' }, { COUNT: 10, BLOCK: 5000 })
// [{ name: 'orders', messages: [{ id: '1731164400000-0', message: { order_id: '42' } }] }], or nullnodeRedis(db) gives a NodeRedisStreams: the methods of node-redis 5 (the redis package). They have camel case names, options as an object, and the answers that node-redis makes from the arrays.
| Method | Answer |
|---|---|
xAdd(key, id, message, { TRIM }?) | the ID |
xAddNoMkStream(key, id, message, { TRIM }?) | the ID, or null when there is no stream |
xLen(key) | a number |
xRange(key, start, end, { COUNT }?) / xRevRange(key, end, start, { COUNT }?) | [{ id, message }] |
xRead(streams, { COUNT, BLOCK }?) | [{ name, messages: [{ id, message }] }] or null |
xDel(key, id | ids) | a number |
xTrim(key, 'MAXLEN' | 'MINID', threshold, { strategyModifier, LIMIT }?) | a number |
xGroupCreate(key, group, id, { MKSTREAM, ENTRIESREAD }?) / xGroupSetId(key, group, id, { ENTRIESREAD }?) | 'OK' |
xGroupDestroy(key, group) / xGroupCreateConsumer(key, group, consumer) | 1 or 0 |
xGroupDelConsumer(key, group, consumer) | the pending count |
xReadGroup(group, consumer, streams, { COUNT, BLOCK, NOACK }?) | as xRead |
xAck(key, group, id | ids) | a number |
xPending(key, group) | { pending, firstId, lastId, consumers: [{ name, deliveriesCounter }] | null } |
xPendingRange(key, group, start, end, count, { IDLE, consumer }?) | [{ id, consumer, millisecondsSinceLastDelivery, deliveriesCounter }] |
xClaim(key, group, consumer, minIdle, id | ids, { IDLE, RETRYCOUNT, FORCE, LASTID }?) | [{ id, message }] |
xClaimJustId(...) | IDs |
xAutoClaim(key, group, consumer, minIdle, start, { COUNT }?) | { nextId, messages, deletedMessages } |
xAutoClaimJustId(...) | the same, with IDs in messages |
xInfoStream(key) | { length, 'last-generated-id', 'entries-added', 'first-entry', ... } |
xInfoGroups(key) | [{ name, consumers, pending, 'last-delivered-id', 'entries-read', lag }] |
xInfoConsumers(key, group) | [{ name, pending, idle, inactive }] |
- The answers are those of the default client (RESP 2, no type mapping).
- A
TRIMneeds itsstrategy, as with a real Redis.
ReplyError
Errors are ReplyError, exported by the two subpaths. It is a subclass of NdbError.
import { ReplyError } from '@thadeu/ndb/ioredis'
try {
await streams.xgroup('CREATE', 'orders', 'billing', '$', 'MKSTREAM')
} catch (err) {
const exists = err instanceof Error && err.message.startsWith('BUSYGROUP')
if (!exists) throw err
if (err instanceof ReplyError) console.warn(err.code) // "group_exists", on the ndb backend only
}messageis the text of Redis, so a check such aserr.message.startsWith('BUSYGROUP')works on the two backends.codeandstatusare those of ndb, andcauseis the ndb error.err instanceof ReplyErroris true only on the ndb backend. The Redis libraries have their own error classes.
| Case | message starts with | code |
|---|---|---|
| The group exists | BUSYGROUP | group_exists |
| No such group, or no stream in a group command | NOGROUP | unknown_group, unknown_collection |
XGROUP without MKSTREAM, and no stream | ERR The XGROUP subcommand requires the key to exist | unknown_collection |
XINFO and no stream | ERR no such key | unknown_collection |
| An ID that is not greater than the last one | ERR The ID specified in XADD is equal or smaller | stream_id_too_small |
| A bad ID | ERR Invalid stream ID | invalid_stream_id |
| The name is a collection that is not a stream | WRONGTYPE | not_a_stream |
| A bad argument list | ERR | invalid_request |
| Something ndb does not have | ERR ndb: ... is not supported | not_supported |
- An error that Redis does not have (
unauthorized,not_ready, a network error) stays as it is. - As in Redis, these give a value and not an error when the stream does not exist:
xlen(0),xrange([]),xread(null),xdel,xtrimandxack(0).
Differences from Redis
- Values are text. Redis keeps strings, so the adapters give every value as a string, and they store the values of
xaddas strings. An entry that other ndb code appended with JSON types comes back as text:42is'42',trueis'true', andnull, an array or an object are JSON text. - Field order. The daemon gives the fields of an entry in name order, not in the order of the
xadd. A field name that starts with_is reserved, and a field name given two times is kept one time. - Stream names. A stream is an ndb collection, so its name follows the rules of a collection name.
orders:euis not a valid name. - One stream for each request.
xreadandxreadgroupwith many streams send one request for each stream. WithBLOCKand more than one stream they throw (not_supported). Before anxreadgroupof>on many streams, the adapters check the group of each stream, so aNOGROUPon one stream gives nothing from the others, as in Redis. - No
COUNT, orCOUNT 0or less. Redis gives every entry. The adapters give at most 10 000 in one read (xrangehas no limit). ACOUNTover 10 000 is read as 10 000. BLOCK 0waits with no limit, as in Redis: the adapter repeats a 30 s read. A blockedxreadon a stream that does not exist looks for it each 500 ms.- Not supported (the call throws):
XREADwith the ID+,XREADGROUP ... CLAIM,XCLAIM ... TIME,XINFO STREAM ... FULL,XGROUP HELPandXINFO HELP, the Redis 8.2 optionsDELREFandACKED, and the idempotentXADDoptions.LASTIDandKEEPREFare read and have no effect. - Not there:
XSETID,XACKDEL,XDELEX, theBuffervariants of ioredis, pipelines andMULTI. The object has only the methods of the tables. xinfo:radix-tree-keysandradix-tree-nodesare0.- A range of
xpendingdoes not take(id. - Errors in the arguments are those of Redis: a negative
BLOCK(ERR timeout is negative),LIMITwithout~, an argument after the end ofxtrim, and a number that is not decimal digits (0x10,5000.0). (idwithout a seq inxrangeis read as in Redis:(1000as a start leaves out only1000-0, and(2000as an end leaves out only2000-<max seq>.
Admin and blobs
await db.health() // GET /_health; throws while recovering
const defs = await db.collections() // GET /_collections
const stats = await db.stats() // GET /_stats
const size = await db.size() // GET /_size
await calls.stats() // GET /calls/_stats
await calls.compact() // POST /calls/_compact
await calls.ensureIndex('dnis') // PUT /calls/_indexes/dnis => ['dnis', 'status']
await calls.dropIndex('dnis')
const blob = await db.putBlob(bytes, 'audio/wav') // { $blob: 'sha256:...', size, type }
await calls.update('c1', { $set: { meta: { recording: blob } } })
const data = await db.getBlob(blob) // Uint8ArrayErrors
Daemon errors are NdbError or a subclass. They have code (the daemon code) and status (the HTTP status; 0 for an error line inside a stream).
| Class | Codes |
|---|---|
UnauthorizedError | unauthorized |
NotFoundError | not_found, unknown_collection |
DuplicateIdError | duplicate_id |
UniqueViolationError | unique_violation |
InvalidQueryError | invalid_query, invalid_pipeline |
CursorError | cursor_invalid, cursor_mismatch, cursor_expired |
WatchLaggedError | watch_lagged |
HistoryTruncatedError | history_truncated |
TtlError | ttl_not_enabled, ttl_derived |
InvalidUpdateError | unknown_operator, no_update, conflicting_update, invalid_field, unindexed_many, empty_filter, invalid_update |
CollectionExistsError | collection_exists |
PitrError | out_of_window, pitr_off |
LookupError | lookup_invalid, lookup_unindexed, lookup_too_large |
NotAStreamError | not_a_stream |
StreamIdError | invalid_stream_id, stream_id_too_small |
UnknownGroupError | unknown_group |
GroupExistsError | group_exists |
NdbError | every other code: not_ready, catalog_conflict, catalog_invalid, direct_collection, invalid_request, group_limit_exceeded, request_too_large, line_too_large, closed, invalid_position, http_status, ... |
import { NdbError } from '@thadeu/ndb'
try {
await calls.get('c1')
} catch (err) {
if (err instanceof NdbError && err.code === 'not_ready') {
// The daemon is still recovering, and readyTimeout ran out.
}
}ErrorCode is the union of every code of the error table, so err.code === 'invalid_path' is type-checked. A test of the repository fails when the daemon gets a code that is not in it.
Testing
ndb has no transactions, so a test cannot roll back. clean(db) truncates the collections that db used since the last clean (a few ms at any size; the definition and the indexes stay), like Ndb::Testing in Rails. It has no test framework in it, so vitest, jest, node --test and bun test all use it the same way:
import { clean, cleanAll, connect } from '@thadeu/ndb'
const db = connect(process.env.NDB_URL, { prefix: `test${process.env.VITEST_POOL_ID ?? 1}_` })
beforeAll(() => cleanAll(db)) // every collection with the prefix
afterEach(() => clean(db)) // the ones this test used- Without
prefix, the prefix isNDB_PREFIX. clean(db, { only, except }): truncate these, or leave some out.cleanAll(db)truncates every engine collection with the prefix ofdb. Direct collections and partitions are left out.prefix(aconnectoption) goes before the name of every collection: each parallel worker gets collections of its own.- A collection that does not exist is not an error.
- A clean runs only in a test run:
NODE_ENV=test(jest,bun test),VITEST,JEST_WORKER_IDorNODE_TEST_CONTEXT(node --test). Anywhere else it throwsUnsafeCleanError.NDB_ALLOW_CLEAN=1forces it. - The SDK does not start a daemon: start it in CI as a service, with
make up, or with Docker, and connect withNDB_URL.
Tests of this package
The tests of this package (vitest) build the daemon from the repository (go build ./cmd/ndb) in a global setup. They start it on a temp unix socket and a free TCP port, wait for GET /_health to return 200, and stop it at the end. To do the same by hand:
ndb -config /nonexistent.yml -data-dir "$(mktemp -d)" -unix /tmp/ndb-test.sock -tcp 127.0.0.1:7070Add -token-file <path> to ask for a bearer token on TCP. Use a short socket path: macOS allows 104 bytes. Use a new collection name per test.