Go
The Go module github.com/thadeu/ndb/sdk/go. Standard library only.
The source and the same guide are in sdk/go.
Install
go get github.com/thadeu/ndb/sdk/go@latestimport ndb "github.com/thadeu/ndb/sdk/go"The module needs a running ndb daemon. See the repository for how to run one. On a dev machine, ndb serve with no flags (not as root) listens on ~/.ndb/ndb.sock and on TCP 127.0.0.1:7070: connect with "unix://" + filepath.Join(home, ".ndb/ndb.sock") (the SDK does not expand ~) or tcp://127.0.0.1:7070. On a server (root, or /var/lib/ndb exists) the socket is /run/ndb/ndb.sock and TCP is off. Releases of this module are tagged sdk/go/vX.Y.Z.
Connect
// Unix socket.
db, err := ndb.Connect("unix:///run/ndb/ndb.sock")
// TCP with a bearer token.
db, err := ndb.Connect("tcp://10.0.0.5:7070", ndb.WithToken("secret"))
// Address and token from the environment: NDB_URL and NDB_TOKEN.
db, err := ndb.Connect("")
// Retry for up to 5 s while the daemon starts or recovers.
db, err := ndb.Connect("unix:///run/ndb/ndb.sock", ndb.WithReadyTimeout(5*time.Second))
defer db.Close()Connectdoes not dial. The first call does.- The address must be
unix:///path,tcp://host:port, orhttps://host[:port](ndb behind a TLS proxy;http://works too). Anything else returns an error. - The token goes only in the
Authorizationheader. A unix socket does not need it. - Every request sends
User-Agent: ndb-go/<version> (go1.26.10; linux/amd64). The version comes from the build info of the app (devwhen the build has none).ndb.WithUserAgent("myapp/1.2")adds the token of your app at the end, so the daemon can tell the apps apart. WithReadyTimeoutsets how long a call retries when the daemon is not ready (HTTP 503not_ready, or the socket is missing or refuses). The default isndb.DefaultReadyTimeout(10 s). Zero turns the retry off. Writes are retried only when the daemon certainly did not process them. Reads are also retried after a dropped connection.- The pool opens at most
ndb.DefaultMaxConns(64) connections. Over that, a call waits for a free one (until itsctxends) instead of opening another.ndb.WithMaxConns(n)changes the limit (0 is no limit);ndb.WithIdleTimeout(d)sets how long an unused connection stays open (90 s). AWatchhas its own connections, out of the limit. - Without a limit, a peak of goroutines opens hundreds of connections at once. Measured on a Mac, 1000 goroutines of 20 counts each over the unix socket: no limit gives p99 35 ms and max 137 ms; the limit of 64 gives p99 9.5 ms and max 12 ms. Over TCP with no limit, some calls failed. In a Docker network (the fsw-esl test), the limit took the peaks of 1 s and 4 s away. Measure it with
NDB_POOL_BENCH=1 go test -run TestPoolBench -v .insdk/go. - Use the unix socket when the app and ndb share a host or a pod: it was 2x faster than TCP in the same test.
*DBis safe for concurrent use.Closecloses idle connections. The handle stays usable.
Define a collection
A collection maps to a Go struct. Tags use the form ndb:"name,option,option".
type Call struct {
ID string `ndb:"_id"`
Status string `ndb:"status,index"`
Caller string `ndb:"caller,unique"`
Duration int `ndb:"duration"`
At time.Time `ndb:"at,time"`
ExpiresAt time.Time `ndb:"expires_at,expires"`
Recording *ndb.Blob `ndb:"recording"`
Version uint64 `ndb:"_v,version"`
Internal string `ndb:"-"`
}
calls, err := ndb.Collection[Call](db, "calls")| Option | Meaning |
|---|---|
index | Index this field. |
unique | Unique value. A second document with it fails with ErrUniqueViolation. |
time | The collection time field. Used by zone maps, $dateTrunc and TTL. |
expires | The field holds the time the document expires. |
version | The field receives _v, the document version, on every read. It is never sent on a write. |
- The primary key is the field named
_id(ndb:"_id"orjson:"_id"). If it is empty on insert, it is not sent and the daemon makes a UUIDv7. For a natural key (email, SKU) keep_idand adduniqueto that field.ndb.NewID()makes the same kind of id on the client. ndb.Encrypted("pem")andndb.Blind("cpf", "numeric")are options ofCollection: the daemon encrypts the field before it stores it, and a blind field is found by value (ndb.Eq("cpf", "390.533.447-05")) and may be inndb.Unique. Reads senddecrypt=1;vault.Encrypted()is a handle whose reads keep the values sealed, and the optionndb.KeepEncrypted()does it for the whole collection. See Encrypted fields.ndb.Index(...)andndb.Unique(...)are options ofCollection, for 1 to 4 fields in order: a compound index or unique key (ndb.Index("caller", "at"),ndb.Unique("account_id", "email")), or one path that is not a field of the struct (ndb.Index("items.sku")). See Indexes.- A field without an
ndbtag uses itsjsonname, or its Go name.ndb:"-"orjson:"-"skips a field. - A
time.Time(or*time.Time) field goes out in UTC with milliseconds, always the same width (2026-11-12T15:00:05.000Z,ndb.TimeLayout), as the other SDKs write it. Atime.Timein a filter (ndb.Gte("next_at", now)) goes out the same way. Where the SDK does not see the field (a nested struct, a map, the operators of an update), usendb.Time: it writes the same layout and reads any RFC 3339 time. See Times. - Only one
timeandexpiresfield are allowed. An unknown option is an error (the oldidoption too). Collectionsends nothing. The first call on the collection sendsensure_collection(PUT /calls) once per name on one*DB. A name with*(a pattern) never declares. A definition that disagrees with the stored one (time, ttl) returnsErrCatalogConflict.
Insert, get, update, delete
// Insert one document (POST /calls/_doc), or many in one transaction (_bulk).
res, err := calls.Insert(ctx, Call{ID: "c1", Status: "answered", Caller: "a", At: time.Now().UTC()})
// res.ID and res.V (the new version).
results, err := calls.InsertMany(ctx, []Call{
{ID: "c2", Status: "failed", Caller: "b", At: time.Now().UTC()},
{ID: "c3", Status: "answered", Caller: "c", At: time.Now().UTC()},
})
// Get by _id (GET /calls/_doc/c1). A missing id returns ErrNotFound.
call, err := calls.Get(ctx, "c1")
// Update one document by _id, in one atomic step. Guard adds conditions on the
// same document: a _v guard is an optimistic lock.
upd, err := calls.Update(ctx, "c1",
ndb.Ops{"$set": {"status": "done"}, "$unset": {"duration": true}},
ndb.Guard(ndb.Eq("_v", call.Version)),
)
// upd.Matched, upd.Modified, upd.Doc (the new document).
if err == nil && upd.Matched == 0 {
// Someone else wrote first, or the document is gone. Read again and retry.
}
// WithRetry does that loop: read, call fn, write the changed fields with the
// _v guard, and read again on a conflict (5 tries, Retries(n)). An error
// from fn writes nothing. After the last try: ErrStaleObject. fn can run
// more than once.
call, err = calls.WithRetry(ctx, "c1", func(c *Call) error {
if c.Status == "done" {
return nil
}
c.Status = "done"
return nil
})
// Lock is the pessimistic lock: one holder at a time, so fn runs once. Use
// it when fn has an effect outside ndb. Another Lock waits (LockWait, then
// ErrLocked); a plain write waits for the release. The changed fields are
// written with the lock, then it is released. LockTTL (default 30 s) must
// cover fn: after it the write is ErrLockExpired.
call, err = calls.Lock(ctx, "c1", func(c *Call) error {
if c.Status == "billed" {
return nil
}
if err := billing.Charge(c.CallID); err != nil {
return err
}
c.Status = "billed"
return nil
})
// Tx is a transaction of the collection: the writes of fn go in one step,
// all or none. With ids, they are locked first (all or none) and given to
// fn. An error from fn writes nothing; ErrRollback writes nothing and Tx
// returns nil. A stale or missing document fails it with ErrTxConflict.
err = calls.Tx(ctx, []string{"c1", "c2"}, func(tx *ndb.Tx[Call], docs map[string]*Call) error {
docs["c1"].Status = "merged"
tx.Update(docs["c1"])
tx.Delete("c2")
tx.Insert(Call{ID: "c3", Status: "new", Caller: "+5511999990003", At: time.Now()})
return nil
}, ndb.LockTTL(5*time.Second))
// Atomic update by query: exactly one of two racing calls wins.
upd, err = calls.UpdateBy(ctx,
ndb.And(ndb.Eq("_id", "c1"), ndb.Gte("duration", 1)),
ndb.Ops{"$inc": {"duration": -1}},
)
// Every match (the query needs the id or an indexed field), or create it.
_, err = calls.UpdateBy(ctx, ndb.Eq("status", "failed"), ndb.Ops{"$set": {"status": "retry"}}, ndb.Many())
_, err = calls.Update(ctx, "c9", ndb.Ops{"$inc": {"duration": 1}}, ndb.Upsert())
// Delete by _id, or by query. Both return how many they removed.
deleted, err := calls.Delete(ctx, "c2")
deleted, err = calls.DeleteBy(ctx, ndb.Eq("status", "failed"), ndb.Many())- Operators:
$set,$setOnInsert(like$set, but only when an upsert creates the document),$unset,$inc,$push(with{"$each": [...]}),$pull,$min,$max. Field names can be dot paths, as they are in the document (the tag names). A key ofOpswithout$is an error before anything is sent. - Nothing matched is not an error:
Matchedis 0.Modifiedis 0 when the operators leave the document as it is. Update(by_id) takesGuard,Upsert()(creates the document with that_id) andExpireIn.UpdateBytakesMany(),Upsert()andExpireIn.Many()needs_idor an indexed field in the query (ErrUnindexedMany).Upsert()onUpdateBycreates the document from the equality fields of the query.- A
Filteris one condition; join several withndb.And.ndb.Eq("_v", n)matches the version. InsertandInsertManyfail withErrDuplicateIDwhen an_idexists.Upsert(ctx, doc)andUpsertMany(ctx, docs)create by_id, or merge into the stored document when the_idexists, in one transaction. See Upsert.
Truncate and drop
err = calls.Truncate(ctx) // every document goes; the definition and indexes stay
err = calls.Drop(ctx) // the collection, its definition and its watch consumers go- Both take the same time for 10 or 10 million documents. Watch consumers get one event with
ndb.OpTruncate(no ID), then the new changes. - A direct-mode collection is
ErrDirectCollection. A collection declared inndb.ymlcomes back, empty, when the daemon restarts after a drop.
Rename, restore and PITR
err = calls.Rename(ctx, "calls_old", false) // false: an existing calls_old is ErrCollectionExists
err = calls.Restore(ctx, time.Now().Add(-time.Hour), "calls_1h_ago")
info, err := calls.PITR(ctx) // GET /calls/_pitrRenamemoves the documents, indexes and watch positions. Withreplacetrue, an existing target goes aside first (with PITR on, it stays restorable for the window). The handle keeps the old name: callCollectionagain with the new name.Restoremakes the collectiontowith this collection as it was at that time, from the PITR window of the daemon (1 day by default). The source does not change. A dropped collection can be restored while it is in the window. Errors:ErrOutOfWindow,ErrPITROff,ErrCollectionExists.PITRInfohasCollection,Window,Oldest,Newest,Snapshots,BytesandDropped.
Expiry (TTL like Redis)
Tag one field expires (a time.Time). The daemon then expires documents one by one.
type Session struct {
ID string `ndb:"_id"`
ExpiresAt time.Time `ndb:"expires_at,expires"`
}
_, err = sessions.Insert(ctx, Session{ID: "s1"}, ndb.ExpireIn(time.Hour))
_, err = sessions.Update(ctx, "s1", ndb.Ops{"$set": {"x": 1}}, ndb.ExpireIn(30*time.Minute))
_, err = sessions.Expire(ctx, "s1", 30*time.Minute) // EXPIRE
_, err = sessions.ExpireAt(ctx, "s1", time.Now().Add(time.Hour))
left, expires, err := sessions.TTL(ctx, "s1") // expires is false when it never expires
_, err = sessions.Persist(ctx, "s1") // PERSISTExpire,ExpireAtandPersistreturn the new version.TTLrounds up to whole seconds. A missing or expired document isErrNotFound.ErrTTLNotEnabled: noexpiresfield.ErrTTLDerived: the expiry comes fromttl+time.ExpireInwins over the value of theexpiresfield in the document.- Disk space of expired documents returns after compaction, and fully only with
wal_retention; see Freeing disk.
Query
// Every document: Query with no filter.
all, err := calls.Query().All(ctx)
rows, err := calls.
Query(
ndb.Eq("status", "answered"),
ndb.Gte("duration", 30),
ndb.In("caller", "a", "b"),
ndb.Not(ndb.Eq("caller", "zzz")),
ndb.Or(ndb.Lt("duration", 5), ndb.Exists("note", false)),
).
Sort(ndb.Desc("at"), ndb.Asc("_id")).
Fields("_id", "status").
Limit(50).
Skip(100).
All(ctx)- Filters:
Eq,Ne,Gt,Gte,Lt,Lte,In,Exists,Regex,ElemMatch,And,Or,Not,RawQuery. The filters in oneQueryare ANDed. Dot paths such as"agent.id"work. ElemMatch(field, filters...)matches when one element of an array matches every filter:ElemMatch("items", Eq("sku", "a"), Gt("qty", 2)). For an array of scalars, use an empty field name:ElemMatch("scores", Gte("", 10), Lt("", 20)).- On the time field, a string that starts with
nowis date math:Gte("ts", "now-7d/d"). Regex(field, pattern, ndb.IgnoreCase, ...)matches strings with Go RE2 (no backreferences, no look-around). An anchored prefix such as^abccan use an index. For other operators, see the query language.- The order of
Sortkeys is the order of the sort. The id is always the last key. Fieldsreturns only these fields. The other struct fields keep their zero value.Page(n, size)setsSkipandLimitfor pagen(from 1).First(ctx)returns the first row, orErrNotFound.- The SDK has no explain call. To see the plan counters, send
"explain": truein a rawfindrequest and read theX-Ndb-Explainheader. See the query language doc.
Queries built elsewhere
For a search form or a saved search, take the JSON as is. ndb.RawQueryJSON (or ndb.RawQuery with a map) is a Filter that mixes with the others; Search takes a whole search body and returns one page.
form, err := ndb.RawQueryJSON([]byte(`{"status":{"$in":["answered","failed"]},"ts":{"$gte":"now-7d"}}`))
rows, err := calls.Query(form, ndb.Eq("dnis", "1000")).Sort(ndb.Desc("ts")).All(ctx)
page, err := calls.Search(ctx, []byte(`{"query":{"status":"answered"},"sort":{"ts":-1},"limit":50,"count":true}`))
page.Rows // []Call
page.Next // cursor for {"after": ...}; "" on the last page
page.Total // with "count": trueRows decode like any other query. The daemon validates the JSON (invalid_query), but it takes any field: for a public form, keep an allowlist of fields and operators, cap limit, and ask for an indexed field or a time window, as you would before sending a form to Elasticsearch _search.
Collection patterns
A collection name with * at the end reads every collection whose name starts with the prefix (see Collection patterns). It declares nothing; writes fail with pattern_read_only. A field tagged ndb:"_coll" gets the name of the collection of each row, and is never sent on writes.
type Log struct {
ID string `ndb:"_id"`
At time.Time `ndb:"at,time"`
Lvl string `ndb:"lvl,index"`
Coll string `ndb:"_coll"`
}
logs, err := ndb.Collection[Log](db, "logs-2026-*")
rows, err := logs.Query(ndb.Eq("lvl", "error")).Sort(ndb.Desc("at")).Limit(50).All(ctx)
n, err := logs.Count(ctx, ndb.Eq("lvl", "error"))Count
n, err := calls.Count(ctx, ndb.Eq("status", "answered")) // POST /calls/_countLimit, Skip and Sort do not change the count.
Cursor pagination
Offset pages get slower as Skip grows. Use a cursor for deep pages.
q := calls.Query(ndb.Eq("status", "answered")).Sort(ndb.Asc("_id")).Limit(100)
page, err := q.All(ctx)
token := q.NextCursor() // "" on the last page
next, err := calls.Query(ndb.Eq("status", "answered")).Sort(ndb.Asc("_id")).Limit(100).After(token).All(ctx)A token works only with the same Query and Sort. Another query shape returns ErrCursorMismatch.
To walk every row, use Each. It follows the cursor, with Limit as the batch size:
for call, err := range calls.Query(ndb.Eq("status", "answered")).Sort(ndb.Asc("_id")).Limit(500).Each(ctx) {
if err != nil {
return err
}
fmt.Println(call.ID)
}Lookup
lookup is a key of the HTTP search (POST /{coll}/_search): in the same call, the daemon reads the documents of other collections that each row points to, and puts them in the row. The HTTP API is the primitive and has every rule (Lookup); this SDK only builds that JSON.
| Relation | The foreign key lives | Match | The row gets |
|---|---|---|---|
| belongs to | in this collection | row.fk == other.pk | an object, or null |
| has many | in the other collection | other.fk == row.pk | an array (100 unless a limit, at most 1000) |
| has one | in the other collection | other.fk == row.pk | an object, or null |
pk is _id unless you say otherwise. The field that is searched on the other side (its pk for a belongs to, its fk for a has many) must be _id or indexed, else the daemon answers lookup_unindexed.
Basic use
Give the struct a field for each relation, with the lookup tag option: a read fills it, a write leaves it out. Its type decodes with encoding/json (json tags).
type Customer struct {
ID string `json:"_id"`
Name string `json:"name"`
}
type Item struct {
ID string `json:"_id"`
OrderID string `json:"order_id"`
SKU string `json:"sku"`
}
type Order struct {
ID string `ndb:"_id"`
CustomerID string `ndb:"customer_id"`
Customer *Customer `ndb:"customer,lookup"`
Items []Item `ndb:"items,lookup"`
}
orders, _ := ndb.Collection[Order](db, "orders")
// belongs to: orders.customer_id -> customers._id
rows, err := orders.Query(ndb.Eq("status", "paid")).
Lookup("customer", ndb.BelongsTo("customers", "customer_id")).
All(ctx)
rows[0].Customer.Name // "Ana" (nil when there is no customer)
// has many: order_items.order_id -> orders._id
rows, err = orders.Query().
Lookup("items", ndb.HasMany("order_items", "order_id")).
All(ctx)
// has one: invoices.order_id -> orders._id
rows, err = orders.Query().
Lookup("invoice", ndb.HasOne("invoices", "order_id")).
All(ctx)Options
Each method returns a new Relation:
ndb.HasMany("order_items", "order_id").
Query(ndb.Gt("qty", 0)). // filter the related documents
Sort(ndb.Asc("pos")). // their order
Limit(50). // per row, 1 to 1000
Fields("sku", "qty"). // keep these fields
Lookup("product", // nested, two levels at most
ndb.BelongsTo("products", "sku").PK("code")) // products.code instead of _idQuery, Sort and Limit are for HasMany and HasOne. The JSON it sends:
{ "lookup": { "items": { "has_many": "order_items", "fk": "order_id", "query": { "qty": { "$gt": 0 } },
"sort": { "pos": 1 }, "limit": 50, "fields": ["sku", "qty"],
"lookup": { "product": { "belongs_to": "products", "fk": "sku", "pk": "code" } } } } }Index
A HasMany("order_items", "order_id") needs an index on order_items.order_id: tag it in the struct of that collection (ndb:"order_id,index") or call items.EnsureIndex(ctx, "order_id").
Errors
ErrLookupInvalid (a bad relation), ErrLookupUnindexed (the other side has no index; the message gives the route), ErrLookupTooLarge (more than 10 000 documents in one answer), ErrUnknownCollection.
Aggregate
The pipeline needs exactly one $group. Before it: $match and one $sort. After it: $match, $sort, $limit, $skip, $project. Accumulators: $count, $sum, $avg, $min, $max, $first, $last.
res, err := calls.Aggregate(ctx, ndb.Pipeline{
{"$match": map[string]any{"duration": map[string]any{"$gte": 1}}},
{"$group": map[string]any{
"_id": map[string]any{
"status": "$status",
"hour": map[string]any{"$dateTrunc": map[string]any{"date": "$at", "unit": "hour"}},
},
"n": map[string]any{"$count": map[string]any{}},
"avg": map[string]any{"$avg": "$duration"},
}},
ndb.SortStage(ndb.Desc("n")),
}, ndb.AggLimit(100), ndb.AggCount())
// res.Rows is []map[string]any. Numbers are float64.
// res.Next is the cursor for the next page ("" on the last one).
// res.Total is set with AggCount.Decode rows into your own type with AggregateAs:
type Row struct {
ID map[string]string `json:"_id"`
N int `json:"n"`
}
typed, err := ndb.AggregateAs[Row](ctx, calls, pipeline)Options: AggLimit, AggSkip, AggAfter(token), AggCount(). $dateTrunc units are minute, hour, day, week and month. Use ndb.SortStage (not a Go map) when the sort has more than one key, so the key order stays.
Watch
A watch streams the changes of a collection to a named consumer. First it sends the events after the last Ack. Then it sends new events as they happen.
for ev, err := range calls.Watch(ctx, "relay", ndb.WatchQuery(ndb.Eq("status", "answered"))) {
if errors.Is(err, ndb.ErrWatchLagged) {
break // call Watch again. It continues after the last Ack.
}
if errors.Is(err, ndb.ErrHistoryTruncated) {
return err // the saved position is too old. Start over.
}
if err != nil {
return err
}
handle(ev.Op, ev.ID, ev.Doc) // ev.Op is ndb.OpPut, ndb.OpDel or ndb.OpTruncate
if err := calls.Ack(ctx, ev); err != nil {
return err
}
}- Delivery is at least once. After a restart an event can come again, so make the work safe to repeat.
Ackstores the position of the event. An olderAckis ignored.- Options:
ndb.FromNow()(a new consumer starts at the end),ndb.Limit(n)(stop after n events),ndb.WatchQuery(filters...). - A delete event has no document (
ev.Docis the zero value). WithWatchQuery, a delete is sent only if the same stream saw an earlier matching put of that id. ErrWatchLaggedmeans the consumer was too slow.ErrHistoryTruncatedmeans the WAL files for its position were deleted (wal_retention).- The stream ends when
ctxis cancelled.
Streams
A stream is an append-only log with consumer groups, with the behavior of Redis Streams. Read Streams for the model and Stream routes for the wire contract. This section shows only the Go calls.
StreamOf
type Order struct {
Order int `ndb:"order"`
Note string `ndb:"note"`
}
orders, err := ndb.StreamOf[Order](db, "orders")StreamOfsends nothing. The first append, or the first group create, makes the stream.Tis a struct (fields map with thendbtags) ormap[string]any. The entry ID and time are inEntry, not inT.WithPrefixonConnectapplies to the stream name too.orders.Name()gives the full name.
Entry
e.ID // "<ms>-<seq>"
e.Time // the ms of the ID, as time.Time
e.Doc // the T
e.Deleted // pending, but no longer in the stream (a trim or a delete)
e.Err // not nil when the doc does not decode into T
e.Raw() // the JSON from the daemon, with _id and _ts
t := ndb.IDTime("1731160000000-0") // the zero time for a bad ID- When
Deletedis true,Docis the zero value. - When the doc does not decode into
T(for example a text in anintfield),Errhas the cause,Docis the zero value andRaw()has the doc.Read,Claim,AutoClaim,Range,RevRangeandTailgive this entry with the other entries; they do not fail. A group read already made it pending, so ack it or send it to a dead letter.Workdoes this for you (see Work).
Create
err = orders.Create(ctx) // PUT /orders with "stream": trueCreatemakes the stream, or checks that the name is one. A collection that is not a stream returnsErrNotAStream.- You do not need it:
AppendandGroup.Createmake the stream too.
Append
id, err := orders.Append(ctx, Order{Order: 42}) // XADD
id, err = orders.Append(ctx, Order{Order: 43},
ndb.WithEntryID("1731160000000-*"),
ndb.WithTrim(ndb.MaxLen(10_000, true)),
ndb.NoCreate(),
)Appendreturns the ID of the entry. With the default durability, the entry is on disk beforeAppendreturns.WithEntryID(id):"<ms>-<seq>"or"<ms>-*". The default is"*": the daemon makes the ID from its clock. An ID that is not higher than the last one returnsErrStreamIDTooSmall. A bad ID returnsErrInvalidStreamID.WithTrim(t)trims the stream after the append, as Trim does.NoCreate()makes the append fail withErrUnknownCollectionwhen the stream does not exist (NOMKSTREAM).
AppendMany
ids, err := orders.AppendMany(ctx, []Order{{Order: 1}, {Order: 2}})- The entries go in order, with one fsync for all. The IDs come back in the same order.
- It takes the same options.
WithEntryIDapplies to the first entry.
Trim
n, err := orders.Trim(ctx, ndb.MaxLen(1000, false)) // XTRIM MAXLEN
n, err = orders.Trim(ctx, ndb.MinID("1731160000000-0", true))
n, err = orders.Trim(ctx, ndb.Trim{MinID: id, Approx: true, Limit: 5000})Trimreturns how many entries it deleted.ndb.MaxLen(n, approx)keeps the newestnentries.ndb.MinID(id, approx)deletes the entries with an ID lower thanid.- With
approxtrue, it deletes only whole blocks of 1 024 entries (~in Redis).Limit(withApprox) is the most entries to delete. retentionon the stream also deletes old entries. See Trim and retention.
Info
info, err := orders.Info(ctx) // XINFO STREAM, XLEN
// info.Length, info.FirstID, info.LastID, info.EntriesAdded, info.MaxDeletedID, info.GroupsRange and RevRange
first, err := orders.Range(ctx, "-", "+", 100) // XRANGE, oldest first
last, err := orders.RevRange(ctx, "+", "-", 10) // XREVRANGE, newest firstRange(ctx, start, end, count): entries with an ID fromstarttoend, both included."-"and"+"are the lowest and the highest ID."(id"leaves that ID out, as in Redis:orders.Range(ctx, "("+last, "+", 100)gives the entries afterlast.- An ID with only the ms is
<ms>-0as a start and<ms>-<max seq>as an end, as in Redis. So"(5"as a start leaves out5-0only, and"(6"as an end leaves out the last ID of ms 6 only. - A bad ID (not
<ms>or<ms>-<seq>in digits,-,+or"(id") returnsErrInvalidStreamID. The SDK checks it before it sends the request. RevRangetakesendfirst, thenstart, asXREVRANGEdoes.count0 uses the default limit of_search.
Tail
entries, err := orders.Tail(ctx, ndb.Tail{After: "0", Count: 100}) // XREAD, from the start
last := ndb.StartNew // "$": only the entries appended after the first call starts
for {
entries, err := orders.Tail(ctx, ndb.Tail{After: last, Count: 100, Block: 5 * time.Second})
if err != nil {
return err
}
for _, e := range entries {
handle(e.Doc)
last = e.ID
}
}Tail gives the entries after an ID, in ID order, without a group (XREAD). It changes no group and no pending list, and nothing is acked.
Fields of ndb.Tail:
| Field | Meaning |
|---|---|
After | Give the entries with an ID greater than this one. Default "0", the start of the stream. ndb.StartNew ($) is the last ID when the call starts: only later entries come. |
Count | The most entries to give. Default 10, at most 10 000. |
Block | Wait up to this long for a new entry when there is none. Zero does not wait. |
Fields | Give only these fields (and _id). |
- An empty answer is an empty slice, not an error.
- To read on, send the ID of the last entry as
After. Do not sendndb.StartNewagain: an entry appended between two calls would be lost. - A blocked
Tail, as a blockedRead, does not count againstWithMaxConns. - A stream that does not exist returns
ErrUnknownCollection.
Delete
n, err := orders.Delete(ctx, id) // XDEL: 1, or 0 when the entry is not there- An ID with only the ms is
<ms>-0. A bad ID returnsErrInvalidStreamID.
Groups
groups, err := orders.Groups(ctx) // XINFO GROUPS
// each: Name, Consumers, Pending, LastDeliveredID, EntriesRead, LagEntriesReadandLagare*int64.Lagis nil when the daemon cannot know it.
Group
g := orders.Group("workers") // sends nothing- A group gives each entry to one consumer of the group. The entry stays pending until it is acked.
Create and Ensure
err = g.Create(ctx, ndb.StartNew) // XGROUP CREATE ... MKSTREAM
err = g.Ensure(ctx, ndb.StartAll) // the same, but no error when the group exists
err = g.Create(ctx, ndb.StartNew, ndb.GroupNoCreate()) // XGROUP CREATE without MKSTREAMstartisndb.StartNew($: only entries appended after now),ndb.StartAll(0: every entry), or an entry ID.Createon a group that exists returnsErrGroupExists.Ensurereturns nil.- Both make the stream when it does not exist.
Create(ctx, start, opts ...GroupOption)takes options.ndb.GroupNoCreate()does not make the stream:CreatereturnsErrUnknownCollectionwhen the stream does not exist.
SetID
err = g.SetID(ctx, ndb.StartAll) // XGROUP SETID: read the stream againSetIDmoves the last delivered ID of the group.
Destroy
err = g.Destroy(ctx) // XGROUP DESTROY- The group and its pending entries go. The entries stay in the stream.
CreateConsumer and DeleteConsumer
created, err := g.CreateConsumer(ctx, "worker-1") // false when it exists
pending, err := g.DeleteConsumer(ctx, "worker-1") // XGROUP DELCONSUMER- A
Readmakes the consumer too. DeleteConsumeralso deletes the pending entries of the consumer, and returns how many it had.
Consumers
cs, err := g.Consumers(ctx) // XINFO CONSUMERS
// each: Name, Pending, IdleMs, InactiveMsRead
entries, err := g.Read(ctx, "worker-1", ndb.Read{Count: 50, Block: 5 * time.Second}) // XREADGROUP
for _, e := range entries {
handle(e.Doc)
}Fields of ndb.Read:
| Field | Meaning |
|---|---|
Count | The most entries to give. Default 10, at most 10 000. |
Block | Wait up to this long for a new entry when there is none. Zero does not wait. |
Pending | Give the consumer's own pending entries, not new ones. |
After | With Pending: only IDs higher than this one. |
NoAck | Do not add the entries to the pending list (at most once). |
Fields | Give only these fields (and _id). |
- New entries become pending for this consumer. Call Ack when the work is done.
- An empty answer is an empty slice, not an error.
- A group that does not exist returns
ErrUnknownGroup. - After a restart, read with
Pending: truefirst to get the entries the consumer took and did not ack.Deletedis true on an entry that was trimmed or deleted.
Ack
n, err := g.Ack(ctx, e.ID, e2.ID) // XACKAckreturns how many IDs were pending. An ID that is not pending counts 0. No IDs sends nothing.
Pending
sum, err := g.Pending(ctx) // XPENDING, the summary
// sum.Count, sum.Lowest, sum.Highest (*string, nil when empty), sum.Consumers ([]{Name, Count})
// sum.Waiting (ndb only): the nacked entries whose retry time has not come- A nacked entry counts in
Count, and for no consumer inConsumers.
PendingList
list, err := g.PendingList(ctx, ndb.PendingQuery{
Consumer: "worker-1",
Idle: time.Minute,
Count: 50,
})
// each: ID, Consumer, IdleMs, Deliveries, RetryAt (*time.Time, nil when not nacked)Fields of ndb.PendingQuery:
| Field | Meaning |
|---|---|
Start, End | The ID range. Default "-" and "+". "(id" leaves that ID out, as in Redis 6.2. A bad ID returns ErrInvalidStreamID. |
Count | The most entries to give. Default 100. |
Consumer | Only the entries of this consumer. |
Idle | Only the entries idle for at least this long. |
Due | ndb only, a *bool. true: only the nacked entries whose retry time passed (a claim takes them now). false: only the nacked entries that still wait. nil: every entry. |
- The entries come in ID order.
- A nacked entry has
Consumer""and aRetryAt(see Nack).
Claim
c, err := g.Claim(ctx, "worker-2", time.Minute, id1, id2) // XCLAIM
// c.Entries ([]Entry[T]), c.Deleted ([]string)Claimmoves the given entries to the consumer when they are idle for at leastminIdle. Each one gets one more delivery.Deletedhas the pending IDs whose entry is no longer in the stream. They leave the pending list.
c, err := g.ClaimIDs(ctx, "worker-2", time.Minute, id1, id2) // XCLAIM ... JUSTID
// c.IDs ([]string), c.Deleted ([]string)ClaimIDsisClaimthat gives only the IDs, in andb.ClaimedIDs. The delivery count of the entries does not change.
AutoClaim
cursor := "0-0"
for {
c, err := g.AutoClaim(ctx, "worker-2", time.Minute, cursor, 100) // XAUTOCLAIM
if err != nil {
return err
}
handleAll(c.Entries)
if c.Cursor == "0-0" {
break
}
cursor = c.Cursor
}AutoClaim(ctx, consumer, minIdle, cursor, count)scans the pending list fromcursorand moves up tocountidle entries to the consumer. An empty cursor is"0-0".c.Cursoris where the next call starts. It is"0-0"when the scan is complete.g.AutoClaimIDs(ctx, consumer, minIdle, cursor, count)isAutoClaimthat gives only the IDs (XAUTOCLAIM ... JUSTID), in andb.ClaimedIDs(Cursor,IDs,Deleted). The delivery count of the entries does not change.
Nack
ndb only: Redis has no delayed retry.
n, err := g.Nack(ctx, "worker-1", time.Minute, e.ID) // give it back in 1 minNack(ctx, consumer, delay, ids...)gives the entries back to the group afterdelay, for a retry, and returns how many it nacked.- Only the IDs pending for this consumer count. Other IDs count 0; they are not an error. So a worker that lost an entry to a claim cannot nack it. No IDs sends nothing.
- The entry stays pending with no owner (
Consumer"") and a retry time (RetryAt) of now +delay. Its delivery count does not change; the claim that takes it again adds 1. - While it waits, no read and no claim gives it: not the own pending read (it is no one's), not
Claim,AutoClaimorClaimDue. AfterRetryAt, any consumer of the group takes it withClaimDue(orClaim, at once:minIdledoes not count). delay0 makes the entry due at once. The most is 30 days. A negative or larger delay returnsErrInvalidRequest.Ackof a nacked entry removes it: it cancels the retry.WorkwithRetrycallsNackfor you.
ClaimDue
ndb only.
c, err := g.ClaimDue(ctx, "worker-2", "0-0", 100)
// c.Entries ([]Entry[T]), c.Cursor ("0-0" when the scan is complete), c.DeletedClaimDue(ctx, consumer, cursor, count)scans the nacked entries fromcursorand moves up tocountof the ones whose retry time passed to the consumer, in ID order. Each one gets one more delivery. An empty cursor is"0-0".- The scan looks only at the nacked entries, not at the entries in flight at live workers. So a worker can call it on a short tick:
Workdoes it everyDueEvery. g.ClaimDueIDs(ctx, consumer, cursor, count)gives only the IDs, in andb.ClaimedIDs. The delivery count of the entries does not change.
Work
Work is the worker loop of the group. Use it in place of Read, Ack and Claim calls of your own.
g := orders.Group("workers")
if err := g.Ensure(ctx, ndb.StartNew); err != nil {
return err
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
err = g.Work(ctx, "worker-1", func(ctx context.Context, e ndb.Entry[Order]) error {
return process(ctx, e.Doc) // nil acks; an error leaves the entry pending
}, ndb.Work{
MaxDeliveries: 5,
DeadLetter: "orders-dead",
OnError: func(err error) { log.Println(err) },
})Work does these steps:
- It reads the consumer's own pending entries first (the ones it took before a restart and did not ack), and runs the handler on them.
- It reads new entries, and waits up to
Blockfor them. - It acks, in one call, each entry the handler did without an error. It acks a
Deletedentry without the handler. - With
Retry, it nacks each entry whose handler failed, with the wait ofRetry. WithoutRetry, the entry stays pending. - Every
DueEvery, it claims the nacked entries whose wait ended (ClaimDue), and runs the handler on them again. - Every
ClaimEvery, it claims the entries of any consumer of the group that are idle forClaimIdle, and runs the handler on them again. A nacked entry that still waits is not in this claim (the list pages on past it), so many waiting entries do not hide the idle ones.
Fields of ndb.Work:
| Field | Default | Meaning |
|---|---|---|
Count | 10 | The most entries per read. |
Block | 5 s | How long a read waits for new entries. |
ClaimIdle | 1 min | Claim the entries pending for this long. A negative value turns the claim off. |
ClaimEvery | ClaimIdle / 2 (at least 1 s) | How often Work looks for idle entries. |
MaxDeliveries | 0 (no limit) | An entry delivered this many times is a poison message. Also with Retry: a failure on this delivery goes to DeadLetter, not to a nack. |
DeadLetter | none | The stream that gets the poison messages and the entries past MaxAge, as ndb.DeadEntry. |
Retry | nil | ndb only. func(deliveries int64) time.Duration: the wait before the next try of an entry whose handler failed on its deliveries-th delivery (1 is the first). Work nacks the entry with this wait. A wait below 0 is 0; above 30 days is 30 days. nil keeps the entry pending for ClaimIdle. Make one with ndb.Tiers or ndb.TierSteps. |
MaxAge | 0 (no limit) | An entry older than this, by the time of its ID (Entry.Time), goes to DeadLetter with reason max_age, without the handler, and is acked: no more tries. Work looks at each delivery, the first one too, and again at a failure: an entry that got too old in the handler is not nacked. |
DueEvery | 1 s with Retry, else off | How often Work claims the nacked entries whose wait ended. A negative value turns it off. A positive value turns it on also without Retry, for the entries that other workers nacked. |
OnError | none | Gets the errors that do not stop Work. |
- Stop:
Workruns untilctxends, then returns nil. The acks of the work that is done still go afterctxends. - The
Handler[T]isfunc(ctx context.Context, e ndb.Entry[T]) error. - Delivery is at least once. Make the work safe to repeat.
- Run many workers with different consumer names to share the work. The group must exist (
Ensure). - A failed read does not stop
Work, also the read of the own pending entries at the start, and also when the group does not exist. It goes toOnError, andWorktries again with a backoff from 100 ms to 5 s. Only the end ofctxstopsWork. - An entry whose doc does not decode into
T(Entry.Err) does not go to the handler. It counts as a handler failure: it goes toOnError, stays pending (or is nacked, withRetry), and afterMaxDeliveriesgoes toDeadLetterwith its raw doc. The other entries of the same read go on as usual. - Crash: the read of the own pending entries at the start is one more delivery. An entry that already had
MaxDeliveriesdeliveries without an ack (the process died in its handler, for example OOM or SIGKILL) goes toDeadLetterat the start, without the handler. So a poison entry that kills the process does not run forever. - Shutdown: when
ctxends and the handler returns an error,Workdoes not count it as a failure. The entry stays pending (no ack, no nack, no dead letter, noOnError), and the next run, or a claim, gets it again. - Restart while an entry waits: a nacked entry has no owner, so the own pending read of the restarted worker does not give it. The due claim of any worker of the group takes it when its wait ends.
- A failed nack goes to
OnErrorand leaves the entry pending: the idle claim takes it afterClaimIdle.
Delayed retry
ndb only. A worker that fails an entry says when the entry comes back, and the wait can grow with the number of deliveries. The wait lives in the pending list of the daemon: the entry is on disk, it is in one place only, and it survives a restart of the worker and of the daemon.
// The fsw-esl outbox: try 1 waits 15 s, tries 2 to 5 wait 1 min, 6 to 11
// wait 10 min, 12 to 23 wait 1 h, then 6 h, until the entry is 15 days old.
err = g.Work(ctx, "jobs-1", func(ctx context.Context, e ndb.Entry[Job]) error {
return postCall(ctx, e.Doc)
}, ndb.Work{
Retry: ndb.TierSteps(
ndb.Step{Wait: 15 * time.Second, UpTo: 1},
ndb.Step{Wait: time.Minute, UpTo: 5},
ndb.Step{Wait: 10 * time.Minute, UpTo: 11},
ndb.Step{Wait: time.Hour, UpTo: 23},
ndb.Step{Wait: 6 * time.Hour},
),
MaxAge: 15 * 24 * time.Hour,
DeadLetter: "postcall_jobs_dead",
OnError: func(err error) { log.Println(err) },
})With this schedule, an entry that always fails runs 81 times: 23 tries in the first 13 h 04 min 15 s, then one try each 6 h. The 82nd delivery is past 15 days: it goes to the dead letter with reason max_age.
ndb.Tiers(waits...) gives one wait per try: the wait after the failure on delivery n is waits[min(n, len(waits))-1], so the last wait repeats.
ndb.Tiers(15*time.Second, time.Minute, 10*time.Minute) // 15 s, 1 min, then 10 min for each next tryndb.TierSteps(steps...) gives the same with bounds. A ndb.Step has:
Wait: the wait before the next try.UpTo: the last delivery of the step. For the failure on deliveryn, the wait is theWaitof the first step withn <= UpTo.UpTo0 is no bound: the step applies to everynthat is left. After the last bound, theWaitof the last step repeats.
So UpTo is a delivery count, and the failure on delivery n is try n: Step{Wait: time.Minute, UpTo: 5} after Step{..., UpTo: 1} covers the tries 2, 3, 4 and 5. A Retry of your own is any func(int64) time.Duration, for example an exponential backoff.
OnErrorgets the read errors, the handler and decode errors, the claim and ack errors, and the poison messages.
DeadEntry and ErrPoison
A poison message is an entry that fails on its MaxDeliveries-th delivery (reason ndb.ReasonMaxDeliveries, "max_deliveries"). An entry past MaxAge goes to the dead letter the same way (reason ndb.ReasonMaxAge, "max_age"). At that failure, Work appends it to DeadLetter (when set), acks it, then sends an error that wraps ndb.ErrPoison to OnError. If the append to DeadLetter fails, the entry stays pending. The append and the ack go also when ctx ended.
dead, err := ndb.StreamOf[ndb.DeadEntry](db, "orders-dead")
rows, err := dead.Range(ctx, "-", "+", 100)
// each Doc: Stream, Group, ID, Deliveries, Reason ("max_deliveries" or "max_age"),
// Doc (json.RawMessage, the entry without _id and _ts)
ndb.Work{OnError: func(err error) {
if errors.Is(err, ndb.ErrPoison) {
alert(err)
}
}}Stream errors
| Sentinel | Code | When |
|---|---|---|
ErrNotAStream | not_a_stream | The name is a collection that is not a stream. |
ErrInvalidStreamID | invalid_stream_id | A bad entry ID. |
ErrStreamIDTooSmall | stream_id_too_small | WithEntryID is not higher than the last ID. |
ErrUnknownGroup | unknown_group | The group does not exist. |
ErrGroupExists | group_exists | Group.Create on a group that exists. |
ErrUnknownCollection | unknown_collection | NoCreate or GroupNoCreate, or a Tail, and the stream does not exist. |
See Errors of the stream routes.
Redis Streams commands (ndbredis)
The module ndbredis has the stream methods of go-redis v9 (XAdd, XReadGroup, XAck, ...) on ndb streams. An app that uses Redis Streams picks its backend in one place, its factory, and keeps its calls. See Move from Redis Streams for the same change in Ruby and TypeScript.
Install ndbredis
go get github.com/thadeu/ndb/sdk/go/ndbredisimport (
"github.com/redis/go-redis/v9"
ndb "github.com/thadeu/ndb/sdk/go"
"github.com/thadeu/ndb/sdk/go/ndbredis"
)- It is a module of its own, with its own
go.mod, because its methods return go-redis types (*redis.StringCmd,*redis.XStreamSliceCmd, ...). The ndb SDK (github.com/thadeu/ndb/sdk/go) stays without dependencies. - It needs go-redis v9.23.0 or later, and the ndb SDK v1.16.0 or later.
- Its release tag is
sdk/go/ndbredis/vX.Y.Z, next to the tag of the SDK (sdk/go/vX.Y.Z).
The factory
Write the interface of the app with only the commands it uses. *redis.Client and *ndbredis.Client both have them:
// Streams is the interface of the app.
type Streams interface {
XAdd(ctx context.Context, a *redis.XAddArgs) *redis.StringCmd
XReadGroup(ctx context.Context, a *redis.XReadGroupArgs) *redis.XStreamSliceCmd
XAck(ctx context.Context, stream, group string, ids ...string) *redis.IntCmd
}
func newStreams(cfg Config) (Streams, error) {
if cfg.Backend != "ndb" {
return redis.NewClient(&redis.Options{Addr: cfg.RedisAddr}), nil
}
db, err := ndb.Connect(cfg.NdbAddr)
if err != nil {
return nil, err
}
return ndbredis.New(db), nil
}The calls of the app do not change:
id, err := streams.XAdd(ctx, &redis.XAddArgs{
Stream: "orders",
Values: map[string]any{"order_id": 42},
}).Result()
res, err := streams.XReadGroup(ctx, &redis.XReadGroupArgs{
Group: "billing",
Consumer: "w1",
Streams: []string{"orders", ">"},
Count: 10,
Block: 5 * time.Second,
}).Result()
if errors.Is(err, redis.Nil) {
// No new entry in 5 seconds.
}
n, err := streams.XAck(ctx, "orders", "billing", id).Result()ndbredis.New(db)returns a*ndbredis.Client. It sends nothing. The app keepsdband closes it. AClientis safe for many goroutines.*ndbredis.Clientsatisfiesredis.StreamCmdableof go-redis v9.23.0, the full list of stream commands. Prefer your own small interface: a later go-redis can add a method toredis.StreamCmdable, and then the app builds only after this module has the method too.
Supported methods
| Command | Methods |
|---|---|
XADD | XAdd (ID, NoMkStream, MaxLen, MinID, Approx, Limit) |
XLEN, XDEL | XLen, XDel |
XRANGE, XREVRANGE | XRange, XRangeN, XRevRange, XRevRangeN (-, +, (id) |
XREAD | XRead, XReadStreams (an ID, 0, $, +) |
XTRIM | XTrimMaxLen, XTrimMaxLenApprox, XTrimMinID, XTrimMinIDApprox, and the four ...Mode methods with KEEPREF |
XGROUP | XGroupCreate, XGroupCreateMkStream, XGroupSetID, XGroupDestroy, XGroupCreateConsumer, XGroupDelConsumer |
XREADGROUP | XReadGroup (>, or an ID for the pending entries; Count, Block, NoAck) |
XACK | XAck |
XPENDING | XPending, XPendingExt |
XCLAIM | XClaim, XClaimJustID |
XAUTOCLAIM | XAutoClaim, XAutoClaimWithDeleted, XAutoClaimJustID |
XINFO | XInfoStream, XInfoStreamFull, XInfoGroups, XInfoConsumers |
Not supported
ndb has no such command or option. The method exists, and its command comes back with an error that errors.Is(err, ndbredis.ErrNotSupported) finds:
XAckDel,XDelEx,XNack,XCfgSet.XAddwithProducerID(IDMP), or with aModethat is notKEEPREF.- The
XTrim...Modemethods withDELREForACKED. XReadandXReadGroupwithMaxCountorMaxSize.XReadGroupwithClaim(useXAutoClaim).XReadandXReadGroupwithBlockon more than one stream.XAddwithValuesthat is a struct (use a map or a list of pairs).
ndbredis errors
Errors have the text of Redis, and they are redis.Error values. errors.Is also finds the ndb error below them.
| Case | Error |
|---|---|
| A read with no entry | redis.Nil |
XAdd with NoMkStream, and no stream | redis.Nil |
| A group that exists | BUSYGROUP Consumer Group name already exists (ndb.ErrGroupExists) |
| A group or a stream that does not exist, in a group command | NOGROUP No such key '...' or consumer group '...' (ndb.ErrUnknownGroup) |
XGroupCreate (no MKSTREAM) and no stream | ERR The XGROUP subcommand requires the key to exist. ... |
XInfoStream, XInfoGroups and no stream | ERR no such key |
| An ID that is not greater than the last one | ERR The ID specified in XADD is equal or smaller than the target stream top item (ndb.ErrStreamIDTooSmall) |
| A bad ID | ERR Invalid stream ID specified as stream command argument (ndb.ErrInvalidStreamID) |
| The name is a collection that is not a stream | WRONGTYPE Operation against a key holding the wrong kind of value (ndb.ErrNotAStream) |
| A command or an option that ndb does not have | ndbredis.ErrNotSupported |
err := streams.XGroupCreateMkStream(ctx, "orders", "billing", "$").Err()
if err != nil && !strings.HasPrefix(err.Error(), "BUSYGROUP") {
return err
}- As in Redis,
XLen,XRange,XDel,XTrim...andXAckgive 0 or an empty list for a stream (or a group) that does not exist. - Any other ndb error (
ndb.ErrUnauthorized, a network error) stays as it is.
Differences from Redis
- Stream names. A stream is an ndb collection: its name has the letters
a-z, the digits,_and-, at most 64 characters, and starts with a letter or a digit.orders:euandOrdersare not valid names. - Field names. A name that starts with
_is reserved (_id,_ts).XAddwith such a name returns an error that wrapsndbredis.ErrNotSupported, and appends nothing. - Values keep their JSON type in ndb.
XAddwrites a Go string,[]byte,time.Timeandencoding.BinaryMarshaleras a JSON string, and every other value as JSON: a number is a number, aboolis a boolean, a map or a slice is an object or an array. So the other SDKs and the queries of ndb see JSON types. - Values come back as text, as Redis gives them:
XMessage.Valueshas a string for each field. A JSON string is its text; every other value is its JSON text ("42","9.5",`{"name":"ana"}`);nullis"". A float has no exponent, as go-redis sends it:1e21is"1000000000000000000000"and1e-7is"0.0000001". One difference: aboolcomes back as"true"or"false"; go-redis sends it to Redis as1or0. - Text only. A value is JSON, so bytes that are not UTF-8 text do not survive. Put binary data in a blob.
- A read without
Countgives at most 10 000 entries for each stream. Redis gives all of them.XRangewithout a count gives all of them (it reads page by page). Blockon one stream only. A read of many streams withoutBlocksends one request for each stream, one after the other; it is not one atomic read.XReadGroupfirst checks that the group exists on every stream, so aNOGROUPerror delivers nothing, as in Redis.Block: 0(wait with no limit, the zero value of go-redis) waits until the context ends. The wait is a sequence of requests of 30 seconds.XReadwithBlockon a stream that does not exist looks for the stream every 100 ms.XReadwith+gives the last entry and never waits.- Trims keep the pending entries (
KEEPREF, the default of Redis). WithApprox, ndb deletes only whole blocks of 1 024 entries. XInfoStreamandXInfoStreamFull: the fields about the memory of Redis (RadixTreeKeys,RadixTreeNodes) and about IDMP are 0. The delivery and seen times ofXInfoStreamFullcome from idle times, so they are correct to some milliseconds.XAutoClaimgives only the entries that are in the stream. The pending IDs whose entry was deleted leave the pending list;XAutoClaimWithDeletedgives them.- No pipelines and no transactions. Each method sends its requests at once.
Admin
err := db.Health(ctx) // nil when ready (GET /_health)
defs, err := db.Collections(ctx) // the catalog (GET /_collections)
stats, err := db.Stats(ctx) // every collection and totals (GET /_stats)
size, err := db.Size(ctx) // disk use (GET /_size)
stats, err = calls.Stats(ctx) // GET /calls/_stats
report, err := calls.Compact(ctx) // POST /calls/_compact
fields, err := calls.EnsureIndex(ctx, "dnis") // PUT /calls/_indexes/dnis
fields, err = calls.DropIndex(ctx, "dnis")The answers are the JSON of the HTTP routes, as map[string]any.
Blobs
Large values live in the blob store. The document keeps only a reference. The same content always gives the same reference.
blob, err := db.PutBlob(ctx, file, "audio/wav") // file is an io.Reader
_, err = calls.Update(ctx, "c1", ndb.Ops{"$set": {"recording": blob}})
call, err := calls.Get(ctx, "c1") // call.Recording is *ndb.Blob
rc, err := db.GetBlob(ctx, *call.Recording)
if err != nil {
return err
}
defer rc.Close()
_, err = io.Copy(dst, rc)A struct field of type ndb.Blob or *ndb.Blob needs no special tag. Blob has Ref, Size and Type.
Errors
The daemon errors are *ndb.Error with Code, Message and Status (the HTTP status). Test them with errors.Is and a sentinel. Use errors.As to read the fields.
if errors.Is(err, ndb.ErrNotFound) {
}
var apiErr *ndb.Error
if errors.As(err, &apiErr) {
log.Println(apiErr.Code, apiErr.Status, apiErr.Message)
}Each code of the error table has a sentinel: ndb.Err and the code in CamelCase, for example ErrNotFound (not_found), ErrInvalidPath (invalid_path), ErrUniqueViolation (unique_violation). ErrStaleObject (stale_object) is the error of WithRetry after its last try; the daemon never sends it. The exceptions are ErrGroupLimit (group_limit_exceeded), ErrTTLNotEnabled and ErrTTLDerived, and ErrPITROff. A test of the repository fails when the daemon gets a code that has no sentinel.
A response that is not a daemon error has the code http_status. A failed connection returns the normal net error.
Testing
ndb has no transactions, so a test cannot roll back. ndb.Testing.Clean truncates the collections that the *DB used since the last clean (a few ms at any size; the definition and the indexes stay), like Ndb::Testing in Rails:
func TestMain(m *testing.M) {
code := m.Run()
ndb.Testing.Clean(context.Background(), db)
os.Exit(code)
}
// or after each test
t.Cleanup(func() { ndb.Testing.Clean(ctx, db) })CleanandCleanAllreturn the names of the collections they truncated.ndb.CleanOnly(names...)truncates these, not the ones used;ndb.CleanExcept(names...)leaves some out.ndb.Testing.CleanAll(ctx, db)truncates every engine collection with the prefix of the DB, for the start of a run. Direct collections and partitions are left out.ndb.WithPrefix("test1_")onConnectputs a prefix before the name of everyCollection: each parallel package or worker gets collections of its own. Without it, the prefix isNDB_PREFIX, so a test that starts the app from env vars only can set it too.- A collection that does not exist is not an error.
- A clean runs only under
go test(testing.Testing()). Anywhere else it returnsndb.ErrUnsafeClean, so it never truncates production data.NDB_ALLOW_CLEAN=1forces it. - The SDK does not start a daemon: start it in CI as a service, with
make up, or with Docker, and connect withNDB_URL.
Tests of this module
The tests of this module build the daemon from the repository (go build ./cmd/ndb) in TestMain. They start it on a temp unix socket and a free TCP port, with a temp data dir, and stop it at the end:
ndb -config /nonexistent.yml -data-dir "$(mktemp -d)" -unix /tmp/ndb-test.sock -tcp 127.0.0.1:7070Add -token-file <path> to ask for a bearer token on TCP. Use a short socket path: macOS allows 104 bytes. Use a new collection name for each test.