Rails
The gem ndb-rails: an ActiveModel layer with Ndb::Record, and streams with Ndb::Streams. It depends on the gem ndb and uses Ndb::Client inside.
The source and the same guide are in sdk/rails.
Optimistic locking by default, pessimistic locking when you ask. Every model uses optimistic locking with no setup: each save sends the version (
_v) the record was read with, aslock_versiondoes in ActiveRecord, without the column. Usewith_retryto try again after a conflict. For pessimistic locking,Model.lock.find(id),lock!andwith_lockwork as in ActiveRecord. See Concurrency.
Install
Needs Ruby 3.2 or newer, activemodel and activesupport 7.1 or newer, and a running ndb daemon. The gem ndb (~> 0.1) is installed with it, as a dependency.
# Gemfile
gem "ndb-rails"bundle add ndb-rails
# or: gem install ndb-railsIn Rails, the gem loads a railtie. It reads config/ndb.yml and, when the app eager loads (production), it declares the collection of every model at boot. In development a model declares its collection on first use. Without Bundler autorequire, use require "ndb/rails". Without Rails, call Ndb::Rails.configure yourself (below).
Connect
Rails: create config/ndb.yml. It has one block per environment. ERB and YAML anchors work.
default: &default
timeout: 30 # seconds, per request
pool_size: 16 # idle connections kept
ready_timeout: 10 # seconds to retry while the daemon starts; 0 turns it off
development:
<<: *default
socket: /tmp/ndb.sock
production:
<<: *default
url: tcp://ndb.internal:7070
token: <%= ENV["NDB_TOKEN"] %>Keys: url, socket, token, timeout, pool_size, ready_timeout, user_agent. Any other key is ignored.
Any app: configure in code, for example in config/initializers/ndb.rb.
Ndb::Rails.configure do |c|
c.socket = "/run/ndb/ndb.sock" # or: c.url = "tcp://10.0.0.5:7070"
c.token = ENV["NDB_TOKEN"]
c.timeout = 30
c.pool_size = 16
c.ready_timeout = 10
c.user_agent = "myapp/1.2" # optional: the token of your app
end- The address is, in order:
url,socket, envNDB_URL, then the default socket/run/ndb/ndb.sock. /run/ndb/ndb.sockis the socket of a server install (root, or/var/lib/ndbexists). A local daemon that you start as a user with no flags (ndb serve) listens on~/.ndb/ndb.sockand on TCP127.0.0.1:7070. In development, setsocket: <%= Dir.home %>/.ndb/ndb.sockorurl: tcp://127.0.0.1:7070.- The address is
unix:///path,tcp://host:port,https://host[:port](ndb behind a TLS proxy), or a bare path that starts with/. - The token is
token, or envNDB_TOKEN. It goes only in theAuthorizationheader. A unix socket does not need it. ready_timeoutis how long a call retries while the daemon is not ready (not_ready, or the socket is missing or refuses). It uses backoff with jitter. A write is retried only when the daemon certainly did not process it. A read is also retried after a dropped connection. A timeout is never retried.- The pool drops its connections after a
fork. - Every request sends
User-Agent: ndb-ruby/<version> (ruby <version>; <platform>) ndb-rails/<version> rails/<Rails.version>, then theuser_agentof the configuration (the token of your app, such asmyapp/1.2), so the daemon can tell the apps apart. Ndb::Rails.clientis anNdb::Clientbuilt from this configuration (Ndb::Rails.configuration). You can also build one:Ndb::Client.new(socket: "/tmp/ndb.sock", token: nil, timeout: 30, pool_size: 16, ready_timeout: 10). ThendbREADME documents every client call.
Define a model
class Call < Ndb::Record
attribute :status, :string, index: true
attribute :caller, :string, unique: true
attribute :duration, :integer
attribute :at, :datetime, time: true
attribute :expires_at, :datetime, expires: true
attribute :recording, :blob
validates :status, presence: true
end| Option or method | Meaning |
|---|---|
_id / id | The primary key is always _id (a string; the daemon makes a UUIDv7 when it is empty). id is an alias of _id, also in where, find_by and exists? (a model with its own id attribute keeps it). For a natural key (email, SKU) keep _id and use unique: true on that attribute. |
created_at / updated_at | Every model has them, with an index, as the timestamps of ActiveRecord. Do not declare them. A save of a new record sets both (a value that you set stays). Each write sets updated_at: save, update, increment!, Relation#update, upsert and upsert_all (created_at only on a new document; one read finds the ids that exist). A save with no change writes nothing. The values are UTC with milliseconds (2026-11-12T15:00:05.000Z). |
index: true | Index this field. For a compound index, call index :account_id, :created_at in the class; with unique: true it is a compound unique key (index :account_id, :email, unique: true). index "items.sku" indexes a path that is not an attribute. |
unique: true | Unique value. A second one raises RecordNotUnique. |
encrypted: true | The daemon encrypts the attribute before it stores it. A filter on it is only exists. See Encrypted fields. |
blind: :numeric | An encrypted attribute that where and find_by find by value, after this normalization (:exact, :lower, :upper, :numeric). It may also be unique: true. |
Secret.encrypted | Reads that keep the encrypted fields sealed ("ndb:e1:k1:..."); the other reads get the values (the gem sends decrypt=1). |
time: true | The collection time field. Used by zone maps, $dateTrunc and TTL. |
expires: true | The field holds the time the document expires. |
ttl 24.hours | Documents expire this long after the time field. Use ttl or expires:, not both: the daemon returns catalog_invalid. |
self.collection_name = "x" | Collection name. Default follows Rails naming (Call becomes calls). |
source "glob" | Declares a read-only direct-mode collection over files. Writes raise ReadOnlyCollection. |
- Attributes are ActiveModel attributes, so every ActiveModel type works (
:string,:integer,:float,:boolean,:datetime,:date, and so on). Times are sent as RFC 3339 UTC strings with milliseconds.:blobis added by this gem. - Validations, callbacks (
validation,save,create,update,destroy), dirty tracking andas_jsonwork as in ActiveModel. readonly?works as in ActiveRecord: when it is true,save,destroyanddeleteraiseNdb::ReadOnlyRecord. Override it in a model (def readonly? = true), or callrecord.readonly!.validates :name, uniqueness: true(oruniqueness: { scope: :app_id }) asks the daemon if another document has the value, and adds:taken("has already been taken") to the field. Two saves at the same time can both pass, so keepunique: trueon the attribute: the unique index is the real guard (Ndb::RecordNotUnique).- The model declares its collection on first use, once per process.
- Mark a base class with
self.abstract_class = trueto share settings.
Insert, find, update, delete
call = Call.create!(_id: "c1", status: "answered", caller: "a", at: Time.now.utc)
Call.create!([{ _id: "c2", status: "failed", caller: "b" }, { _id: "c3", status: "answered", caller: "c" }])
call = Call.new(_id: "c4", status: "answered", caller: "d")
call.save # false when invalid
call.save! # raises Ndb::RecordInvalid
Call.where(status: "failed") # a list (lazy relation)
Call.find_by(status: "failed") # one record or nil: the daemon gives 1 row (limit 1)
Call.find_by!(status: "x") # the same, raises Ndb::NotFound when missing
call = Call.find("c1") # by id, raises Ndb::NotFound
Call.first # the lowest _id; Call.last: the highest (limit 1, not the whole list)
Call.order(:created_at).last(3) # the last 3 in that order, oldest firstfirst and last with no order follow _id. An _id that the daemon makes (a UUIDv7) grows with time, so last is the newest. An _id that the app makes (SecureRandom.hex, an email) does not, so last is not the newest: use order(:created_at).last.
call.update!(status: "done") # $set of the changed fields only, filtered by id and version
call.version # the document version
call.increment!(:duration) # atomic $inc on the daemon, refreshes the attribute and the version
call.decrement!(:duration, 5)
call.touch # updated_at = now, one atomic write, no validations; after_touch runs
call.touch(:seen_at, time: t) # also the named datetime attributes
call.update_columns(status: "x", tries: 2) # atomic $set, cast; no validations, no callbacks, updated_at stays
call.update_column(:status, "x")
call.toggle(:active) # flips a boolean (not saved); toggle! saves it without validations
call.update_attribute(:status, "x") # save without validations, with callbacks
call.saved_change_to_status? # after a save: saved_changes, status_before_last_save, too
call.will_save_change_to_status?
call.cache_key # "calls/c1" ("calls/new" for a new record)
call.cache_key_with_version # "calls/c1-<updated_at>-<_v>": changes on every write
Call.insert_all([{ id: "c7", status: "new" }, { id: "c8", status: "new" }]) # one _bulk, no validations/callbacks
Call.update_counters(["c1", "c2"], tries: 1, credits: -1) # atomic $inc by id, sets updated_at
Call.increment_counter(:tries, "c1")
Call.decrement_counter(:tries, "c1")
call.destroy # callbacks, filtered by id and version
call.destroy! # raises Ndb::RecordNotDestroyed when a before_destroy callback aborts
call.delete # by id only: no callbacks, no version guard
Call.destroy("c1") # find, destroy, return the record (an Array of ids returns an Array)
Call.delete(["c2", "c3"]) # one delete by id, no callbacks, returns the count
# Dynamic finders, for declared attributes and id. The ! variant raises NotFound.
Call.find_by_status("failed")
Call.find_by_status_and_caller("failed", "a")
Call.where(caller: "a").find_by_status!("failed")
Call.find_or_initialize_by(_id: "c9") { |c| c.status = "new" } # unsaved when missing
Call.find_or_create_by(_id: "c9") { |c| c.status = "new" } # atomic: concurrent callers get one document
Call.find_or_create_by!(_id: "c9", status: "new") # raises RecordInvalid
# Atomic writes on every match of a where (it needs the id or an indexed field).
# update returns the number of modified documents. It takes attributes, as Active Record
# (a $set, cast by the attribute types), the operators of the client, or both.
# Unlike Active Record it runs no validation and no callback (as update_all).
Call.where(status: "failed").update(status: "retry", queued_at: Time.current)
Call.where(status: "failed").update("$set" => { "status" => "retry" }) # the braces are optional
Call.where(id: "c1").update(status: "retry", "$inc" => { "tries" => 1 }) # the attributes join the $set
Call.where(status: "retry").delete_all # one delete, no callbacks, returns how many (delete is an alias)
Call.where(status: "retry").destroy_all # loads and destroys each record with its callbacks, returns them
# A limit, offset, page or cursor narrows a write to those rows, as in Active Record:
# the relation reads their ids, then writes with the where and "_id" $in those ids.
Call.where(status: "retry").order(:created_at).limit(100).delete_all # at most 100
Call.where(status: "retry").in_batches(of: 500) { |batch| batch.update(status: "queued") }
Call.where(status: "retry").update_all(status: "queued") # update_all is an alias of update
# More Active Record relation methods.
Call.where(status: "new").new(caller: "a") # status "new" comes from the where (also build, create, create!)
Call.where(status: "new").find_or_create_by(caller: "a")
Call.first!; Call.last!; Call.take; Call.take! # ! raises Ndb::NotFound
Call.find_sole_by(caller: "a") # NotFound, or Ndb::SoleRecordExceeded for more than one
Call.where(status: "x").ids # the ids
Call.order(created_at: :desc).pick(:status) # pluck of the first row
Call.distinct.pluck(:status) # each value once
Call.select(:status).to_a # records with only status (and _id) loaded
Call.none # no request: [], count 0, writes 0
Call.destroy_by(status: "x"); Call.delete_by(status: "x")
Call.order(:a).reorder(b: :desc); Call.where(status: "a").rewhere(status: "b")
Call.where(status: "a").limit(5).unscope(:where, :limit)Every load asks for the version (
_v).save,updateanddestroyput the id and_vin the filter of the write. If another writer changed the document first, the filter matches nothing and you getNdb::StaleObjectError(codestale_object).record.with_retry { |r| ... }reads the document again, yields the record and saves it. OnNdb::StaleObjectErrorit reads again and yields again (5 tries,with_retry(retries: 10)), then raises it. An exception in the block saves nothing. Nothing waits on the daemon, and the block can run more than once. See with_retry.rubyWallet.find("w1").with_retry do |w| raise InsufficientFunds if w.balance_cents < 500 w.balance_cents -= 500 endlockis pessimistic locking, with the names of ActiveRecord. One request holds the record; anotherlockof it waits (wait:, default 10 s, thenNdb::Locked), and a write from any other place waits for the release. So the code runs once: use it when the code has an effect outside ndb. See Lock.ruby# With a block: the lock ends with the block, also on an exception. Payment.lock.find("p1") do |p| next if p.status == "refunded" Gateway.refund(p.charge_id) p.update!(status: "refunded") end payment.with_lock { |p| ... } # the same, on a loaded record (it is read again) # Without a block: the lock ends at the first save or destroy (also when it # raises), or at unlock!. A save that returns false keeps it. payment = Payment.lock.find("p1") payment.update!(status: "refunded") # written with the lock, then released payment.lock! # lock a loaded record and read it again payment.unlock! # release without a write Payment.lock(ttl: 2.minutes.to_i, wait: 0).find_by(charge_id: "ch_1")Outside a transaction, prefer the block. Every lock also ends after
ttl:(default 30 s): a process that dies holds nothing, and a save after it raisesNdb::LockExpired(the record then holds no lock, and its next save is a plain save).lock.findtakes one id;lock.find_byandlock.firstfind the record, then lock it and read it again.Model.transaction { }writes everything of the block in one step, all or none:save,update,create,destroy,increment!anddecrement!run their validations and callbacks, but are sent together at the end of the block. An exception writes nothing;raise Ndb::Rollbackstops the block with no error (it returnsnil).lock: [records or ids]withttl:andwait:takes every lock in one call (no deadlock), andlock!inside the block joins it. One collection per block (Ndb::TransactionErrorotherwise). See transaction.rubyAccount.transaction(lock: [from, to], ttl: 5) do raise Ndb::Rollback if from.balance < amount from.update!(balance: from.balance - amount) to.update!(balance: to.balance + amount) endafter_commit,after_create_commit,after_update_commit,after_save_commit,after_destroy_commit(andon:) run when the write is durable: after the write, or after theModel.transactionblock commits.after_rollbackruns when the block writes nothing.touchandupdate_columnsrun no commit callbacks.insert_all(andinsert_all!, the same) stampscreated_atandupdated_at. An_idthat exists raisesRecordNotUnique(codeduplicate_id).find_or_create_byis atomic. It checks validations and runs thesaveandcreatecallbacks, then sends one upsert: the attributes you pass are the filter, and the other attributes go in$setOnInsert, so a writer that created the document first is never overwritten and you get that document. An invalid record is returned unsaved (the!form raisesNdb::RecordInvalid). Only the attributes you pass must be equality values.delete_allanddestroy_allon a model without awhereare refused by the daemon (Ndb::EmptyFilter);delete_allneeds the id or an indexed field in thewhere. To empty the whole collection, useCall.truncate!.Call.truncate!removes every document, with no callbacks, and keeps the definition and indexes.Call.drop!removes the collection, its definition and its watch consumers; the next use of the model declares it again. Both take the same time for 10 or 10 million documents, and watch consumers get one"truncate"event.Setting an attribute to
niland saving removes the field from the document.A duplicate id raises
RecordNotUnique(codeduplicate_id).Call.upsert(id: "c1", status: "done")andCall.upsert_all([...])write by_idlike ActiveRecord: they create the document, or merge the given attributes into the stored one (the others stay), in one transaction. No validations or callbacks run. See Upsert.persisted?,new_record?,destroyed?,idandversionwork as in ActiveRecord.
begin
call.update!(status: "done")
rescue Ndb::StaleObjectError
call = Call.find(call.id)
retry
endExpiry (TTL like Redis)
Give the model an expires: true attribute. The daemon sets it for you.
class Session < Ndb::Record
attribute :user, :string
attribute :expires_at, :datetime, expires: true
end
Session.create!({ _id: "s1", user: "u" }, expire_in: 1.hour) # also create, save, update (seconds, a Duration or "30m")
session = Session.find("s1")
session.expire(30.minutes) # EXPIRE: sets expires_at in the object, other unsaved changes stay
session.ttl # => 1799, -1 (never expires), or nil (missing or expired)
session.persist_ttl! # PERSIST: removes the expirypersist_ttl!is not calledpersist!: ActiveModel already usespersisted?for something else.expire_in:oncreate,saveandupdategoes in the same write.expire_inwins over theexpires_atvalue. After a write withexpire_in, load the record again (Session.find) to seeexpires_at.expireandpersist_ttl!raiseNdb::NotFoundfor a missing or expired document, andNdb::TtlNotEnabledwhen the model has noexpires: trueattribute.- The class-level
ttl 24.hoursstays: every document expires that long after its time field.expireandpersist_ttl!raiseNdb::TtlDerivedfor it. - Disk space of expired documents returns after compaction, and fully only with
wal_retention; see Freeing disk.
Query
Relations are lazy and immutable. The request goes out when you call to_a, first, each, count and so on.
to_ndb gives the JSON body that a relation sends to the daemon (POST /{collection}/_search), with no request, like to_sql in ActiveRecord:
Call.where(status: "failed").order(created_at: :desc).limit(10).to_ndb
# => {"query":{"status":"failed"},"sort":{"created_at":-1},"limit":10,"versions":true}
puts Call.where(status: "failed").to_ndb(pretty: true)Call.where(status: "answered")
Call.where(status: %w[answered failed]) # $in
Call.where(duration: 30..60) # $gte and $lte
Call.where(duration: 30..) # $gte
Call.where(duration: nil) # $exists: false
Call.where(at: 1.day.ago..)
Call.where("agent.id" => 7) # dot path
Call.where(duration: { "$gt" => 10 }) # any daemon operator, as a Hash
Call.where.not(status: "failed") # $ne, $nin, $exists
Call.where(status: "failed").or(Call.where(duration: 60..))
Call.where("$or" => [{ status: "failed" }, { duration: { "$gt" => 50 } }])
Call.where(status: "answered")
.order(at: :desc, _id: :asc)
.limit(50)
.offset(100)
.to_a
Call.where(status: "answered").pluck(:_id) # one field: an array of values
Call.where(status: "answered").pluck(:_id, :status) # several: an array of arrays
Call.where(status: "answered").first(5)
Call.where(status: "failed").exists?
Call.exists?("c1") # by id; exists?(status: "failed") takes conditions- Chained
wherecalls are ANDed. See the query language for the full operator list. - The order of the
orderkeys is the order of the sort. The id is always the last key. page(n).per(size)sets offset paging. The default page size is 25.pluckasks the daemon for only those fields.- There is no
explainon a relation. Use the client:Ndb::Rails.client.search(coll, "query" => {...}, "explain" => true)returns aPagewithplan(theX-Ndb-Planheader:id,index:<field>,scan,snapshot) andexamined.
Scopes
A scope is a named query, as in ActiveRecord. A scope and a class method of the model can also be called on a relation: the relation is the all of the call.
class Message < Ndb::Record
scope :recent, -> { order(created_at: :desc) }
scope :top, ->(count) { order(score: :desc).limit(count) }
def self.failed = where(status: "failed")
end
Message.where(app_id: app.id).recent.top(10)
Message.recent.failed- A relation runs only the methods that the model declares. The methods of
Ndb::Record(truncate!,upsert, ...) are not run on a relation. - A scope cannot have the name of a relation method (
count,open, ...):ArgumentError.
Regular expressions
A Ruby Regexp works in where, where.not and or. The gem ndb turns it into $regex and $options (the daemon uses RE2). Ruby ^ and $ match at each line, so m is always sent. /i is i, /m is s, /x is x. Look-around, backreferences, atomic groups, possessive quantifiers, \h, \G, \R and \X raise Ndb::InvalidQuery before the request.
Call.where(caller: /^55/)
Call.where(caller: /^55/i).where.not(status: /fail/)
Call.where(caller: /^55/).or(Call.where(caller: /^11/))
Call.where(caller: { "$regex" => "^55" }) # a String: no `m`, and a literal ^prefix uses the indexA Regexp is a scan filter. For a prefix search that uses the index, pass the String form. See the ndb README for the full mapping.
Queries built elsewhere
For a search form or a saved search, take the JSON as is. where_raw takes a where Hash and chains like where, without the Rails conventions: an Array stays an Array (equal to the whole array), where where(tags: [...]) means $in. search takes a whole search body (the query, sort, limit, after and count of POST /{coll}/_search).
form = JSON.parse(params[:q])
Call.where_raw(form).where(dnis: "1000").order(ts: :desc).limit(50)
page = Call.search("query" => form, "sort" => { "ts" => -1 }, "limit" => 50, "count" => true)
page.records # Call records, with their version
page.next_cursor # for "after"; nil on the last page
page.total # with "count" => trueRows decode like any other query. The daemon validates the JSON (invalid_query), but it takes any field: for a public form, keep an allowlist of fields and operators, cap limit, and ask for an indexed field or a time window, as you would before sending a form to Elasticsearch _search.
Collection patterns
A model whose collection_name is a collection pattern reads every matched collection. It declares nothing at boot, and its writes raise Ndb::Error with code pattern_read_only. record.source_collection is the collection a record came from.
class LogArchive < Ndb::Record
self.collection_name = "logs-2026-*"
attribute :at, :datetime, time: true
attribute :lvl, :string
end
LogArchive.where(lvl: "error").order(at: :desc).limit(50).map(&:source_collection)
LogArchive.where(lvl: "error").countCount
Call.where(status: "answered").countCursor pagination
relation = Call.where(status: "answered").order(:_id).limit(100)
rows = relation.to_a
token = relation.next_cursor # nil on the last page
Call.where(status: "answered").order(:_id).limit(100).after(token).to_aA token works only with the same where and order. Another query shape raises InvalidCursor.
To walk every row, use find_each. It follows the cursor in batches (default 1000):
Call.where(status: "answered").find_each(batch_size: 500) { |call| puts call._id }
Call.in_batches(of: 500) { |batch| batch.each { |call| ... } }Lookup
lookup is a key of the HTTP search (POST /{coll}/_search): in the same call, the daemon reads the documents of other collections that each row points to, and puts them in the row. The HTTP API is the primitive and has every rule (Lookup); this SDK only builds that JSON.
| Relation | The foreign key lives | Match | The row gets |
|---|---|---|---|
| belongs to | in this collection | row.fk == other.pk | an object, or null |
| has many | in the other collection | other.fk == row.pk | an array (100 unless a limit, at most 1000) |
| has one | in the other collection | other.fk == row.pk | an object, or null |
pk is _id unless you say otherwise. The field that is searched on the other side (its pk for a belongs to, its fk for a has many) must be _id or indexed, else the daemon answers lookup_unindexed.
In Rails the relations are associations on the model, with the Rails names and conventions, and includes sends them.
Basic use
class User < Ndb::Record
attribute :name, :string
has_many :cdrs # cdrs.user_id -> users._id
has_one :profile # profiles.user_id -> users._id
end
class Cdr < Ndb::Record
attribute :user_id, :string
attribute :duration, :integer
belongs_to :user # cdrs.user_id -> users._id
end
User.where(plan: "pro").includes(:cdrs, :profile).each do |user|
user.cdrs.sum(&:duration) # loaded with the users, in the same call
end
Cdr.order(at: :desc).limit(50).includes(:user).map { |cdr| cdr.user.name }Each association is a method, made by the declaration. Without includes, it loads on first use, with one more call for each record (the N+1 of ActiveRecord):
cdr = Cdr.find("x1")
cdr.user # finds the user now: User.find_by(_id: cdr.user_id)
cdr.user # the same object, no callhas_many is a relation
user.cdrs is an Ndb::Relation, as in ActiveRecord: lazy and chainable. A builder on it is a new query, scoped to the user:
user = User.find("u1")
user.cdrs.where(status: "answered").order(at: :desc).limit(10).to_a
user.cdrs.count # a count in the daemon
user.cdrs.each { |cdr| ... } # loads once, then reads from memory
User.includes(:cdrs).each do |u|
u.cdrs.size # loaded with the users: no call
u.cdrs.where(status: "missed").count # a builder queries again
endStale associations
An association is kept on the record for the key it was read with:
cdr.user # Ana
cdr.user_id = "u2"
cdr.user # the foreign key changed: reads again, Bruno
cdr.reload # reads the document; every association loads again on next useWhat is inferred
Nothing needs to be said when the names follow Rails:
| Declaration | Class | Foreign key | Primary key | Match |
|---|---|---|---|---|
User has_many :cdrs | Cdr ("cdrs".singularize.camelize) | user_id, on each CDR | _id, on the user | cdrs.user_id == users._id |
User has_one :profile | Profile | user_id, on the profile | _id | profiles.user_id == users._id |
Cdr belongs_to :user | User | user_id, on the CDR | _id, on the user | cdrs.user_id == users._id |
The foreign key of a has_many or has_one is named after the model that declares it (User → user_id), and lives on the other side; the one of a belongs_to is named after the association (:user → user_id) and lives on this side. The collection is the collection_name of the class.
When the names differ
A value you pass always wins over the inference:
class User < Ndb::Record
has_many :cdrs, fk: :account_id # cdrs.account_id -> users._id
has_many :calls, class_name: "CDR", fk: :account_id # an acronym class
end
class Cdr < Ndb::Record
belongs_to :caller, class_name: "User", fk: :account_id
belongs_to :plan, fk: :plan_code, pk: :code # plans.code instead of _id
endfk: and foreign_key: are the same option, as are pk: and primary_key:; passing both names of one option is an ArgumentError.
Options
| Option | On | |
|---|---|---|
class_name: | all | the model of the other side |
fk: / foreign_key: | all | the foreign key field |
pk: / primary_key: | all | the field it points to (default _id) |
sort: | has_many, has_one | sort: { at: -1 } |
limit: | has_many | per record, 1 to 1000 (default 100) |
where: | has_many, has_one | filter the other side: where: { status: "answered" } |
dependent: | has_many, has_one | on destroy of this record: :destroy destroys each other record (callbacks run), :delete_all deletes them in one call (no callbacks) |
has_many :recent_cdrs, class_name: "Cdr", sort: { at: -1 }, limit: 20, where: { status: "answered" }Nested
Two levels, as in ActiveRecord:
Order.includes(:customer, items: :product).to_aIndex
has_many and has_one put the foreign key of the other model (cdrs.user_id) in the indexes of that model. So Cdr.where(user_id: id).delete_all works from the first use of Cdr, when the model with the has_many is loaded. With lazy loading (development), declare it on the other model too: attribute :user_id, :string, index: true. A belongs_to with pk: other than _id needs that field indexed on the other model (attribute :code, :string, index: true).
Limits
- Associations read; they do not write: no
user.cdrs << cdr,build,dependent:orcounter_cache. - No
joins(...).where(filtering users by their CDRs); query the CDRs and readuser_id. Ndb::LookupErrorfor a bad relation, a missing index or more than 10 000 documents in one answer.
Aggregate
Relations have simple group calculations. Each one is one aggregate request.
Call.group(:status).count # => {"answered" => 2, "failed" => 1}
Call.group(:status).sum(:duration)
Call.group(:status, :caller).count # keys are arrays
Order.where(status: "paid").unwind(:items).group("items.sku").sum("items.qty") # one row per element of items ($unwind)
Order.unwind(:items).count # the elements, not the orders; unwind is for the aggregates only
Call.where(status: "answered").average(:duration)
Call.minimum(:duration)
Call.maximum(:duration)
Call.sum(:duration)For a full pipeline, use the client. The pipeline needs exactly one $group. Accumulators: $count, $sum, $avg, $min, $max, $first, $last.
page = Ndb::Rails.client.aggregate(Call.collection_name,
pipeline: [
{ "$match" => { "duration" => { "$gte" => 1 } } },
{ "$group" => {
"_id" => {
"status" => "$status",
"hour" => { "$dateTrunc" => { "date" => "$at", "unit" => "hour" } }
},
"n" => { "$count" => {} },
"avg" => { "$avg" => "$duration" }
} },
{ "$sort" => { "n" => -1 } }
],
limit: 100,
count: true)
page.rows # array of Hash
page.total # with count: true
page.next # the cursor of the next page; nil on the last page$dateTrunc units are minute, hour, day, week and month. Optional "timezone": a name such as "Europe/Rome".
Watch
A watch streams the changes of a collection to a named consumer: first the events after its last ack, then new events as they happen. The watch API is on the client (Ndb::Client).
client = Ndb::Rails.client
begin
client.watch(Call.collection_name, "relay", query: { status: "answered" }) do |event|
# event has "pos", "op" ("put", "del" or "truncate"), "_id", "_v" and "doc"
handle(event)
client.ack(Call.collection_name, "relay", event["pos"])
end
rescue Ndb::WatchLagged
retry # the consumer was too slow. watch continues after the last ack.
rescue Ndb::HistoryTruncated
raise # the saved position is too old. Start over.
endquery:takes the same conditions aswhereon a model (Hash and Range). Only matching events come. A delete event has no"doc"; it is sent only if this stream saw an earlier matching put of that id.from: "now"starts a new consumer at the end.limit: nends the stream after n events.- Without a block,
watchreturns an Enumerator. - Delivery is at least once. After a restart an event can come again, so make the work safe to repeat.
HistoryTruncatedmeans the WAL files for the position were deleted (wal_retention).- A watch holds one connection until it ends.
Streams
A stream is an append-only log with consumer groups, with the behavior of Redis Streams. A class in app/streams appends to a stream, consumes it, or does both. There is no base class: the class has include Ndb::Streams. See Streams for the model. Ndb::Stream (singular) is the low-level client of the gem ndb; see the Ruby SDK.
Ndb::Streams needs no Rails: require "ndb/streams" loads it in any Ruby program. Then the client comes from NDB_URL and NDB_TOKEN, or from Ndb::Streams.client = Ndb::Client.new(...).
A writer
# app/streams/order_event.rb
class OrderEvent
include Ndb::Streams
end
OrderEvent.append(order_id: 42) # => "1731164400000-0"- A class without a
performmethod is a writer. - The stream name comes from the class name:
OrderEventisorder_events,Billing::OrderEventisbilling_order_events, andCategoryiscategories.streamsets another name. - The name uses the ActiveSupport inflections (
underscoreandpluralize).ndb/streamsloads them, so the name is the same with and without Rails. An inflection of the app (config/initializers/inflections.rb) changes the name: set the name withstreamwhen it must not change. The class keeps its name after the first call. - A class name needs no suffix.
A consumer
# app/streams/billing_paid.rb
class BillingPaid
include Ndb::Streams
stream :order_events, group: :billing
def perform(event)
Invoice.create!(order_id: event[:order_id])
end
end- A class with a
performmethod is a consumer. It needsgroup:. - Each group gets every entry one time. Give each service (billing, mail, search) its own group. The threads and the processes of one group share the entries.
- One class has one stream and one group. Put shared code in a module.
- A class can do both: it appends with
appendand it consumes withperform. - The consumers run in a process of their own: see Run the consumers.
stream and its options
stream :order_events, group: :billing, concurrency: 2, max_deliveries: 5
stream OrderEvent, group: :billing # the stream of another class
stream group: :billing # the name from the class namestream(name = nil, **options). The name is a Symbol, a String, or a class that has include Ndb::Streams. Every option has a default:
| Option | Default | Meaning |
|---|---|---|
group: | none | The consumer group. Required in a class with perform. |
concurrency: | 1 | The threads of this class in each ndb streams process. |
max_deliveries: | none (no limit) | After this many deliveries, the entry goes to the dead letter and is acked. |
dead_letter: | "<stream>_dead" when max_deliveries or max_age is set | The stream that gets the poison entries. false: no dead letter stream. |
claim_idle: | 1.minute | An entry pending for this long is tried again. Seconds or a duration. |
retry: | none | The wait before the next try of a failed entry: an Array of seconds or durations (the last one repeats), or a callable ->(deliveries) { seconds }. See Delayed retry. |
max_age: | none | Seconds or a duration. An entry older than this (by the time of its ID) goes to the dead letter (reason max_age) and is acked, with no retry. |
start: | :new | Where a new group starts. :new: only the entries appended after the group is made. :all: from the start of the stream. |
discard_on: | [] | Exception classes. An error of these classes acks the entry without a retry. |
max_len: | none (keep everything) | Keep only the newest N entries. It is an exact trim on each append. |
retain: | none (keep everything) | Delete the entries older than this: "30d", 30.days, or seconds. It is the retention of the collection. |
- An option that is not in the table, or a bad value, raises
ArgumentErrorwhen the class loads. - A second
streamcall in the same class (or in a subclass) merges its options with the first ones. start:is read only when the group is made. A change ofstart:does not move a group that exists.OrderEvent.stream_nameandOrderEvent.stream_optionsgive the name and every option with its value.
append
OrderEvent.append(order_id: 42) # one entry => its ID
OrderEvent.append([{ order_id: 1 }, { order_id: 2 }]) # many entries => their IDs
OrderEvent.append('{"order_id":42}') # a JSON string (an object or an array)
OrderEvent.enqueue(order_id: 42) # the same method
OrderEvent.perform_async(order_id: 42) # the same method- The payload is a Hash (one entry), an Array of Hash (many entries, in one request) or a JSON string of one of the two. Any other value raises
ArgumentError. An empty Array returns[]and sends nothing. enqueueandperform_asyncare aliases ofappend.- The payload is JSON. A Time or a Date is written as a String, and it comes back as a String. See The contract between languages.
- A field name that starts with
_is reserved. - With
max_len:, each append trims the stream.
In a transaction. In an ActiveRecord transaction, the append waits for the commit. A rollback drops it.
Order.transaction do
order.update!(status: "paid")
OrderEvent.append(order_id: order.id) # => nil; sent after the commit
end- In a transaction,
appendreturnsnil: the entry has no ID yet. - This needs Rails 7.2 or newer (
ActiveRecord.after_all_transactions_commit). With an older Rails, or without ActiveRecord, the append is sent at once. - It is not an outbox. The append is a request after the commit: if the process dies between the commit and the request, the entry is lost. If the daemon is down, the error comes from the
commit, after the data of the transaction is saved.
last
OrderEvent.last # the newest entry, or nil
OrderEvent.last(10) # the 10 newest entries, oldest first- Each entry is an event.
last,findandrangeread without a group: they change no group, andevent.deliveriesisnil. - A stream that does not exist gives
nilor[].
find
OrderEvent.find("1731164400000-0") # the entry with this ID, or nilrange
OrderEvent.range # every entry, oldest first
OrderEvent.range(1.hour.ago, Time.now, count: 100)
OrderEvent.range("1731164400000-0", "+")range(from = "-", to = "+", count: nil): the entries fromfromtoto, both included, oldest first.- Each end is an entry ID or a Time. A Time as
fromis the first ID of that millisecond; astoit is the last ID of that millisecond. count:is the most entries. Without it, every entry of the range comes.- A stream is also a collection: for a filter on the fields, use
Ndb::Rails.client.searchon the stream name.
Stream count
OrderEvent.count # => 1280- How many entries the stream has now. A stream that does not exist gives 0.
perform and the event
def perform(event)
event[:order_id] # one field, by Symbol or String
event.key?(:total) # true when the entry has the field
event.to_h # the payload, without "_id" and "_ts"
event.id # "1731164400000-0"
event.time # the Time of the ID (UTC)
event.deliveries # 1 on the first delivery, 2 on the first retry
end- The runner makes a new instance of the class for each entry and calls
perform(event).performcan be private. performthat returns acks the entry.performthat raises leaves it pending: see Failure and retry.- In a Rails app, each
performruns in the Rails reloader (Rails.application.reloader.wrap), so a changed class loads again in development and the ActiveRecord connections go back to the pool. The options (discard_onand the others) come from the class after the reload. event.to_his a copy. With ActiveSupport, it reads a Symbol and a String as the same key.- There are no typed fields. The payload comes as JSON gives it: a number is a number, a date is a String.
- Delivery is at least once: after a crash or a retry, an entry can come again. Make
performsafe to repeat. - A deleted entry (a trim or a delete while it was pending) is acked without a
perform.
perform_now
Runs perform on one payload now, in the caller's thread, with no request: nothing is written to ndb and nothing is acked. It takes the path of a delivery (the Rails reloader, around_perform, discard_on) and returns the value of perform (nil when discard_on took the error). Use it in a unit test, or to run one entry by hand in a console.
PostCallJob.perform_now(call_id: 42) # the fields as keywords
PostCallJob.perform_now({ "call_id" => 42 }) # a Hash
PostCallJob.perform_now('{"call_id":42}') # a JSON string
PostCallJob.perform_now(call_id: 42, _id: "1731163600000-0", _deliveries: 3)_idand_deliveriessetevent.idandevent.deliveries(default: an ID of now, and 1). They are not payload fields: a name that starts with_is reserved in a stream.- An error of
perform(not one ofdiscard_on) goes to the caller. There is no retry, no dead letter and no group. - A class without
perform(a writer) raisesArgumentError, and so does an Array (one entry only). - To test the whole path, with the append and the consumers of the stream, use
Ndb::Streams.inline!(see Test the streams).
around_perform
class BillingPaid
include Ndb::Streams
stream :order_events, group: :billing
around_perform do |event, block|
Current.set(tenant: event[:tenant]) { block.call }
end
def perform(event) = Invoice.create!(order_id: event[:order_id])
end- The block runs around each
perform, on the instance (selfis the consumer object). It gets the event and a callable.block.callcontinues. Withoutblock.call,performdoes not run and the entry is acked. - Many
around_performblocks run in the order of their declaration; the first one is the outer one. around_performis the only hook. There is nobefore_performorafter_perform.
Failure and retry
stream :order_events,
group: :billing,
claim_idle: 30.seconds,
max_deliveries: 5,
discard_on: [ActiveRecord::RecordNotFound]- An error in
performleaves the entry pending. The error goes to the log (Rails.logger), and the thread continues with the next entry. Any exception is an error here, also one that is not aStandardError(NotImplementedError,SystemStackError). OnlySignalException(withInterrupt) andSystemExitgo on and stop the thread. - Retry. Without
retry:, a thread of the group claims the entries that are pending forclaim_idleor longer, of any consumer, and runs them again. It looks for them everyclaim_idle / 2(at least 1 second). So a retry comes betweenclaim_idleand 1.5 timesclaim_idleafter the failure. Withretry:, the wait comes from the schedule (see Delayed retry). discard_on. An error of one of these classes (or of a subclass) is a warning in the log, and the entry is acked. Nothing goes to the dead letter.max_deliveries. An entry that fails on itsmax_deliveries-th delivery is poison. At that failure, the thread appends the entry to the dead letter stream, acks it and writes a warning. Withmax_deliveries: 5,performruns 5 times at most.- The dead letter is the stream
<stream>_dead(or the name indead_letter:). Each entry of it is{stream, group, id, deliveries, reason, doc};reasonis"max_deliveries"or"max_age". Withdead_letter: false, a poison entry is only acked and logged. - After a restart, each thread reads its own pending entries first. That read counts as one delivery, as in Redis.
- A crash in
perform. Whenperformstops the process (an out-of-memory kill, a segfault), the entry is not acked and gets no failure. Withmax_deliveries, the read of the pending entries after the restart sends an entry with more thanmax_deliveriesdeliveries to the dead letter, withoutperform. So a poison entry that crashes the process stops aftermax_deliveriesruns.
Ndb::Rails.client.stream("order_events_dead").range.each do |entry|
entry.doc # {"stream" => "order_events", "group" => "billing", "id" => "...", "deliveries" => 5, "reason" => "max_deliveries", "doc" => {...}}
endLimits:
- Without
max_deliveriesormax_age, an entry that always fails is tried again with no end. - The dead letter has no consumer of its own. Read it with the low-level client, or write a class with
stream :order_events_dead.
Delayed retry
ndb only (it uses nack of the daemon; see the Ruby client). The fsw-esl outbox schedule: try 1 waits 15 s, tries 2 to 5 wait 1 min, 6 to 11 wait 10 min, 12 to 23 wait 1 h, then 6 h, for at most 15 days:
# app/streams/post_call_job.rb
class PostCallJob
include Ndb::Streams
stream :postcall_jobs,
group: :jobs,
retry: [
{ wait: 15.seconds, up_to: 1 },
{ wait: 1.minute, up_to: 5 },
{ wait: 10.minutes, up_to: 11 },
{ wait: 1.hour, up_to: 23 },
{ wait: 6.hours }
],
max_age: 15.days
def perform(event)
PostCall.deliver(event[:call_id])
end
end- A failed
performgives the entry back to the group after the wait for its delivery count: 15 s after the first failure, 1 min after failures 2 to 5, and so on; 6 h after each failure from the 24th. An entry that always fails runs 81 times in 15 days. A plain list (retry: [15.seconds, 1.minute, 6.hours]) is one wait for each try, and the last one repeats. It does not come back before that wait, also after a restart, and any thread of the group takes it when it is due (in about 1 second). retry:also takes a callable,retry: ->(deliveries) { 2**deliveries }, orNdb::Stream.tiers(...)(with steps, see the Ruby client).max_age: 15.days: an entry older than 15 days goes topostcall_jobs_deadwith the reasonmax_age, withoutperform. An entry that fails when it is older than that goes there at once, with no more wait.- With
max_deliveries:too, an entry that fails on its last delivery goes to the dead letter, not to a retry. discard_on:still acks without a retry.bundle exec ndb streamswrites the schedule of each consumer to the log at the start:PostCallJob retry: 15s x1, 1m x4, 10m x6, 1h x12, then 6h; max age 15d. A consumer withoutretry:writesretry: after claim_idle (1m).- The dead letter is a normal stream, and nothing delivers its entries again. Send one back with
append(a new entry: new ID, delivery count 0, new max age):
dead = Ndb::Rails.client.stream("postcall_jobs_dead")
dead.range.each do |entry|
next unless entry.doc["reason"] == "max_age"
PostCallJob.append(entry.doc["doc"])
dead.delete(entry.id)
endFor a rule of the app, a class can consume the dead letter: stream :postcall_jobs_dead, group: :replay with a perform that appends event[:doc] to the source. See The dead letter.
- Inline mode (
Ndb::Streams.inline!) runsperformone time, in the caller's thread. A failure raises to the caller, as withoutretry:: the schedule andmax_agedo not apply inline.
ApplicationStream
# app/streams/application_stream.rb
class ApplicationStream
include Ndb::Streams
stream claim_idle: 30.seconds, max_deliveries: 5
around_perform { |event, block| Current.set(tenant: event[:tenant]) { block.call } }
end
class BillingPaid < ApplicationStream
stream :order_events, group: :billing
def perform(event) = Invoice.create!(order_id: event[:order_id])
end- A subclass starts with the stream name, the options and the
around_performblocks of its parent. Its ownstreamcall merges with them. - A parent without a stream name gives none: the name of each subclass comes from its own class name.
- A subclass of a consumer class is a consumer too, with the same stream and group. Give it its own
group:, or the two classes are aDuplicateGroup.
Run the consumers
bundle exec ndb streams # every consumer class of app/streams
bundle exec ndb streams --only BillingPaid -c 4 # one class, 4 threads
bundle exec ndb streams -e production
bundle exec ndb streams -r ./consumers.rb # without RailsThe ndb command comes with the gem ndb; streams needs the gem ndb-rails. In a Rails app (a directory with config/environment.rb), it loads the app, then loads the classes of app/streams in its own thread (in the reloader of the app) before it reads them, also when the app does not eager load. It starts the threads; each thread makes its group, and the command runs until a signal.
| Flag | Meaning |
|---|---|
-r, --require PATH | A file that loads the stream classes. Required without Rails. With Rails, it loads after the app. |
-e, --environment NAME | Sets RAILS_ENV. |
--only A,B | Run only these consumer classes. A name that is not a consumer class is an error. |
-c, --concurrency N | The threads of each class. It replaces concurrency: of every class. |
-h, --help | Print the help. |
- A wrong flag, or a wrong config of a class, prints
ndb streams: <message>and exits with status 2. Any other error at the boot of the app (also anArgumentError) is raised with its backtrace (status 1). - Stop.
SIGTERMandSIGINTstop the process: each thread ends its current entries, acks them and stops. A read waits 2 seconds at most, so an idle process stops in about 2 seconds. A signal during the start starts no more threads. - A second signal. In the first 2 seconds after the first signal, a second signal does nothing (a terminal and Puma can send two). After that, a second signal interrupts the current
performcalls: the entries done before them are acked, and the interrupted entries stay pending. A third signal ends the process at once, with status 1. - The daemon is down. Each thread makes its group, then reads. When the daemon is down, the thread writes the error to the log and tries again, with a wait of up to 5 seconds. The process does not exit, also when the daemon is down at the start.
- A thread that ends. When a thread ends before the stop (an exception outside
perform, aThread#kill), the process writes an error and starts the thread again. - Consumer names. Each thread is a consumer with a stable name:
NDB_CONSUMER(default: the hostname), then:and the thread index (web-1:0). After a restart with the same name, a thread reads its own pending entries first. Two processes on one host must have differentNDB_CONSUMERvalues. With the same name they are one consumer for the daemon, and each one also reads the pending entries of the other at its start. - Any other command (
bundle exec ndb serve, ...) goes to the daemon binary: see The ndb executable.
Defaults in config/ndb.yml. The streams: block has two keys:
production:
streams:
concurrency: 4
only: [BillingPaid]concurrencyreplacesconcurrency:of every class, as-cdoes.onlyis the default of--only. A flag wins over the file.- The file says nothing about what a process does: the command does that.
With Puma. The plugin runs the same command next to Puma:
# config/puma.rb
plugin :ndb- After Puma boots, the plugin starts
bin/ndb streamswhen the app has abin/ndbfile, orbundle exec ndb streams. It is a child process, not threads in Puma. - The process has its own process group, so a Ctrl-C in the terminal goes to Puma only. When Puma stops or restarts, the plugin sends one
SIGTERMto the process, waits up to 30 seconds, then kills it. - When the consumers stop by themselves, Puma stops too. So a broken consumer is not hidden.
- Without
NDB_STREAMS_IN_PUMA, the plugin runs only in development (RAILS_ENV,RACK_ENVorAPP_ENV; none of them is development).NDB_STREAMS_IN_PUMA=true(or1,yes,on) turns it on, andfalse(or0,no,off) turns it off. Any other value is an error at the start of Puma. - In production, a process of its own (
bundle exec ndb streams) is the usual choice: it scales and restarts apart from the web process.
Groups at boot
- Any process of the Rails app (web, console, a job worker) loads
app/streamsat boot and makes the missing groups, in a background thread. The boot does not wait for ndb. - So a group exists before the first
ndb streamsprocess runs, and a group withstart: :newgets the entries that the web process appends from then on. - When the daemon is down, the thread tries 5 times (it waits 1, 2, 4 and 8 seconds), then writes a warning.
ndb streamsmakes the groups again at its start, and does not wait for this thread: it loadsapp/streamsitself. - A stream with
retain:is made here too, with its retention. A stream withoutretain:is made by the first append or by its first group. - Limit: with
start: :new, the entries appended before the group is made are not delivered to it. For a new consumer of an old stream, usestart: :all. - Without Rails,
Ndb::Streams.ensure_groups!makes them.
Notifications
ActiveSupport::Notifications gets two events:
| Event | Payload |
|---|---|
append.ndb_streams | stream (the name), class, count (the entries of this append) |
perform.ndb_streams | stream, group, class, event |
ActiveSupport::Notifications.subscribe("perform.ndb_streams") do |name, start, finish, _id, payload|
Rails.logger.info("#{payload[:class]} #{payload[:event].id} #{((finish - start) * 1000).round} ms")
endperform.ndb_streamscoversperformand itsaround_performblocks. Whenperformraises, the payload hasexception, as for every notification.
Test the streams
# test/test_helper.rb
require "ndb/rails/minitest"
Ndb::Streams.inline!PostCallJob.perform_now(...)runs one consumer on one payload, with no ndb: see perform_now.Ndb::Streams.inline!: an append writes the entry, then callsperformof each consumer class of that stream, in the same thread. Nondb streamsprocess is necessary.Ndb::Streams.inline = falseturns it off.- In inline mode, an error in
performgoes to the caller ofappend(an error ofdiscard_onis only logged).event.deliveriesis 1. There is no retry and no dead letter, and the groups are not used. - The first inline
appendloadsapp/streams(in the reloader of the app), so the consumers are there also right after the boot, when the app does not eager load. - In a transaction, the
performcalls run after the commit, with the append. ndb/rails/minitestandndb/rails/rspectruncate the streams that a test appended to, with the collections (see Testing). The stream names get the prefix of the parallel worker.
class CheckoutTest < ActiveSupport::TestCase
test "it appends the event" do
Checkout.call(order)
assert_streamed OrderEvent, order_id: order.id
assert_no_streamed OrderEvent, order_id: 0
end
endassert_streamed(klass, **fields)passes when the stream of the class has an entry with these fields. Without fields, any entry passes. It returns the events that match.assert_no_streamed(klass, **fields)passes when no entry has these fields.- The values are compared as JSON: a Symbol is equal to its String.
- The two read the newest 1 000 entries of the stream.
# spec/rails_helper.rb
require "ndb/rails/rspec"
expect(OrderEvent).to have_streamed(order_id: 42)
expect(OrderEvent).not_to have_streamed(order_id: 0)Stream errors
| Class | When |
|---|---|
Ndb::Streams::MissingGroup | A class has perform and its stream has no group:. |
Ndb::Streams::DuplicateGroup | Two consumer classes have the same group on one stream. |
Ndb::Streams::ConfigurationError | The parent of the two. Also: a class without a name and without stream :name. |
- All are subclasses of
ArgumentError. - The two group errors are not raised when the class loads: the check runs when the consumers are listed.
ndb streamsprints the message and exits with status 2. The boot of another process writes a warning. In inline mode, theappendraises. - An unknown option or a bad value of
streamraisesArgumentErrorwhen the class loads. - The errors of the daemon are the
Ndb::Errorclasses of the gemndb: see Stream errors of the Ruby SDK.
Blobs
Large values live in the blob store. The document keeps only a reference ($blob, size, type).
call = Call.new(_id: "c1", status: "answered", caller: "a",
recording: File.open("call.wav")) # an IO or a String
call.save! # uploads the blob, then saves
call.recording.ref # "sha256:..."
call.recording.size
loaded = Call.find("c1")
loaded.recording.read # all the bytes
loaded.recording.each_chunk { |chunk| out.write(chunk) } # 1 MiB rangesA blob behaves like a read-only IO (read, each_chunk, eof?, rewind). A loaded blob reads the daemon in 1 MiB ranges, so memory stays small. The same content always gives the same reference. You can also use Ndb::Blob.new(io_or_string, content_type: "audio/wav").
Method reference
Every public method of the gem, grouped. The names and return values follow ActiveRecord; what is not in these lists does not exist (no joins, transaction, dependent:, migrations).
Declare a model (class methods)
| Method | |
|---|---|
attribute :name, :type, index:, unique:, time:, expires:, encrypted:, blind: | an ActiveModel attribute; index/unique declare an index or a unique key; time is the time field; expires holds the expiry of each document; encrypted/blind make it an encrypted field |
self.collection_name = "calls" / collection_name | the collection (default: the plural model name). A pattern ("logs-2026-*") reads many collections and writes none |
ttl 24.hours | documents expire this long after the time field |
source "/var/log/app/*.ndjson" | a direct collection: the app files are the data, ndb only reads |
belongs_to, has_many, has_one | associations, see Lookup |
self.abstract_class = true / abstract_class? | a base class with no collection |
primary_key | always "_id" |
time_field, expires_field, indexed_fields, unique_fields, collection_definition | what the model declares |
collection_pattern? | true for a pattern |
ensure_collection! | declares the collection in the daemon (done by itself on first use, and at boot by the Railtie) |
Query (on the model and on a relation)
| Method | Returns |
|---|---|
all | a relation of every document |
where(hash), where.not(hash), where_raw(hash), or(relation) | a relation. A Range is $gte/$lte, an Array is $in, nil is a missing field; where_raw sends the Hash as is |
order(:a, b: :desc), order("a, b DESC"), limit(n), offset(n), page(n), per(n), after(cursor) | a relation. A String order takes only "field" and "field asc|desc", split by commas; a SQL expression raises ArgumentError |
includes(:a, b: :c) | a relation that loads associations in the same call |
reorder(...), rewhere(hash), unscope(:where, :order, :limit, :offset, :page, :after, :group, :includes, :select, :distinct) | a relation with that part replaced or removed |
select(:a, :b), distinct | records with only these attributes (and _id) loaded; distinct.pluck gives each value once. select { } with a block is Enumerable#select |
none | an empty relation: no request, [], 0, writes 0 |
group(*fields) | a relation for count/sum/... by group |
find(id), find(id1, id2) | a record, or an Array; Ndb::NotFound when missing |
find_by(hash), find_by!(hash), find_by_<attr>[_and_<attr>](values) | a record or nil (!: Ndb::NotFound) |
first!, last!, take, take(n), take! | as first/last; take is in no given order; ! raises Ndb::NotFound |
sole, find_sole_by(hash) | the one record; Ndb::NotFound for none, Ndb::SoleRecordExceeded for more |
ids, pick(:a, :b) | the primary keys; the pluck of the first row (or nil) |
to_ndb, to_ndb(pretty: true) | the JSON body of the search, as a String; no request |
first, first(n), last, last(n) | a record or nil, or an Array. The daemon gives only n rows. With no order, the order is _id (first and last are its two ends). On a relation with limit, offset or page, or a loaded one, last uses the loaded rows. |
to_a, each, map... (Enumerable), load, loaded?, records | the records (one call, then memory) |
size, length, empty? | from the loaded records, else a count |
count, exists?, pluck(:a, :b) | a number, a boolean, an Array of values. On a relation with limit, offset, page or after, count counts only those rows (limit(2).count is at most 2) |
sum(:f), average(:f), minimum(:f), maximum(:f) | a number (or a Hash by group) |
find_each(batch_size:), in_batches(of:) | every record, page by page with the cursor |
next_cursor | the cursor of the next page |
search(body) | a raw search body; SearchResult (records, next_cursor, total) |
to_query | the JSON body the relation sends |
Write
| Method | |
|---|---|
Model.create(attrs), create!(attrs), create([attrs, ...]) | insert with validations and callbacks (! raises) |
Model.upsert(attrs), upsert_all([attrs, ...]) | create or merge by _id, no validations or callbacks |
Model.find_or_create_by(hash), find_or_create_by!(hash) | atomic (an upsert) |
Model.find_or_initialize_by(hash) | a found record, or a new one not saved |
Model.destroy(id), Model.delete(id) | by id, with or without callbacks |
relation.new(attrs), build, create, create!, find_or_create_by, find_or_initialize_by | a record with the equality attributes of the where, merged with attrs |
Model.destroy_by(hash), Model.delete_by(hash) | where(hash).destroy_all / .delete_all |
relation.update(attrs or ops), update_all | one atomic update on every match: attributes (status: "x", a $set), operators ("$inc" => { "n" => 1 }), or both; no callbacks; the number modified |
relation.delete_all, relation.destroy_all | delete every match (no callbacks / with callbacks). With limit, offset, page or after, update and delete_all write only those rows |
Model.truncate!, Model.drop! | empty the collection / remove it (keeps / removes the definition) |
A record (instance methods)
| Method | |
|---|---|
save(validate: true, expire_in: nil), save! | insert or update the changed fields; Ndb::StaleObjectError when another write changed it |
update(attrs), update!(attrs) | assign and save |
destroy, destroy!, delete | remove, with or without callbacks |
reload | read the document again: attributes, version, associations |
increment!(:n, by), decrement!(:n, by) | an atomic $inc |
expire(30.minutes), persist_ttl!, ttl | the expiry of this document |
create_unless_exists(attrs), create_unless_exists!(attrs) | insert only when the _id is free |
id, id=, _id | the primary key (id is an alias) |
version | _v, the version the record was read with |
persisted?, new_record?, destroyed?, to_key, == | state and identity, like ActiveRecord |
changed?, changes, <attr>_changed?, valid?, errors, as_json, to_json | from ActiveModel (Dirty, Validations, Serializers) |
source_collection | the collection a record came from, when the model reads a pattern |
customer, items, ... | one method per association: a record or nil (belongs_to, has_one), a relation (has_many) |
Blobs
| Method | |
|---|---|
attribute :recording, :blob | a large value stored apart, referenced by the document |
Ndb::Blob.new(io_or_string, content_type:) | a blob to assign; it uploads on save |
blob.read(n), each_chunk, rewind, eof? | stream the content |
blob.uploaded?, upload!, to_reference | its state and its sha256: reference |
Tests
Ndb::Testing.clean!, clean_all!, worker!, touched; ndb/rails/minitest (Minitest), ndb/rails/rspec, ndb/rails/database_cleaner. See Testing.
The low layer
Ndb::Rails.client is the Ndb::Client of the gem: watch, ack, aggregate, restore, rename, pitr, ensure_index and every route of the HTTP API. See the Ruby SDK.
Errors
The transport and daemon errors are defined by the gem ndb. They all live in the Ndb module (Ndb::Error, Ndb::NotFound, ...). For an ActiveRecord-like rescue, this gem adds one alias: Ndb::RecordNotUnique is Ndb::UniqueViolation (the same class). Ndb::StaleObjectError, Ndb::RecordInvalid, Ndb::RecordNotSaved and Ndb::RecordNotDestroyed exist only here, and they inherit from Ndb::Error. Every error has code (the daemon code) and status (the HTTP status).
| Class | Code |
|---|---|
Unauthorized | unauthorized |
NotReady | not_ready |
NotFound | not_found |
UnknownCollection | unknown_collection |
StaleObjectError | stale_object (raised by the gem, not sent by the daemon) |
InvalidUpdate and its subclasses | unknown_operator, no_update, conflicting_update, invalid_field, unindexed_many, empty_filter |
RecordNotUnique | unique_violation, duplicate_id |
CatalogConflict | catalog_conflict |
CatalogInvalid | catalog_invalid |
ReadOnlyCollection | direct_collection |
InvalidQuery | invalid_query, invalid_request, invalid_pipeline |
InvalidCursor | cursor_invalid, cursor_mismatch, cursor_expired |
WatchLagged | watch_lagged |
HistoryTruncated | history_truncated |
TtlNotEnabled, TtlDerived | ttl_not_enabled, ttl_derived |
RecordInvalid | ActiveModel validation failed (has record) |
RecordNotSaved | save! returned false without errors, or reload/update of a record that is not saved |
RecordNotDestroyed | destroy! when a before_destroy callback aborts (has record) |
ConnectionError | cannot reach the daemon. Subclasses: ConnectError (nothing was sent), TimeoutError |
The gem ndb maps more codes to classes (Forbidden, InvalidDocument, Closed, ...); the full list is on Ruby errors. Any other code is a plain Ndb::Error. A response that is not a daemon error has the code http_status.
rescue Ndb::RecordNotUnique => e
e.code # "unique_violation"
e.status # 409Testing
ndb has no transactions, so a test cannot roll back like use_transactional_tests. Instead, the gem truncates the collections a test used, after it: a few ms per collection whatever its size, and the definition and indexes stay. It knows what a test used from the models (every query and write of a model marks its collection).
Minitest
# test/test_helper.rb
ENV["RAILS_ENV"] ||= "test"
require_relative "../config/environment"
require "rails/test_help"
require "ndb/rails/minitest"Every ActiveSupport::TestCase (models, controllers, integration, system tests) cleans after each test. To keep the data of a test case:
class ImportTest < ActiveSupport::TestCase
self.ndb_clean = false
endRSpec
# spec/rails_helper.rb
require "ndb/rails/rspec"Each example cleans after itself. ndb_clean: false on an example or a group keeps its data:
it "imports the file", ndb_clean: false do
# ...
endDatabaseCleaner
For a suite that already uses DatabaseCleaner (2.x, the database_cleaner-core gem):
require "ndb/rails/database_cleaner"
DatabaseCleaner[:ndb].strategy = :touched # the default: what the test used
DatabaseCleaner[:ndb].clean_with(:truncation) # every collection, before the suite
DatabaseCleaner[:ndb].strategy = :truncation, { except: %w[plans] }
# RSpec
config.before(:suite) { DatabaseCleaner[:ndb].clean_with(:truncation) }
config.around { |example| DatabaseCleaner[:ndb].cleaning { example.run } }| Strategy | Cleans |
|---|---|
:touched | the collections the tests used since the last clean (fast; the default) |
:truncation | every engine collection of this worker, or only:; less except: |
Both truncate: ndb has no :transaction or :deletion strategy.
Parallel tests
The daemon is shared by every worker, so each one gets collections of its own: the models read and write test<n>_calls instead of calls.
- Rails
parallelize:ndb/rails/minitestsets it inparallelize_setup. - parallel_tests (
TEST_ENV_NUMBER):ndb/rails/minitestandndb/rails/rspecread it. - By hand:
Ndb::Testing.worker!(2), orconfig.collection_prefix = "ci_"inNdb::Rails.configure. Without it,collection_prefixisNDB_PREFIX.
The prefix covers the models (and their associations). A collection written with the raw client (Ndb::Rails.client) is not prefixed and not cleaned unless you pass it: Ndb::Testing.clean!(only: %w[raw_events]).
The API
| Method | |
|---|---|
Ndb::Testing.clean!(only: nil, except: []) | truncate what the tests used since the last clean (or only these) |
Ndb::Testing.clean_all!(except: []) | truncate every engine collection of this worker (its prefix) |
Ndb::Testing.worker!(n) | collections of their own for worker n (test<n>_) |
Ndb::Testing.touched | the collections used since the last clean |
- Safeguard: a clean refuses to run outside the test environment (
Ndb::Testing::UnsafeClean), so a test helper loaded by mistake against the development daemon deletes nothing.NDB_ALLOW_CLEAN=1forces it. - Direct collections (
source) and patterns are never cleaned: their data is the app files. - The test daemon: run it with
pitr: off. A truncate with PITR on keeps the old data in the trash for the window, so a long suite would fill the disk. A data dir in a temp dir (-data-dir "$(mktemp -d)") starts each run empty.-tcp ''turns off the default TCP listener (127.0.0.1:7070), so the test daemon does not take the port of your local daemon.
ndb serve -config test/ndb.yml -data-dir "$(mktemp -d)" -unix /tmp/ndb-test.sock -tcp '' # test/ndb.yml: pitr: offThe tests of this gem
The tests of this gem build the daemon from the repository (go build ./cmd/ndb), start it on a short temp unix socket (macOS allows 104 bytes), wait for GET /_health to return 200, and stop it at exit. Run them with bundle exec rake test in sdk/rails; its Gemfile uses the gem ndb from ../ruby.