honker
honker is a SQLite extension + language bindings that add
Postgres-style NOTIFY/LISTEN semantics to SQLite, with built-in
durable pub/sub, task queue, and event streams, without client polling
or a daemon/broker. Any language that can
SELECT load_extension('honker') gets the same features.
honker replaces queue-table polling with a single-digit-microsecond
PRAGMA data_version read. The default watcher checks every 1 ms,
giving push-like semantics and single-digit-millisecond cross-process
delivery; raise the watcher interval when lower idle CPU matters more
than lowest-latency wakeups.
If SQLite is your primary datastore, the queue should live in the same
file. INSERT INTO orders and queue.enqueue(...) can commit in the
same transaction. Rollback drops both.
See honker.dev for guides and API details, and Binding support for what each binding supports.
Simon Willison highlighted honker as a SQLite implementation of the transactional outbox pattern.
Alpha software. Better than experimental, not beta-quality yet.
Quick Start
pip install honkerimport honker
db = honker.open("app.db")
emails = db.queue("emails")
with db.transaction() as tx:
tx.execute("INSERT INTO orders (user_id) VALUES (?)", [42])
emails.enqueue({"to": "[email protected]"}, tx=tx)
async for job in emails.claim("worker-1"):
send_email(job.payload)
job.ack()The enqueue is atomic with the order insert. A worker in another process wakes when the transaction commits.
What It Does
- Notify/listen across processes on one SQLite
.dbfile - Durable at-least-once queues with retries, delayed jobs, priority, visibility timeouts, dead-letter rows, and task result storage
- Durable streams with per-consumer offsets
- Time-trigger scheduling with cron and
@every <duration>expressions - Named locks, rate limits, and transactional outbox helpers
- SQL functions through a SQLite loadable extension
- Thin bindings for Python, Node.js, Rust, Go, Ruby, Bun, Elixir, C++, .NET / C#, Java/JVM, and Kotlin
Deliberately not included: workflow DAGs, task chains/groups/chords, multi-writer replication, or distributed locking across machines.
Why
SQLite is increasingly the database for shipped projects. Those projects eventually need pub/sub and a task queue. The usual answer is "add Redis
- Celery." That works, but it introduces a second datastore with its own backup story, a dual-write problem between your business table and the queue, and a broker to run.
Honker takes the approach that if SQLite is the primary datastore, the queue should live in the same file. The queue is just rows in a table with a partial index. Every binding uses the same schema and extension, so one language can enqueue work and another can claim it.
Design
Honker is built around three pieces:
- ephemeral pub/sub with
notify()/listen() - durable streams with per-consumer offsets
- at-least-once queues with visibility timeouts and retries
All three are INSERTs inside your transaction. Put queue.enqueue(...),
stream.publish(...), or notify(...) beside the write that created the
work. Commit lands both rows. Rollback drops both rows.
SQLite has no server-side push channel, so honker uses a shared watcher.
The stable backend reads PRAGMA data_version every millisecond; when
the counter changes, listeners re-read indexed SQLite state.
If you use your app's existing SQLite file, honker wakes workers on every commit to that file. Most wakes will not find work for a given queue or channel. That overtriggering is on purpose: one indexed SELECT is cheap, while a missed wake is a correctness bug. The stable semantics are:
- wake on committed updates
- ignore rolled-back work
- re-read SQLite state after every wake
- use file-backed SQLite databases, not
:memory:
Optional source-build backends also exist for kernel file events and WAL shared-memory reads. See Binding support for which bindings expose backend options and what CI proves.
Honker is single-machine and file-backed. SQLite's locking model is
designed for one host writing one database file; two servers writing the
same .db over NFS is not a Honker deployment strategy.
Prior Art
pg_notify
gives Postgres fast triggers, but no retry or visibility timeout.
pg-boss and
Oban are the Postgres-side gold standards
we're chasing on SQLite. Huey is an
excellent SQLite-backed Python task queue. If you already run Postgres,
use the Postgres tools, as they are excellent.
The transactional outbox idea also owes a lot to Brandur Leach's Transactionally Staged Job Drains in Postgres: write the job row in the same transaction as the business row, then let a worker deliver it after commit.
Bindings
| Ecosystem | Package / path | Notes |
|---|---|---|
| Python | pip install honker |
Batteries-included package; includes the Python API and loadable extension in release wheels |
| Node.js | npm install @russellthehippo/honker-node |
Native Node binding |
| Ruby | gem install honker |
Native gem with precompiled platforms where available |
| .NET / C# | dotnet add package Honker |
NuGet package with bundled runtime assets |
| Rust | honker, honker-core, honker-extension |
Core engine and Rust wrapper |
| Elixir | Hex package honker |
Extension-backed Elixir binding |
| Go, Bun, C++, JVM, Kotlin | in packages/ |
Maintained in-tree bindings |
| SQLite | honker-extension |
Loadable extension for any SQLite 3.9+ client |
The detailed parity table lives in BINDINGS.md. Language-specific install and API notes live in each package README.
SQL Extension
Any SQLite client that can load extensions can use honker directly:
.load ./libhonker_ext
SELECT honker_bootstrap();
INSERT INTO _honker_live (queue, payload) VALUES ('emails', '{"to":"alice"}');
SELECT honker_claim_batch('emails', 'worker-1', 32, 300); -- JSON array
SELECT honker_ack_batch('[1,2,3]', 'worker-1'); -- DELETEs; returns count
SELECT honker_sweep_expired('emails'); -- count moved to dead
SELECT honker_lock_acquire('backup', 'me', 60); -- 1 = got it, 0 = held
SELECT honker_lock_release('backup', 'me'); -- 1 = released
SELECT honker_rate_limit_try('api', 10, 60); -- 1 = under, 0 = at limit
SELECT honker_rate_limit_sweep(3600); -- drop windows >1h old
SELECT honker_cron_next_after('0 3 * * *', unixepoch()); -- 5-field cron
SELECT honker_cron_next_after('*/2 * * * * *', unixepoch()); -- 6-field cron
SELECT honker_cron_next_after('@every 5s', unixepoch()); -- interval schedule
SELECT honker_scheduler_register('nightly', 'backups',
'0 3 * * *', '"go"', 0, NULL); -- periodic task
SELECT honker_scheduler_tick(unixepoch()); -- JSON: fires due
SELECT honker_scheduler_soonest(); -- min next_fire_at
SELECT honker_queue_next_claim_at('emails'); -- next run/reclaim deadline
SELECT honker_stream_publish('orders', 'k', '{"id":42}'); -- returns offset
SELECT honker_stream_read_since('orders', 0, 1000); -- JSON array
SELECT honker_stream_save_offset('worker', 'orders', 42); -- monotonic upsert
SELECT honker_stream_get_offset('worker', 'orders'); -- offset or 0
SELECT honker_result_save(42, '{"ok":true}', 3600); -- save w/ 1h TTL
SELECT honker_result_get(42); -- value or NULL
SELECT honker_result_sweep(); -- prune expired
SELECT notify('orders', '{"id":42}');
SELECT honker_enqueue('emails', '{"to":"[email protected]"}', NULL, NULL, 0, 3, NULL);The extension shares tables with the language bindings, so a Python worker can claim jobs written by SQL, Node, Ruby, Go, or any other binding.
How long has this job been running
_honker_live.claimed_at is when the CURRENT claim started. Nothing
else answers that: created_at includes queue wait, run_at is when
the job became ready, and claim_expires_at moves on every heartbeat.
No language binding exposes claimed_at yet, so read it in SQL:
SELECT id, queue, unixepoch() - claimed_at AS running_s
FROM _honker_live
WHERE state = 'processing'
AND claim_expires_at >= unixepoch(); -- required, see belowclaimed_at is only meaningful while claim_expires_at >= unixepoch().
When a claim lapses and nobody reclaims the job — attempts exhausted, no
worker on that queue, queue drained — honker does not touch the row.
worker_id, claim_expires_at and claimed_at all stay put and all go
stale together, and the next claim overwrites all three. Without the
claim_expires_at filter, unixepoch() - claimed_at keeps counting up
for an attempt nobody is running.
claimed_at is NULL until the first claim, and stays NULL for jobs that
were already in flight when an existing database was upgraded: the
migration adds the column without backfilling, because a backfill would
date those rows to upgrade time and read as a claim that never happened.
They pick up a real value on their next claim.
Architecture
- One
PRAGMA data_versionwatcher perDatabase; the default Rust-backed watcher cadence is 1 ms and can be raised - Counter change fans out a wake to each listener/worker/subscriber
- Subscribers re-read SQLite state with indexed SELECTs
- 100 subscribers still share one watcher
- Idle listeners run zero queue/notification SELECTs
Queue claim is one UPDATE ... RETURNING through a partial index:
(queue, priority DESC, run_at, id) WHERE state IN ('pending','processing').
Ack is one DELETE. Retry-exhausted jobs move to _honker_dead, so
claim speed depends on pending/processing jobs, not old queue history.
The language bindings default to WAL because it gives concurrent readers
with one writer and efficient fsync batching. Other journal modes still
work. Correctness and cross-process wake do not depend on WAL; the wake
path is SQLite's own data_version counter.
ORMs And Frameworks
Honker does not ship framework plugins. Load the extension on your
framework or ORM connection, run honker_bootstrap(), and call SQL
functions inside the ORM's transaction.
That works with SQLAlchemy, SQLModel, Django, Drizzle, Kysely, sqlx, GORM, ActiveRecord, Ecto, Hibernate, jOOQ, MyBatis, and Exposed. See the ORM guide at honker.dev/guides/orm.
Performance
On a modern laptop, honker handles thousands of messages per second. Cross-process wake latency is set by the watcher cadence, which defaults to 1 ms. Measure on your hardware with:
python bench/wake_latency_bench.py --samples 500
python bench/real_bench.py --workers 4 --enqueuers 2 --seconds 15Development
make test # Rust + Python + Node fast path
make test-all # broader suite, including slower tests
make build # build Python package + loadable extension
cargo build --release -p honker-extensionRepo layout:
honker-core/ # shared Rust engine
honker-extension/ # SQLite loadable extension
packages/ # language bindings
tests/ # cross-package integration tests
bench/ # benchmarks
Docs
License
Apache-2.0 OR MIT. See LICENSE.
