Show HN: Sidequest.js – Background jobs for Node.js using your database

docs.sidequestjs.com

81 points by merencia 2 days ago

Hey HN,

I'm the maintainer of node-cron (5M+ downloads/month), and I recently built Sidequest.js, a background job runner for Node.js inspired by Oban (Elixir) and Sidekiq (Rails).

It solves some common problems I saw with libraries like node-cron:

- Jobs don’t block your API: they run in isolated worker threads

- No Redis or vendor lock-in: use Postgres, MySQL, SQLite, or MongoDB

- Supports retries, uniqueness, concurrency, snoozing, prioritization

- Comes with a CLI and a simple dashboard

- Works great in monoliths and doesn’t require extra infra

Quick start (no signup needed): https://docs.sidequestjs.com/quick-start

GitHub: https://github.com/sidequestjs/sidequest

Would love feedback or feature suggestions. Happy to answer any questions here!

kristianc a day ago

I'm not sure I follow with the LGPL requirement.. how do you envision downstream users complying with the relinking requirement, particularly in cases where Sidequest is orchestrating jobs across tightly-coupled backend infrastructure?

  • yonixw 11 hours ago

    My guess is the 2-level separation. You WILL need to make some part in your system LGPL by using "Sidequest.js" tightly, but then expose very simple APIs (like /start /status etc..) that will stop the LGPL from "infecting" other parts, as they can replace the "linking" with anything that will support those simple APIs.

    Which is a good way to get improvements other are doing that relate to your source code through LGPL's source exposing, while not forcing it everywhere (GPL case). Especially, for backend libs.

    And since AGPL will essentially make it non viable for SaaS (as network separation won't do), LGPL is what left.

  • shrikrishna a day ago

    AFAIK, the relinking requirement only applies if you distribute the software to someone.

    If you’re running Sidequest entirely on your own infrastructure to orchestrate jobs across your backend, you’re not distributing the software at all, you’re providing a service. The tight coupling does not itself trigger extra obligations. What matters legally is distribution, not architecture.

    Edgecase is if you give your software to a customer to run on their own servers (self‑hosted deployment/docker image shipped to customer). In those cases, you would need to allow them to replace Sidequest.js (ie, not obfuscating it away).

    Someone more knowledgeable can correct me, if I'm wrong

    • yonixw 11 hours ago

      Layman here also, but I think you are correct about GPL and LGPL (this case), but not for AGPL which adds a requirement that: "... If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source"

      https://www.gnu.org/licenses/gpl-faq.html#AGPLv3ServerAsUser

spiffytech a day ago

This is great! I sometimes have projects that could use SQLite, but they're just big enough to need a persistent job queue. Great to have an option for that.

I also like that there's a dashboard. That's really important when a project gets serious, and a surprising number of job queue libraries don't have admin tools.

sleiben 21 hours ago

Looks really interesting, especially with the dashboards.

Just to validate an idea here: I’m using k8s (20+ services) and trying to stick to the 12factor design pattern by having all the baking/companion services, also like cron jobs deployed from within the service directory. Right now I’m using k8s cron services for cron jobs and log the steps and observe with DataDog. Using k8s cron services feels right somehow but the observability with DataDog not. So, would it make sense for me, to deploy a baking k8s web service running sidequest instead?

  • giovaniguizzo 7 hours ago

    Well, Sidequest comes with a comprehensive built-in dashboard that provides real-time monitoring, job management, and performance analytics, so there's that.

    You can probably deploy Sidequest a job processor for all your 20+ services. Each service can enqueue the jobs and you can run as many machines as you want with Sidequest just to run the jobs. You maintain the 12-factor principle - your services remain stateless and delegate scheduled work to Sidequest.

    The only requirement here is that the scripts to be executed must be in the same path in all machines. As long as you follow that, you can deploy it and run your jobs :)

seymon a day ago

How does Sidequest compare to Graphile Worker https://worker.graphile.org/ ?

  • giovaniguizzo 7 hours ago

    Interesting lib!

    Graphile Worker: PostgreSQL only

    Sidequest: Multiple backends (PostgreSQL, MySQL, SQLite, MongoDB)

    Graphile Worker: No built-in dashboard - you need external monitoring

    Sidequest: Comprehensive built-in web dashboard for job monitoring

    Graphile Worker: Single queue with job prioritization

    Sidequest: Multiple queues with individual: i) Concurrency limits; ii) Priority levels; iii) State management (active/paused); iv) Isolated workloads.

    Graphile Worker: Direct PostgreSQL integration, very lightweight

    Sidequest: Worker threads for non-blocking processing, more comprehensive job lifecycle management

    Graphile Worker: Optimized for PostgreSQL performance (3ms latency)

    Sidequest: Balanced performance with rich feature set

    I hope that helps answering your question.

drewrbaker 21 hours ago

This looks like exactly what I need, but our DB is Postgres on Supabase and I don’t think their version supports transaction locking…

  • giovaniguizzo 7 hours ago

    Well, you don't need to worry. Sidequest only locks the rows it's gonna use, i.e., the job rows it manages. Any other data in your DB is not touched by Sidequest, so you should be safe :)

anonzzzies a day ago

Ah!Nice, was looking for that as all are either redis or postgres (and we don't use postgres and prefer not to use redis). So thanks for that.

MutedEstate45 a day ago

Really like your approach of using existing Postgres/MySQL instead of dragging in Redis. It feels genuinely drop-in, but still Sidekiq-class. I know it's a bit early to ask about production patterns, but I was curious: if the worker thread flood hits the same Postgres that serves the web API, how do the job-fetch queries avoid contending with OLTP traffic? Does Sidequest follow Oban's advisory-lock approach or use a different throttling strategy?

  • giovaniguizzo a day ago

    Hello! One of the creators of Sidequest here.

    Great question!

    Sidequest uses transaction-level row locks (`SELECT ... FOR UPDATE SKIP LOCKED`) when fetching jobs. This means each worker thread only locks the specific job rows it’s about to process, minimizing lock contention and avoiding blocking other queries. This pattern is inspired by Oban’s advisory-lock approach, but instead of using explicit advisory locks, Sidequest relies on the database’s built-in row locking mechanism.

    The only drawback is that Sidequest will require one or two connections to your DB. If you enqueue jobs from within other jobs, then each job that requests an enqueue will also connect to your DB (lazily connected upon request - if your job does not enqueue, no connection is created). However, you can configure the number of concurrent workers per queue and globally, so you control how much load Sidequest puts on your database.

    I hope that answers your question :)

    • sorentwo 12 hours ago

      Oban doesn't use advisory locks for fetching jobs (unless there is uniqueness involved)—it uses `FOR UPDATE SKIP LOCKED` as well to pull jobs.

    • MutedEstate45 a day ago

      Thanks for the clarification. That's a clean approach. I just stared your repo. Looking forward to seeing where sidequest.js goes :)

nip a day ago

Looks really neat! Starred on GitHub!

If you have heard of pg-boss [1], how does sidequest compare to it? I’m about to embark on some « jobification » on some flows and I’d love to have your opinion (possibly biased, but still)!

[1] https://github.com/timgit/pg-boss

  • merencia a day ago

    Thanks for the question! I just checked out pg-boss. Solid library if you're fully on Postgres. Sidequest.js takes a broader and more flexible approach. Key differences:

    Database agnostic: Sidequest isn't tied to Postgres. It also works with MySQL, MongoDB, and SQLite, which helps if your stack isn’t Postgres-based.

    Job isolation: Jobs run in worker threads, so heavy tasks won’t block your main Node.js process. Express APIs stay responsive.

    Distributed coordination: Designed to run across multiple instances. One node picks up the job, and if it fails, another can retry. This is built-in.

    Built-in dashboard: Includes a web UI to inspect, retry, or cancel jobs.

    More than queues: Supports cron jobs, uniqueness constraints, per-queue concurrency, and configuration. Some of this overlaps with pg-boss, but the intent behind Sidequest is to provide a complete solution for job processing.

    If you just need simple queues in a Postgres-only setup, pg-boss is great. If you want more flexibility, tooling, and backend options, Sidequest may be a better fit.

    • nip a day ago

      Thanks for the thorough reply!

      I'm all in with Postgres, but the job isolation + built-in dashboard seem really appealing. I'll definitely give it a try!

      Keep up the great work, love to see such high quality codebase / documentation / tooling!

shireboy a day ago

Excellent! I’m a .NET developer who dabbles in Node and have been looking for a Hangfire alternative for a while. This looks like just what I would want.

mousetree a day ago

I'm a big fan of ActiveJob in Rails. I was considering building a version inspired by it in Node but now it looks like I don't have to. Thank you for building this.

hersko a day ago

This looks really cool.

stephenlf 12 hours ago

This is awesome. Nice and simple. Thanks.