Name: queue_classic
Owner: Rainforest QA
Description: Simple, efficient worker queue for Ruby & PostgreSQL.
Created: 2016-01-12 16:54:07.0
Updated: 2016-01-15 01:06:46.0
Pushed: 2017-07-08 00:14:33.0
Size: 636
Language: Ruby
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Simple, efficient worker queue for Ruby & PostgreSQL
IMPORTANT NOTE REGARDING VERSIONS
This README is representing the current work for queue_classic 3.1. You can find the README for other versions:
queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.
There are 2 ways to use queue_classic.
The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method invocation defined by the first argument. The set of arguments will be encoded as JSON in the database.
is method has no arguments.
nqueue("Time.now")
is method has 1 argument.
nqueue("Kernel.puts", "hello world")
is method has 2 arguments.
nqueue("Kernel.printf", "hello %s", "world")
is method has a hash argument.
nqueue("Kernel.puts", {"hello" => "world"})
is method has an array argument.
nqueue("Kernel.puts", ["hello", "world"])
is method uses a non-default queue.
eue = QC::Queue.new("priority_queue")
eue.enqueue("Kernel.puts", ["hello", "world"])
There is also the possibility to schedule a job at a specified time in the future. It will not be worked off before that specified time.
ecifying the job execution time exactly.
nqueue_at(Time.new(2024,01,02,10,00), "Kernel.puts", "hello future")
ecifying the job execution time as an offset in seconds.
nqueue_in(60, "Kernel.puts", "hello from 1 minute later")
There are two ways to work jobs. The first approach is to use the Rake task. The second approach is to use a custom executable.
Require queue_classic in your Rakefile.
ire 'queue_classic'
ire 'queue_classic/tasks'
Start the worker via the Rakefile.
ndle exec rake qc:work
Setup a worker to work a non-default queue.
EUE="priority_queue" bundle exec rake qc:work
Setup a worker to work multiple queues.
EUES="priority_queue,secondary_queue" bundle exec rake qc:work
In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
This example is probably not production ready; however, it serves as an example of how to leverage the code in the Worker class to fit your non-default requirements.
ire 'timeout'
ire 'queue_classic'
edQueue = QC::Queue.new("failed_jobs")
s MyWorker < QC::Worker
f handle_failure(job, e)
FailedQueue.enqueue(job[:method], *job[:args])
d
er = MyWorker.new
('INT') {exit}
('TERM') {worker.stop}
do
b = worker.lock_job
meout::timeout(5) { worker.process(job) }
In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using PSQL(1) or use a database migration script.
m install queue_classic
eatedb queue_classic_test
port QC_DATABASE_URL="postgres://username:password@localhost/queue_classic_test"
by -r queue_classic -e "QC::Setup.create"
by -r queue_classic -e "QC.enqueue('Kernel.puts', 'hello world')"
by -r queue_classic -e "QC::Worker.new.work"
Declare dependencies in Gemfile.
ce "http://rubygems.org"
"queue_classic", "~> 3.0.0"
Add the database tables and stored procedures.
s generate queue_classic:install
le exec rake db:migrate
If you use Rails 4.2+, all you need to do is to set config.active_job.queue_adapter = :queue_classic in your application.rb. Everything else will be taken care for you. You can now use the Active Job functionality from now.
Just for your information, queue_classic detects your database connection and uses it.
Alternatively, you can use the Rake task to prepare your database.
eating the table and functions
ndle exec rake qc:create
opping the table and functions
ndle exec rake qc:drop
Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used.
If you don't want to use the automatic database connection, set this environment variable to false: export QC_RAILS_DATABASE=false
Note on using ActiveRecord migrations: If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. rake db:reset), your application must be configured with config.active_record.schema_format = :sql in config/application.rb. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: postgres://username:password@localhost/database_name. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. QueueClassic will maintain its own connection to the database. This may double the number of connections to your database.
If you are upgrading from a previous version of queue_classic, you might need some new database columns and/or functions. Luckily enough for you, it is easy to do so.
You just need to run those lines, which will copy the new required migrations:
s generate queue_classic:install
le exec rake db:migrate
This rake task will get you covered:
dating the table and functions
ndle exec rake qc:update
All configuration takes place in the form of environment vars. See queue_classic.rb for a list of options.
If you are running PostgreSQL 9.2 or higher, queue_classic will use the json datatype for storing arguments. Versions 9.1 and lower will use the 'text' column. If you have installed queue_classic prior to version 2.1.4 and are running PostgreSQL >= 9.2, run the following to switch to using the json type:
r table queue_classic_jobs alter column args type json using (args::json);
By default queue_classic does not talk very much. If you find yourself in a situation where you need to know what's happening inside QC, there are two different kind of logging you can enable: DEBUG and MEASURE.
This will output the time to process and that kind of thing. To enable it, set the QC_MEASURE:
rt QC_MEASURE="true"
You can enable the debug output by setting the DEBUG environment variable:
rt DEBUG="true"
If you think you have found a bug, feel free to open an issue. Use the following template for the new issue:
If you have general questions about how to use queue_classic, send a message to the mailing list:
https://groups.google.com/d/forum/queue_classic
ndle
eatedb queue_classic_test
port QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
ke
Copyright © 2010 Ryan Smith
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.