codeclimate/sqskiq

Name: sqskiq

Owner: Code Climate

Description: Efficient Ruby background processing using Amazon SQS

Forked from: ricardolazaro/sqskiq

Created: 2017-01-26 21:32:33.0

Updated: 2017-01-26 21:32:35.0

Pushed: 2017-01-26 22:44:48.0

Homepage:

Size: 75

Language: Ruby

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Sqskiq

High performance, Sidekiq-like Amazon SQS messages consumer. Currently only supports Rails 3.x.

SQS is complete message solution powered by Amazon, including monitoring, alarms, redundancy, etc. Due to its particularities, we decided to build a message consumer from scratch, to better handle costs, latency and others.

Getting Start

1. Add sqskiq to your Gemfile:

   'sqskiq'
   

2. Add an initializer config/initializers/sqskiq.rb with:

   iq.configure do |c|
   configuration = {
   aws_access_key_id: 'AWS_ACCESS_KEY_ID'
   aws_secret_access_key: 'AWS_SECRET_ACCESS_KEY'
   
   
   

3. Add a worker in app/workers to process messages asynchronously:

   s HardWorker
   clude Sqskiq::Worker
   
   skiq_options queue_name: :queue_test
   
   f perform(message)
   # do something
   d
   
   

   The message received by this worker is an instance of AWS::SQS::ReceivedMessage

   Configure the parallelism using the param processors: sqskiq_options queue_name: :queue_test, processors: 30 Currently, the min number of processors is 2 and the default is 20. Any unacceptable value will end up using the default.
   

   Configure how long sqskiq will delay new fetches using the param empty_queue_throttle in seconds: sqskiq_options queue_name: :queue_test, empty_queue_throttle: 20 #for 20 seconds of delay If the queue got empty, sqskiq will wait empty_queue_throttle seconds to perform a new fetch. This can drastically reduce your sqs costs.

4. Start Sqskiq and consume the queue:

   s runner HardWorker.run
   

5. Publish message

   Worker.perform_async(params)
   

Deploy

Use a Procfile (like Heroku does) to start your workers. In your Procfile, add:

eue_test: rails runner 'HardWorker.run'

Tips and Limitations

• Currently, the minimum number of workers is 2
• If your worker uses Mongoid, ActiveRecord, etc… make sure to disable data caches (or at least clean the data after each execution), otherwise your workers will explode.
• Workers will automatically retry a job if the execution raises an exception
• If the execution does not raise an exception, workers assume the execution ran cleanly and will remove the message from queue
• Be aware of the costs of using SQS, even while under the service's free tier

Future and TODO's

• Implement a better retry policy (today the message will be retried forever)
• User can configure only one processor
• Handling batches
• Better database integration with automatic cache clean