Mike Wil banks | Barnes & Noble Message Queues : A Primer
Housekeeping… • Talk – Slides wil be posted after the talk. • Me – Sr. Web Architect Manager at NOOK Developer – Prior MNPHP Organizer – Open Source Contributor – Where you can find me: • Twitter: mwil banks G+: Mike Wil banks • IRC (freenode): mwil banks Blog: http://blog.digitalstruct.com • GitHub: https://github.com/mwil banks
Agenda • Message Queues? • Protocols and Software • Picking a Message Queue • Best Practices
A Definition “Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter- thread communication within the same process. They use a queue for messaging – the passing of control or of content.”
What is messaging? “Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.”
Producer Messages Consumer Tas Mess k ages Messages General Anatomy Producer creates a message and pushes it to the queue; the consumer reads from the queue and processes the message.
Describing Message Queues l Pub/Sub
l FIFO buffer
l Push / Pul
l A way to communicate between
applications / systems. l A way to decouple components.
l A way to offload work.
Why to use a Message Queue l Offload Heavy Work
l Integration with Legacy Systems
l Asynchronous Processing
l Paral el Processing
l Process consistency
Unix Foundations l “Write programs that work together.”
l “Do it in the background.”
Why it matters l Web systems need to be geared to run things
asynchronously. l Distribution of load
l System integrity
You’ve seen them before; they are used in most applications to help them scale. MESSAGE QUEUE EXAMPLES
When to make use of message queues in case you’re wondering. GENERIC USE CASES
Video Processing Resampling, Audio Overlay, Type Conversion…
Analytics Web Server Logs, Log Aggregation, PHP Errors, etc.
Integrations Save local first; push second.
AMQP STOMP XMPP Vendor Specific PROTOCOLS
AMQP Advanced Message Queuing Protocol
Overview of AMQP l AMQP Working Group (Community and
Vendor) l Platform agnostic protocol.
l Completely open, interoperable and broadly
applicable. l Many severs available and many client
How it Works Producer, Exchange, Queue, Consumer
How it Works l AMQP utilizes exchanges, queues and
bindings. l An exchange are routers with routing tables.
l A binding defines the routing rules.
l A queue is where the messages wait for a
Understanding Exchanges l Fanout Exchange
l No routing keys involved. Any message that is
sent to the exchange is sent to al queues bound to that exchange. l Direct Exchange
l Routing keys involved. A queue binds to the
exchange to request messages that match a routing key exactly. l Topic Exchange
l Routing keys involved. A queue binds to the
exchange to request messages that match a routing key pattern.
Implementations www.rabbitmq.com Very popular and common message queue owned by VMware. qpid.apache.org Long standing project; apache foundation. www.openamq.org Long standing project; ZeroMQ partner, no news since 2009.
Building a Queue l An exchange, queue and bindings must be defined
first. Publishing can then commence after. l Create the queue
l Create the exchange
l Bind to the queue.
Persistence? l Default behavior is no persistence.
l How important are the messages?
l Just about al items have a level of persistence if you
would like them to survive on reboot. l Mark exchanges, queues and messages as DURABLE.
PECL AMQP l Extension compatible with AMQP specification
0-9-1. l pecl instal amqp
STOMP Simple (or Streaming) Text Orientated Messaging Protocol
Overview l Simple protocol
l Behaviors fol ow very simple commands. l Most message queues can communicate over
/queue/ msg Connect Send Disconnect /queue/ S T msg O Connect Subscribe Disconnect M P P H S P E R V Read E R Ack How It Works l When you send in a message, you tel it which
queue to go to. l When you subscribe you request a queue.
Sever Implementations activemq.apache.org One of the oldest message queues existing; a apache foundation project activemq.apache.org/apollo Next generation ActiveMQ www.rabbitmq.com Very popular and common message queue owned by VMware. www.jboss.org/hornetq Supported by Red Hat Middle Ware division, picking up steam.
PECL Stomp l pecl instal stomp
l That was easy
XMPP Extensible Messaging and Presence Protocol (Although not real y a “Message Queue”)
Messages • Formatting – JSON or XML are great options. • Please no serialized PHP objects. • Message Size – Only as large as necessary. • Don’t send a binary object through the queue.
Workers • Dumb as possible! – Single message type • Easier to scale. – Single operation • Easy to debug and far more flexible.
PHP Daemons • Prevent Memory Leaks – Detection – Cycle Workers • Handle Signals – Properly shutdown! – Watch out for OS service kil s • Sleeping is good J
Database Worker Server Web Server Message Queue Environment Integration Web Server -> Message Queue Worker Server -> Message Queue
Database Web Server Worker Server Worker Server Message Queue Send to 1 Message Queue High Availability Insert multiple message queue servers + multiple worker nodes. Each worker node can connect to as many message queue servers as necessary.