What are we going to cover? • What is Messaging? • ZeroMQ v/s other Messaging solution • Basics of ZeroMQ • Coding examples • Quiz
What is Enterprise Messaging ? • Enterprise messaging is a design patern for loosely coupled system to communicate asynchronously. • This type of architecture is foundation for “shared-nothing” model. • Great example of this model in computation is also known as “Actor Model”.
Broad classification of Messaging System • Broker based Messaging System. – AMQP based system, Implementation RabbitMQ, ActiveMQ – JMS – Java Message Service. – STOMP – The Simple Text Oriented Messaging Protocol – MQTT – Light Weight protocol for Sensor networks • Broker-less Messaging system – ZeroMQ, Nanomsg etc.
AMQP – RabbitMQ • RabbitMQ is open source message broker software that implements the Advanced Message Queuing protocol. This is writen in Erlang language. • Basic components of AMQP Producer, Exchange, Queue and Consumer
Basic AMQP Model Publishes Publisher Routes Queue Consumes Consu s mer r1 Exch Ex a ch nge g Ro R u o te t s Queue e Consumes Consu s mer rn
In Nutshell - ZeroMQ • Not MOM ( no broker, no queues, etc..) • “Super” Socket library that acts as concurrency framework • Carries messages across various transport, inproc, IPC, TCP and multicast. • Connect N-to-N via fan-out, pub-sub, pipeline. Request-reply. • Asynchronous I/O for scalable multicore message-passing apps. • 40+ languages including C. C++, JAVA, .NET, Python.
In Nutshell - ZeroMQ.. • Is a library that supports certain network communication paterns using socket. • The “MQ” part comes in because ZeroMQ uses queues internal y to buffer messages so that you don’t block your application when sending data. • When you say, socket.send(…), ZeroMQ actual y enqueues a messages to be sent later by a dedicated communication thread.
In nutshell - ZeroMQ • It does not have overhead of an over- engineered protocol such AMQP • It can make use of efficient transports such as reliable multicast or Y-suite IPC ytansport • It makes use of intelligent message batching. This al ows ZMQ to efficiently utilize a TCP/IP connection by minimizing not only protocol overhead but system cal s.
ZeroMQ – Message Encapsulation • Encapsulates communication into Messages which may be composed of multiple parts. • Rather than asking ZeroMQ to receive certain number od f bytes from socket, you ask ZeroMQ to receive a single message.
ZeroMQ - Scalability • A single ZeroMQ socket can for example connect to multiple end points and automatically load balance messages over them or it can work as some sort of Fan-in, collecting messages from sources through a single socket. • ZeroMQ follows brokerless design so there is no single point of failure. Combine this with the simplicity and performance and you get something that you can use to make your application distributed.
ZeroMQ Socket – Request- Reply Patern • Most basic patern is client/server model, where a client sends a request and server replies to that request. • But… • Request type of socket can connect to many servers and request would be distributes to al connected server.
ZeroMQ Request-Reply Socket Patern depiction Zm Z q clcient n t Zm Z q serve v rs r Request # 1 REQ E REP E Reply #1 Requests are load balanced across server Request # 2 REP E Reply # 2 REP E REP E
Demo • ZMQ Request/Reply Patern
ZeroMQ – Pub-Sub Patern – Publish/Subscribe is classic patern where senders of messages, called publisher do not program the messages to be sent directly to specific receives, called subscribers. Messages are published without knowledge of what or if any subscribes exists.
Pub-Sub Socket Depiction Pub Pu Sock oc e k t e -1 t Pub Pu Sock oc e k t e -2 t Ze Z ro r M o Q Bus u Sub u Sock o e ck t e -1 t Sub b Sock oc e k t e -2 t Sub Sock S e ock t e -3 t Publisher sockets are unaware of number of Subscribers. Subscriber socket can subscribe to multiple of Publisher Socket.
Demo • ZMQ Publisher/Subscriber Demo
ZeroMQ Socket – Push/Pull Patern • Push and Pull sockets let you distribute messages to multiple workers arranged in pipeline. A Push socket wil distribute sent messages to its al Pull clients evenly. This is equivalent to producer/consumer model but the results computed by consumers are not sent upstream but downstream to another pull/consumer socket.
ZeroMQ Push-Pul Patern Pr P od r uce c r Consu s mers r Re R sul u t tColl Col ecto ct r o PUL PU L L PU P S U H PU P S U H H PUL PU L L PU P S U H PU P L U L PUL PU L L PU P S U H
Demo • ZMQ Push/Pull Socket demo
ZMQ Devices • ZMQ – sockets are either bind or connect • General patern is stable part (server) bind and dynamic part connects to it. • What do you do when both parts are dynamic ? • Enter ZMQ device
Types of Devices.. • Queue • Forwarder • Streamer
These devises can bind to 2 different ports and forward messages from one end to The other. The forwarding device can become stable point in your network where Each component can connect to. Moving Part Moving Part Stable Part Cli Cl ent n Se S rve v r Cli Cl en e t n t ZM Z Q De D v e i v ce Se S rve v r Cli Cl en e t n t Se S rve v r
Queue Device This is the intermediary that sits between clients and servers, forwarding request to server and relaying replies back to client. The ZMQ device takes (ZMQ.QUEUE) and The two sockets bound to wel known ports. This is Request/Response broker. REP REQ E RE XREP XREQ REQ E RE R P E Queue u REQ E RE R P E
Forwarding Device Just like Queue, Which is like the request-reply broker, For forwarder like the pub-sub Proxy server. It al ows both publishers and subscribers to be moving parts and it self becomes stable hub interconnecting them. FORWARDER collects messages from set of Publishers to and forwards these to a set of subscribers. SUB PU P B U SU XREP XREQ PU P B U SUB U For F w or a w rd r e d r PU P B U SUB U
Who is using ZeroMQ in production • Storm platform – communication channel between spouts and bolt. • Loggly – Cloud based logging platform • Dotcloud – Open source RPC platform cal ed ZeroRPC based on ZeroMQ messaging. • This is very popular messaging framework for high frequency financial trading.
Which Messaging is beter ? Brokered or Broker-less • Both messaging paterns have advantages and disadvantages. • In Enterprise, we see both being used to solve messaging problem which are multidimensional. • ZeroMQ out-performs other messaging solution in performance, as it is bare-bone framework
Which are other Interesting Open Source Messaging project ? • Kafka – LinkedIn’s open source message bus, Apache project, used by Tweeter also and many many companies • NullMQ – Operates over WebSocket, on communication protocol based on STOMP. ZeroMQ semantics in Web Browser. • NanoMSG-
I know you have questions… • I may not have answer.. But please ask.. • I wil find out answers If I don’t know…