Introduction to RabbitMQ
AMQP, or Advanced Message Queuing Protocol, is an open standard for application layer protocols, designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, and the sender of the message does not need to know the existence of the message user, and vice versa.
The main features of AMQP are message oriented, queues, routing (including point-to-point and publish/subscribe), reliability, and security.
RabbitMQ is an open source AMQP implementation, written in Erlang language on the server side, supports a variety of clients, such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., and supports AJAX. It is used to store and forward messages in distributed systems, and performs well in terms of ease of use, scalability, high availability, etc.
Below, we will focus on some basic concepts in RabbitMQ. Understanding these concepts is the basis for using RabbitMQ well.
ConnectionFactory、Connection、Channel
ConnectionFactory, Connection, and Channel are the most basic objects in the API provided by RabbitMQ. Connection is a socket link for RabbitMQ, which encapsulates part of the logic related to the socket protocol. ConnectionFactory is the manufacturing plant of Connection.
Channel is the most important interface we deal with RabbitMQ. Most of our business operations are completed in the Channel interface, including defining Queue, defining Exchange, binding Queue and Exchange, publishing messages, etc.
Queue
Queue is an internal object of RabbitMQ, used to store messages, represented by the figure below.
All messages in RabbitMQ can only be stored in the Queue. The producer (P in the figure below) produces the message and finally delivers it to the Queue. The consumer (C in the figure below) can obtain the message from the Queue and consume it.
Multiple consumers can subscribe to the same Queue, and the messages in the Queue will be evenly distributed to multiple consumers for processing, rather than each consumer receiving all the messages and processing them.
Message acknowledgment
In actual applications, it may occur that the consumer receives a message in the Queue, but it crashes (or other accidents) without processing, which may cause the message to be lost. To avoid this, we can ask the consumer to send a receipt to RabbitMQ after consuming the message. RabbitMQ will only remove the message from the Queue after receiving the message receipt (Message acknowledgment). If RabbitMQ does not receive the receipt and detects that the consumer's RabbitMQ connection is disconnected, RabbitMQ will send the message to other consumers (if multiple consumers exist) for processing. There is no timeout concept here. No matter how long a consumer processes a message, it will not cause the message to be sent to other consumers unless its RabbitMQ connection is disconnected.
Another problem will arise here. If our developers forget to send a receipt to RabbitMQ after processing the business logic, this will lead to serious bugs - more and more messages are piled up in the Queue; after the consumer restarts, they will repeatedly consume these messages and execute the business logic...
Message durability
If we hope that even if the RabbitMQ service is restarted, we can set both Queue and Message to durable, so that our RabbitMQ messages will not be lost in most cases. But it still cannot solve the occurrence of low-probability loss events (for example, the RabbitMQ server has received a message from the producer, but the RabbitMQ server is powered off before it can be persisted). If we need to manage such low-probability events, then we need to use transactions. Since this is only a brief introduction to RabbitMQ, we will not explain RabbitMQ-related transactions here.
Prefetch count
We mentioned earlier that if multiple consumers subscribe to messages in the same Queue at the same time, the messages in the Queue will be spread to multiple consumers. At this time, if the processing time of each message is different, it may cause some consumers to be busy all the time, while others will quickly finish the work at hand and be idle all the time. We can limit the number of messages sent by Queue to each consumer each time by setting prefetchCount. For example, when we set prefetchCount=1, Queue will send a message to each consumer every time; after the consumer processes this message, Queue will send another message to the consumer.
Exchange
In the previous section we saw that the producer delivers messages to Queue, which is actually something like this will never happen in RabbitMQ. The reality is that the producer sends the message to Exchange (the exchange, X in the figure below), and Exchange routes the message to one or more Queues (or discards).
What logic does Exchange route messages to Queue? This will be introduced in the Binding section.
There are four types of Exchange in RabbitMQ, and different types have different routing policies, which will be introduced in the Exchange Types section.
routing key
When a producer sends a message to Exchange, he or she usually specifies a routing key to specify the routing rules for the message. This routing key needs to be used in conjunction with Exchange Type and binding key before it can finally take effect.
When the Exchange Type and binding key are fixed (these contents are usually fixedly configured during normal use), our producers can specify the routing key to determine where the message flows.
The length limit set by RabbitMQ for routing key is 255 bytes.
Binding
In RabbitMQ, Exchange is associated with Queue through Binding, so that RabbitMQ knows how to correctly route messages to the specified Queue.
Binding key
While binding Exchange and Queue, a binding key is generally specified; when the consumer sends a message to Exchange, a routing key is generally specified; when the binding key matches the routing key, the message will be routed to the corresponding Queue. This will be explained in the Exchange Types chapter.
These Bindings allow the same binding key when binding multiple Queues to the same Exchange.
binding key does not take effect in all cases, it depends on the Exchange Type. For example, the Fanout type Exchange ignores the binding key and routes the message to all Queues bound to the Exchange.
Exchange Types
The commonly used Exchange Types of RabbitMQ include fanout, direct, topic, and headers (the AMQP specification also mentions two types of Exchange Types, system and custom, respectively, and will not be described here). The following will be introduced separately.
fanout
The Fanout type Exchange routing rule is very simple, it will route all messages sent to the Exchange to all Queues bound to it.
In the above figure, all messages sent by the producer (P) to Exchange (X) are routed to the two Queues in the figure and are eventually consumed by the two consumers (C1 and C2).
direct
Direct type Exchange routing rules are also simple, it will route messages to those Queues whose binding key exactly matches the routing key.
As an example, if we send a message to Exchange with routingKey="error", the message will be routed to Queue1 (-S9b..., which is the Queue name automatically generated by RabbitMQ) and Queue2 (-Agl...); if we send a message with routingKey="info" or routingKey="warning", the message will only be routed to Queue2. If we send messages in other routingKeys, the messages are not routed to both Queues.
topic
As mentioned earlier, the direct type Exchange routing rules are to exactly match the binding key and the routing key, but this strict matching method cannot meet the actual business needs in many cases. Exchange of topic type extends on matching rules. It is similar to the direct type Exchage and also routes messages to Queues that match binding keys and routing keys. However, the matching rules here are somewhat different, and it convention:
- The routing key is a string separated by the period ". " (we call each independent string separated by the period ". ", called a word), such as "", ", "
- binding key is also a string separated by the period number ". ", just like routing key
- There can be two special characters "*" and "#" in the binding key, which are used to make fuzzy matches, where "*" is used to match a word and "#" is used to match multiple words (can be zero)
As an example, the routingKey="" message will be routed to Q1 and Q2 at the same time, the routingKey="" message will be routed to Q1, the routingKey="" message will be routed to Q2, and the routingKey="" message will be routed to Q2 (it will only be delivered to Q2 once, although the routingKey and the two bindingKeys of Q2 match); the routingKey="", routingKey="orange" and routingKey="" messages will be discarded because they do not match any bindingKey.
headers
Exchange of headers type does not rely on the matching rules of routing key and binding key to route messages, but matches them based on the headers attribute in the sent message content.
Specify a set of key-value pairs when binding Queue and Exchange; when a message is sent to Exchange, RabbitMQ will retrieve the headers of the message (also in the form of a key-value pair) and compare whether the key-value pairs in it exactly match the key-value pair specified when binding Queue and Exchange; if it exactly matches, the message will be routed to the Queue, otherwise it will not be routed to the Queue.
This type of Exchange has not been used (but it should be very useful), so I won't introduce it.
RPC
MQ itself is based on asynchronous message processing. In the previous example, all producers (P) sent the message to RabbitMQ and did not know whether the consumer (C) processed successfully or failed (even if there is a consumer to process the message).
But in actual application scenarios, we may need some synchronization processing, and we need to synchronize and wait for the server to complete my message processing before performing the next step of processing. This is equivalent to RPC (Remote Procedure Call). RPC is also supported in RabbitMQ.
The mechanism for implementing RPC in RabbitMQ is:
- When the client sends a request (message), two values are set in the MessageProperties, which define 14 properties in the AMQP protocol. These properties will be sent along with the message) replyTo (a Queue name, which tells the server to send the message notifying me to this Queue after the processing is completed) and correspondenceId (the identification number of this request, the server needs to return this attribute after the processing is completed. The client will understand which request was successfully executed or failed based on this id)
- The server receives the message and processes it
- After the server processed the message, a reply message will be generated to the Queue specified by replyTo, and the correlationId attribute will be included.
- The client has previously subscribed to the replyTo specified Queue. After receiving the server's reply message, it analyzes which request was executed based on the correlationId attribute, and performs subsequent business processing based on the execution result.
Summarize
This article introduces the concepts that I personally think are the most important. By making full use of these functions provided by RabbitMQ, we can handle most of our asynchronous business.
The basic concepts of this article may be difficult to understand and digest, and combined with actual application code, it should be easier to absorb. Therefore, the examples to write next will include actual business application scenario analysis, why use RabbitMQ to implement it, and how to implement it with RabbitMQ.
Several official website demos:/