1. IO multiplexing
O multiplexing technique is to use an intermediary that can monitor multiple IO blocks at the same time to monitor these different IO objects, and any one or more of these monitored IO objects that have messages returned will trigger this intermediary to return these messaged IO objects for their messages.
The advantage of using IO multiplexing is that processes can handle multiple IO blocking at the same time with a single thread as well. Compared with the traditional multi-threaded/multi-process model, I/O multiplexing has less system overhead, and the system does not need to create new processes or threads or maintain them, which reduces the maintenance workload of the system and saves system resources.
Python provides selector module to realize IO multiplexing. At the same time, on different operating systems, the type of this intermediary selector is different, the current common ones are, epoll, kqueue, devpoll, poll,select, etc.; kqueue (BSD, mac support), devpoll (solaris support) and epoll implementation is basically the same, epoll in the Linux2.5+ kernel implementation, Windows systems only implement the select.
1.1. epoll, poll, select comparison
Select and poll use polling to detect whether all monitored IOs have returned data, requiring constant traversal of each IO object, which is a time-consuming operation that is less efficient. poll is superior to select in that select limits the maximum number of monitored IOs to 1024, which is clearly insufficient for servers that require a large number of network IO connections; while poll has no limit on this number. poll has no limit on this number. But this also faces the problem that when using polling to monitor these IOs, the larger the number of IOs, the more time each poll consumes. This is a problem that polling cannot solve.
epoll was born to solve this problem, first of all, he did not have the maximum number of monitoring IO limitations, and do not use the polling method to detect these IO, but the use of event notification mechanism and callbacks to get these messages returned to the IO object, only the "active" IO will take the initiative to call the callback function. This IO will be handled directly without polling.
2. Basic use of the selector module
import selectors
import socket
# Create a socketIO object that listens for request messages.
sock = ()
(("127.0.0.1", 80))
()
slt = () # Use the system default selector, select for windows, epoll for linux
# Add this socketIO object to the select to monitor
(fileobj=sock, events=selectors.EVENT_READ, data=None)
# Loop through messages
while True:
# select method: poll the selector and when at least one IO object returns a message, it will return the IO object with the message.
ready_events = (timeout=None)
print(ready_events) # Ready IO objects
break
ready_events is a list (representing all data-receivable IO objects registered to this select), and each tuple in the list is:
SelectorKey object:
- fileobj:registered socket object
- fd:file descriptor
- data: the parameter we pass when registering, can be any value, bound to a property, easy to use later.
mask value
- EVENT_READ : Indicates readable; its value is actually 1;
- EVENT_WRITE: indicates writable; its value is actually 2;
- Or a combination of both
Example:
[(SelectorKey(fileobj=< fd=456, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 80)>, fd=456, events=1, data=None),
1)]
To process this request, you only need to use the corresponding method of the socket, which is used to receive the requested connection, and use the accept method to process the request.
When the request is accepted, a new client will be generated, and we will put it into the selector and monitor it together. When a message comes in, the handle_request() function handles it if it is a connection request, and the handle_client_msg() function handles it if it is a client's message.
There are two types of sockets in the select, so we need to determine which type of socket is returned when activated, and then call different functions to make different requests. If there are many types of sockets in the select, it will not be possible to determine this. The solution is to bind the handler function to the corresponding selectkey object, using the data parameter.
def handle_request(sock:, mask): # Process new connections
conn, addr = ()
(False) # Set non-blocking
(conn, selector.EVENT_READ, data=handle_client_msg)
def handle_client_msg(sock:, mask) # Processing messages
data = ()
print(())
sock = ()
(("127.0.0.1", 80))
()
slt = ()
(fileobj=sock, events=selectors.EVENT_READ, data=handle_request)
while True:
ready_events = (timeout=None)
for event, mask in ready_events:
(, mask)
# differentsocketdifferdatafunction (math.)，Use your own bounddatafunction (math.)调用，And then you'll be able to put your ownsocketas a parameter。It will be possible to handle different types ofsocket。
The use of data above is a good solution to the above problem, but note that functions (or callable objects) bound to the data attribute will eventually be called using the () method, and these functions should accept the same arguments.
to this article on the Python multiplexing selector module is introduced to this article, more related Python selector module content, please search for my previous posts or continue to browse the following related articles I hope you will support me in the future!