What the importlib module does
Module, a single py file Package containing multiple modules (py files)
Dynamically import modules so that you don't have to write so much import code. Typical example: automatic synchronization service, each website has a py file. The main process receives a synchronization task, according to the name to dynamically import the corresponding py file, so you do not have to write so much import code. (somewhat similar to java's factory methods)
However, importlib doesn't solve the problem of me modifying the py source code online and then making the changes take effect without restarting the process. In this case, you can use reload()
reload method
To prevent the problem of two modules importing each other, Python defaults to importing all modules only once. If you need to re-import a module, Python2.7 can use reload() directly, and Python3 can use the following methods:
Method 1: Basic method from imp import reload reload(module)
Method 2: Following the routine, you can import imp (module) like this
Method 3: Look, there's something to be found, so there's also this import importlib (module)
Method 4: According to the Divine Principle, this is certainly possible from importlib import reload reload(module)
In a multi-process program, the reload of one process cannot affect the reload of another process.
Example:
# Start multiple processes in the master process
def begin():
""" Launching multiple processes """
plist = []
for i in xrange(Num_process):
p = Process(target=pre_run)
()
(p)
# This process listens for redis messages, and when it receives a message, it executes the reload method.
p = Process(target=reload_spider)
()
(p)
for p in plist:
()
# Listen to redis and execute the reload method.
def reload_spider():
""" listen for file changes, auto-reload """
rconn = (host=settings.REDIS_HOST, port=settings.REDIS_PORT, db=settings.REDIS_DB, password=settings.REDIS_PW)
while True:
try:
key = 'reload-spider'
value = (key)
print value
if value == '1':
crawler_module = importlib.import_module('crawlers.%s' % 'temp'.lower())
reload(crawler_module)
crawlerClass = getattr(crawler_module, 'temp'.upper())
print 'class in reload_spider: %s' % ()
# (key)
except Exception, e:
pass
(3)
Another process prints a variable inside the py file
crawler = get_crawler_from_factory(mq_service, message) print
It turns out that doing a reload in one process doesn't change the variables in the other process. What about in the same process...
Multiple threads in the same process, if any one thread performs a reload operation, all other threads are affected.
def pre_run():
t = (target=reload_spider, name='LoopThread')
()
# ()
""" Multi-threading inside each process """
pool = ThreadPool(Num_Thread)
# Initialize the mq channel
mq_service = RabbitMqService()
def callback(ch, method, properties, body):
# Message Acknowledgement
mq_service.input_channel.basic_ack(delivery_tag=method.delivery_tag)
# Get the name of the current thread
current_process_name = multiprocessing.current_process().name
('Current process name: %s - pid: %s' % (current_process_name, ()))
('Process %s, received message: %s' % (current_process_name, body))
# Receive a task message, throw it to the thread pool #
pool.apply_async(run, (properties, body, mq_service))
# Start listening to the entry channel
mq_service.receive(callback)
The reload_spider listens for messages in the redi, and if there is a reload flag, performs a reload operation.
def reload_spider():
""" listen for file changes, auto-reload """
rconn = (host=settings.REDIS_HOST, port=settings.REDIS_PORT, db=settings.REDIS_DB, password=settings.REDIS_PW)
while True:
try:
key = 'reload-spider'
value = (key)
print value
if value == '1':
crawler_module = importlib.import_module('crawlers.%s' % 'temp'.lower())
reload(crawler_module)
crawlerClass = getattr(crawler_module, 'temp'.upper())
print 'class in reload_spider: %s' % ()
# (key)
except Exception, e:
pass
(3)
Upon testing, the variables introduced in other threads, also changed.
Of course, it is best to use mq or redis blocking queue to listen to messages.
Above this Python reload module implementation method is all that I have shared with you, I hope to give you a reference, and I hope you support me more.