The logs don't print any obvious errors either, they just show a connection to rabbitmq and then it shuts down
[2019-09-11 06:08:45,729: INFO/Beat] beat: Starting...
[2019-09-11 06:08:45,731: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-09-11 06:08:45,732: INFO/Beat] Writing entries (0)...
[2019-09-11 06:08:45,740: INFO/MainProcess] mingle: searching for neighbors
[2019-09-11 06:08:45,771: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-09-11 06:08:45,771: INFO/Beat] Writing entries (0)...
[2019-09-11 06:08:46,748: INFO/MainProcess] mingle: all alone
[2019-09-11 06:08:46,749: INFO/MainProcess] beat: Shutting down...
[2019-09-11 06:08:46,749: INFO/Beat] Writing entries (0)...
[2019-09-11 06:08:46,749: INFO/Beat] Writing entries (0)...
Checking the logs of rabbitmq, here it shows that rabbitmq connects successfully, and then disconnects, (but I still think it's a rabbitmq problem, because I have no problem with the same environment on another host)
=INFO REPORT==== 15-Oct-2019::02:17:54 ===
connection <0.305.0> (127.0.0.1:39786 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'=INFO REPORT==== 15-Oct-2019::02:17:56 ===
connection <0.310.0> (127.0.0.1:39788 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'=WARNING REPORT==== 15-Oct-2019::02:17:57 ===
closing AMQP connection <0.305.0> (127.0.0.1:39786 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection=INFO REPORT==== 15-Oct-2019::02:17:58 ===
closing AMQP connection <0.310.0> (127.0.0.1:39788 -> 127.0.0.1:5672, vhost: '/', user: 'guest')=INFO REPORT==== 15-Oct-2019::02:17:58 ===
closing AMQP connection <0.302.0> (127.0.0.1:39780 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
After trying reinstalling services like erlang and rabbitmq and tossing them around for a day, the next day I switched gears and viewed the celery source code from the
Find Shutting down... This is the location of the logging output.
def stop(self, wait=False):
info('beat: Shutting down...')
# Add a code here
''''''
import traceback
info("start shutdown traceback ====")
traceback.print_stack()
''''''
self._is_shutdown.set()
wait and self._is_stopped.wait() # block until shutdown done.
Start celery again and get the following log
[2019-10-16 10:15:15,037: INFO/Beat] beat: Starting...
[2019-10-16 10:15:15,039: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:15,082: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:15:15,082: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:15,394: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:15:15,394: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:16,201: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:15:16,201: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:16,859: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:15:16,859: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:23,337: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-10-16 10:15:28,102: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:31,348: INFO/MainProcess] mingle: searching for neighbors
[2019-10-16 10:15:32,357: INFO/MainProcess] mingle: all alone
[2019-10-16 10:15:32,362: WARNING/MainProcess] invalid syntax (, line 13)
[2019-10-16 10:15:33,372: INFO/MainProcess] start shutdown traceback ====
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "", line 15, in <module>
execute_from_command_line()
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py"
, line 381, in execute_from_command_line ()
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py"
, line 375, in execute self.fetch_command(subcommand).run_from_argv()
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/djcelery/management/commands/celery
.py", line 21, in run_from_argv ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 793, in
execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv)))
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 311, in e
xecute_from_commandline return self.handle_argv(self.prog_name, argv[1:])
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 785, in
handle_argv return (command, argv)
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 717, in
execute ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 179, in
run_from_argv return self(*args, **options)
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 274, in _
_call__ ret = (*args, **kwargs)
[2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 212, in
run state_db=self.node_format(state_db, hostname), **kwargs
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 21
2, in start ()
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 25
4, in stop self._shutdown(warm=True)
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 26
8, in _shutdown (self, terminate=not warm)
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 179, in
stop description=what, propagate=False,
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 142, in
restart self.send_all(parent, method, description, propagate=propagate)
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 155, in
send_all fun(parent, *args)
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 378, in
stop return ()
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 559, in stop
()
[2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 499, in stop
traceback.print_stack()
[2019-10-16 10:15:33,375: INFO/MainProcess] beat: Shutting down...
[2019-10-16 10:15:33,375: INFO/Beat] Writing entries (0)...
[2019-10-16 10:15:33,376: INFO/Beat] Writing entries (0)...
[root@test operations]# [2019-10-16 10:15:51,178: INFO/Beat] Writing entries (0)...
[2019-10-16 10:16:30,352: INFO/Beat] Writing entries (0)...
[2019-10-16 10:16:46,806: INFO/Beat] Writing entries (0)...
[2019-10-16 10:18:56,064: INFO/Beat] Writing entries (0)...
[2019-10-16 10:19:35,231: INFO/Beat] Writing entries (0)...
[2019-10-16 10:19:51,689: INFO/Beat] Writing entries (0)...
You can see that the stop method is executed in the start method in /usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py, where the beat: Shutting down is printed... Characters
def start(self):
"""Starts the workers main loop."""
'''
Modify the start method to remove the try, in order to find the error location
'''
(self)
#try:
# (self)
#except WorkerTerminate:
# ()
#except Exception as exc:
# ('Unrecoverable error: %r', exc, exc_info=True)
# ()
#except (KeyboardInterrupt, SystemExit):
# ()
Then start celery again to get the following log
[2019-10-16 10:19:57,241: INFO/Beat] beat: Starting...
[2019-10-16 10:19:57,244: INFO/Beat] Writing entries (0)...
[2019-10-16 10:19:57,287: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:19:57,287: INFO/Beat] Writing entries (0)...
[2019-10-16 10:20:00,218: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:20:00,218: INFO/Beat] Writing entries (0)...
[2019-10-16 10:20:01,023: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:20:01,023: INFO/Beat] Writing entries (0)...
[2019-10-16 10:20:01,685: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:20:01,685: INFO/Beat] Writing entries (0)...
[2019-10-16 10:20:05,548: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-10-16 10:20:10,307: INFO/Beat] Writing entries (0)...
[2019-10-16 10:20:13,561: INFO/MainProcess] mingle: searching for neighbors
[2019-10-16 10:20:14,570: INFO/MainProcess] mingle: all alone
[2019-10-16 10:20:14,571: WARNING/MainProcess] Traceback (most recent call last):
[2019-10-16 10:20:14,571: WARNING/MainProcess] File "", line 15, in <module>
[2019-10-16 10:20:14,571: WARNING/MainProcess] execute_from_command_line()
[2019-10-16 10:20:14,571: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py"
, line 381, in execute_from_command_line[2019-10-16 10:20:14,572: WARNING/MainProcess] ()
[2019-10-16 10:20:14,572: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py"
, line 375, in execute[2019-10-16 10:20:14,572: WARNING/MainProcess] self.fetch_command(subcommand).run_from_argv()
[2019-10-16 10:20:14,572: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/djcelery/management/commands/celery
.py", line 21, in run_from_argv[2019-10-16 10:20:14,572: WARNING/MainProcess] ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
[2019-10-16 10:20:14,572: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 793, in
execute_from_commandline[2019-10-16 10:20:14,573: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2019-10-16 10:20:14,573: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 311, in e
xecute_from_commandline[2019-10-16 10:20:14,573: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2019-10-16 10:20:14,573: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 785, in
handle_argv[2019-10-16 10:20:14,574: WARNING/MainProcess] return (command, argv)
[2019-10-16 10:20:14,574: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 717, in
execute[2019-10-16 10:20:14,574: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2019-10-16 10:20:14,574: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 179, in
run_from_argv[2019-10-16 10:20:14,574: WARNING/MainProcess] return self(*args, **options)
[2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 274, in _
_call__[2019-10-16 10:20:14,575: WARNING/MainProcess] ret = (*args, **kwargs)
[2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/", line 212, in
run[2019-10-16 10:20:14,575: WARNING/MainProcess] state_db=self.node_format(state_db, hostname), **kwargs
[2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 20
5, in start[2019-10-16 10:20:14,575: WARNING/MainProcess] (self)
[2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 123, in
start[2019-10-16 10:20:14,576: WARNING/MainProcess] (parent)
[2019-10-16 10:20:14,576: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 374, in
start[2019-10-16 10:20:14,576: WARNING/MainProcess] return ()
[2019-10-16 10:20:14,576: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/", line 28
0, in start[2019-10-16 10:20:14,576: WARNING/MainProcess] (self)
[2019-10-16 10:20:14,577: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/", line 123, in
start[2019-10-16 10:20:14,577: WARNING/MainProcess] (parent)
[2019-10-16 10:20:14,577: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/", line 64
2, in start[2019-10-16 10:20:14,577: WARNING/MainProcess] c.update_strategies()
[2019-10-16 10:20:14,577: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/", line 43
9, in update_strategies[2019-10-16 10:20:14,578: WARNING/MainProcess] [name] = task.start_strategy(, self)
[2019-10-16 10:20:14,578: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/app/", line 448, in s
tart_strategy[2019-10-16 10:20:14,578: WARNING/MainProcess] return instantiate(, self, app, consumer, **kwargs)
[2019-10-16 10:20:14,578: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/utils/", line 53,
in instantiate[2019-10-16 10:20:14,578: WARNING/MainProcess] return symbol_by_name(name)(*args, **kwargs)
[2019-10-16 10:20:14,578: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/kombu/utils/__init__.py", line 96,
in symbol_by_name[2019-10-16 10:20:14,578: WARNING/MainProcess] module = imp(module_name, package=package, **kwargs)
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/importlib/__init__.py", line 127, in import_modul
e[2019-10-16 10:20:14,579: WARNING/MainProcess] return _bootstrap._gcd_import(name[level:], package, level)
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 983, in _find_and_load
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap_external>", line 724, in exec_module
[2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap_external>", line 860, in get_code
[2019-10-16 10:20:14,580: WARNING/MainProcess] File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
[2019-10-16 10:20:14,580: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[2019-10-16 10:20:14,580: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/", line 13
[2019-10-16 10:20:14,580: WARNING/MainProcess] from import to_timestamp
[2019-10-16 10:20:14,580: WARNING/MainProcess] ^
[2019-10-16 10:20:14,580: WARNING/MainProcess] SyntaxError
[2019-10-16 10:20:14,580: WARNING/MainProcess] :
[2019-10-16 10:20:14,580: WARNING/MainProcess] invalid syntax
You can see that the from import to_timestamp line when importing the package reported an error In kombu, the async directory was changed to asynchronous, and because after python 3.7, async became a keyword and could not be referenced as a package, so it reported an error, so async was changed to asynchronous
Start celery again and get the following log
[2019-10-16 10:27:14,497: INFO/Beat] beat: Starting...
[2019-10-16 10:27:14,500: INFO/Beat] Writing entries (0)...
[2019-10-16 10:27:14,545: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:27:14,545: INFO/Beat] Writing entries (0)...
[2019-10-16 10:27:14,949: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:27:14,949: INFO/Beat] Writing entries (0)...
[2019-10-16 10:27:15,029: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:27:15,029: INFO/Beat] Writing entries (0)...
[2019-10-16 10:27:15,748: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:27:15,748: INFO/Beat] Writing entries (0)...
[2019-10-16 10:27:16,427: INFO/Beat] DatabaseScheduler: Schedule changed.
[2019-10-16 10:27:16,427: INFO/Beat] Writing entries (0)...
[2019-10-16 10:27:22,485: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-10-16 10:27:24,618: INFO/MainProcess] mingle: searching for neighbors
[2019-10-16 10:27:25,626: INFO/MainProcess] mingle: all alone
[2019-10-16 10:27:25,648: WARNING/MainProcess] /usr/local/python3/lib/python3.7/site-packages/djcelery/:133: UserWarning: Usi
ng leads to a memory leak, never use this setting in production environments! warn('Using leads to a memory leak, never '
[2019-10-16 10:27:25,648: WARNING/MainProcess] celery@ ready.
Once celery@ ready. appears, you can be sure that celery is working properly
I hope the above compilation helps, and I thank you for learning and supporting me.