2026-03-31 14:49:00,497 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://192.168.229.161:44509'.

2026-03-31 14:49:00,495 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://192.168.229.178:39827'.

2026-03-31 14:49:00,492 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://192.168.229.154:41261'.

2026-03-31 14:49:00,492 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://192.168.229.175:38383'.

2026-03-31 14:49:00,489 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://192.168.229.140:40949'.

2026-03-31 14:47:00,899 - distributed.scheduler - ERROR - 5/5 nanny worker(s) did not shut down within 30s: {'tcp://192.168.229.161:44509', 'tcp://192.168.229.140:40949', 'tcp://192.168.229.178:39827', 'tcp://192.168.229.154:41261', 'tcp://192.168.229.175:38383'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 8729, in check_worker_ttl await self.restart_workers( File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 5/5 nanny worker(s) did not shut down within 30s: {'tcp://192.168.229.161:44509', 'tcp://192.168.229.140:40949', 'tcp://192.168.229.178:39827', 'tcp://192.168.229.154:41261', 'tcp://192.168.229.175:38383'}

2026-03-31 14:47:00,898 - distributed.scheduler - ERROR - 5/5 nanny worker(s) did not shut down within 30s: {'tcp://192.168.229.161:44509', 'tcp://192.168.229.140:40949', 'tcp://192.168.229.178:39827', 'tcp://192.168.229.154:41261', 'tcp://192.168.229.175:38383'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 5/5 nanny worker(s) did not shut down within 30s: {'tcp://192.168.229.161:44509', 'tcp://192.168.229.140:40949', 'tcp://192.168.229.178:39827', 'tcp://192.168.229.154:41261', 'tcp://192.168.229.175:38383'}

2026-03-31 14:47:00,896 - distributed.scheduler - INFO - Lost all workers

2026-03-31 14:47:00,896 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.229.175:38383 name: simple-default-worker-0be49a89fe (stimulus_id='check-worker-ttl-1774968390.8939364')

2026-03-31 14:47:00,896 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.229.154:41261 name: simple-default-worker-9547b074c8 (stimulus_id='check-worker-ttl-1774968390.8939364')

2026-03-31 14:47:00,896 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.229.178:39827 name: simple-default-worker-b3181b05c2 (stimulus_id='check-worker-ttl-1774968390.8939364')

2026-03-31 14:47:00,896 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.229.140:40949 name: simple-default-worker-629095a6ce (stimulus_id='check-worker-ttl-1774968390.8939364')

2026-03-31 14:47:00,896 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.229.161:44509 name: simple-default-worker-64bfb4e1ec (stimulus_id='check-worker-ttl-1774968390.8939364')

2026-03-31 14:47:00,895 - distributed.scheduler - ERROR - Workers ['tcp://192.168.229.161:44509', 'tcp://192.168.229.140:40949', 'tcp://192.168.229.178:39827', 'tcp://192.168.229.154:41261', 'tcp://192.168.229.175:38383'] did not shut down within 30s; force closing

2026-03-31 14:46:30,894 - distributed.scheduler - INFO - Restarting 5 workers: ['tcp://192.168.229.161:44509', 'tcp://192.168.229.178:39827', 'tcp://192.168.229.140:40949', 'tcp://192.168.229.154:41261', 'tcp://192.168.229.175:38383'] (stimulus_id='check-worker-ttl-1774968390.8939364'

2026-03-31 14:46:30,894 - distributed.scheduler - WARNING - Worker failed to heartbeat for 513s; attempting restart: <WorkerState 'tcp://192.168.229.178:39827', name: simple-default-worker-b3181b05c2, status: running, memory: 0, processing: 0>

2026-03-31 14:46:30,894 - distributed.scheduler - WARNING - Worker failed to heartbeat for 513s; attempting restart: <WorkerState 'tcp://192.168.229.175:38383', name: simple-default-worker-0be49a89fe, status: running, memory: 0, processing: 0>

2026-03-31 14:46:30,894 - distributed.scheduler - WARNING - Worker failed to heartbeat for 513s; attempting restart: <WorkerState 'tcp://192.168.229.161:44509', name: simple-default-worker-64bfb4e1ec, status: running, memory: 0, processing: 0>

2026-03-31 14:46:30,894 - distributed.scheduler - WARNING - Worker failed to heartbeat for 513s; attempting restart: <WorkerState 'tcp://192.168.229.154:41261', name: simple-default-worker-9547b074c8, status: running, memory: 0, processing: 0>

2026-03-31 14:46:30,893 - distributed.scheduler - WARNING - Worker failed to heartbeat for 513s; attempting restart: <WorkerState 'tcp://192.168.229.140:40949', name: simple-default-worker-629095a6ce, status: running, memory: 0, processing: 0>

2026-03-10 15:55:35,644 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.229.178:39827

2026-03-10 15:55:35,644 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.229.178:39827 name: simple-default-worker-b3181b05c2

2026-03-10 15:55:35,512 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.229.154:41261

2026-03-10 15:55:35,512 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.229.154:41261 name: simple-default-worker-9547b074c8

2026-03-10 15:55:35,492 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.229.140:40949

2026-03-10 15:55:35,492 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.229.140:40949 name: simple-default-worker-629095a6ce

2026-03-10 15:55:35,491 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.229.161:44509

2026-03-10 15:55:35,491 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.229.161:44509 name: simple-default-worker-64bfb4e1ec

2026-03-10 15:55:35,483 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.229.175:38383

2026-03-10 15:55:35,247 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.229.175:38383 name: simple-default-worker-0be49a89fe

2026-02-24 11:31:30,892 - distributed.scheduler - INFO - Registering Worker plugin shuffle

2026-02-24 11:31:30,892 - distributed.scheduler - INFO - dashboard at: http://192.168.145.50:8787/status

2026-02-24 11:31:30,892 - distributed.scheduler - INFO - Scheduler at: tcp://192.168.145.50:8786

2026-02-24 11:31:30,873 - distributed.scheduler - INFO - -----------------------------------------------

2026-02-24 11:31:30,728 - distributed.scheduler - INFO - State start