Forums

TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated.

Hi

I have deployed a python based flask api where I run model already trained. But unfortunately, I am getting below error in the log file:


2022-01-25 17:27:52,915: /usr/local/lib/python3.8/dist-packages/joblib/externals/loky/backend/resource_tracker.py:120: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some folders/sempahores might leak. 2022-01-25 17:27:52,915: warnings.warn('resource_tracker: process died unexpectedly, ' 2022-01-25 17:27:52,935: /usr/lib/python3.8/multiprocessing/resource_tracker.py:96: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak. 2022-01-25 17:27:52,935: warnings.warn('resource_tracker: process died unexpectedly, ' 2022-01-25 17:27:52,953: Exception on /predict [POST] Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2051, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1501, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1499, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1485, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) File "/home/antoinemle/OpenClassrooms-P7/api.py", line 48, in predict client_input = transformer.transform(client_input) File "/home/antoinemle/.local/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 748, in transform Xs = self._fit_transform( File "/home/antoinemle/.local/lib/python3.8/site-packages/sklearn/compose/_column_transformer.py", line 606, in _fit_transform return Parallel(n_jobs=self.n_jobs)( File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 1054, in call self.retrieve() File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 933, in retrieve self._output.extend(job.get(timeout=self.timeout)) File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result return future.result(timeout=timeout) File "/usr/lib/python3.8/concurrent/futures/_base.py", line 439, in result return self.__get_result() File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self._exception joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.


When I run my code though flask in local I do not have any problem. Would you know how I can solve it? Many thanks!

The most likely cause of that is that your model used too much memory, so we terminated the process. If you have opted in to "Resource Warnings" emails

thanks for your reply, is there a way to bypass this issue?

Fixed it!

It was a timeout issue in the request.post(). I added a parameter "timeout=120" . Sorry having bothered you.

Regards.

No problem, glad you solved it!