Hey hope that this finds you well Leaked Semaphore object in

Question

Hey hope that this finds you well Leaked Semaphore object interrupts pipeline while node seem healthy I m running a pipeline everything goes well and then I get ```segmentation fault kedro run Users cyril verluise opt anaconda3 envs my project lib python3 9 multiprocessing resource tracker py 216 UserWarning resource tracker There appear to be 1 leaked semaphore objects to clean up at shutdown``` and the pipeline is interrupted What is strange is that if I run the current node when the pipeline stops in isolat everything goes well Note also that I m not implementing multiprocessing myself Any idea I ll keep googling on my side just in case anyone has already seen that before Set up kedro 0 18 13

Nok Lam Chan · Answer

Are you using parallelrunner

Nok Lam Chan · Answer

or is this the libraries that you are using

Cyril Verluise · Answer

no not using parallelrunner

Cyril Verluise · Answer

gt is this the libraries that you are using might be the case

Cyril Verluise · Answer

you mean that <https docs kedro org en stable kedro runner ParallelRunner html> if yes then no I m not using ParallelRunner

Nok Lam Chan · Answer

Then it is one of the libraries that you are using Kedro only use multi processing when you use `ParallelRunner`

Juan Luis · Answer

ugh is this <https github com kedro org kedro issues 1100>

Nok Lam Chan · Answer

Can you show your `settings py `

Nok Lam Chan · Answer

The issue that < Juan Luis> show may related but that should be fixed 1 year ago

Nok Lam Chan · Answer

Any chance that you are importing the `ShelveStore`

Cyril Verluise · Answer

Thanks No not using ShelveStore

Cyril Verluise · Answer

So I guess that I have identified the node creating the issue It contained a call to tqdm which is said to be responsible for such errors in some cases I removed that call I still have an error but the error does not refer to a specific multiprocessing explicitly it s only `segmentation fault kedro run`

Cyril Verluise · Answer

not sure it helps much

Nok Lam Chan · Answer

Ah do you run into the error if you just run the python function without kedro

Cyril Verluise · Answer

nope

Nok Lam Chan · Answer

<https github com tqdm tqdm issues 1197> Not sure can you remove the tqdm from your code it s the library that used to render your progress bar

Cyril Verluise · Answer

All nodes work fine in isolation

Cyril Verluise · Answer

gt can you remove the tqdm from your code yes I did It did not fix the issue only changed the error message

Nok Lam Chan · Answer

Hmm I am still unsure whether it is a Kedro issue or some 3rd party libraries problem

Nok Lam Chan · Answer

gt All nodes work fine in isolation With or without Kedro

Cyril Verluise · Answer

both

Nok Lam Chan · Answer

I see then I am more inclined to some libraries issue rather than Kedro s

Nok Lam Chan · Answer

Are you using some more exotic computation libraries

Cyril Verluise · Answer

I m using pdal <https pdal io en latest > to ingest and process lidar data That seems to be the node causing the issue if I run the following nodes it works well if I run them with the lidar ingestion processing node i get the errror

Cyril Verluise · Answer

I suspect there is some multiprocessing or so happening in pdal

Nok Lam Chan · Answer

Lidar data sounds fun grin So it could be the libraries problem as I do see people reporting seg fault issues 21 issues <https github com PDAL PDAL issues 3848>

Cyril Verluise · Answer

ohoho

Cyril Verluise · Answer

looks we have our top candidate here

Cyril Verluise · Answer

thanks a lot for the help Sorry to disturb you for non kedro issues

Nok Lam Chan · Answer

Did you manage to find the issue

Cyril Verluise · Answer

hmm pdal is not the most straightforward tool to be fair In short I m not using the filters mentioned in the issue but still find it most likely that it s the cause Now fixing the issue is another story