https://kedro.org/ logo
Join Slack
Powered by
I am using custom hooks, and testing `ParallelRunn...
# questions
s
I am using custom hooks, and testing 
ParallelRunner
Is it expected that most hooks are not executed at all with 
ParallelRunner
? By my testing, we'll only see:
Copy code
after_context_created
after_catalog_created 
before_pipeline_run
after_pipeline_run
OR
on_pipeline_error
Is there any way to "enable" remaining hooks when using 
ParallelRunner
?
šŸ‘€ 2
d
One sec
Almost certainly it's because 
hook_manager
isn't getting passed: https://github.com/kedro-org/kedro/pull/5587
I don't know if you want to try installing Kedro from that branch, or patching that line in your installation, and seeing if it works for you
Actually, let me double-check, since it seems tests fail due to pickling errors, maybe there was a reason this wasn't done (but it still seems very weird that plugin behavior would be inconsistent)
s
AttributeError: Can't pickle local object 'PluginManager.add_hookcall_monitoring.<locals>.traced_hookexec' Same error as your gh action. This with a trivial hook that just prints out process name I had a more elaborate hook I first had to disable because its 
self
attributes were not pickleable šŸ˜…
d
Ah... looking through the annals, it seems like hooks don't work with 
ParallelRunner
šŸ˜ž ā€¢ https://github.com/kedro-org/kedro-viz/issues/1801#issuecomment-2962810932 ā€¢ https://github.com/kedro-org/kedro/issues/4602
s
Is this actually documented as 4602 suggested? At least casual search didn't reveal this important fact.
d
I don't see it. šŸ˜ž I can raise a documentation PR to make this more clear, unless you would like to take a stab!
s
I can draft a quick addition tomorrow
šŸ’› 2
6 Views
Open in Slack
PreviousNext
Powered by