Matthias Roels
09/14/2023, 11:38 AMNok Lam Chan
09/14/2023, 12:23 PMMatthias Roels
09/14/2023, 12:34 PMNok Lam Chan
09/14/2023, 1:08 PMDeepyaman Datta
09/14/2023, 1:29 PMpipelines/
directory) contain either Spark or pandas/sklearn code, and not both? In that case, +1 to what @Nok Lam Chan said, and just specifying dependencies per pipeline.Matthias Roels
09/14/2023, 7:06 PMNok Lam Chan
09/14/2023, 9:09 PMMatthias Roels
09/15/2023, 5:03 AMNok Lam Chan
09/15/2023, 2:04 PMbefore_pipeline_run
so that you only instantiate SparkHook for Spark pipeline?