Matthias Roels09/14/2023, 11:38 AM
Nok Lam Chan09/14/2023, 12:23 PM
Matthias Roels09/14/2023, 12:34 PM
Nok Lam Chan09/14/2023, 1:08 PM
Deepyaman Datta09/14/2023, 1:29 PM
directory) contain either Spark or pandas/sklearn code, and not both? In that case, +1 to what @Nok Lam Chan said, and just specifying dependencies per pipeline.
Matthias Roels09/14/2023, 7:06 PM
Nok Lam Chan09/14/2023, 9:09 PM
Matthias Roels09/15/2023, 5:03 AM
Nok Lam Chan09/15/2023, 2:04 PM
so that you only instantiate SparkHook for Spark pipeline?