Andrew Stewart
02/01/2023, 1:35 AMDeepyaman Datta
02/01/2023, 3:09 AMpyspark.pandas
in production
• Write code in pandas, use modin
to scale
• Write code in Fugue, choose your backend
• Write code in Ibis, choose your backend
For Kedro, I'd recommend one of the first options, and you can potentially look at https://github.com/mzjp2/kedro-dataframe-dropin (very out of date) to see how this could be achieved
Or you can set up a Spark env 🙂datajoely
02/01/2023, 8:36 AMWilliam Caicedo
02/01/2023, 8:58 AMAndrew Stewart
02/01/2023, 5:18 PMsettings.py
?datajoely
02/01/2023, 5:23 PMcontext.env
and do your condition there?Andrew Stewart
02/01/2023, 5:29 PMcontext
is passed into the hookdatajoely
02/01/2023, 5:29 PMsettings.py
is evaluated before the env
is known