Afiq Johari
10/17/2023, 2:46 PMnodes.py
and then verify their output. Typically, these functions
rely on data specified in the catalog.yml
as parameters.
I'm currently using Kedro's IPython environment, which allows me to load data using catalog.load('datasetname')
. However, I find it a bit confusing to figure out how to run the functions I've defined for a specific pipeline. I use %reload_kedro
to refresh my Kedro IPython environment.
I'm aware that it's possible to run nodes (slicing them), but I'm wondering if it's also possible to run only a specific function before fully defining the nodes.
I'd greatly appreciate any insights or best practices for carrying out these incremental updates effectively.Nok Lam Chan
10/17/2023, 2:49 PMif it’s also possible to run only a specific function before fully defining the nodes.What do you mean?
Hugo Rebeix
10/17/2023, 3:14 PMMemoryDataset
as outputs and test-run them from the Kedro Notebook and inspect the results. I usually use it that way:
res = session.run(nodes=["node_im_testing"])
If the outputs are not as I want or if my node fails you can then debug the cell that contains the previous line with breakpoints inside your node.
When I'm sure my node works I then connect the outputs to the catalog or to the next nodes, clear my notebook and I'm done 😉. Hope it helps!Afiq Johari
10/17/2023, 3:16 PMShift+Enter
the lines of code that I want and it will goes to the Python terminal.
But when I'm within kedro iPython environment, I'm not sure how to do this kind of incremental changes to my codes and test them.kedro jupyter notebook
at the moment. But yeah, ideally, I want to update my functions within the nodes.py and use something like Shift+Enter
to see the output of the function.Hugo Rebeix
10/17/2023, 3:25 PM%reload_kedro
enables to reload everything without restarting the kernel)Nok Lam Chan
10/17/2023, 3:36 PMTypically in a one script python file, I can justIf I understand correctly, it won’t work if it’s a function? If you want to achieve the same in any interactive terminal/Jupyter. You have two choices: 1. Use a debugger (it’s built exactly for this purpose and have more advance feature) 2. Copy &paste the function and remove the indent etc.the lines of code that I want and it will goes to the Python terminal.Shift+Enter
Afiq Johari
10/17/2023, 3:41 PMNok Lam Chan
10/17/2023, 3:41 PMHugo Rebeix
10/17/2023, 3:42 PM