Florian d
08/11/2023, 12:36 PMsession
or context
“aware” that they were initialised using kedro ipython
? As in is there any attribute that can identify this?datajoely
08/11/2023, 12:37 PMNok Lam Chan
08/11/2023, 12:39 PMmarrrcin
08/11/2023, 12:40 PMFlorian d
08/11/2023, 12:41 PMkedro ipython
sessions because the hook we use to update the catalog won’t be executed.kedro ipython/jupyter
datajoely
08/11/2023, 12:41 PMexecution_context
property to the session right?marrrcin
08/11/2023, 12:43 PMFlorian d
08/11/2023, 12:44 PMafter_context_created
hook and expand that for the IPython use case if the context is aware or can be made aware that it is an ipython session. That way in the ipython case we can support datasets that only exist remotely to be downloadabledatajoely
08/11/2023, 12:44 PMNok Lam Chan
08/11/2023, 12:44 PMI realised that the hooks wont support the case of using it inCan you explains a little bit why is this the case?sessions because the hook we use to update the catalog won’t be executed.kedro ipython
Florian d
08/11/2023, 12:48 PMbefore_pipeline_run
hook (setting a download
flag to True) because for pipeline runs we need to know which datasets are “root” datasets. We obtain this information from the pipeline
argument in that hook spec. If users use kedro run
it will download the “root” datasets but leave intermediate and output datasets untouched. (some other things happen too).
However, for kedro ipython
if the dataset was not downloaded in a previous pipeline run it does not exist locally and the catalog/dataset does not know it should be downloadedNok Lam Chan
08/11/2023, 12:53 PMkedro ipython
catalog alone?Florian d
08/11/2023, 12:58 PMkedro ipython
the pipeline use-case already worksNok Lam Chan
08/11/2023, 12:59 PMafter_context_created
hook looks like? I am unsure why IPython is important here. If an user start from a terminal (not IPython), wouldn’t you still want it to be loaded?Florian d
08/11/2023, 1:10 PMpython
?Nok Lam Chan
08/11/2023, 1:17 PMFlorian d
08/11/2023, 1:18 PM