hey i am using the latest version of Kedro 0 18 4 i am tryin Kedro #questions

hey, i am using the latest version of Kedro (0.18...

dor zazon

01/10/2023, 9:56 AM

hey, i am using the latest version of Kedro (0.18.4). i am trying to use the session.load_context() function and the functions returns :

datajoely

01/10/2023, 10:00 AM

have you got any custom hooks enabled?

dor zazon

01/10/2023, 10:03 AM

dor zazon

01/10/2023, 10:03 AM

i was able to run it two days ago

dor zazon

01/10/2023, 10:04 AM

i just install the pyarrow package

dor zazon

01/10/2023, 10:04 AM

and something went broken

datajoely

01/10/2023, 10:22 AM

That shouldn’t have affected things, could you post the stack trace

dor zazon

01/10/2023, 10:23 AM

Untitled.cpp

datajoely

01/10/2023, 10:30 AM

I think this bit of dynamic pipeline-ing here is causing the issues:

Copy code

│ /Users/dorzazon/Documents/workspace/egged-kedro/egged/src/egged/pipelines/data_processing/pipeli │
│ ne.py:28 in create_pipeline                                                                      │
│                                                                                                  │
│   25 │   │   │   )])                                                                             │
│   26 │   pipelines = []                                                                          │
│   27 │   # get catalog                                                                           │
│ ❱ 28 │   catalog = _get_catalog()                                                                │
│   29 │   for dataset in catalog.load('params:dataset_names'):                                    │
│   30 │   │   pipelines.append(pipeline(pipe=template,                                            │
│   31 │   │   │   │   │   │   │   │   │   │   inputs={"dataset": dataset},

datajoely

01/10/2023, 10:31 AM

it’s best to access the catalog live via hooks

dor zazon

01/10/2023, 10:32 AM

how can i find information on how to acess catalog live via hooks?

datajoely

01/10/2023, 10:32 AM

https://kedro.readthedocs.io/en/stable/hooks/introduction.html

dor zazon

01/10/2023, 10:35 AM

but, what is the problem with acessing the catalog like i did?

dor zazon

01/10/2023, 10:36 AM

it worked for me two days ago

dor zazon

01/10/2023, 10:36 AM

now something is broken

dor zazon

01/10/2023, 10:40 AM

and, which hook should i use to mimic what is did?

dor zazon

01/10/2023, 10:41 AM

this is my create_pipeline functiom:

dor zazon

01/10/2023, 10:41 AM

Copy code

template = pipeline(
    [
        node(
            func=preprocess_df,
            inputs=["dataset", 'params:dataset_config', 'params:col_names_config'],
            outputs="preprocessed_dataset_name",
            name="preprocess_df_node"
        )])
pipelines = []
# get catalog
catalog = _get_catalog()
for dataset in catalog.load('params:dataset_names'):
    pipelines.append(pipeline(pipe=template,
                                    inputs={"dataset": dataset},
                                    parameters={"params:dataset_config": f'params:{dataset}',
                                                'params:col_names_config': 'params:col_names_config'},
                                    outputs={"preprocessed_dataset_name": f'preprocessed_{dataset}'},
                                    namespace=f'preprocessed_{dataset}'))
# return all pipelines
final_pipeline = pipelines[0]
for pipe in pipelines[1:]:
    final_pipeline += pipe

5 Views

Open in Slack

Previous Next