Hi friends when I have a parameters yml ```data science acti Kedro #questions

Hi friends, when I have a parameters.yml ```data_s...

Jonas Kemper

03/15/2023, 12:03 PM

Hi friends, when I have a parameters.yml

Copy code

data_science:
  active_modelling_pipeline:
    model_options:
      test_size: 0.2...

and I load it via

Copy code

conf_loader = kedro.config.ConfigLoader(".")
parameters = conf_loader['parameters']

that returns me

Copy code

{'data_science': {'active_modelling_pipeline': {'model_options': {'test_size': 0.2,

. When in another place, I

Copy code

data_catalog = DataCatalog.from_config(catalog, credentials)
data_catalog.add_feed_dict(parameters)

this won't work, because eventually that'll land me

Copy code

ValueError: Pipeline input(s) {'params:data_science.candidate_modelling_pipeline.model_options.random_state', ...} not found in the DataCatalog

What's the intermediate step that I'm missing?

Jonas Kemper

03/15/2023, 12:08 PM

I figured it will probably have something to do with https://github.com/kedro-org/kedro/blob/a6919efb2bec81e864fe7fa02849394bcc7ecd17/tests/pipeline/test_modular_pipeline.py#L122 ... but didn't get much further than that.

Merel

03/15/2023, 12:09 PM

@Jonas Kemper what are you trying to do?

Jonas Kemper

03/15/2023, 12:10 PM

Working on an implementation of KedroService, where we're loading catalog and parameters via an s3 URL that get's passed in through a FastAPI endpoint. We want to use configloader to load those config files and then pass them into a pipeline/runner.

Merel

03/15/2023, 12:13 PM

And what is this step for? Why are you adding parameters into the catalog?

Copy code

data_catalog = DataCatalog.from_config(catalog, credentials)
data_catalog.add_feed_dict(parameters)

Jonas Kemper

03/15/2023, 12:14 PM

I probably shouldn't. 🙈 Eventually I'll need them to end up in my

runner.run(pipeline, data_catalog)

somehow, don't I?

Jonas Kemper

03/15/2023, 12:15 PM

(pipeline comes from

pipeline = register_pipelines()[pipeline_name]

)

Merel

03/15/2023, 12:18 PM

hmmm.. I don’t know on top of my head how to do this. But I’ll have a think.

🙏 1

Jonas Kemper

03/15/2023, 12:18 PM

Thank you. 🙂

Nok Lam Chan

03/15/2023, 12:32 PM

Are you using all these component without a Kedro project? And the problem here is the parameters isn’t loaded into catalog?

Jonas Kemper

03/15/2023, 12:32 PM

Yes.

Nok Lam Chan

03/15/2023, 12:33 PM

Hm, let me try

🥳 1

Nok Lam Chan

03/15/2023, 12:39 PM

The problem is the

params:

syntax is something handled by

KedroContext

(search for KedroContext._get_feed_dict) When you use

add_feed_dict

, it really just store the dictionary as A MemoryDataSet

👀 1

💡 1

Jonas Kemper

03/15/2023, 12:40 PM

Can you think of any straight-forward workaround?

Nok Lam Chan

03/15/2023, 12:41 PM

You are using a Kedro Project so I assume KedroSession/context isn’t available?

Jonas Kemper

03/15/2023, 12:42 PM

No, not currently.

Jonas Kemper

03/15/2023, 12:42 PM

Probably not a good idea to try and hijack

from kedro.framework.context.context import _get_feed_dict

Nok Lam Chan

03/15/2023, 12:51 PM

It would be easier if it’s possible to just create

KedroContext

and let it create the

catalog

etc

👍 1

Nok Lam Chan

03/15/2023, 12:53 PM

Otherwise, you may either fake a

KedroContext

to just use the

context._get_feed_dict

method or just copy that piece of code_

Nok Lam Chan

03/15/2023, 12:53 PM

But I think having KedroContext will be ideal

Jonas Kemper

03/15/2023, 12:53 PM

Hmhm, I understand, let me evaluate that option and then report back to you here. :)

Nok Lam Chan

03/15/2023, 1:00 PM

https://github.com/kedro-org/kedro/issues/2169#issuecomment-1447001865 This may be relevant, some of our community take a different approach by creating a WebServerRunner . It describes the high-level approach but there isn’t code for reference.

👀 1

3 Views

Open in Slack

Previous Next