https://kedro.org/ logo
#questions
Title
# questions
c

Christoph Imler

01/25/2024, 12:59 PM
I have a kedro-boot or in general a kedro deployment question. I have build a set of kedro pipelines to train and test a model, now i want to deploy predictions in a service. The service we are using dont have allot of flexibility, we can only download wheels / packages, install and run them. We can not create any folders or pull code from git. So I need to package my project. In addition to run if as a module i also want to be able to send in inputs and parameters in my code. So i have been exploring Kedor-boot, and the code below works as long as it is run from within the project repository.
Copy code
from kedro_boot.booter import boot_session

kedro_boot_session = boot_session()

client = kedro_boot_session.run(
        name="get_client_app_pipeline", inputs=None
    )

data = kedro_boot_session.run(
        name="get_raw_date", inputs={"client": client}, parameters={"database": "db_raw", "table": "datatable"}
    )

print(data.head())
If i package my project, install it in another environment and try to run this code i get an error below:
Copy code
RuntimeError: Could not find the project configuration file 'pyproject.toml' in /Users/christoph.imler/Documents/Work/EquinorCognite/test_deployment. If you have created your project with Kedro version <0.17.0, make sure to update your project template. See 
<https://github.com/kedro-org/kedro/blob/main/RELEASE.md#migration-guide-from-kedro-016-to-kedro-0170> for how to migrate your Kedro project.
Is there a way to run a pipeline like kedro-boot from a packaged project alone? The pipeline dont need any data other than inputs provided as arguments
K 1
t

Takieddine Kadiri

01/25/2024, 1:19 PM
It’s indeed support only booting a kedro project. We plan to support booting kedro package in the next release. See the corresponding issue
🥳 1
c

Christoph Imler

01/25/2024, 2:08 PM
That would be a great feature! Let me know if I can help with testing:)
👍 1
n

Nok Lam Chan

01/25/2024, 2:13 PM
Would like to understand more about the constrains.The reason why Kedro separate Config and wheel are largely influenced by thw Twelve Factor App principles. https://12factor.net/ However, you can still choose to package your config into your project if that's desired. As for the
client
injection, https://kedro-org.slack.com/archives/C03RKP2LW64/p1705928537669549?thread_ts=1705925153.315459&amp;cid=C03RKP2LW64 would be the more kedro way to solve it.
c

Christoph Imler

01/25/2024, 3:05 PM
We are working on a custom data platform which has its own wrapper around APIs, cloud functions and dashboards, so we only write code for this platform, we can't change anything fundamental. It might be that Kedro is unsuitable for this platform, but it has been really helpful to use it while we developed the models and pipelines locally. To cite your reply from 3 days ago @Nok Lam Chan "Creating a custom dataset may sounds intimidating ... " , that is why i have not solved it as you mentioned with
AbstractDataset
. I still dont see how i can inject the client in runtime using this.
n

Nok Lam Chan

01/25/2024, 3:12 PM
Thanks @Christoph Imler for the clarification. I understand that platform has their limitations. To your questions, the solution that I give takes a different path. Instead of injecting a new client, you make an explicit node to initialise this client (just as you did already in local pipeline). In a production environment, you can have it initialised as a different dataset.
Copy code
# base/catalog.yml
my_client:
  type: LocalClientDataset

# prod/catalog.yml
my_client:
  type: ProdClientDataset
To run it with your local setup , you simply do
kedro run
, when running the production one, you use
kedro run --env=prod
instead
See more about configuration environment here if you are interested.
So essentially you will be running the same pipeline, just different definition of catalgo.
t

Takieddine Kadiri

01/25/2024, 8:02 PM
> you can still choose to package your config into your project For the record and to complete this @Nok Lam Chan point, you can have a sort of application configs by pointing the base env to a src based config. If you're using OmegaConfigLoader it's become easier to enable that : • Create a conf folder in src/your_package/conf • In settings.py point the base env to your src conf
CONFIG_LOADER_ARGS = {
"base_env": Path(__file__).parent / "conf"
}
• Add your src/your_package/conf to package_data through pyproject.toml or setup.py your wheel will contains now the base configs, and you can always override them with your environment/runtime configs
c

Christoph Imler

01/25/2024, 9:47 PM
Thanks! I really appreciate the great replies:) I hope to test it all tomorrow.
@Takieddine Kadiri got to test the new version of kedro boot today. It works great on my use case!
🥳 2
t

Takieddine Kadiri

02/01/2024, 5:11 PM
Thank you for the feedback ! We plan to release it soon, we just need to update the Readme and some examples
🥳 1
👍 1
5 Views