I have a kedro-boot or in general a kedro deployment question. I have build a set of kedro pipelines to train and test a model, now i want to deploy predictions in a service. The service we are using dont have allot of flexibility, we can only download wheels / packages, install and run them. We can not create any folders or pull code from git. So I need to package my project. In addition to run if as a module i also want to be able to send in inputs and parameters in my code. So i have been exploring Kedor-boot, and the code below works as long as it is run from within the project repository.

from kedro_boot.booter import boot_session

kedro_boot_session = boot_session()

client = kedro_boot_session.run(
        name="get_client_app_pipeline", inputs=None
    )

data = kedro_boot_session.run(
        name="get_raw_date", inputs={"client": client}, parameters={"database": "db_raw", "table": "datatable"}
    )

print(data.head())

If i package my project, install it in another environment and try to run this code i get an error below:

RuntimeError: Could not find the project configuration file 'pyproject.toml' in /Users/christoph.imler/Documents/Work/EquinorCognite/test_deployment. If you have created your project with Kedro version <0.17.0, make sure to update your project template. See 
<https://github.com/kedro-org/kedro/blob/main/RELEASE.md#migration-guide-from-kedro-016-to-kedro-0170> for how to migrate your Kedro project.

Is there a way to run a pipeline like kedro-boot from a packaged project alone? The pipeline dont need any data other than inputs provided as arguments

It’s indeed support only booting a kedro project. We plan to support booting kedro package in the next release. See the corresponding issue

That would be a great feature! Let me know if I can help with testing:)

Would like to understand more about the constrains.The reason why Kedro separate Config and wheel are largely influenced by thw Twelve Factor App principles. https://12factor.net/ However, you can still choose to package your config into your project if that's desired. As for the

client

injection, https://kedro-org.slack.com/archives/C03RKP2LW64/p1705928537669549?thread_ts=1705925153.315459&cid=C03RKP2LW64 would be the more kedro way to solve it.

We are working on a custom data platform which has its own wrapper around APIs, cloud functions and dashboards, so we only write code for this platform, we can't change anything fundamental. It might be that Kedro is unsuitable for this platform, but it has been really helpful to use it while we developed the models and pipelines locally. To cite your reply from 3 days ago @Nok Lam Chan "Creating a custom dataset may sounds intimidating ... " , that is why i have not solved it as you mentioned with

AbstractDataset

. I still dont see how i can inject the client in runtime using this.

Thanks @Christoph Imler for the clarification. I understand that platform has their limitations. To your questions, the solution that I give takes a different path. Instead of injecting a new client, you make an explicit node to initialise this client (just as you did already in local pipeline). In a production environment, you can have it initialised as a different dataset.

# base/catalog.yml
my_client:
  type: LocalClientDataset

# prod/catalog.yml
my_client:
  type: ProdClientDataset

To run it with your local setup , you simply do

kedro run

, when running the production one, you use

kedro run --env=prod

instead

> you can still choose to package your config into your project For the record and to complete this @Nok Lam Chan point, you can have a sort of application configs by pointing the base env to a src based config. If you're using OmegaConfigLoader it's become easier to enable that : • Create a conf folder in src/your_package/conf • In settings.py point the base env to your src conf

CONFIG_LOADER_ARGS = {

"base_env": Path(__file__).parent / "conf"

}

• Add your src/your_package/conf to package_data through pyproject.toml or setup.py your wheel will contains now the base configs, and you can always override them with your environment/runtime configs

Thanks! I really appreciate the great replies:) I hope to test it all tomorrow.

Thank you for the feedback ! We plan to release it soon, we just need to update the Readme and some examples