Hi kedronistas :) I have a question about customization: In my company we have our own cookiecutter template resp. folder structure and naming conventions. I am struggling integrating kedro capabilities into our existing template. E.g. we don't follow the "src" naming convention but "<pkg_name>". How can I configure kedro so it knows about that and looks e.g. for the kedro_cli.py in there? So to wrap up: is it possible and if yes, what is best practice, to configure kedro to integrate well into an existing repo structure without loosing kedro functionality?

Hey @fmfreeze; I was actually just looking at this due to your comment on another thread. In short, I don't think it's straightforward to maintain Kedro CLI functionality while changing your project layout.

I think it's worth trying it out. if I understand correctly,

[tool.kedro]

in

pyproject.toml

would remain untouched, and as long as there's a

settings.py

, everything should be fine? but I might be missing things

Thank you for those super-quick responses :) Having no cli functionality out of the box and run pipelines in python is ok to me. But when it comes to kedro-viz, things might get even trickier?! So when no way exists (yet?) to tell kedro where which files are stored where, I think I have to start finding arguments to argue with my boss why we should rearrange future project structure around kedro (and not vice versa, as intended).

You can configure the source directory location, but it is still going to require a

src

folder under there I think.

I misspoke on this/misread the code; it will default to

src

is

source_dir

isn't provided, but you should be able to override it.

But when it comes to kedro-viz, things might get even trickier?!

I don't think Kedro-Viz will be any worse; it basically also delegates the context-loading to

kedro.framework.startup.bootstrap_project

, so as long as that works, you could be fine.

in other words:

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,7 @@
 [tool.kedro]
 package_name = "spaceflights"
 project_name = "Spaceflights"
+source_dir = "spaceflights"
 kedro_init_version = "0.18.8"
 
 [tool.isort]

this moves you from a

src

layout to a "flat" layout (instead of

src/spaceflights/settings.py

, you'd have

spaceflights/settings.py

)

@Juan Luis I tried your suggestion. Adding the "source_dir" works fine for renaming the "src" folder, but I cannot reproduce your "flat hierarchy" statement. In the spaceflights example, what do you change in the file-/folder structure to get a flat(ter) hierarchy?

@fmfreeze I did

mv src/spaceflights/ .

, hence having a

spaceflights

directory at the root of the project. is that what you were attempting?

It is not exactly what I was attempting, but also that I cannot reproduce. (

ModuleNotFoundError: No module named spaceflights

).

interesting, that happened to me yesterday the first time I tried it, and now I'm reproducing it again. while I figure this out, @fmfreeze what would be your ideal

tree

output? here's mine at the moment:

.
├── README.md
├── conf
│   ├── ...
├── data
│   ├── ...
├── pyproject.toml
├── setup.cfg
└── spaceflights
    ├── __init__.py
    ├── __main__.py
    ├── __pycache__
    ├── pipeline_registry.py
    ├── pipelines
    └── settings.py

Thx @Juan Luis for helping out. Your tree suggestion would be nice already. An ideal solution would be even more flexible to achieve a more granular structure. E.g. having settings.py & pipeline_registry.py in it's own kedro subfolder. But that would just be a nice-to-have to avoid confusion among our scientists (which are no SW Devs - and don't have to be :)

okay, I was sending you the wrong path - this is the actual

source_dir

for my tree above:

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,7 @@
 [tool.kedro]
 package_name = "spaceflights"
 project_name = "Spaceflights"
+source_dir = "."
 kedro_init_version = "0.18.8"

(by using

source_dir = "spaceflights"

Kedro was looking in the wrong place - I'm not sure how I got it working yesterday)

Done https://github.com/kedro-org/kedro/issues/2553#issue-1694559376

thanks for your patience @fmfreeze!

What is the part you mean by hardcoded? What would you change?

what I meant is that the code I highlighted is part of Kedro and is what prevents users from customizing where the

settings.py

and

pipeline_registry.py

- so, to allow for the customization you proposed @fmfreeze, Kedro would need to make those lines more generic and allow for some configuration, similar to what

source_dir

does. does this make sense?

Yes, it makes sense and would be fine. Your sugesstion with

source_dir

works also nice. 👍

great to know! 🥳