I’m working on a big project that is about to hit it’s next phase. We are using kedro and we have a large single kedro project. To give you an idea on how big, we have about 500+ catalog entries, 500+ nodes in different kedro pipelines (we disabled the default sum of all pipelines as it is too large to use). Now I know the general guideline is to split your project in several smaller ones if it becomes too big, but I need some advice/opinions on this. I’ll explain more details in the comments. Thanks!

Someone will reply to you shortly. In the meantime, this might help:

The project we have is a single solution rolled out to 50 countries across EMEA. For each country, we have the same code base, nodes and pipelines. But different configuration (parameters) that we store in different folders in

conf/

, i.e. every country is a

kedro_env

. Hence, splitting the project based on country won’t reduce the size and complexity. The only logical split I see is based on grouping certain pipelines and nodes and moving those to separate kedro projects, roughly with the following dependencies

— B — C
A—|
      — D — E

These projects would share some code but that’s not too bad the handle. The biggest challenge is that would they share an extensive amount of config. So a change to a param would then need to be replicated in several projects. Any advice on how you would solve this would be awesome.

To help reduce your catalog size, can you use kedro dataset factory? https://docs.kedro.org/en/stable/data/kedro_dataset_factories.html

Hi @Matthias Roels, If its just about sharing a param, you can consider saving the params in an s3 bucket as a CSV (maybe) and load it in a Kedro Hook like

before_pipeline_run

. But as @Alexandre Ouellet mentioned, kedro dataset factories are a great way to reduce the config complexity. Thank you

The way we manage these is using a monorepo management framework called Alloy, it was actually initially developed to help manage Kedro projects like this at scale. With Kedro projects we separate our functional logic from pipeline logic, and the functional logic is stored within Python packages, and the the pipeline logic within separate Python packages. In Alloy these packages are called

components

. With Alloy there's also the concept of

apps

, these are .yml files which declare combinations of

components

and relevant config and metadata. We then use Alloy to assemble

apps

as needed for different contexts. Often different regions, or business units have their own app which is a specific combination of components relevant to them. By managing the code in this way there's really clear separations of concern and it's super easy to scale development. More info: https://medium.com/quantumblack/engineering-solutions-for-reuse-1ff5a81d8611

Speaking of (and related I believe) : is there a way to manage a dataset of about 1 million files in AzureML? The files are about 4k each, and are entirely independent from each other

@Alexandre Ouellet would

PartitionedDataset

work for you? (feel free to open a separate thread to keep this one focused)