[Finished editing] Hello team ; I am trying to use Kedro Catalog and IO functions in standalone mode (i.e. w/o using a full Kedro structure). However I have difficulties "anchoring" the ConfigLoader object ; let me give an example (referencing file from root) : in

my_project_package/utils.py

:

from pathlib import Path

def get_project_directory(config=None) -> Path:
    """Returns the path to the project root directory"""
    return Path(__file__).parent

in

exploration/some_file.py

:

from kedro.config import TemplatedConfigLoader
from <http://kedro.io|kedro.io> import DataCatalog
from my_project_package.utils import (
   get_project_directory, # Enables to get project directory from anywhere
   )

# Catalog
conf_loader = TemplatedConfigLoader(
   get_project_directory() # Enables to get project directory from anywhere
   / "conf"
   )
catalog_conf = conf_loader.get("catalog.yml")
catalog = DataCatalog.from_config(catalog_conf)

# Load data
test_df = catalog.load("test_data")

Executing

exploration/some_file.py

: • will work from project root • Won't work from within

exploration

folder, with the following error :

[Errno 2] No such file or directory

Any clue why and how I can make this work from anywhere ?

Hey, I think I can help with that 🙂 Was just dealing with this recently https://github.com/kedro-org/kedro/issues/2819

whatever the answer is today - I want to mention making this workflow neater is a high priority for the team today

https://github.com/kedro-org/kedro/issues/2819#issuecomment-1669771871 This is the root cause - and temporary workaround.

Thanks for all the answers ! I see there is no easy solution yet 😅 ; @Nok Lam Chan regarding that solution with

_convert_paths_to_absolute_posix

, isn't it going to be an issue if I want to use the catalog to I/O from different places depending on something specified in config (e.g. I want to be able to load from either local or a DataLake) ?

This is two separate problem, though both related to the path 1. Load from local (not from root) -

_convert_paths_to_absolute_posix

2. Load from s3, this was handled by

DataCatalog

and

fsspec

under the hood, it shouldn’t matter where your

root

is in this case