Vincent Liagre
08/30/2023, 3:06 PMmy_project_package/utils.py
:
from pathlib import Path
def get_project_directory(config=None) -> Path:
"""Returns the path to the project root directory"""
return Path(__file__).parent
in exploration/some_file.py
:
from kedro.config import TemplatedConfigLoader
from <http://kedro.io|kedro.io> import DataCatalog
from my_project_package.utils import (
get_project_directory, # Enables to get project directory from anywhere
)
# Catalog
conf_loader = TemplatedConfigLoader(
get_project_directory() # Enables to get project directory from anywhere
/ "conf"
)
catalog_conf = conf_loader.get("catalog.yml")
catalog = DataCatalog.from_config(catalog_conf)
# Load data
test_df = catalog.load("test_data")
Executing exploration/some_file.py
:
• will work from project root
• Won't work from within exploration
folder, with the following error : [Errno 2] No such file or directory
Any clue why and how I can make this work from anywhere ?Nok Lam Chan
08/30/2023, 3:12 PMdatajoely
08/30/2023, 3:12 PMNok Lam Chan
08/30/2023, 3:13 PM```from kedro.framework.context.context import _convert_paths_to_absolute_posix
config_loader = ConfigLoader("../conf")
conf_catalog = config_loader["catalog"]
conf_catalog = _convert_paths_to_absolute_posix(
project_path=Path("../").resolve(), conf_dictionary=conf_catalog
)
catalog = DataCatalog.from_config(conf_catalog)
catalog.load("companies")```
You will need this
_convert_paths_to_absolute_posix
if you are not starting from the root directory - this will be solved by https://github.com/kedro-org/kedro/issues/2965 (coming soon)Vincent Liagre
08/30/2023, 3:25 PM_convert_paths_to_absolute_posix
, isn't it going to be an issue if I want to use the catalog to I/O from different places depending on something specified in config (e.g. I want to be able to load from either local or a DataLake) ?Nok Lam Chan
08/30/2023, 3:33 PM_convert_paths_to_absolute_posix
2. Load from s3, this was handled by DataCatalog
and fsspec
under the hood, it shouldn’t matter where your root
is in this case