Is there a way to access the config values of my conf/base/globals.yml inside a node? I know I can load them in by creating a ConfigLoader, however since they are used to create the DataCatalog, I assume they were read into memory already and maybe accessible some other way?

Do you want all the config values or just one particular one? If its just one you can put add a value to your

parameters.yml

like:

important_global: ${important_global_value}

Then access that value in your node. One way is like this:

node(foo, "params:important_global", "return_dataset")

Either way, can you explain what you’re trying to accomplish with this? There might be something more straightforward.

So we fill globals with values per environment, such that we can load in our data catalog in different environments (as I think its intended). So it contains some S3 bucket paths, and other resource names that are different per environment. I’d like to be able to also access these during runtime in my node, just like I would with other values from parameters.yml. But I rather not create duplicate values (similar values in globals.yml and parameters.yml).

Totally makes sense! Duplicating them is definitely not ideal. I guess I’m asking what you want to do with the values in the node? There might be some other solution to that problem.

If you need access to things that have to do with the I/O or file system, a better solution might be to create a custom dataset (or subclass an existing one) to use that information. E.g., let's say you wanted to add a column to your dataset with the name of the s3 bucket. Why not create a dataset that subclasses the

pandas.CSVDataSet

and adds the column? During the I/O, all the info in the catalog entry is visible to the dataset class.