Hi All, Another noob question... So we're currently using databricks as the compute engine, and we would like to use azure secrets to grab data from our snowflake instance. Anyone here have done this? would really appreciate the help. I've seen codes but not sure if its a node or a function. TIA!

the best way to do this is a

before_pipeline_run

or

after_context_created

hook, pick up the credentials and then inject them into the catalog object

https://docs.kedro.org/en/latest/hooks/common_use_cases.html#use-hooks-to-load-external-credentials This may be applicable

@datajoely I was able to use credentials, but I'm just wondering how will I grab data from Snowflake from the catalog? I was thinking of having a SQL file within a folder, then the path will be stored in a catalog. but not sure about how the mechanism of calling the SQL file will work...

So for this you should use the new SnowparkTableDataSet which uses snowflake’s new dataframe aPI