Hello Kedro team :raised_hands::skin-tone-2: Over...
# questions
Hello Kedro team 🙌🏻 Overall TLDR: How can I run a piece of code after a node finished and a dataset was saved, while still being able to obtain some information about the node which run and where the dataset was saved. Long form: I am trying to use Weights and biases (WandB) with Kedro to track the output of dataset nodes. WandB creates an artifact of a dataset by getting its hash and if this changes then it’s a new version of that dataset. This means the dataset needs to exist at the time I want to log it in WandB. As at the end of a node in Kedro the data is not yet saved, I looked into the
hook, but even at this stage it’s still not saved. The
hook could work but it does not have the catalog information and therefore I do not know where the dataset was saved to get its hash with WandB.
hook would work but this runs very late and I want to track each node individually. The only working options I found, is to create custom datasets from
which will log each file individually after it was saved. But this would add a lot of overhead to write for each file type. Is there another way? Thank you very much!
🤔 1
🔥 1
That’s a nice one!
😄 1
See • https://github.com/kedro-org/kedro/pull/2296https://github.com/kedro-org/kedro/issues/2258 Is the dataset hook not enough? #2258 was created exactly to extend the use case for wandb. There was some discussion to create kedro-wandb extension previously IIRC.
There is an example WandbHook in the issue as well.
Thank you very much for your reply! This is useful but wont work exactly for my use case. In that PR, the user is saving the full data as an artifact, while I am mostly looking for where the data was saved (the path in the catalog) to get its hash. I am not quite sure if I can obtain the catalog information from the
variable of
. The reason I am doing this is because I do not want to upload the entire data to WandB 🙂
What exactly do you need from catalog?
Copy code
  type: pandas.CSVDataSet
  filepath: path/to/file.csv
for the specific dataset I just saved out of a node. The above will be the output of a node. In the above it would be 1 file, but for different nodes I will have a folder of large data. I want to make WandB log the hash of a file or directory after it was saved and after a node finished.
is a great place to run my wandb logger, but I am not sure how to obtain the filepath.
If some argument is not available, you can always use the fact that Hook is a stateful object.
Copy code
class MyHook:

def after_catalog_create(self, catalog):
  self.catalog = catalog

def after_dataset_saved(self, dataset):
   self.catalog  # it doesn't exist in the signature but because of the execution order, this is available
👍 1
I don’t remember if this is included in our docs already, I check GH quickly but I couldn’t find the issue Cc @Juan Luis
Oh! This would do it! Trying now
K 1
The above catalog would give me the entry names only of the catalog using
and can load the data using
but I cant see to be able to get the filepath of an entry?
got it!
@Cãtãlin Alexandru Can you share what you did at the end? It will help other users in the future, we also have our Slack archive which hopefully at some point can be googled.
this 1
👍 2
Will do and get back to this in afternoon! For the above message crossed, it was just me not knowing where to find the data in the catalog object. Got it with the debugger. Will write a more comprehensive msg later
🥳 1
Thank you everyone for your help! The issue was that I could not run a piece of code (weights and biases) after a node ends and get a hash of the dataset created. The best way to approach this is to use the
hook. It runs after a node ends and AFTER the dataset was saved, which is the important part for me. The
hook will run after a node ends but before dataset is saved which won’t work for me.
hook doesn’t have the catalog unfortunately to obtain the path of where the dataset was saved, but this can be fixed with Nok’s suggestion. See below something which works for me and how to obtain the file path of the dataset saved:
Copy code
class WandBLogging:
    """Uses Weights and Biases to log artifacts after a node/run ends."""

    def __init__(self):
        self.catalog = None

    def after_catalog_created(self, catalog):
        self.catalog = catalog

    def after_dataset_saved(self, dataset_name, data, node):
        filepath = str(self.catalog.datasets.__dict__[dataset_name]._filepath)
K 2
This is awesome! Thanks for sharing this back to the community :) Optionally you can use the c`catalog.__get_dataset` method
🙌 1
Thank you! Will test and update the above when i can if its simpler 🙂 what i wrote was the first thing i saw in the debugger
👍🏼 1