Hello all. I want to log the number of rows for the datasets at each step of my pipeline. Do you know a good way to do that ? I’ve seen this great pull request to about logging the number of columns and rows. Is this already possible in the current module ? I didn’t find much documentation about this.

Thanks @Nicolas Oulianov. This isn't in the current module yet. But would be reviewed during backlog grooming. How does logging the number of rows for the datasets help your workflow? is it mainly for debugging? Pls add your comment to the ticket so we can consider it during the session.

Yes, it’s for debugging. The goal is to notice big drop of rows during one data transformation step. For example, after one node, I may see that my number of lines drops by 30% when it’s supposed to stay the same. How to monitor this with the currently available tools ?

I can think of two ways: • Logging this information via a

before_dataset_saved

hook • Using the Experiment Tracking feature and viewing the trends there

Thank you @datajoely ! The hook would be great to do that. Do you have an example of implementation ? Eg link to kedro project on github I’ve read through the docs but can’t figure out how to start setting it up.

these are the exp tracking docs

@Nicolas Oulianov this feature has been implemented on the latest Kedro-Viz release. Can you confirm if this solves your pain point and provide feed back. Thanks.

Hi Nero ! Thanks for sharing this update. Looks great on the demo, thanks for shipping it ! However, on my code: • By default, it shows N/A rows and columns for my ExcelDataSet and CSVDataSet • The preview feature doesn’t work with PartitionedDataset