Is there some flag in `kedro viz` to disable datas...
# questionsm
marrrcin
12/18/2023, 9:48 AMIs there some flag in
kedro viz.exists()n
Nok Lam Chan
12/18/2023, 9:57 AMAre you on the latest version of viz? I don't recall it checking datasets
m
marrrcin
12/18/2023, 9:58 AMYes, 6.7.0
marrrcin
12/18/2023, 9:59 AMIt goes into
kedro_viz.server.populate_datadata_access_manager.resolve_dataset_factory_patternscatalog.exists(dataset_name)s
Simon Wolf
12/18/2023, 9:59 AMHey I had the same problem, so I would also be interested. In some other cases the check of data took too long and kedro-viz was canceled after 60s timeout. In this case, it would also be good to deactivate the check. My interim solution was my own dataset class in which I adapted the _exists function...
🙈 1
m
marrrcin
12/18/2023, 10:01 AMI imagine sth like `kedro viz --skip-datsets`would be nice
👍 1
s
Simon Wolf
12/18/2023, 10:06 AMOther interim solution would be to go back to kedro-viz version 6.6.1. If I remember correctly, the data check did not yet exist in this version
👍🏼 1
m
marrrcin
12/18/2023, 10:08 AMThat’s actually super convenient, thanks @Simon Wolf
marrrcin
12/18/2023, 10:13 AMFYI @Nero Okwa
n
Nero Okwa
12/18/2023, 10:42 AMInteresting, I don't recall this either. FYI @Rashida Kanchwala
Nero Okwa
12/18/2023, 10:44 AM@Simon Wolf please provide more context on the other cases when the data check too long and Kedro-Viz cancelled.
s
Simon Wolf
12/18/2023, 11:05 AMI use a custom PartitionedDataset to load tables from databricks hive storage. In this case the _list_partitions function takes a while, because there are many tables in the database. executing the spark function to list all tables takes a while. Probably also because the tables are distributed in the cluster. The _exists() function normally simply calls bool(self._list_partitions()) for a PartitionedDataset. and if the list_partitions() takes too long, the whole thing doesn't work. I could also imagine that this could be a problem in other scenarios (using the standard PartitionedDataset or other single Dataset Classes) if the data is not stored locally and there are therefore delays when loading/listing files...
a
Ankita Katiyar
12/18/2023, 12:00 PMI don’t know if it’s been released yet but this has been resolved - https://github.com/kedro-org/kedro-viz/issues/1645
👍🏽 1
👍 2
r
Rashida Kanchwala
12/18/2023, 12:18 PMYes we will be releasing the fix for it hopefully today in Kedro-viz 7.0
m
marrrcin
12/18/2023, 12:19 PMIt will probably not cover the latency problem though
r
Rashida Kanchwala
12/18/2023, 6:59 PMTrue. Looping @Ravi Kumar Pilla
r
Ravi Kumar Pilla
12/18/2023, 7:18 PMThe check was introduced in kedro viz 6.7.0 to discover factory patterns but I see the problem here with the latency. Quick solution could be as @marrrcin suggested to have a flag to disable this discovery. But, this will also disable the dataset factory pattern discovery. This needs further discussion with the team and I will have a look at it. Thank you
👍🏽 1
Ravi Kumar Pilla
12/18/2023, 11:00 PMHi All, Thank you for your patience. We had an internal discussion with the team and decided to drop dataset factory pattern discovery implementation for kedro viz 7.0.0. This removes the dataset existence check and will resolve the issues mentioned in this thread. However, this will also remove the support for dataset factory patterns from experiment tracking. We will add this to our backlog and work on it.
👀 1
👍 1
2 Views