Hi fellows, is there a way to avoid to write the

stats.json

file when running pipelines? I couldn't find the information in the documentation.

It is from viz, we have a pending ticket to move that into a .viz folder so it’s less intrusive. Would that be sufficient?

To give a bit context, I am fighting Terraform, Airflow and AWS at the same time which leads to struggles with read/write permissions. My pipeline fails because it can not write

stats.json

which is why I wanted to remove it. I can also manage two virtual environments, the production one not including

kedro-viz

which should solve the problem too, I guess.

Make total sense, in this case

DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",)

should helps. | I did a quick search in docs.kedro.org and found nothing, so I think we should definitely add a mention somewhere in the docs. Unfortunately we can't have comments in the JSON file itself, so probably somewhere in the docs?

Could we add an example in the section above? As an example on how to use it.

I think so, do you want to start a PR for this? I think the best thing to do is add an example in kedro-viz documentation, then link the example in the section above.

I can, sure.

Thanks!

Hi @Flavien, I am not sure why the pipeline run fails as it should ideally spit out a warning as below instead of an error. However, as you and Nok pointed out -

DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",)

should help resolve the issue. Thanks @Nok Lam Chan

Logs for kedro run with only read permissions on project dir
INFO   Completed 20 out of 20 tasks                                                                                          sequential_runner.py:90
[03/04/24 11:59:02] INFO   Pipeline execution completed successfully.                                                                                        runner.py:119
          INFO   Loading data from ingestion.prm_spine_table_clone (MemoryDataset)...                                                                        data_catalog.py:483
          WARNING Unable to write dataset statistics for the pipeline: [Errno 13] Permission denied: 'stats.json'

Thank you

@Ravi Kumar Pilla, my apologies, my message was not clear. The pipeline does not fail and is similar to your example.