Hi Team! I have a PR(https://github.com/kedro-org/kedro-plugins/pull/1008/files) to add

s3://

support to spark dataset along with

s3a://

on a user request. I know s3a is recommended for spark but would love to take your viewpoint on would you really want that and for what use case? is it for EMRFS?

Hi @Anu Arora so this is breaking some of the tests - I think we'd need to fix this to get the change in. I think you are right the historical reasons for things liek

s3a

s3n

etc don't exist anymore https://github.com/kedro-org/kedro-plugins/actions/runs/13262221663/job/37021194513?pr=1008

yes checking though i have only changed in spark_dataset file which passes and fail is happening here which i didn’t even touch.

FAILED tests/databricks/test_base_table_dataset.py::TestBaseTableDataset::test_save_schema_spark - kedro.io.core.DatasetError: Failed while saving data to dataset BaseTableDataset(database=test, dataframe_type=spark, format=delta, table=test_save_spark_schema, version=None, write_mode=overwrite)