hey Kedro Community, I was wondering if someone ma...
# questions
m
hey Kedro Community, I was wondering if someone may be able to help with something. We have a pipeline which runs just fine in our
dev
env, but when deploying and running in
test
, we are getting the following error:
Copy code
DatasetError: Failed while loading data from dataset ParquetDataset(filepath=psi-test-data/***********/data/02_intermediate/formatted_transactions_df.parquet, load_args={}, protocol=s3, save_args={}).
An error occurred (PreconditionFailed) when calling the GetObject operation: At least one of the pre-conditions you specified did not hold
What's weird is that the error is not consistent. Sometimes the node responsible runs through just fine, other times it errors out, without anything (at least what we can see) changing. Any help would be massively appreciated! @Tom McHale
n
looks like it's related to the object store (s3 maybe?)
try upgrade the
fsspec
/
boto
version to see if this solve the problem. Could also be some weird race condition with multiple runs, cache on fsspec when listing directory
🙏 1