hey guys,
Anyone know how to pass AWS role arn fo...
# questionsf
Fazil B. Topal
09/13/2023, 3:42 PMhey guys,
Anyone know how to pass AWS role arn for kedro? Ive been hunting down s3fs and other stuff but not much luck. I can work with local aws profile but in cloud I have optiona to pass ARN but couldn't see anyway to pass this. Any help is much appreciated
d
datajoely
09/13/2023, 3:47 PMmaybe worth building a custom dataset using this?
https://aws-sdk-pandas.readthedocs.io/en/stable/api.html
f
Fazil B. Topal
09/13/2023, 3:48 PMWe're using parquet dataset directly, I just don't know if we can use ARN as is or not supported so that I need to do custom dataset
d
datajoely
09/13/2023, 3:57 PMf
Fazil B. Topal
09/13/2023, 4:00 PMI just did, im getting this guy:
Copy code
AioSession.__init__() got an unexpected keyword argument 'role_arn'd
datajoely
09/13/2023, 4:00 PMwhat version of fsspec do you have installed?
datajoely
09/13/2023, 4:01 PMoh sorry I’m being silly
datajoely
09/13/2023, 4:01 PMthat’s arrow not fsspec
f
Fazil B. Topal
09/13/2023, 4:01 PM Copy code
"2023.6.0"Fazil B. Topal
09/13/2023, 4:02 PMI feel like there has to be way to pass this guy without me doing boto stuff 😄
d
datajoely
09/13/2023, 4:02 PMimage.png
datajoely
09/13/2023, 4:02 PMin the fsspec code they are doing some ARN regex
datajoely
09/13/2023, 4:02 PMmaybe its just a bucket prefix?
f
Fazil B. Topal
09/13/2023, 4:03 PMNo idea, I think looks like they just exrtract the raw bucket name here, don't care about the arn
Fazil B. Topal
09/13/2023, 4:04 PMIs there a way to invoke the pyarrow s3fs like you linked?
d
datajoely
09/13/2023, 4:04 PM😞
f
Fazil B. Topal
09/13/2023, 4:04 PMDo I need to do anything special there
d
datajoely
09/13/2023, 4:04 PMIs there a way to invoke the pyarrow s3fs like you linked?no I don’t think so