Hi all, I am setting up data science and MLOps for a large media company that already uses AWS. We’re transferring from no mlops to a Kedro+mlflow+doltlab setup. I want the pipelines to run on AWS batch because we are training custom LLM’s, so using spot instances and automaticc resource provisioning is quire crucial. However, the latest Kedro documentations states that the AWS batch deployment pattern is now deprecated. And i can see that my implementation no longer works on AWS after updating. I really want to use some of the discussed features, like the default dataset (since I make heavy use of modular pipelines), and AWS batch requires this, and keeping the pipelines and the datacatalog in sync is very brittle. So i want to keep updating kedro. But i also want to keep using AWS batch. Is there a strong reason why the batch support was deprecated? Or is this simply a matter of updating the implementation. I could update mine, and open a pull request to update the docs, if this implementation will keep receiving support. Or is there a fundamental reason why Kedro 0.19 and AWS batch no longer work together?

I believe there's no fundamental desire to explicitly drop support for AWS Batch, and it's more to signal that it hasn't been tested in a while/we haven't tested it/it's not currently being maintained. @Jo Stichbury or somebody else can correct me if that wasn't the intention. Maybe the word "deprecated" also is a bit strong, since it feels like an intentional break. That being said, we felt that maintaining so many of these deployment guides that weren't automatically tested in any way and could break at any time was not ideal. So, to answer your questions:

Is there a strong reason why the batch support was deprecated? Or is this simply a matter of updating the implementation.

No strong reason; simply a matter of updated the implementation. Also, you mention 0.19, which isn't even released yet (and may be some time).

I could update mine, and open a pull request to update the docs, if this implementation will keep receiving support.

Generally welcome contributions, although I'll let somebody else chime in on whether the best thing is to update the guide that we don't intend to maintain. I think we've generally found what some others have done with deployment plugins (like

kedro-azureml

) to be a more effective way of keeping this knowledge. FYI @Juan Luis @Merel @Yetunde if you have any opinion on the right way to make the contribution here 🙂

Hi @Hugo Evers I agree with @Deepyaman Datta's analysis. We were concerned that the docs above were untested and we couldn't in good faith leave them without some indication that they were likely to be out of date. Perhaps "deprecated" is too strong for the message we intended, which was "we discourage you from committing to using this approach and later finding out that the docs are flawed" but I'm not sure how else to convey the message. Maybe something like "We haven't tested these docs for correctness and, even if they work now, we cannot guarantee to maintain them". I am more than happy to review contributions to update the docs and am still looking into how we present docs pages to reflect better when they were last updated/what version they were last tested against, since I think that will help guide choice.

hi @Hugo Evers, thanks to this feedback @Jo Stichbury went ahead and replaced "deprecated" with "outdated documentation that needs review", hope that clarifies things!

great, I found my own error, i was cobbling PR’s together to get the kedro functionality i wanted, and I got a really ugly error, indicating to me that something broke. However, that was an error on my part.

So is there plan to update the AWS batch deployment docs?

Did you run into an issue following the docs? I'm not aware of any plan to update the docs at this time, but if you do deploy to AWS Batch and modify or use different methods, feel free to suggest an improvement to the docs!

Hi @Nan Dong, we don’t plan to update the AWS batch deployment docs soon. However there are people in the Slack with experience deploying it. If you run into any issue we could try to help too.

the AWS batch deployment works, but its quite barebones. looking back i would have used the prefect aws batch agent instead