Hello @marrrcin & Kedro community I'm currently using the Kedro-SageMaker plugin to run a Kedro pipeline on SageMaker. My Kedro project has multiple parameters (using several parameters YAML files). While executing

kedro sagemaker run -i <image name>

, I encountered this error:

ClientError: ValidationError for UpdatePipeline operation: Unable to parse pipeline definition. Model validation failed - container ProcessingEnvironmentMap length (132) exceeds maximum limit (100).

In my scenario, the number of parameters surpasses the 100 limits. How can I best manage this situation? Is there an option in SageMaker to increase the parameter limit? Looking forward to your suggestions.

Hi @Vijay Jawaharlal, I’m happy that you’re using our plugin! Please add an issue on our GitHub and explain it in more details - preferably with some example project so that we could replicate it. Thanks!

I have added an issue in your repo. Unfortunately, I can't share my code. but it is easy to reproduce with spaceflight example. Here are the steps to reproduce the issue. - Add a dummy_parameter.yml file under conf/base/parameters/ in the kedro space flight example code. (check dummy_pamarameter.yml file content below) - Simply run

kedro sagemaker run --auto-build

dummy_parameter.yml file content:

dummy_params:
  parameter1: 1
  parameter2: 2
  parameter3: 3
  parameter4: 4
  parameter5: 5
  .
  .
  .
  .
  .
 parameter100: 100

Great, thank you! We will update you on GH