Hello everyone! I'm new at Kedro. I hope you are all doing well. I'm trying to run Kedro by connecting from Windows 10, using the Databricks extension in Visual Studio Code (Python 3.10.1 virtual environment). When I execute any pipeline, it seems there is a connection for a few seconds, but then I get this error:

SparkConnectGrpcException: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INVALID_ARGUMENT

details = "Missing required field 'UserContext' in the request."

debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Missing required field \'UserContext\' in

the request.", grpc_status:3, created_time:"2023-11-24T13:06:22.386119+00:00"}"

It seems there is a solution (https://community.databricks.com/t5/data-governance/databricks-connect-version-13-0-0-throws-exception-with-details/m-p/5148#M77), but I don't quite understand in which part of the framework it can be configured. Thank you very much.

One thing about databricks I learned is that it is extremely sensitive to versions. Make sure the databricks-connect version matches the databricks version of your cluster