Hey Team, I hope you are doing well! Do any of you have already worked with SAP Datasphere and Kedro? We use hana-ml. We would need to load the data from Kedro pipeline to SAP Datasphere DB, using customised queries when loading data as input. Have you already used CustomDataSet to do so? Thanks a lot! 🙂 cc @Louis de Charsonville

Hey. I haven't use SAP Datasphere before, how would the connection normally looks like in Python? Is there a custom Python SDK or can it work with SQL alchemy?

Hey! In python on DataBricks, it looks like this:

from hana_ml import dataframe

def create_conn_sap_dwc(env_conn:str = 'PROD'):
    #Datasphere PRod Connection
    if env_conn=='PROD':
        dbHost=DBHOST_PROD
        display (DBHOST_PROD)
        dbPort=DBPORT_PROD
        dbUser=DBUSER_PROD
        dbPwd=dbutils.secrets.get(scope=SCOPE_PROD,key=KEY_PROD)
        display (dbPwd)
        conn = dataframe.ConnectionContext(dbHost, dbPort, dbUser, dbPwd, encrypt=True, sslValidateCertificate=True)
        return conn

cc = create_conn_sap_dwc()

sql_code = "MY SQL CODE"

result = cc.sql(sql_code)

df = result.collect()

two things that you may try: 1. Check if the conneciton is compatible with SQLalchemy, most database are and you can read it via

pandas.SQLQueryDataset

as long as you get the connection right. 2. ^ if above doesn't work, you can make a custom dataset inherit the above. I won't expect there to be too much changes, the main thing here is just getting the connection right and then passing sql around.

Thanks a lot Nok! I'll try and let you know. 🙂

amazing!