Is it possible to reload a data within a function/node? For example

node(func=regenerate, inputs="mydata_sql",outputs="mydata_excel")

def regenerate(mydata):
     # run SQL stored procedure that impacts the table of mydata_sql in the database
     # reload mydata because the stored procedure will have changed mydata
     return mydata # convert it to excel file

Unfortunately, recreating the data transformation of the stored procedure in Python may not be straightforward. That's why I depend on the stored procedure to transform/update my data.

Hi Afiq, I think that reloading data within a node isn't a best practice in Kedro. However, have you considered achieving this by dividing the process into multiple nodes step by step?

@Dmitry Sorokin Basically, what I want to do with Kedro is the ability to update my SQL tables by executing some stored procedures. So the expected output from these stored procedures would be the updated SQL tables. Once the SQL tables are updated, I want to output them as Excel files. I think the main challenge I have is to create a pipeline with the right inputs and outputs so that when I execute the pipeline, it will always start with executing the stored procedures, return the SQL table, and then output the SQL table as Excel.

So if I understand correctly - you want to load

mydata_sql

but you need to make sure the store_proc get executed?

@Nok Lam Chan the

regenerate

takes in

parameters

but these

parameters

are actually for the

stored procedures

(SP) At the moment, there's no plan to migrate the SP to Python, hence why we rely on the SP. The latest code iteration I have is to reload the data within the node Added this to

nodes.py

credentials = config_loader.get("credentials.yml")
catalogs = config_loader.get("catalog.yml")
thedata= DataCatalog.from_config(catalogs, credentials)

def regenerate(mydata):
  # run SP
  # reload sql data after SP execution
  mydata= thedata.load("mydata_sql")
return mydata