Hello Team I want to save a df back to a Snowpark Table data Kedro #questions

Hello Team, I want to save a df back to a Snowpark...

Akshata Patel

10/14/2024, 1:38 PM

Hello Team, I want to save a df back to a Snowpark Table dataset object, but im running into this error

Copy code

DatasetError: Failed while saving data to data set SnowparkTableDataset(...).
'DataFrame' object has no attribute 'write'

Code snippet in thread, please let me know if there is a way to do this 😄 Thanks so much!

👀 1

Akshata Patel

10/14/2024, 1:41 PM

Copy code

def push_output_to_snowflake(
    classified_text: pd.DataFrame,
    required_cols: list[str],
) -> pd.DataFrame:
    """ingests classified text and saves into Snowflake

    args:
        x
    return:
        classified_text: pd.DataFrame
    """

    # clean up the classified dataframe
    classified_text = process_dataframe(classified_text, required_cols)
    return classified_text

Copy code

node(
                func=push_output_to_snowflake,
                inputs=[
                    "combined_classification",
                    "params:post_processing.message_output_cols",
                ],
                outputs="snowflake_message_classification_table",
)

in catalog.yml

Copy code

test_pipe.snowflake_message_classification_table:
  type: kedro_datasets.snowflake.SnowparkTableDataset
  table_name: x
  database: x
  schema: x
  credentials: snowflake_creds
  save_args:
    mode: append
    column_order: name
    table_type: ''

Ravi Kumar Pilla

10/14/2024, 2:15 PM

I see the

push_output_to_snowflake

function returns a pandas dataframe. Instead can you convert it to Snowpark df before saving it back to Snowflake ?

Akshata Patel

10/14/2024, 4:37 PM

will try that so that would be using the

Session.builder.configs

to create a session and then using

write_pandas

Ravi Kumar Pilla

10/14/2024, 4:55 PM

I was suggesting to convert the pandas df to snowpark_df in your

push_output_to_snowflake

function as SnowparkTableDataset loads and saves Snowpark dataframes. So the function will return a df of type

snowflake.snowpark.DataFrame

. You can try using -

snowpark_df = session.create_dataframe(pandas_df, schema=schema)

👀 1

Akshata Patel

10/14/2024, 6:03 PM

Any recommended way to create a session? im trying to use Session.builder.configs but that doesnt seem to work with omegaconfig resolver and oauth (as I need to fetch the token from snoflake container service)

Ravi Kumar Pilla

10/14/2024, 7:14 PM

Hi Akshata,

Session.builder.configs(connection_params).create()

should create the session. Could you tell us what is not working or if there are any errors that are thrown ? I did not fully understand if the issue is with getting the token or something else. Also, I suppose your application should have already created a singleton session somewhere, unless this is the first time you are creating a session.

3 Views

Open in Slack

Previous Next