hello 🙂 I would like to use your spark.SparkStreamingDataSet with a Kafka integration. Due to project constraints, I am working with Kedro 0.19.14, and the following jars (

spark.jars.packages: org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.4,org.apache.spark:spark-token-provider-kafka-0-10_2.12:3.2.4

) From what i have understood, my dataset definition in the catalogue should be something like this:

data:
  type: spark.SparkStreamingDataSet
  file_format: kafka
  load_args:
    options:
      subscribe: my-topic
      kafka.bootstrap.servers: kafka:0000
      startingOffsets: earliest

however, I cannot navigate around the error :

DatasetError: Failed while loading data from data set 
SparkStreamingDataset(file_format=kafka, filepath=., load_args={'options': 
{'kafka.bootstrap.servers': kafka:0000, 'startingOffsets': earliest, 
'subscribe': my-topic}}, save_args={}).
schema should be StructType or string

Would you have any example projects or extra docs to point me to ? Thanks a bunch 😊

Hey Laure, You can checkout below project/docs: https://github.com/deepyaman/kedro-streaming https://kedro.org/blog/kedro-dataset-for-spark-structured-streaming https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-8.1.0/api/kedro_datasets/spark.SparkStreamingDataset/

thank you ! 😊 I have found these and they are quite useful, especially the first one that has kafka in the pipeline (the others do not if i am not mistaken) any idea where this schema issue is coming from though?

If you read this https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-8.1.0/api/kedro_datasets/spark.SparkStreamingDataset/#kedro_dataset[…]eamingDataset "Please note that a schema is mandatory for a streaming DataFrame"

I had it originally but removed it after getting the following error 😓

DatasetError: Failed while loading data from data set 
SparkStreamingDataset(file_format=kafka, filepath=., 
load_args={'kafka.bootstrap.servers': kafka:0000, 'startingOffsets': earliest, 
'subscribe': my-topic}, save_args={}).
requirement failed: Kafka source has a fixed schema and cannot be set with a 
custom one

I had understood the schema argument was necessary when tied to the file_format json or csv i.e. spark streaming not coupled to kafka.

Hey, what version of

kedro-datasets

are you using?

in the example project I also see there's an absence of the schema argument

https://github.com/deepyaman/kedro-streaming/blob/develop/conf/base/catalog.yml

FWIW this is a hackathon project from 2021. 🙂 It implements it's own dataset that predates whatever is in

kedro-datasets

, so the configuration may not line up. I wasn't very involved in the version that is in

kedro-datasets

, but it looks like I commented that it was OK to require schema? 😅 https://github.com/kedro-org/kedro-plugins/pull/198#discussion_r1209233565 I would consider streaming support in Kedro to be rather experimental at best. This doesn't answer your question, but do you know how you want to use streaming with Kedro? As in, are you looking to interactively run something streaming (more like a demo) or deploy some streaming tables? One other option you could consider is using the Ibis dataset, depending on what you're looking to do, as Ibis also has Spark Streaming backend support. I haven't tried it with Kedro.

Hello, thank you both for your input! 🙏 😊 • we are trying to do: deploy some streaming tables (monitor some tables on a PostGres and if there are new inputs, run a preprocessing pipeline streaming version) • our kedro dataset version:

kedro-datasets 1.8.0

if there are new inputs, run a preprocessing pipeline streaming version

What do you mean by this? I would say the main difference with a streaming pipeline is that it's always running. Do you just mean a deployed streaming pipeline, and if you get new events in Kafka it will process them?

yes exactly. it is constantly running and picks up changes

You can make the PR to https://github.com/kedro-org/kedro-plugins, where kedro-datasets lives

thanks, keep you posted then 🙂