Hi all ive been playing around with Kedro and I wo...
# plugins-integrations
Hi all ive been playing around with Kedro and I would love to migrate my current project to it. I am however, having a lot of trouble with the following • Connecting to the Databricks Delta Tables without having spark. (Trying to connect on my local IDE) • Understanding how am I supposed to run the same pipeline on a bunch of different documents. ◦ Example: I have a folder in Azure Blob storage, and I have a bunch of documents there that needs to be run through the pipeline. Can I have it point to the folder instead of the individual documents?
@Richard Purvis maybe you encountered question 1? On question 2, likely what you need is a
(docs). In short, how it works is that you define it like this:
Copy code
  type: partitions.PartitionedDataset
  path: <s3://my-bucket-name/path/to/folder>  # path to the location of partitions
  dataset: pandas.CSVDataset  # shorthand notation for the dataset which will handle individual partitions
And that means: 1. Go to the folder specified in
2. Read all items as individual datasets (in this case
) 3. On load, it would return a
dict[str, object]
thing where
is a filename and
is whatever your
would read - in example above it would be
Hi @Benjamin Cheung, there is a pandas delta table dataset, see if that works. You can probably use transcoding to access from either pandas or spark.
🚀 1