Toni - TomTom - Madrid
07/10/2023, 9:28 AMNok Lam Chan
07/10/2023, 9:58 AMformat
into the load_args
?
def load(self, path=None, format=None, schema=None, **options):
ββ"Loads data from a data source and returns it as a class`DataFrame`.
.. versionadded:: 1.4.0
Parameters
----------
path : str or list, optional
optional string or a list of string for file-system backed data sources.
format : str, optional
optional string for format of the data source. Default to βparquetβ.This is an excerpt from Spark documentation, we use DataFrameReader under the hood, so whatever Spark support should work out of the box.
Toni - TomTom - Madrid
07/11/2023, 3:30 PMNok Lam Chan
07/11/2023, 7:32 PMToni - TomTom - Madrid
07/21/2023, 8:25 AM>> textFiles = sc.wholeTextFiles(dirPath)My point is that it kedro lacks RDD capabilities π . Generally speaking this should be a must when working with Spark (not all big data is structured data). KMl files o XLM files are an example of this. Thanks a lot for your help! π€