Toni - TomTom - Madrid07/10/2023, 9:28 AM
Nok Lam Chan07/10/2023, 9:58 AM
def load(self, path=None, format=None, schema=None, **options):
“”"Loads data from a data source and returns it as a class`DataFrame`.
.. versionadded:: 1.4.0
path : str or list, optional
optional string or a list of string for file-system backed data sources.
format : str, optional
optional string for format of the data source. Default to ‘parquet’.This is an excerpt from Spark documentation, we use DataFrameReader under the hood, so whatever Spark support should work out of the box.
Toni - TomTom - Madrid07/11/2023, 3:30 PM
Nok Lam Chan07/11/2023, 7:32 PM
Toni - TomTom - Madrid07/21/2023, 8:25 AM
>> textFiles = sc.wholeTextFiles(dirPath)My point is that it kedro lacks RDD capabilities 😅. Generally speaking this should be a must when working with Spark (not all big data is structured data). KMl files o XLM files are an example of this. Thanks a lot for your help! 🤗