Hi everyone, Hope you’re all doing well 🙂 I am stumbling on something that seems odd (to me). So… As you all already know

pd.read_csv()

has a

nrows

param that takes an

int

as argument but is set by default to

None

, in which case, the full dataset is loaded. Therefore, I thought that I could leverage that in order to give myself the ability to experiment / iterable quickly by downsampling my datasets at runtime with something like:

"raw_{table}":
  type: pandas.CSVDataset
  filepath: data/01_raw/tables/raw_{table}.csv
  load_args:
    nrows: "${runtime_params:nrows, None}"

While this works perfectly with

kedro run --params nrows=100

, when leaving the param unspecified, I end up with

DatasetError: Failed while loading data from data set
CSVDataset(filepath=/Users/marc/DODOBIRD/DODO_CODE/kedro-etl/data/01_raw/tables/raw_users.csv, load_args={'nrows': None}, protocol=file, save_args={'index': False}).
'nrows' must be an integer >=0

and yet running

pd.read_csv("data.csv", nrows=None)

simply returns the full dataset as expected. Is this a bug ? or am I missing something / doing something wrong. Thanks for your input, M.

Could you put a breakpoint in the pandas validate_integer function to see what the "val" is arriving as? Because that definitely seems strange

Try -

nrows: "${runtime_params:nrows, null}"

From the traceback it looks like it's passing None correctly though, right? But let's see if that fixes it 🙂

Thx to you both for your messages. I’m trying the

null

instead of

None

thing right now. 👍🏼

to answer my own question, it seems strings get printed in the dataset repr

Thanks for being more curious / thorough than me and digging deeper into this 😉

I was actually looking into something different and saw that the "nonsense" was being represented without the quotes and remembered this thread 😆