Hi!! Have a question about a smart way of implementing the following. I have a pipeline that with a custom api dataset. This api has to be called with a date_from and date_to parameters. The values of these depends on a checkpoint json. Such that each time only new data is requested. Now this is where it is problematic because there is a dependency between datasets. I could of course do the api call in a pure node call. However i really like that I have it in the catalog. Especially because it allows me to easily swap it for a json dataset in testing. What is the best way of approaching this?
Never mind used the lazy approach described above worked really well ☺️