Hi, we are getting ready to deploy our first kedro project, but we see one challenge ahead. We are getting our initial raw dataset from a data service where we need an authorized client to access. When we run our pipeline locally we authorize the client personally is the first node of our pipeline and store it in our catalog, no problems there. But in the service where we are going to host the pipeline, an authorized client is provided, but how do we now pass this inn to our packaged pipeline? Can we pass it as an argument som how?

from <package_name>.__main__ import main
from service import client
main(
    ["--pipeline", "__default__", client]
)  # or simply main() if you don't want to provide any arguments

Hi @Christoph Imler, this is a great question. Generally, authorisation/connections are done with hook, occasionally they are being passed as an object and pass between nodes. Is it possible to re-written this in a hook fashion? How would it looks like when it's getting run in a local mode? You mention this get stored in the Catalog, can you show the relevant entry in your

catalog.yml

? Is this an object that can be pickled?

Here are snippits of the code below. The client is stored as MemoryDataset, I have not tried using pickle, but it might also work. I have not worked with hooks before, will look into that as well. As I am writing, I realise that we are not able to recreate the folder hierarchy used by the data catalog in the service we are going to use, is it possible to change the catalog to run everything in memory when we deploy it and still have the folders localy? nodes.py

def get_client():
    client = CogniteClient(
        ClientConfig(
            credentials=oauth_provider,
            project=COGNITE_PROJECT,
            base_url=f"https://{CDF_CLUSTER}.<http://cognitedata.com|cognitedata.com>",
            client_name="test",  # a name to identify your session
            debug=False,
            max_workers=100,
            timeout=60 * 5,  # 5 minutes
        )
    )
    _ = client.iam.token.inspect()
    return client

catalog.yml

### Pipeline 1
client:
  type: MemoryDataset

It's best to separate your environment, i.e. you can have

prod

environment that everything runs in Memory

But then a Hook is my best chance of getting the client in?

Correct me if I am wrong, I think it's not so straight forward to pass this into DataCatalog because it requires it to be something saved on disk. Allow injecting data into a `KedroSession` run #2169 is an ticket that discuss this issue but we don't have anything implemented yet. @Takieddine Kadiri may have opinion on this but this may diverge from the standard kedro since this is a plugin perspective.

I am accessing the client in some of the nodes, but its not saved on disk only memory, but it still needs to be passed in somehow

If you actually need this in your node. The best option that I can think of is a custom dataset Essentially you want to create the connection conditionally base on the runtime environment pseudocode:

if env == prod:
  from service import client
else:
  from local import client
  ...

What about Kedro-boot? Have you tried it?

It's a community developed plugin. I haven't tried it myself but I have tagged the author in this thread.

Hi Christoph, If i understand correctly, you need to inject data to your kedro project at runtime, and this data is actually a Client object. Here is a solution path that you can explore: • As adviced by @Nok Lam Chan Develop a AbstractDataset that load such Client object. The client object could now be setted as a dataset • Take a look at kedro-boot-examples. It demonstrate through some examples, how you can use kedro-boot to inject data into kedro project and perform multiple pipeline runs.

Thanks, I was looking into that earlier, but its nice to know that this is the right track. It gets complicated fast.

You can open an issue in the kedro-boot repo. We plan to release a new version in a week or two that suport kedro 0.19.x

Great! Thanks