https://kedro.org/ logo
Join Slack
Powered by
Is there a way to formally validate your proficien...
# questions
a
Is there a way to formally validate your proficiency in Kedro through certifications? For instance, similar to Datacamp where completing a course earns you a digital certification of completion, which can be beneficial for showcasing skills on LinkedIn or appending to your resume.
👀 2
a
No, afaik.
🥳 1
n
Level 1: spaceflights Level 2: anything in the "Advance xxx" section Level 3: modular pipelines / hooks Level 4: Kedro deployment Level 99: submit a PR to kedro Level ?: Dynamic pipelines
K 1
We did have internal Kedro training with beginner/intermediate level but that's not really a certification. cc @Juan Luis
j
Level 9000: explaining the difference between a modular pipeline and a normal pipeline 🧌
🤯 2
🫠 1
we considered issuing certificates for a while but it requires some effort on our side and it's not currently a priority https://github.com/kedro-org/kedro-devrel/issues/66 but @Afiq Johari please go ahead and add me on LinkedIn, happy to endorse your Kedro abilities 😄 https://www.linkedin.com/in/juanluiscanor/
K 3
🥳 1
y
@Nok Lam Chan I'd probably rank hooks above deployment and submitting a PR haha
👀 1
n
What makes hook difficult? 👀👀
y
For me it’s probably just lack of experience with them
👀 1
a
Hooks are not really difficult on their own, but they are kind of magic in a way that you're not sure how and when exactly they are called (pluggy), need some extra wizardry in pyproject.toml to work (that's filled for you, but still) and for me personally I have problems picturing how they work together with parallel runners in python like multi-procesing if they need some shared internal state
n
https://docs.kedro.org/en/latest/hooks/introduction.html Have you seen the diagram in the docs? Indeed for parallel runner you will need some shared memory state to work, though this is the tricky part of multiprocess itself that you have to worry about locks etc. Hooks itself doesn't require pyproject.toml, it's only needed if you need to ship that as a plugin that is enabled automatically.
For me the first time I came across this concept was from Keras callback API, where you can pass some callback to record training loss etc and save them as a csv without touching the training code.
a
Yes I did, but there still were some surprises or tricky ways to pass parameters.
I guess it's a matter of transparency - also plugins can add their own hooks and it's not always clear the order in which things get called, or it's not apparent that you need to unregister and re-register hook in your settings to get them to be called after your hooks.
👀 1
and I don't claim that hooks are difficult, or hard to get into, just that I'm not surprised that some people find them to be as I had my share of surprises when using them
n
it's not apparent that you need to unregister and re-register hook in your settings to get them to be called after your hooks.
Does it work if you import your hook explicitly in 
settings.py
? In general hooks shouldn't interact with each other (think about pytest plugins). If this is the case I think this is not a good behavior to rely on.
a
We had use case where we had to force kedro-mlflow hook to happen after our own hook and we had to unregister mlflow hook and re-register for that to work.
👀 1
n
a
no, TIL it exists 😄
👍🏼 1
6 Views
Open in Slack
PreviousNext
Powered by