Hey team, i'm researching how to deploy kedro in dagster. I found out 2 ways by now that works properly: 1.- Running as a subprocess

@op
def run_kedro_terminal(context: OpExecutionContext):
    process = Popen("kedro run", stdout=PIPE, stderr=STDOUT, shell=True)
    with process.stdout:
        log_subprocess_output(context, process.stdout)
    process.wait()

2.- Running in code

@op
def kedro_run(context: OpExecutionContext):
    project_entry = Path(os.getcwd())
    bootstrap_project(project_entry)
    with KedroSession.create(project_path=project_entry) as session:
        session.run()

By now i'm not able to show the kedro logs on the 2nd option, i found out that

logger = get_dagster_logger()

enables me to log properly inside code/function of kedro, but i haven't found a way to show kedro logs for nodes and pipelines execution sadblob (edited)

Hey @Camilo López getting a Dagster tutorial out has been an our radar for a while, so working out what works would be very helpful

Hey @datajoely cool, happy to collaborate with this im just starting with this new project with kedro, minio and dagster

Both Kedro and Dagster are using Python standard

logging

module.

lookslike you can put the logger names that you need to capture on dagster.yml

python_logs:
  managed_python_loggers:
    - my_logger
    - my_other_logger

I tried with no results:

python_logs:
  managed_python_loggers:
    - kedro
    - kedro_handler

can you post your existing logging.yml?

is just spaceflights logging.yml

version: 1

disable_existing_loggers: False

formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: <ext://sys.stdout>

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: logs/info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  error_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: ERROR
    formatter: simple
    filename: logs/errors.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  rich:
    class: rich.logging.RichHandler

loggers:
  kedro:
    level: INFO

  minio_spaceflights:
    level: INFO

root:
  handlers: [rich, info_file_handler, error_file_handler]

can you change your handlers at the bottom and switch

rich

with

console

lookslike you can put the logger names that you need to capture on dagster.yml

what's the name of your logger? you mentioned the log inside node is logged correctly.

i changed from rich to console didnt work 😞 . Idk what's the logger name, inside the node works if i do the following:

logger = get_dagster_logger()

Are you using Kedro-docker or you write your own docker?

i wrote my own one, yes i'm running 0.19

I don't think it's possible. Cc @Ahdra Merali is the markup syntax

rich

specific?

you can do so with rich environment vars directly https://rich.readthedocs.io/en/stable/console.html#environment-variables

@datajoely have you perhaps made any progress on the dagster+kedro tutorial idea mentioned above? Also, @Camilo López / @Nok Lam Chan do you perhaps know of any GitHub repos (or other material) one can use to consult ?

@George p unfortunately I don't, I know there are teams using it internally but I don't recall we have any blog post to showcase. I will ask around

Hey! I haven’t but we can do something im open to it!. The main struggle i found out was just the logger connection mentioned above. I'll pin you this week @Nok Lam Chan wdyt?

if its open source can we showcase the repo?

It would be fantastic if it can be open sourced, at a bare minimal a blog post to summarise what it takes, the general approach and some code snippets would be a great start.

would love to have a look at the repo yes!

Consider me an amateur, but i am happy to help out as well 🙌🏻

Thanks @George p Curious to see examples here too :) I’ll play around myself