Trevor
06/12/2023, 4:52 PMMemoryDataSet
returned by nodes as pieces of larger scripts. Up until now, I've only needed to import main
and that has worked for our purposes so farNok Lam Chan
06/12/2023, 4:54 PMTrevor
06/12/2023, 4:57 PMmain()
in whatever script I put it in until this point which exits the script. Not sure what I'm doing wrong, but I saw old documentation and examples of others doing it but it was around ver=~17 and things like load_context()
were still around (even though that case wouldn't be a package)Nok Lam Chan
06/12/2023, 5:24 PMTrevor
06/12/2023, 5:31 PMfrom my_pipeline.__main__ import main
main()
print('here!')
I haven't addressed the issue of getting the return value from the pipelineNok Lam Chan
06/12/2023, 5:33 PMTrevor
06/12/2023, 5:43 PM__main__.py
is the default main that is generated when you create a new kedro project with kedro new
and hasn't been modified in any way. Unless it get's changed when you do a kedro package
? I'll check thoughmain()
. It looks like the session solution you mentioned in that github issue will solve my problem though? Just import KedroSession
from kedro.framework.session
and configure_project
from kedro.framework.project
?__main__.py
and the one generated when the project is created yields no differences:__main.py__
:
import importlib
from pathlib import Path
from kedro.framework.cli.utils import KedroCliError, load_entry_points
from kedro.framework.project import configure_project
def _find_run_command(package_name):
try:
project_cli = importlib.import_module(f"{package_name}.cli")
# fail gracefully if cli.py does not exist
except ModuleNotFoundError as exc:
if f"{package_name}.cli" not in str(exc):
raise
plugins = load_entry_points("project")
run = _find_run_command_in_plugins(plugins) if plugins else None
if run:
# use run command from installed plugin if it exists
return run
# use run command from `kedro.framework.cli.project`
from kedro.framework.cli.project import run
return run
# fail badly if cli.py exists, but has no `cli` in it
if not hasattr(project_cli, "cli"):
raise KedroCliError(f"Cannot load commands from {package_name}.cli")
return project_cli.run
def _find_run_command_in_plugins(plugins):
for group in plugins:
if "run" in group.commands:
return group.commands["run"]
def main(*args, **kwargs):
package_name = Path(__file__).parent.name
configure_project(package_name)
run = _find_run_command(package_name)
run(*args, **kwargs)
if __name__ == "__main__":
main()
Nok Lam Chan
06/13/2023, 10:40 AMclick
from kedro.framework.project import configure_project
from kedro.framework.session import KedroSession
configure_project(package_name)
with KedroSession.create(env=env, conf_source=conf_source) as session:
result = session.run() # result is a dict of result that you are interested