https://kedro.org/ logo
#questions
Title
# questions
c

Cyril Verluise

03/20/2023, 9:53 PM
Hello there, I hope that this finds you well. Thanks for the awesome work! Sorry to come with an issue. Issue I'm trying to set up kedro pipeline test as part of my CI/CD using GH action. Everything goes well until I receive a git related error
Copy code
TypeError: HEAD is a detached symbolic reference as it points to 
'dc15ea87ce9d917bafb09d5d7bddb2aaf44f5989'
Full error log and GH action config in thread. What I tried I have tried to checkout with fetch depth 0 but this did not fix the issue (I had a similar issue when building doc from GH action which was fixed using the above trick). Environment kedro version: 0.18.6 OS: ubuntu latest Any ideas?
GH action config
Copy code
name: requirements-code-quality-tests
on:
  pull_request:
    branches:
      - master
      - develop

jobs:
  check-requirements-code-quality-tests:
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]

    steps:
    - name: "๐Ÿก Checkout repository"
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
    - name: "๐Ÿ Install python"
      uses: actions/setup-python@v4
      with:
        python-version: "3.8"
        cache: "pip" # caching pip dependencies
    - name: "๐Ÿงฑ Check install cerrotorre dependencies"
      run: |
        pip install -r src/requirements.txt
        pip install -r src/requirements-dev.txt
        pip install -e src/
    - name: "๐ŸŽจ Linting checks"
      run: |
        black --check src/
        isort --check src/
        pylint src/
    - name: "๐Ÿงช Check tests"
      run: pytest
    - name: "๐Ÿƒ Check pipelines"
      run: |
        kedro run
        kedro run --pipeline evaluate_allotment
Error log
Copy code
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Traceback (most recent call last) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/bin/kedro:8 in <module>               โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra โ”‚
โ”‚ mework/cli/cli.py:211 in main                                                โ”‚
โ”‚                                                                              โ”‚
โ”‚   208 โ”‚   """                                                                โ”‚
โ”‚   209 โ”‚   _init_plugins()                                                    โ”‚
โ”‚   210 โ”‚   cli_collection = KedroCLI(project_path=Path.cwd())                 โ”‚
โ”‚ โฑ 211 โ”‚   cli_collection()                                                   โ”‚
โ”‚   212                                                                        โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor โ”‚
โ”‚ e.py:1130 in __call__                                                        โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra โ”‚
โ”‚ mework/cli/cli.py:139 in main                                                โ”‚
โ”‚                                                                              โ”‚
โ”‚   136 โ”‚   โ”‚   )                                                              โ”‚
โ”‚   137 โ”‚   โ”‚                                                                  โ”‚
โ”‚   138 โ”‚   โ”‚   try:                                                           โ”‚
โ”‚ โฑ 139 โ”‚   โ”‚   โ”‚   super().main(                                              โ”‚
โ”‚   140 โ”‚   โ”‚   โ”‚   โ”‚   args=args,                                             โ”‚
โ”‚   141 โ”‚   โ”‚   โ”‚   โ”‚   prog_name=prog_name,                                   โ”‚
โ”‚   142 โ”‚   โ”‚   โ”‚   โ”‚   complete_var=complete_var,                             โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor โ”‚
โ”‚ e.py:1055 in main                                                            โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor โ”‚
โ”‚ e.py:1657 in invoke                                                          โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor โ”‚
โ”‚ e.py:1404 in invoke                                                          โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor โ”‚
โ”‚ e.py:760 in invoke                                                           โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra โ”‚
โ”‚ mework/cli/project.py:455 in run                                             โ”‚
โ”‚                                                                              โ”‚
โ”‚   452 โ”‚   with KedroSession.create(                                          โ”‚
โ”‚   453 โ”‚   โ”‚   env=env, conf_source=conf_source, extra_params=params          โ”‚
โ”‚   454 โ”‚   ) as session:                                                      โ”‚
โ”‚ โฑ 455 โ”‚   โ”‚   session.run(                                                   โ”‚
โ”‚   456 โ”‚   โ”‚   โ”‚   tags=tag,                                                  โ”‚
โ”‚   457 โ”‚   โ”‚   โ”‚   runner=runner(is_async=is_async),                          โ”‚
โ”‚   458 โ”‚   โ”‚   โ”‚   node_names=node_names,                                     โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra โ”‚
โ”‚ mework/session/session.py:303 in __exit__                                    โ”‚
โ”‚                                                                              โ”‚
โ”‚   300 โ”‚   def __exit__(self, exc_type, exc_value, tb_):                      โ”‚
โ”‚   301 โ”‚   โ”‚   if exc_type:                                                   โ”‚
โ”‚   302 โ”‚   โ”‚   โ”‚   self._log_exception(exc_type, exc_value, tb_)              โ”‚
โ”‚ โฑ 303 โ”‚   โ”‚   self.close()                                                   โ”‚
โ”‚   304 โ”‚                                                                      โ”‚
โ”‚   305 โ”‚   def run(  # pylint: disable=too-many-arguments,too-many-locals     โ”‚
โ”‚   306 โ”‚   โ”‚   self,                                                          โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra โ”‚
โ”‚ mework/session/session.py:295 in close                                       โ”‚
โ”‚                                                                              โ”‚
โ”‚   292 โ”‚   โ”‚   if `save_on_close` attribute is True.                          โ”‚
โ”‚   293 โ”‚   โ”‚   """                                                            โ”‚
โ”‚   294 โ”‚   โ”‚   if self.save_on_close:                                         โ”‚
โ”‚ โฑ 295 โ”‚   โ”‚   โ”‚   self._store.save()                                         โ”‚
โ”‚   296 โ”‚                                                                      โ”‚
โ”‚   297 โ”‚   def __enter__(self):                                               โ”‚
โ”‚   298 โ”‚   โ”‚   return self                                                    โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro_viz โ”‚
โ”‚ /integrations/kedro/sqlite_store.py:67 in save                               โ”‚
โ”‚                                                                              โ”‚
โ”‚   64 โ”‚   โ”‚   engine, session_class = create_db_engine(self.location)         โ”‚
โ”‚   65 โ”‚   โ”‚   Base.metadata.create_all(bind=engine)                           โ”‚
โ”‚   66 โ”‚   โ”‚   database = next(get_db(session_class))                          โ”‚
โ”‚ โฑ 67 โ”‚   โ”‚   session_store_data = RunModel(id=self._session_id, blob=<http://self.to|self.to> โ”‚
โ”‚   68 โ”‚   โ”‚   database.add(session_store_data)                                โ”‚
โ”‚   69 โ”‚   โ”‚   database.commit()                                               โ”‚
โ”‚   70                                                                         โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro_viz โ”‚
โ”‚ /integrations/kedro/sqlite_store.py:51 in to_json                            โ”‚
โ”‚                                                                              โ”‚
โ”‚   48 โ”‚   โ”‚   โ”‚   โ”‚   try:                                                    โ”‚
โ”‚   49 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   import git  # pylint: disable=import-outside-toplev โ”‚
โ”‚   50 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚                                                       โ”‚
โ”‚ โฑ 51 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   branch = git.Repo(search_parent_directories=True).a โ”‚
โ”‚   52 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   value["branch"] = branch.name                       โ”‚
โ”‚   53 โ”‚   โ”‚   โ”‚   โ”‚   except ImportError as exc:  # pragma: no cover          โ”‚
โ”‚   54 โ”‚   โ”‚   โ”‚   โ”‚   โ”‚   logger.warning("%s:%s", exc.__class__.__name__, exc โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/git/repo/ โ”‚
โ”‚ base.py:896 in active_branch                                                 โ”‚
โ”‚                                                                              โ”‚
โ”‚    893 โ”‚   โ”‚   :raises TypeError: If HEAD is detached                        โ”‚
โ”‚    894 โ”‚   โ”‚   :return: Head to the active branch"""                         โ”‚
โ”‚    895 โ”‚   โ”‚   # reveal_type(self.head.reference)  # => Reference            โ”‚
โ”‚ โฑ  896 โ”‚   โ”‚   return self.head.reference                                    โ”‚
โ”‚    897 โ”‚                                                                     โ”‚
โ”‚    898 โ”‚   def blame_incremental(self, rev: str | HEAD, file: str, **kwargs: โ”‚
โ”‚    899 โ”‚   โ”‚   """Iterator for blame information for the given file at the g โ”‚
โ”‚                                                                              โ”‚
โ”‚ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/git/refs/ โ”‚
โ”‚ symbolic.py:309 in _get_reference                                            โ”‚
โ”‚                                                                              โ”‚
โ”‚   306 โ”‚   โ”‚   โ”‚   to a reference, but to a commit"""                         โ”‚
โ”‚   307 โ”‚   โ”‚   sha, target_ref_path = self._get_ref_info(self.repo, self.path โ”‚
โ”‚   308 โ”‚   โ”‚   if target_ref_path is None:                                    โ”‚
โ”‚ โฑ 309 โ”‚   โ”‚   โ”‚   raise TypeError("%s is a detached symbolic reference as it โ”‚
โ”‚   310 โ”‚   โ”‚   return self.from_path(self.repo, target_ref_path)              โ”‚
โ”‚   311 โ”‚                                                                      โ”‚
โ”‚   312 โ”‚   def set_reference(                                                 โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
TypeError: HEAD is a detached symbolic reference as it points to 
'dc15ea87ce9d917bafb09d5d7bddb2aaf44f5989'
Error: Process completed with exit code 1.
d

Deepyaman Datta

03/20/2023, 10:04 PM
Does your pipeline depend on
kedro-viz
? Can you try without installing it in your CI environment? I don't have time to look at this just this minute, but I think it has to do with experiment tracking having some dependency on git...
c

Cyril Verluise

03/20/2023, 10:27 PM
Thanks! this looks correct. But, if I don't install kedro-viz, my
settings.py
is going to raise an issue. Setting up the store requires kedro-viz form what I understand https://docs.kedro.org/en/stable/visualisation/experiment_tracking.html#set-up-the-session-store
Does it mean that experiment tracking is not compatible with running kedro from with GH action?
d

Deepyaman Datta

03/21/2023, 1:19 AM
I see, issue is a GitPython issue because you're on detached head. I didn't read it properly before. Maybe you can try something like this? https://github.com/actions/checkout/issues/124#issuecomment-586664611 Not 100% sure about why the fetch depth thing didn't work. Otherwise, I would consider running a version with that git metadata fetching step patched out, because you aren't actually needing to do tracking. TBH, I'm not very familiar with experiment tracking and some of these dependencies, maybe @Antony Milne have you run across this?
c

Cyril Verluise

03/21/2023, 8:24 AM
thanks, yes, will try with #124. Would make sense that fetch.depth=0 did not work because it was a pull request. Will let you knoe. Thanks a lot @Deepyaman Datta!
a

Antony Milne

03/21/2023, 10:10 AM
Hi @Cyril Verluise, thanks for this very clear bug report with excellent error log! Iโ€™ve never seen this before but I can see where the problem is. Ultimately this is a bug with kedro-viz since the
SQLiteStore
shouldnโ€™t fail due to a git error. Itโ€™s already designed to work in the case that git isnโ€™t installed but we didnโ€™t anticipate git throwing a
TypeError
. So we should change that
except
just to catch anything. In the short term, before we release a fix, thereโ€™s several ways you could fix this yourself: 1. Copy and paste
SQLiteStore
into your own codebase, apply the fix there and then point to that in settings.py rather than the kedro-viz one 2. Try to make it so that the underlying
TypeError
no longer gets triggered at all, e.g. by following what @Deepyaman Datta suggested above 3. Change your settings.py to no longer use
SQLiteStore
on CI. You can probably achieve this without having to have two different versions for settings.py (one for your project vs. one for CI) by only enabling
SQLiteStore
in settings.py according to an environment variable thatโ€™s set in github actions:
if os.environ.get("CI"): SESSION_STORE_CLASS = SQLiteStore
``````
c

Cyril Verluise

03/21/2023, 10:13 AM
thanks! very clear. Will try that and let you know which approach worked/was chosen. Thanks
๐Ÿ‘ 1
a

Antony Milne

03/21/2023, 10:15 AM
I am curious - do you actually want to keep run experiment tracking and look at the results in CI? Or itโ€™s just there because you use it on your project already and want to run the same pipeline in CI?
c

Cyril Verluise

03/21/2023, 10:34 AM
just there because I'm using it in the project and want to run the same pipeline in CI
a

Antony Milne

03/21/2023, 10:35 AM
Cool, in that case Iโ€™d probably recommend solution 3 above because then you donโ€™t have the (very small) overhead of writing to the
SQLiteStore
at all in CI. You could then remove kedro-viz as a CI requirement also.
๐Ÿ‘ 1
c

Cyril Verluise

03/21/2023, 10:57 AM
thx
๐Ÿ‘ 1
hey, solution 3 worked like a charm. full snippet with GH actions is
Copy code
if not os.environ.get("GITHUB_ACTIONS"):
    SESSION_STORE_CLASS = SQLiteStore
    SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}
Should I raise an issue on kedro viz repo so that we keep track of it? Thanks a lot @Antony Milne and @Deepyaman Datta! Cheers!
a

Antony Milne

03/21/2023, 10:43 PM
perfect, thank you for this @Cyril Verluise. If you donโ€™t mind raising an issue on kedro-viz then that would be great, thank you!
โœ… 1
4 Views