https://kedro.org/ logo
Join Slack
Powered by
Hello there, I hope that this finds you well. Tha...
# questions
c
Hello there, I hope that this finds you well. Thanks for the awesome work! Sorry to come with an issue. Issue I'm trying to set up kedro pipeline test as part of my CI/CD using GH action. Everything goes well until I receive a git related error
Copy code
TypeError: HEAD is a detached symbolic reference as it points to 
'dc15ea87ce9d917bafb09d5d7bddb2aaf44f5989'
Full error log and GH action config in thread. What I tried I have tried to checkout with fetch depth 0 but this did not fix the issue (I had a similar issue when building doc from GH action which was fixed using the above trick). Environment kedro version: 0.18.6 OS: ubuntu latest Any ideas?
GH action config
Copy code
name: requirements-code-quality-tests
on:
  pull_request:
    branches:
      - master
      - develop

jobs:
  check-requirements-code-quality-tests:
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]

    steps:
    - name: "🏡 Checkout repository"
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
    - name: "🐍 Install python"
      uses: actions/setup-python@v4
      with:
        python-version: "3.8"
        cache: "pip" # caching pip dependencies
    - name: "🧱 Check install cerrotorre dependencies"
      run: |
        pip install -r src/requirements.txt
        pip install -r src/requirements-dev.txt
        pip install -e src/
    - name: "🎨 Linting checks"
      run: |
        black --check src/
        isort --check src/
        pylint src/
    - name: "🧪 Check tests"
      run: pytest
    - name: "🏃 Check pipelines"
      run: |
        kedro run
        kedro run --pipeline evaluate_allotment
Error log
Copy code
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /opt/hostedtoolcache/Python/3.8.16/x64/bin/kedro:8 in <module>               │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra │
│ mework/cli/cli.py:211 in main                                                │
│                                                                              │
│   208 │   """                                                                │
│   209 │   _init_plugins()                                                    │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                 │
│ ❱ 211 │   cli_collection()                                                   │
│   212                                                                        │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor │
│ e.py:1130 in __call__                                                        │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra │
│ mework/cli/cli.py:139 in main                                                │
│                                                                              │
│   136 │   │   )                                                              │
│   137 │   │                                                                  │
│   138 │   │   try:                                                           │
│ ❱ 139 │   │   │   super().main(                                              │
│   140 │   │   │   │   args=args,                                             │
│   141 │   │   │   │   prog_name=prog_name,                                   │
│   142 │   │   │   │   complete_var=complete_var,                             │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor │
│ e.py:1055 in main                                                            │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor │
│ e.py:1657 in invoke                                                          │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor │
│ e.py:1404 in invoke                                                          │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/click/cor │
│ e.py:760 in invoke                                                           │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra │
│ mework/cli/project.py:455 in run                                             │
│                                                                              │
│   452 │   with KedroSession.create(                                          │
│   453 │   │   env=env, conf_source=conf_source, extra_params=params          │
│   454 │   ) as session:                                                      │
│ ❱ 455 │   │   session.run(                                                   │
│   456 │   │   │   tags=tag,                                                  │
│   457 │   │   │   runner=runner(is_async=is_async),                          │
│   458 │   │   │   node_names=node_names,                                     │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra │
│ mework/session/session.py:303 in __exit__                                    │
│                                                                              │
│   300 │   def __exit__(self, exc_type, exc_value, tb_):                      │
│   301 │   │   if exc_type:                                                   │
│   302 │   │   │   self._log_exception(exc_type, exc_value, tb_)              │
│ ❱ 303 │   │   self.close()                                                   │
│   304 │                                                                      │
│   305 │   def run(  # pylint: disable=too-many-arguments,too-many-locals     │
│   306 │   │   self,                                                          │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro/fra │
│ mework/session/session.py:295 in close                                       │
│                                                                              │
│   292 │   │   if `save_on_close` attribute is True.                          │
│   293 │   │   """                                                            │
│   294 │   │   if self.save_on_close:                                         │
│ ❱ 295 │   │   │   self._store.save()                                         │
│   296 │                                                                      │
│   297 │   def __enter__(self):                                               │
│   298 │   │   return self                                                    │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro_viz │
│ /integrations/kedro/sqlite_store.py:67 in save                               │
│                                                                              │
│   64 │   │   engine, session_class = create_db_engine(self.location)         │
│   65 │   │   Base.metadata.create_all(bind=engine)                           │
│   66 │   │   database = next(get_db(session_class))                          │
│ ❱ 67 │   │   session_store_data = RunModel(id=self._session_id, blob=<http://self.to|self.to> │
│   68 │   │   database.add(session_store_data)                                │
│   69 │   │   database.commit()                                               │
│   70                                                                         │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/kedro_viz │
│ /integrations/kedro/sqlite_store.py:51 in to_json                            │
│                                                                              │
│   48 │   │   │   │   try:                                                    │
│   49 │   │   │   │   │   import git  # pylint: disable=import-outside-toplev │
│   50 │   │   │   │   │                                                       │
│ ❱ 51 │   │   │   │   │   branch = git.Repo(search_parent_directories=True).a │
│   52 │   │   │   │   │   value["branch"] = branch.name                       │
│   53 │   │   │   │   except ImportError as exc:  # pragma: no cover          │
│   54 │   │   │   │   │   logger.warning("%s:%s", exc.__class__.__name__, exc │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/git/repo/ │
│ base.py:896 in active_branch                                                 │
│                                                                              │
│    893 │   │   :raises TypeError: If HEAD is detached                        │
│    894 │   │   :return: Head to the active branch"""                         │
│    895 │   │   # reveal_type(self.head.reference)  # => Reference            │
│ ❱  896 │   │   return self.head.reference                                    │
│    897 │                                                                     │
│    898 │   def blame_incremental(self, rev: str | HEAD, file: str, **kwargs: │
│    899 │   │   """Iterator for blame information for the given file at the g │
│                                                                              │
│ /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/git/refs/ │
│ symbolic.py:309 in _get_reference                                            │
│                                                                              │
│   306 │   │   │   to a reference, but to a commit"""                         │
│   307 │   │   sha, target_ref_path = self._get_ref_info(self.repo, self.path │
│   308 │   │   if target_ref_path is None:                                    │
│ ❱ 309 │   │   │   raise TypeError("%s is a detached symbolic reference as it │
│   310 │   │   return self.from_path(self.repo, target_ref_path)              │
│   311 │                                                                      │
│   312 │   def set_reference(                                                 │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: HEAD is a detached symbolic reference as it points to 
'dc15ea87ce9d917bafb09d5d7bddb2aaf44f5989'
Error: Process completed with exit code 1.
d
Does your pipeline depend on 
kedro-viz
? Can you try without installing it in your CI environment? I don't have time to look at this just this minute, but I think it has to do with experiment tracking having some dependency on git...
c
Thanks! this looks correct. But, if I don't install kedro-viz, my 
settings.py
is going to raise an issue. Setting up the store requires kedro-viz form what I understand https://docs.kedro.org/en/stable/visualisation/experiment_tracking.html#set-up-the-session-store
Does it mean that experiment tracking is not compatible with running kedro from with GH action?
d
I see, issue is a GitPython issue because you're on detached head. I didn't read it properly before. Maybe you can try something like this? https://github.com/actions/checkout/issues/124#issuecomment-586664611 Not 100% sure about why the fetch depth thing didn't work. Otherwise, I would consider running a version with that git metadata fetching step patched out, because you aren't actually needing to do tracking. TBH, I'm not very familiar with experiment tracking and some of these dependencies, maybe @Antony Milne have you run across this?
c
thanks, yes, will try with #124. Would make sense that fetch.depth=0 did not work because it was a pull request. Will let you knoe. Thanks a lot @Deepyaman Datta!
a
Hi @Cyril Verluise, thanks for this very clear bug report with excellent error log! I’ve never seen this before but I can see where the problem is. Ultimately this is a bug with kedro-viz since the 
SQLiteStore
shouldn’t fail due to a git error. It’s already designed to work in the case that git isn’t installed but we didn’t anticipate git throwing a 
TypeError
. So we should change that 
except
just to catch anything. In the short term, before we release a fix, there’s several ways you could fix this yourself: 1. Copy and paste 
SQLiteStore
into your own codebase, apply the fix there and then point to that in settings.py rather than the kedro-viz one 2. Try to make it so that the underlying 
TypeError
no longer gets triggered at all, e.g. by following what @Deepyaman Datta suggested above 3. Change your settings.py to no longer use 
SQLiteStore
on CI. You can probably achieve this without having to have two different versions for settings.py (one for your project vs. one for CI) by only enabling 
SQLiteStore
in settings.py according to an environment variable that’s set in github actions: 
if os.environ.get("CI"): SESSION_STORE_CLASS = SQLiteStore
``````
c
thanks! very clear. Will try that and let you know which approach worked/was chosen. Thanks
👍 1
a
I am curious - do you actually want to keep run experiment tracking and look at the results in CI? Or it’s just there because you use it on your project already and want to run the same pipeline in CI?
c
just there because I'm using it in the project and want to run the same pipeline in CI
a
Cool, in that case I’d probably recommend solution 3 above because then you don’t have the (very small) overhead of writing to the 
SQLiteStore
at all in CI. You could then remove kedro-viz as a CI requirement also.
👍 1
c
thx
👍 1
hey, solution 3 worked like a charm. full snippet with GH actions is
Copy code
if not os.environ.get("GITHUB_ACTIONS"):
    SESSION_STORE_CLASS = SQLiteStore
    SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}
Should I raise an issue on kedro viz repo so that we keep track of it? Thanks a lot @Antony Milne and @Deepyaman Datta! Cheers!
a
perfect, thank you for this @Cyril Verluise. If you don’t mind raising an issue on kedro-viz then that would be great, thank you!
✅ 1
35 Views
Open in Slack
PreviousNext
Powered by