Aiir — our small AI experiment that didn’t work out

Today, at ioki’s Android coding dojo, we tried to use an LLM to summarize why our CI (GitHub Action) was failing.

Basically, we just glued a bunch of existing CLI tools together into a new one. 😅

How does it work?

1. ./aiir [PR_NUMBER]
2. Using gh (the official GitHub CLI), it fetches the checks for the given PR number and looks for jobs in the FAILURE state
3. It then fetches the logs from each failed job via gh run view
4. Finally, it sends the logs — together with a prompt — to ollama run

The tool expects both gh and ollama to be installed.
Additionally, gh needs to be aware of the git remote, so it’s best to place and run the tool inside the git repository.

Our results with qwen3:0.4b, qwen3:4b, and qwen3:8b? Pretty bad! 😂

The LLM output was cluttered with CI warnings and other unrelated “problems”. While the real errors were mentioned, they got lost in the sheer amount of unrelated text. In our tests, we knew why the CI failed. But if you don’t, the output isn’t helpful at all.

What was the greater goal of that experiment?

We hoped the LLM would give us a helpful summary of the CI errors, which we could then pass further to the GitHub Copilot CLI to fix them for us. 😳

But since the first step didn’t work as expected — and we also ran out of time — we stopped there.

If you want to play around with it, or even improve it, you can find the aiir code here:

The aiir source code

Nerdy details

1. The name aiir stands for Artificial Intelligence Issue Reporter — clever, right? 😉
2. After we figured out how to approach it, we used GitHub Copilot’s Agent mode (GPT 4.1) to write the entire codebase. We didn’t touch or change a single line