Intro

Another midnight I’m working on my side project; my child and wife are sleeping a few meters away.

I’m asking Claude to help out: write a plan, implement a simple feature. I do a review and the result is quite clean. Modified 9 classes in a few minutes.

But did the model follow practices? Project design, patterns, separation of concerns? Do we have documentation, do we know what and how in the project?

Act I

1. The Rules of Software Development

At school and during your career you learn a lot. The first teaches you the basics: algorithms, introduction to a few types of languages, and the applicable mathematics.

You will probably learn more in practice: the principles, guidelines and patterns.

Single Responsibility, Interface Segregation, Dependency Inversion Principle.

These keep your code more structured and inherently safer.

SRP helps to have fewer side effects, where it is also worth using a functional programming style, even if you use an OOP language.

Interfaces are useful to have clear boundaries: black-boxing modules, or simply implementations, with a single front-facing contract.

Dependency inversion to depend only on the contracts and their defined interface. Again, black-boxing and side-effect free behaviour. You shouldn’t care if behind an interface dependency you consume a single, multiple or a facade implementation.

Some of these can be enforced, but overall you just keep them in mind and design accordingly to leave maintainable code behind. Maintainable, readable, not just for others, but even for yourself 1-2 years later.

2. The Learnings about AI

When I started to use AI in the early days, it felt like a more advanced search engine. You throw a problem at it, and usually you get back a pretty good answer or direct code. It shortened the loop where you google, read the docs or a Stack Overflow page, and find what you’re looking for.

Intelligent autocomplete: a whole file is generated, first as a more sophisticated data class generator, later whole mock classes. Mentioning mocks, I started to rely a lot on it to create at least the most obvious test cases, then just extend them. These were good gains already in 2023.

The question is the balance. What is the point when you lean on the tool too much and you slip on details?

Discipline, Awareness!

3. Ramping Up

I remember the first time I ran a 2-3 billion param model offline. Only on CPU, after an OpenVINO conversion of the model.

Later I started a project running on a Raspberry Pi 5. 8 GB of memory was enough to run a 1.5B model for routing requests to the appropriate cloud model and a 0.6B model for embedding.

Ramping up with agentic development was gradual and fast. At first I started to use, then build my own agentic skills. Same with agents: first I inspected ready-made solutions, then, incorporating some of the practices I liked, I started to build my own agentic pipeline.

I also found some like-minded colleagues in work, who were working on and exploring the same problems. We encourage each other and share our recent findings around AI.

My agents evolved into a pipeline named “Forge”.

The shape of Forge starts with an optional preliminary skill to create a design doc. Then the pipeline starts with a Planner agent, who interviews the user and explores the horizon with them. This is the most essential step; it requires a multi-step reasoning model. Then comes many more steps with coder, QA, tester and documenting agents.

The key factor is to separate agents and contexts. Distill everything at its agent level, write it into a file. This helps on 2 fronts: one is an extra documentation step, the other is context separation. Another important feature is the approval gates. These enforce the hands-on approach from the user. This is also multi-fold: it forces review to avoid bugs and keeps you more in the loop.

There is a potential pitfall when you’re not churning the code. You are doing the design with the agent. You’re reviewing the code, but still, it’s not written by you; there’s a detachment. It should sink in, you still own the code.

Then many versions of the pipeline came; making it right the first time is not obvious.

Actually, I’m happy I was constrained to local models in some way.

It started as an evaluation process for local agents. To my surprise it turned into heavy rework on my pipeline. But let me humbly mention: while it’s my main driver for agentic pipeline work, it’s still a work in progress.

These weaker models became differential testers of my agents. Use templates and validate if you want reliable documentation.

Define DOs and DON’T DOs as well. Mind the length, as reads pollute the context, and agent directives even earlier.

Still, eventually a prose instruction can slip; put into code what you can.

During the development I picked up a lot of skills. With Pocock’s /teach skill I even improved my learning speed where I needed it.

• RAG over Qdrant. A collection per kind of thing: notes, mail, transcripts.

  
  

• A graph store, because my notes have relationships that a similarity search keeps finding indirectly and calling coincidence.

  
  

• DuckDB analytical SQL store for my own metrics and data, next to a standard relational DB.

  
  

Also, to give credit, I used and customized the grill-me skill from Pocock. In my custom skill, at one point it became a multi-question tool with grouping/branching. To my surprise, not much later Matt also updated his skill in a somewhat similar fashion.

My takeaway here is that in some way many people are side-by-side in the game around AI and sometimes have the same influences

It is really charming to explore a field where everyone is experimenting.

You can also find many fields to work on. I also started to understand LLM models’ inner workings and structure when I was transforming the models into OpenVINO.

• There is a lot of room to refine/retrain existing models. I had plans to work in this field as well, however recent news may make it unnecessary. Before the release and fame of Jev and similar generic decision models, I planned to train a custom classifier decision model. Now I’m testing these newer decision models to see if they satisfy my requirements out of the box.

  
  

• You can work on open-source model inference (llama.cpp, vLLM, …). New models are released more and more frequently, and the hardware is specialized, not unified. The major versions of new LLM models often come with a different architecture, which needs changes on the inference side as well, not to mention further fine-tunings. There are many experimental forks of llama.cpp specialized for one model and specific hardware. I personally found many versions targeting solely Strix Halo with Qwen3.8-Flash-Next specialized forks of llama.cpp, and this is just one hardware-model combination! This requires a lot of work from enthusiasts and professionals and clearly shows there is interest and activity in these communities.

  
  

  Also there is a lot of higher-level software, which does even more, either integrating or connecting inference engines (llama-swap, LM Studio, vLLM, …).

  
  

• to interact with the agents, you have AI platform UI applications (AnythingLLM, OpenWebUI, …).

  
  

• for development, structural work with an agentic pipeline/workflow, you have to choose a harness. Here you also have many options with closed and open-source harnesses. You have the obvious sidecars: Claude has Claude Code, OpenAI has Codex. Now even DeepSeek and GLM have their own: DeepSeek and ZCode. The open-source palette ranges from the barebone Pi to OpenClaw/Hermes, which play rather in the personal assistant field.

  
  

  Harnesses are important differentiators above the models. Obviously, different use cases give different results, especially if providers cherry-pick metrics, but the model provider’s native harness doesn’t always deliver the highest scores.

  
  

• With AI platforms and harnesses you open a whole new and wide software/tooling bucket.

  
  

  Skills, Agents, MCPs: skills and agents can contain simple prose text, or can be extended/driven with scripts as well to have more deterministic results. MCPs are there to integrate any kind of endpoint/service to interact with. These aid and improve the capabilities. You can find many repositories and collections with various skills and workflows/pipelines for any activity: design, planning, coding, DevOps.

  
  

• Desktop applications: these also opened a new frontier. They’re capable of interacting at the operating system UI level, with some restrictions: accessing file managers or manipulating applications directly. You write or, even better, speak what you want, and given enough access credentials the task will be done on your user interface without manual interaction.

  
  

It’s shocking and futuristic how easy it is to make it happen: automate a full dev pipeline, where a Jira ticket triggers a flow, where the feature will be planned, created, tested, documented, deployed, and the ticket closed without any human interaction.

Obviously quality and safety are a big risk factor in this case. But the possibility is crazy.

Along this idea I created a version of Forge, where the pipeline called “nightly” picks up tasks and runs them through an automated process, from task to ready-made feature. It lands in a new worktree and builds on its own feature branch through the full Forge pipeline with no human gates, always choosing the recommended answer if there’s a question to answer.

It leaves one report behind to review in the morning: an LLM-written executive summary, a deterministic table generated from the agents’ handoff logs, then decisions needing attention, what autopilot decided at medium or low confidence. This is the first level. However, here you have just a vague idea of what was built. Next to the regular documentation of Forge, I persist the whole process at the lowest level: you can track and read through all the tool calls and actions during the run if you want to drill down.

If the process went sideways? Forge has the capability to pick up the work from any step of the pipeline, and the worktree branch has a separate commit for every step. So you can easily roll back to the latest healthy phase and kick off manual mode to steer the AI pipeline, or just implement it manually if you will.

With decent local hardware tokens are free, and speed is not a factor because of the nightly automation.

Act II

1. Birth of a Project

The project is Raz, a self-hosted personal assistant. A software, that runs at home, connects to my calendar and to my mail, remembers and distills what I discussed with it, and briefs me in the morning and at night. Still now runs only on a Raspberry Pi 5. To keep privacy over all the personal data it’s capable of anonymizing data.

It has built-in directives/commands to instruct it and make it feature-rich before you would hit the model.

To my surprise, markdown turned out to be a document format that sits well with LLM models. This was suitable, since I use Obsidian Zettelkasten-style notes. These were very convenient for my assistant to consume.

I built a RAG retrieval, which was an essential part of getting results out of my data. I worked from common sense and gut feeling. Then I read about Karpathy’s autoresearch; I loved the simplicity, felt the potential, and related it to my project, where I had a lot of knobs but only guessed the right settings.

I created a golden set: pick a Qdrant sample, generate a natural language question for it.

Phase 1 was to pick the vector type based on MRR; this resulted in using hybrid vectors.

Then I started the multi-step process to refine the weights. After some trivial errors, I started to get real results: 0.311 MRR for the dense-only baseline I had been running for months, 0.447 with hybrid, 0.522 with hybrid and tuned weights.

Surprisingly, the best settings were with time decay off. A lot of my notes and docs have no time bound on relevance, the calendar works with a rolling window, and conversations so far don’t show the need for decay – I see the need to change this in the future. The only exception is mail, but that was not part of this fine-tune session.

The markdown enrichment with links provides an Obisidian compatible read access with links, so all the connections are explorable manually as well.
The qdrant and graphDB provides exceptional retrievability, depending on the actual use cases.

2. Throughput with Audit

I have a daily job and a family with a small child. The hours available for my own project are naturally very limited.

Given the constraints, I was naturally pushing the boundaries to 10x my productivity. I started to focus on executive decisions and planning, and handed the raw coding over to the agent.

Throughput was growing, while my comprehension at midnight was often the constraint.

Development was two-fold. On one side the project grew in features; on the other I built tools, improved the pipeline, or created an eval harness and Self-Learning Skills.

I kept the disciplines:

• Documenting from intent to execution and outcome
• Reviewing code
• Testing

The project evolved well and my thirst for more and more challenging features did not end.

The features kept coming: mail, calendar, briefs, notifications, ingestion, a routing cascade between models, a nightly pipeline.

However, while no tests were failing, bugs were still creeping in. No big ones, but enough to sidetrack me and make me question the quality of my work.

I did reviews, but sometimes the scope was overly big. While a good model had enough context to track it, I on the other side didn’t always.

Also, when you run through a review you won’t spot what was missing; you only see what was changed/created. The untouched but missed spots are unseen by a review (unless you really have every module and its code in your head, which you don’t).

The AI and I both have the chance to recap why a feature was built as it is (ADR), and how and what was created (feature summary and changelog).

But again, for me as a human, reading through it all and keeping it in mind is not that easy at high velocity.

The part I’m not proud of: there was a point where I slipped into a vibecoding phase. I believed my overview was complete, that I could track this speed of work, and I trusted the LLM more and more.

Even if the models had the context size to track the project and features, there was a point where I started to feel estranged from my project.

I wasn’t short of context or tokens. I was short of comprehension, and I relied too much on an agent who is good, but eventually bounded by its driver.

3. Context Bloat

I had the assistant features I wanted. It is a well-working project, despite the occasional challenges/bugs. Every software has them.

But as I was thinking about releasing it to the public, I realised this shape is not good enough, and in the long term not the software I want to maintain the way it was. As much as I tried to organize the code, sadly I ended up with a monolith, which started to cost.

The source of the problem was also the nature of the software cycle: I did a lot of feature switches until I found the way to go. Choosing the right type of RAG or classification. Building a summarization feature which not just shortens, but enriches the data. These were needed experiments, and at the same time I was learning, digging deep into these topics, which I knew before by name only.

Multiple spikes left residue: incorrect design or even dead code.

These eventually messed with hygiene, module concern principles and the overall architecture.

To be honest, to buy speed I was sloppy on my own rules, which I learned during my career.

Act III

1. How Work Changed

I see big industry-wide experiments running. Even in case of eventual maturity, there probably won’t be 2 companies using AI the same way.

Some use it for coding, some for QA, cybersecurity, some for infra and tooling, or simply for documentation, and any combination of these. Looking at the constant change of available tools, everyone is changing their in-company rules and utilities.

One shape of use is aiding the software development lifecycle. AI can help to formulate tickets or requirement documentation. In software development it’s used to create, debug and document.

The other is integrating AI features into applications. This feels like it’s coming everywhere; it’s an annoying marketing buzz if you ship any software.

What works and is easy: for a POC or wireframe, you don’t necessarily need the cleanest code. You need fast iterations to brainstorm with the business using clickable throwaway software. Once you reach the point where you see value, and business and tech reach the same understanding, you step up to the real software development cycle and discipline level.

What would definitely change if you want to be professional and safe: Rules, Guardrails, Security.

2. Enforce the Rules, Build the Guardrails

Here comes the solution for my project rewrite: what I set up and will enforce to get the best quality out of AI.

For AI to make fewer mistakes, I needed a strategic solution with high quality enforced. So now I’m doing a rebuild from the monolith with a strangler refactor methodology. Two architectural choices drive it:

• A fat core with vertical modules.
  
  I want optional modules, while having base functionalities and an extensible core.

  
  

• Ports and adapters,
  
  so the domain stops knowing about external technologies, like the specific database.
  
  I have multiple database requirements in play and multiple models; this shape feels right for this need.

  
  

Besides my project-internal choices, what matters from the AI perspective? To enforce rules, not just remember them:

• Hard rules about which module may import which.
  
  Hard enforced rules are for both AI and humans. You can’t miss them if you have a hard check like a git pre-commit/pre-push hook.
  
  And SRP is not just for classes or functions.

  
  

• A linter,
  
  to keep the code as readable as possible.

  
  

• External warnings on specific module changes.
  
  Diff size is not the really important metric; a core functionality change should be a higher concern than a utility function.

  
  

• Modules sized small enough to read in one sitting.
  
  A manageable context for both AI and humans.

  
  

Rules alone are not enough; you also need tooling to aid you:

• A toolwarden in front of the destructive calls, and observability, so the system can tell me what it did.
• Containers, for safety, to keep AI from spreading over the whole computer.
• Counters kept in files, never in context.

3. How to Relate

It’s like a junior developer mindset: it has to discover everything from scratch, but it can do so, very fast, with a very big internal library. Its decisions can be influenced the right way with instructions and skills.

Because of this big internal library it can act like a senior, but without specific discipline and opinion. As an analogy: it’s as if you sent a request for advice to a Slack channel and random people answered. Seniority varies based on which channel/model you choose to ask. Taste or choice can vary depending on who is answering, as the model’s choice of answer is non-deterministic.

And the more you work with it, the more familiar the weak spots look.

Have you noticed the point where you can’t keep track of a feature as it spreads wide through modules and utilities?

Context rot. AI models are not so different from humans. They also work best when you don’t spread the context out too much.

I hit this before the model did: in my reviews the model still had the context, I was the one short of comprehension.

Have you read about why checklists exist in surgery and aviation? It’s not because people are incompetent; things just slip. If you rely on AI to keep a counter or checklist in prose, it will slip on it.

Did you have a period when you learned something and had the urge to apply it to your work, your software, even if it wasn’t the best fit? “If all you have is a hammer…”

In my experience AI models sometimes fall for this fallacy, too. Give them a skill and they will use it. That’s why my instructions/skills/agents usually contain a “do” and a “don’t do” section as well.

Like when my eventual monolith became full with freshly learned things, what was shiny at a point.

It also tends to agree with you. It’s always a good discipline to ask it to challenge or debate you, or its own plan. Have it make a pro-con list; when it evaluates explicitly, it will likely reach a better decision.

So a high level of curation is required to get the desired outcome.

Coding is cheap. What I always said: “the basic elements of coding, like assignments, conditions and iterations, can be taught even to a monkey”. Seeing problems and translating them into solutions, overseeing and designing systems is another step. And yes, you can use AI for this as well, but you have to be extremely cautious when laying down the foundation of a house.

If you consider all of these examples, eventually AI is not so different from us. So relate to it as you would to a fast, well-read junior on your team: give it clear rules, check its work, and while listen the offfered options, keep the design decisions in your own hands. Because

like it or not, you own it, and you’re responsible for it.

My Future Outlook

While AI is powerful, it’s far from perfect. We’re still improving it and exploring the best use cases and how it can serve more and more purposes. One thing is certain: it will be an integral part of our lives, with both good and bad effects.

My view is that we are still far from AGI, and reaching it is probably constrained by our hardware technology at this point.

We see it can be malicious even with good intent, when we see the leading AI companies having accidents with models breaking out of their environments (the irony: as far as I read, this was also partly human sloppiness). If you built a powerful thing, use comparable smart guardrails.

Where I’m uncertain is how it will affect us: will it accelerate our capabilities or brain-rot us on a long-term?