From Chatbot to Self-Hosted AI Infrastructure

My AI setup started as a model choice: GPT or Claude, local or hosted, bigger context window, cheaper tokens. That still matters, but it is not enough once an assistant can read notes, call tools, remember preferences, and interact with infrastructure.

This post covers the shape of the setup. The deeper details belong in the follow-up posts.

The setup in layers

At home I run Hermes as the agent layer on my main Linux machine. It talks to model providers, but it also has access to local tools, memory, scheduled jobs, and messaging. Around that sits a Docker-based infrastructure stack for knowledge retrieval, memory, search, monitoring, secrets, reverse proxying, and voice experiments with Whisper, Piper, XTTS, and Ollama.

I think about it in three layers:

  • Agent layer: Hermes, model routing, tool use, scheduled jobs, Telegram, and Claude review.
  • Context layer: Obsidian, the local knowledge API, Qdrant, and memory.
  • Boundary layer: secrets vault, MCP access rules, reverse proxy/authentication rules, monitoring, Home Assistant, and voice.

The model layer

A better model helps. It reduces the amount of babysitting and makes the assistant better at following a plan.

The useful part is that the setup is model agnostic. Hermes is the layer I work with; the model behind it can change. Today that might be GPT or Claude through a hosted provider. Tomorrow it could be Mistral, OpenRouter, a local Ollama model, or whatever the new best model happens to be. The notes, tools, permissions, and workflows do not need to change with it.

Most of the value comes from that surrounding workflow. For example:

  • Hermes can inspect files instead of guessing from memory.
  • It can search my Obsidian notes through a local knowledge layer.
  • It can use MCP tools for services like Gitea, Grafana, NetBox, NPM, and a secrets vault.
  • It can ask Claude for a second review before I publish or commit something substantial.

So Hermes works from the actual file or note, not only from memory.

Retrieval instead of a bigger prompt

My notes live in Obsidian. That is where infrastructure notes, plans, diary context, and working thoughts end up. The simple option is to paste a pile of markdown into a prompt and hope the model finds the right paragraph. That works until the pile gets big enough to become its own problem.

The retrieval layer changes the shape of that. Notes are split into smaller chunks, each chunk gets an embedding, and Qdrant stores those embeddings in a vector database. When I ask a question, the knowledge API can search for chunks that are close to the meaning of the question, not just for exact matching words. It then returns a small set of cited snippets instead of dumping the whole vault into the prompt.

So the roles stay separated:

  • Obsidian stays the human workspace.
  • The knowledge API decides which cited chunks are relevant.
  • Qdrant handles vector search over those chunks.
  • Memory stores durable preferences and recurring facts.

That is enough detail for this overview. I will cover the context layer in a follow-up post, including how the vector database knows where context lives and why that is different from just adding more text to every prompt.

Memory is useful, but I do not treat it as truth. If the assistant is going to change something important, it should check a note, a config file, or the running system first.

Boundaries for tools

Tools are also where most of the risk sits.

A tool that summarizes a note is low risk. A tool that can change infrastructure, access secrets, or control something in the house needs different defaults. In my setup that means:

  • Secrets stay in a secrets vault and are exposed through an MCP server only when needed.
  • Internal services sit behind reverse proxy and authentication rules.
  • MCP tools are treated differently depending on what they can change.
  • Monitoring and logs matter because I want to see what happened afterwards.
  • Voice and Home Assistant need stricter defaults than a terminal-only workflow.

For me, the parts worth keeping close are the control points: notes, memory, secrets, logs, automations, and the rules around them. The model itself does not always have to run on my own hardware.

What this series will cover

I am splitting the details into three smaller posts:

  1. The agent layer
    How Hermes, model routing, tool use, scheduled jobs, and Claude review fit together.

  2. The context layer
    How Obsidian, local retrieval, Qdrant, and memory changed how I use my own notes.

  3. The boundary layer
    How self-hosting, secrets, voice, observability, and home automation ended up in the same design discussion.

I may add a separate architecture reference with a diagram and component map. That belongs in its own post so the main articles stay readable.