showers

Separation of Repo and State

code for this post srewohs/homelab @ v0.1.0

New here? The About page covers what the lab is and why this blog exists.

“Paranoid is what they call people who imagine threats against their life. I have threats against my life.”

Garak, Star Trek: Deep Space Nine

I wanted to share my homelab setup with others. More specifically, I wanted to share how my services are configured in a meaningful and reproducible way. This is part of the reason I decided to move toward keeping my configuration as code, and it has the added benefit of serving as a form of self-documentation and timeline for my work.

My goal was to have a place I could point someone to after talking about a new service I had set up or a specific approach I took. Making my Ansible code public is also a good way to keep myself honest about writing things properly.

The issue I ran into was that the code lived in the same repo as everything that identifies my network, plus the credentials that unlock it. My understanding is that this is normal for an Ansible repo, but it’s also why I felt that “publish and pray” wasn’t an acceptable solution.

Copies don’t work (for me at least)#

The first solution I considered was a sanitized public copy of the repo. You can strip out the addresses and secrets, push the result somewhere public, and then repeat whenever the code changes (and of course never get lazy or forget to do this step). I wasn’t really a fan of this approach because it would mean a manual scrubbing step before every push, and the two copies would inevitably get out of sync.

I know myself well enough to see that a step I have to remember every time is a step I’ll eventually skip.

Split by kind of content instead#

So instead of two copies, the approach is two repos that hold different things:

In hindsight this is pretty obvious. It’s effectively how shared Ansible roles already work, where the code is public and everyone brings their own inventory. I just had a Miles Morales moment and had to do my own thing.

This works largely because of how Ansible finds variables. It loads group_vars/ and host_vars/ from the directory the inventory file is in, so pointing a playbook at the private inventory brings every lab value along with it. The two repos sit next to each other:

<parent>/
  homelab/            public: roles, playbooks, examples
  homelab-private/    private: inventory.yml, group_vars/, the vault

and playbooks run from the public one:

cd homelab
ansible-playbook -i ../homelab-private/inventory.yml playbooks/deploy-traefik.yml

Because there is nothing sensitive being kept in the public repo, there’s nothing to sanitize and nothing to keep in sync. There is only one copy of each item, and the risk of leaking secrets or lab details is minimized. The README has the setup if you’d like to do the same. If you find something I’ve overlooked, issues and pull requests are welcome.

What had to change in the code#

Splitting the files was quick, but the code itself had my lab baked into it in a few places. The old history also mixed code and lab data in every commit, so the public repo started fresh from git init.

Role defaults. Some roles used my real domain, addresses and user names as defaults. Those moved into private group_vars, leaving each role with a neutral placeholder or nothing at all. Where there’s no safe default, the role checks that the value is set and fails with a message saying what’s missing, rather than running with a placeholder that silently breaks something. From the Traefik role’s defaults:

# Required, no defaults (the role asserts they are set):
#   traefik_acme_email     Let's Encrypt account email
#   traefik_base_domains   list of domains; each gets a wildcard certificate
#   traefik_services       list of {name, url}; routed as <name>.<domain>

ansible.cfg. Mine had personal settings in it, including the inventory path, the vault password file, and host key checking turned off. Ansible reads only one config file and doesn’t merge them, so personal settings can’t be layered on top of a shared file. The public ansible.cfg now holds only portable settings, and the personal ones are environment variables in a gitignored .envrc, loaded by direnv:

export HOMELAB_PRIVATE="$PWD/../homelab-private"
export ANSIBLE_INVENTORY="$HOMELAB_PRIVATE/inventory.yml"
export ANSIBLE_VAULT_PASSWORD_FILE="$HOME/.ansible-vault-pass"

With ANSIBLE_INVENTORY set, the -i flag goes away.

Examples. The examples/ directory has a fake inventory with every variable and vault variable the roles expect, using documentation addresses and CHANGEME values, ready to copy into your own private repo.

Guardrails#

This split is only as good as the next commit to the public repo, and unfortunately it’s very easy to paste a real address into a README or a role default without noticing. The quote above might be slightly dramatic, but every commit gets the TSA special anyway.

A private denylist. Before the split, I ran a secret scanner over the old repo’s entire history. It found nothing, which didn’t mean much: scanners generally look for things shaped like secrets, and my addresses, host names, and domain don’t fit that description. So a pre-commit hook checks staged files, and the commit’s author and committer, against a list of those strings. The list lives in the private repo, since publishing a list of the things you’re trying to hide would be a bit counterproductive. The hook fails closed: if it can’t find the list, it blocks the commit.

gitleaks^2. gitleaks runs as a pre-commit hook and again in GitHub Actions over the full history on every push. A local hook can be skipped, but CI can’t. Since CI can’t see the private denylist, it only catches generic secrets like API keys.

A vault leak check. Yes, I’m wearing a tinfoil hat. No, I won’t take it off. The last check before any push lives in the private repo, because it needs the vault. It decrypts the vault in memory and searches the public repo’s files and every commit in its history for each secret value. A hit prints only the variable name and file, so the secret itself never appears on screen. There is also a self-test mode that plants a value to prove the check still catches it.

What’s not done#

Most of the lab is still set up by hand, and those parts will come over to Ansible one at a time. The benefit of this work now is that they land somewhere I can share with peace of mind.

Found a mistake, or something I overlooked? Open an issue on the repo.