If you can’t remember what something does, cut it off. If you know remember it, put it back on the document it.

Ansible and Nix. Code is the document.

Everyone will have their own system.

I save all my credentials in Bitwarden/Vaultwarden and take notes in Joplin.

The good thing about YOUR homelab is that YOU’RE taking notes solely for YOURSELF and only YOU know how YOU work and how YOU organize YOUR thoughts.

I have a repo for the infra files (compose files and terraform files just for playing). I store the docs in the same repo in MD files. As for the secrets, I’m using docker swarm, so I can store the needed passwords there. otherwise Vaulwarden is my go to, <ad> self hosted, lightweight password manager, compatible with bitwarden clients </ad> I’m a little paranoid if the note-service got db corruptions, I might loose too much info, so git is the way (personal opinion).

edit: add the related MD file next to the compose file, one folder per service, the source and the doc will be coupled in one place.

Document everything as if it were a step by step tutorial you will give to someone so that they can duplicate your deployment without any prior knowledge. I’ll even include urls to sites I consulted with to achieve production deployment.

ETA: I absolutely care nothing about points. Up voting and down voting used to be a way to weed out bad info. So it always leaves me wondering 'Did I give erroneous advice? What was the reason for the down vote? I mean, if you down voted and said ‘I down voted you because I hate your guts’, I can deal with that.

I’ve been documenting my homelab experiments, set ups, configurations, how-to’s, etc in both Trilium and Silverbullet. I use Silverbullet more as a wiki and Trilium for journal style notes. I just got into self hosting earlier this year, so I’m by no means an expert or authority on any of this.

So my Silverbullet set up contains most of my documentation on how to get things set up. I have sections for specific components of the homelab (Proxmox general set up, general networking, specific how tos for getting various VMs and LXCs set up for specific applications, specific how tos on getting docker stacks up and running, etc.)

I didn’t document shit the first two times I set up and restarted my entire homelab, but by the third time I learned. And from there I basically just wrote down what I did to get things running properly, and then reviewed the notes afterword to make sure I understood what I wrote. This is never a perfect process, so in the following attempts of resetting my server, I’ve updated sections or made things more clear so that when I’m coming at this 8 months later I can follow my guide fully and be up and running.

Some of my notes are just copy pasted directly from tutorials I originally followed to get things set up. This way I just have an easily accessible local copy.

When I troubleshoot something, I document the steps I take in Trilium using the journal feature, so I can easily track the times and dates of when I did what. This has helped me out immensely because I forget what the fuck I did the week before all the time.

I learned all this through trial and error. You’ll figure out what needs to be documented as you go along, so don’t get too caught up trying to make sure you have a perfect documentation plan in place before deploying anything.

I’m one of those people who never really took notes on things or wrote shit down for most my life. Mostly because I’ve been doing shit that doesn’t require extensive documentation, so it was a big learning curve.

Edit: Forgot to mention that I also have a physical paper journal that I’ve scrawled various notes in. I found it easier to take quick notes on paper while I’m in the middle of working on something, then I transcribe those notes digitally in either Silverbullet or trilium.

You’re on the right track. Like everything else in self-hosting you will learn and develop new strategies and scale things up to an appropriate level as you go and as your homelab grows. I think the key is to start with something immediately achievable, and iterate fast, aiming for continuous improvement.

My first idea was much like yours, very traditional documentation, with words, in a document. I quickly found the same thing you did, it’s half-baked and insufficient. There’s simply no way to make make it match the actual state of the system perfectly and it is simply inadequate to use English alone to explain what I did because that ends up being too vague to be useful in a technical sense.

My next realization was that in most cases what I really wanted was to be able to know every single command I had ever run, basically without exception. So I started documenting that instead of focusing on the wording and the explanations. Then I started to feel like I wasn’t capturing every command reliably because I would get distracted trying to figure out a problem and forget to, and it was duplication of effort to copy and paste commands from the console to the document or vice versa. That turned into the idea of collecting bunches of commands together into a script, that I could potentially just run, which would at least reduce the risk of gaps and missing steps. Then I could put the commands I wanted to run right into the script, run the script, and then save it for posterity, knowing I’d accurately captured both the commands I ran and the changes I made to get it working by keeping it in version control.

But upon attempting to do so, I found that just a bunch of long lists of commands on their own isn’t terribly useful so I started to group all the lists up, attempting to find commonalities by things like server or service, and then starting organize them better into scripts for different roles and intents that I could apply to any server or service, and over time this started to develop into quite a library of scripts. As I was doing this organizing I realized that as long as I made sure the script was functionally idempotent (doesn’t change behaviors or duplicate work when run repeatedly, it’s an important concept) I can guarantee that all my commands are properly documented and also that they have all been run – and if they haven’t, or I’m not sure, I can just run the script again as it’s supposed to always be safe to re-run no matter what state the system is in. So I started moving more and more to this strategy, until I realized that if I just organized this well enough, and made the scripts run automatically when they are changed or updated, I could not only improve my guarantees of having all these commands reliably run, but also quickly run them on many different servers and services all at once without even having to think about it.

There are some downsides of course, this leaves the potential of bugs in the scripts that make it not idempotent or not safe to re-run, and the only thing I can do is try to make sure they don’t happen, and if they do, identify and fix these bugs when they happen. The next step is probably to have some kind of testing process and environment (preferably automated) but now I’m really getting into the weeds. But at least I don’t really have any concerns that my system is undocumented anymore. I can quickly reference almost anything it’s doing or how it’s set up. That said, one other risk is that the system of scripts and automation becomes so complex that they start being too complex to quickly untangle, and at that point I’ll need better documentation for them. And ultimately you get into a circle of how do you validate the things your scripts are doing are actually working and doing what you expect them to do and that nothing is being missed, and usually you run back into the same ideas that doomed your documentation from the start, consistency and accuracy.

It also opens an attack vector, where somebody gaining access to these scripts not only gains all the most detailed knowledge of how your system is configured but also the potential to inject commands into those scripts and run them anywhere, so you have to make sure to treat these scripts and systems like the crown jewels they are. If they are compromised, you are in serious trouble.

By now I have of course realized (and you all probably have too) that I have independently re-invented infrastructure-as-code. There are tools and systems (ansible and terraform come to mind) to help you do this, and at some point I may decide to take advantage of them but personally I’m not there yet. Maybe soon. If you want to skip the intermediate steps I did, you might even be able to skip directly to that approach. But personally I think there is value in the process, it helps defining your needs and building your understanding that there really isn’t anything magical going on behind the scenes and that may help prevent these tools from turning into a black box which isn’t actually going to help you understand your system.

Do I have a perfect system? Of course not. In a lot of ways it’s probably horrific and I’m sure there are more experienced professionals out there cringing or perhaps already furiously warming up their keyboards. But I learned a lot, understand a lot more than I did when I started, and you can too. Maybe you’ll follow the same path I did, maybe you won’t. But you’ll get there.

Ansible is my config and documentation in one.

It’s reproducible, idempotent and I don’t need anything else.

I write all code myself, that makes it even easier to read.

Whenever I set something up I usually make a markdown file listing the commands and steps to take. I do this as I am setting things up and familiarizing myself, so once I’m done, I have a start to finish guide.

Raw text/markdown files will be readable until the end of time.

I created something similar to this. It got a lot of love during interviews later down the line. https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fi.redd.it%2Fvmd34mabi4r91.jpg&f=1&ipt=2dde77fd04d48156bc514ad4b1f090c8473f4e666ead0e16906eeed55a79aca6