nixos, mostly. It has a very steep learning curve but if it breaks I’m just “whatever, rollback now, fix properly later”.

I actually don’t remember off the top of my head, huh. The output chain is not the one I use often. I’d think nft add rule tracing filter output ip protocol icmp icmp type echo-reply meta nftrace set 1 would do it. Just make sure the priority is low enough again.

systemd will try to save any kernel panics in pstore via systemd-pstore, if it’s enabled in your kernel. I’d check /var/lib/systemd/pstore and see if anything in there on the next boot.

Can you elaborate on ping “running”? Do you get actual icmp replies coming back? Because there’s no code path I can imagine where a ping would cause hdd activity (on a normally running system).

If ssh times out (and you don’t do anything fancy with the firewall), then it’s not sshd dead, it’s sshd not being able to respond. Grab a tcpdump for dst port 22 from your local machine while RPi is stuck and see if you get any replies whatsoever or it’s just retransmits going into the void.

My first rough suspicion would be ram abuse. Something eats up all ram and the system locks up and semi-dies. Pstore would have OOMs. You could run a local script for telemetry recording too to see if ram use spikes up before the system gets unresponsive.

It is accepted just fine then. You might want to check the reverse, start in top of the output chain and trace the echo reply.

nftables offers a very decent debugging interface. First, you add a rule to trace the packet (a new chain with high priority works best). Usually I’d suggest to add those rules by hand instead of relying on declarative configs:

nft add table ip ping_trace
nft 'add chain ip ping_trace prerouting { type filter hook prerouting priority -301; policy accept; }'

Then you add the actual tracing rule there with the shape to match the incoming traffic:

nft 'add rule ip ping_trace prerouting icmp type { echo-request, echo-reply } meta nftrace set 1'

now you can run nft monitor trace and see the decisions made for the matching packets. Remember to delete the ping_trace table afterwards to clean up.

First on how to get an ASN: you can buy it for reasonably cheap from a LIR. Some will even toss a free /48 with that. Happy to offer names in private so that there’s no advertising. Expect a budget quoted above.

Once you have an ASN, you need to get an upstream - actually two as RIPE mandates at least two (otherwise why’d you need an ASN). Some LIRs would offer transit with ASN purchase. You can upstream via your ISP, if they allow you to (that’s very rare). Another option is a tunnel (there are free and paid ones) or a VM somewhere (some cloud providers offer to set up bgp with VMs they host). Generally, free ones are enough for basic stuff. Not much bandwidth and oftentimes IPv6 only, but you don’t pay anything either. Besides, you can ask around in various network related chats. Practically, I can offer ip transit with some marginally low burstable bandwidth, and that’s pretty common. You can look/ask around https://discord.gg/ipv6 for example.

For getting ASN to your homelab you’re looking at a tunnel option, most probably. Great if you have static ipv4 - allows you to use more common tunnels, but is still doable with a floating IP (e.g. check bgptunnel).

BGP anycast person here. If you have any presence in the region RIPE operates in then the pricing is within the homelab reach. ASN and a block of /48 would be about 70 EUR/year.

Alternatively, something like route64 would happily tunnel you IPs they announce for about 2 EUR/month.

I’ve used VM in my homelab for a while - it’s genuinely a smaller ram footprint, especially on the ingress path. You can even scale query separately if required.

I’d suggest VictoriaMetrics over pure Prometheus - better ram footprint is pretty useful nowadays.

While matter technically requires IPv6 there are non-confirming devices (glares at Hue) that will only do IPv4. It will work with most of the matter controller networks, because in the end it’s a conscious decision to disable IPv4 there.

You really want the ECC ram and the motherboard/cpu combo that supports it.

If your note’s type is JSON (or TW’s native dictionary), you can query it as such in filters.

My search problem is that I rely on metadata a lot. It’s natural for me to want a UI that renders machine readable metadata in a way that my brain can process and that requires rich querying capabilities.

I tried them all and, so far, TW wins for me, with orgmode being second close (I like orgmode in vim, but it had some fatal rendering flaws and I don’t feel like using emacs just for notes).

tiddlywiki has one of the most insane search engines from this list. They have a whole filters syntax that can express pretty much anything imaginable, no? I went back to TW from Obsidian because I was tired from Obsidian’s trivial search functionality.

Let’s untangle those problems. I have a similar setup so I just want to share some ideas to show that you don’t need to copy keys.

If I’m traveling or I wipe my device or get a new one, I would have to add the new key to many servers as authorized keys

If you oftentimes access ssh from untrusted systems you’re kind of in a bad spot to begin with. The best thing you can have is a yubikey on a keychain. Everything else means you leak secret material (a password or a key) to a machine you don’t inherently trust.

Also, I want a key backed up in case of disaster since all of my devices are in my home most of the time

Again, something that you can easily solve with a hardware key [in a safe]. But realistically, in case of a disaster a local shell password login should be good enough?

I’d recommend you to think about what attacks are you trying to prevent by using a shared private key. I’m not saying it’s a bad concept, inherently having it in your password manager (like 1Password that even has ssh-agent support) is pretty common. The problem with just the keys is that it’s non-trivial to expire them if needed. You might be indeed better off with some web based authentication that you can access from any place which would ask you secret questions/send you a text message or do whatever 2FA you deem sufficient and mint you a short-lived certificate for ssh.

Not an answer, but I’m curious: what’s wrong with just having several ssh keys, one per device?

You can absolutely run your own CA and even get your friends to trust it.

I’m in a same boat, honestly.

Matrix has decent clients but managing a matrix instance is a world of pain, especially if you federate. And its resource use is really bad then: a single user instance can easily demand 4gb ram if you are in a couple popular chatrooms. Key propagation is oftentimes broken. Clients all have mixed support of features.

Xmpp is a joy to host, but there are no decent clients for iOS.

IRC is easy to host, but the IRCv3 coverage for clients is also meh.

I was looking for something that I could throw at casual people with relative ease and there’s just not a thing. Even the “techy” chat is in discord nowadays.

I’ll chime in: simplicity. It’s much easier to keep a few patches that apply to local OS builds: I use Nix, so my Mastodon microVM config just has an extra patch line. If there’s a new Mastodon update, the patch most probably will work for it too.

Yes, I could build my own Docker container, but you can’t easily build it with a patch (for Mastodon specifically, you need to patch js pre-minification). It’s doable, but it’s quite annoying. And then you need to keep track of upstream and update your Dockerfile with new versions.

One thing about grafana, though, is that you get logs, metrics and monitoring in the same package. You can use loki as the actual log store and it’s easy to integrate it with the likes of journald and docker.

Yes, you will have to spend more time learning LogQL, but it can be very handy where you don’t have metrics (or don’t want to implement them) and still want some useful data from logs.

After all, text logs are just very raw, unstructured events in time. You may think that you only look into them very occasionally when things break and you would be correct. But if you want to alert on them, oftentimes that means you’re going from raw logs to structured data. Loki’s LogQL does that, and it’s still ten times easier to manage than the elastic stack.

VictoriaMetrics has its own logging product too, now, and while I didn’t try it yet, VM for metrics is probably the best thing ever happened since Prometheus. Especially for resource constrained homelabs.

Storage box networking can be hit and miss. It’s ok for incremental uploads, but I went through hell and back to get the initial backup finish, which makes me wonder what it would take to download it in case I have to.

Scp breaks off once in a while, and WebDAV terminates the session. I didn’t try smb as I feel it’s a rather weird protocol for the public internet. In the end, I figured it’s not the networking per se, it’s something with the timeouts on the remote, and I was able to finish the backup using a Hetzner-hosted server as a jumpbox.

But it’s cheap, yeah.