I’ve been doing a lot of traffic engineering studies now that I have my own ASN; apparently traffic engineering is a massive field with many interesting problems you never get to experience unless you’re doing network operations.

I have several upstreams and IXPs terminated on my router. What are they? Upstreams are providers that offer me transit, aka the broad “access to the rest of the internet.” They basically agree to transit packets from my network to networks they can reach (and they always can reach more) for free, or for a small fee (or for a very large fee if you have lots of traffic).

I finally passed my ham exam and now I can do more exciting radio transmissions than just an 802.15.4 mesh. With the license, I got access to the awesome 44net, too. 44net is a community for the licensed radio operators to experiment and learn using real reachable IPv4 addres...

I’ve been facing a strange problem with my kata setup that manifests in my inability to use more than one vCPU.

The normal flow is that you use k8s’s limits.cpu to set the number of vCPUs; however, that doesn’t work for me:

Error: failed to create containerd task: failed to create shim task: failed to hot add vCPUs: only 0 vCPUs of 3 were added

This error comes from hotplugAddCPUs function, that talks to QEMU over the QMP socket and first queries the number of spare CPUs and then tries to add more (if needed). The QMP socket is a file descriptor, though, so you can’t access it directly. There is some code that allows you to expose a separate, unix socket, which is only enabled in debug mode, as you can easily break kata state by manipulating the guest directly.

---

Intermission

I figured that I only publish anything to my blog when it reaches the level of polish I expect from a good technical article. That means that most of my notes never leave emacs. It’s, frankly, boring. I will experiment with turning more of my personal notes into posts even though they might seem like ramblings. You know where to provide the feedback if you don’t like it (or if you do).

Kata Containers is a lightweight VM container runtime that allows to run your workloads with better isolation. Setting them up on NixOS with k3s is slightly more intricate than it should be, so here’s a quick guide.

First, allow k3s to access a whole bunch of /dev/* nodes that are used for VM bootstrapping. K3s runs the containerd as part of its server. If you have an external containerd, adjust the service name accordingly.

In the New Year episode of “farcaller breaks things and then fixes them” I will tell you about envoy and its lua scripting capabilities. I have a bit of a tricky website setup in here—the old website is wordpress, and the new one is static files. What I want is to serve the static files (off their S3 bucket web endpoint provided by garage), unless the response is 404. In that case I want to use a fallback backend (wordpress), but only if that doesn’t return 404 either (if both are a miss I want the 404 page coming from the static website). I also want to always hit wordpress if there’s a magic query param or a cookie, as a fallback mechanism.

It’s been a rough month. Winters are always rough with the lack of sunlight, but it was just getting progressively worse. I was suffering from brain-rot. I needed a brain twister.

A year ago, Shinyuu asked me what it would take to make a simple bank app for her roleplay project. With the requirements being extremely trivial, I figured I’d challenge myself to write a bank using a blockchain. That’s how dracones blockchain became a thing. It was a very non-trivial rust project with a huge codebase (mostly coming from the underlying substrate blockchain), all the complexity of Ethereum, with WASM sprinkled on top. It was a fun project to do, and I somewhat enjoyed tearing out the “expensive” consensus and figuring out how to replace it with a trivial Raft-based state machine. In the long run, it showed that blockchains are mostly useless as databases.

Cilium has this fancy feature where it can filter the outgoing DNS traffic. Basically, you’d have a policy like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: block-dns spec: end...

Mastodon developers insist on having the max post length being limited to 500 characters. While it’s their software and they can do whatever they want, it’s also open source, so I can do whatever I want with it.

Previously, that meant re-building both the frontend and the backend, because the limit was explicit in both.

With Masotodon 4.3.0, a new change landed, letting the fronted deduce the limit from the backend’s configuration. That means the patch is now much slimmer:

Why Clojure?

I’ve been working with Clojure on and off for soon a year. I find the language fun, and I find expressing my thoughts in it outright joyful. While I’d spend way less time programming some things I did lately in Go or TypeScript, they don’t give me the same feeling of satisfaction with the result.

Clojure isn’t all rainbows and unicorns, though. I struggle with the JVM clojure a lot (I still yet to figure an idiomatic websocket-based RPC there). I posted about my issues with promises in ClojureScript, too.

When I moved from Synology as a NAS solution, I migrated to a self-made NixOS setup. This was mostly because of the extra flexibility I required. I switched to ZFS as my primary storage filesystem, but due to potential issues with encryption I opted-out of the native ZFS encryption and proceed with ZFS-on-LUKS. There are downsides to this, especially around remote replication, but I value the data on my NAS and I don’t want to be in the spot where I have to do a full restore (especially if my primary backup relies on ZFS itself).

Working with Clojure, I discovered several issues with how promises interoperate. It made me wonder, what actually is a Javascript promise?

Apparently, there’s a whole website dedicated to explaining what a promise is. From it, we can figure a straightforward definition:

“promise” is an object or function with a then method whose behavior conforms to this specification. In Javascript, Promise.resolve(true) instanceof Promise holds true, but that doesn’t mean it is the only acceptable promise. Studying Clojure’s promesa, I learned that anyone can implement a Promise-like object. After all, the only thing it needs to have is a then method.

NixOS is awesome in how it allows to specify the entire system configuration declaratively. Paired with tools like colmena one can manage a whole fleet of NixOS machines in a single, coherent repository.

That comes with downsides, when the system’s configuration has to change during an emergency. Recently, I had an ISP outage that made NextDNS completely inaccessible. My whole LAN is based on the premise that my DNS server at 192.168.53.53 can resolve the queries, and it was failing to reach NextDNS. Normally, I’d just rebuild the system with something else (e.g. CoreDNS) to survive the networking issues, but unfortunately I don’t have any reliable derivation of CoreDNS cached, and I couldn’t reach cache.nixos.org to download it either. I was stuck in a loop of needing the DNS to run the nixos-rebuild switch that would unbork the DNS.

Being a hosted language, Clojure heavily relies on the underlying language and its runtime. While the JVM Clojure is praised for being so integrated with JVM that it’s easier to use Java code from Clojure than from Java itself, the same doesn’t exactly hold true for ClojureScript.

In the modern Javascript world, everything is async. Many APIs are awaitable even if there’s no explicit need for them to be, because chaining promises is easy. Unfortunately, ClojureScript doesn’t provide any first-class support for those.

It looks like every time I get onto a new ingress, the second thing I try to figure out is the whole OAuth thing. Now that I’m toying with Traefik in my homelab, the time has came to set up OAuth out there too, so that I could verify it’s Homekit submitting the temperature measurements into VictoriaMetrics.

The requirements I have are very simple. User interactions should have transparent authentication, i.e. if I go to grafana.homelab.example.com or traefik.homelab.example.com/dashboard/ that needs to take a round-trip to Auth0 first. Some backends have native support for that, e.g. in case of grafana or argocd the ingress doesn’t have to do anything. Other backends like traefik’s own dashboard or victorametrics’s insert daemon expect the whole authentication and authorization to be done outside.

Historically, when I wanted to send some part on my home traffic from my non-primary IP address, I’d use VPN. My whole NAS’s subnet was pretty much routed through wireguard at all times. Now that my new NAS is kubernetes-first, I re-evaluate some of those decisions and make things better. In this post I will explain how I managed to send only some of the pods’ traffic over VPN.

Now, there are many solutions to this problem, one of the most prominent of which is gluetun, a docker container sidecar that does VPN. I very much don’t like the idea of individual pods setting up VPN connections, though, because the traffic goes dark in my netflow that I collect from my RouterOS setup. Luckily, it’s not too hard to have RouterOS do the routing decisions for kubernetes!

I needed a simple S3-compatible API today, and it seems that Minio is very much an enterprise thing nowadays. Instead of trying to figure how to spin it up, I looked elsewhere and found Garage, an actually simple S3-compatible server.

Garage comes with a helm chart to simplify its installation on kubernetes. Unfortunately, and typically for helm, some values that I need to be overridden aren’t exposed through values.yaml. In my case, there’s no way to add extra attributes to the Service, and I use that to do cross-cluster routing with cilium.

VictoriaMetrics is a fantastic alternative to prometheus, especially in a home lab where resources are constrained. It’s several times more efficient with its RAM usage while being pretty much fully PromQL compatible (with a few nice extras, too).

One of the nice features of VM are retention filters, allowing to set up different retentions for different metrics (this feature is available only in the enterprise version, though). This allows longer retention for important historical data like the average HDD temperature over the last 5 years vs. dropping the very exciting and very useless (most of the time) kubelet metrics within days.

I run the node_exporter on my NAS and I’m generally curious about the temperatures inside. There’s a metric called node_hwmon_temp_celsius—it exports the temperature measurements of all the sensors you have, but it has a bit of hard to digest set of labels: node_hwmon_temp_c...

k3s is a “Lightweight Kubernetes”, that’s “built for IoT”. Apparently IoT is no longer about devices with kilobytes of RAM, because there’s no way you can run the whole kubernetes stack that thin.

People kept asking about k3s not being lightweight for a while. The official profiling page sums up the k3s requirements around 1.6Gb RAM and 6% of a CPU core. For tiny VMs with a whooping 2Gb RAM total, k3s alone will eat most of it, so it sounds that the least practical setup is 4Gb RAM nodes. Is that actually so?

I’m moving my kubernetes setups to dual stack IPv4+IPv6, and, naturally, I want cilium to handle the BGP part of that so I could just use the LoadBalancer services’ assigned IPv6 addresses directly. For that, of course, the VM host node needs to know where to route them, and this is where BGP comes into play.

Cilium natively supports BGP using gobgp. Host-side, I tried a few options, but I settled on bird2. Gobgp would be nice, I suppose, but it cannot handle the FIB natively, and instead relies on Zebra to do that, and after trying to make them talk to each other for a couple hours, I just gave up.

Clojure is fun. It’s pretty hard to write (like rust), but you actually enjoy the process (unlike rust). The syntax takes a bit getting used to with the general “method target args” approach, but it’s not too bad, just something slightly unusual.

There’s nothing like TypeScript, but guardrails allow to type check in debug builds (optionally in release, too). Those type checks can get very elaborate, way more so than in TS given they are just the same valid Clojure.

Energy bills are getting weird lately and I decided I need to know what’s my homelab’s energy usage is to make some educated decisions. I went shopping for smart plugs that can measure the usage and came back with Eve Energy.

The shop listing showed a screenshot of the iOS app where you can see usage in Watts. Seemed good enough. The device came in either HomeKit or Matter, and I decided to grab the latter, as it seemed more future-proof. This was my first Matter device, and I didn’t have a Thread Border Router, so I grabbed an Apple HomePod Mini along. back then I didn’t know what a rollercoaster of IoT I’m signing up for.

Migrations are tricky business. If you think of the Kubernetes model, the client API supports versioned objects. You can e.g. store an ExternalSecret version v1beta1 and then read it as an ExternalSecret v1alpha1. Kubernetes APIs do that by allowing upgrading and downgrading the resource versions.

Why is it practically useful? The client of your API can still support interacting with it even if the object versions are newer—of course it’s only practical for reads, as mutations will destroy whatever the new data is there.

When I based the initial implementation of tinyvmm’s networking on networkd I thought it was a genuinely good idea. Networkd knows how to manage networks and it’s much easier to talk INI files than kernel syscalls. It also had a neat DBUS API, meaning I could control it programmatically. And I didn’t even need that much—tinyvmm only needs to manage bridges and tap devices for now.

It even worked. The code was messy, sure, but I got my networking up and running with no major incidents. That is, until I noticed I can’t curl from the VM to a different machine. I could, but not always. One in 5-6 curl runs would get stuck, hanging forever. A quick check with wrk showed 5% loss.

(advisory link)

This story starts like many similar ones: I wanted to figure the best way of integration my automation with ArgoCD and ended up discovering that my code works despite the logic saying it wouldn’t. OAuth is fun like that.

I have ArgoCD configured with Auth0 as the OIDC provider. It works just fine for human interactions, but I wanted to add some machinery code that’d do automation on top of what ArgoCD does, and what better option is there than Auth0’s machine-to-machine project? Surely, that requires a new audience, and ArgoCD didn’t seem to support several of those at a quick skimming of the docs, so I went out to study the code.

Prologue MetaNotes originally started as a create-react-app template. CRA is the simplest way to get your React application up and running, after all. The original prototype also included a ReactNative version, and to maintain both in the same repo I opted in for a monorepo ...

An overly lengthy prologue

I have vast collections of notes I amassed over the years, spanning from text files in SVN, webpage snips in Evernote, semi-structured notes in Bear, and, lately, TiddlyWiki. Of course, there’s paper too—I keep at least two bullet journal styled notebooks and otherwise scribble on whatever I have at hand.

This knowledge base is vast and helps me to offload things I don’t need to remember all the time, but it also creates a problem of navigating it. Did I save that article in bear? Is that video I needed to check later in my YT watch later list, TiddlyWiki, or in my paper bullet journal’s TODO list?

Isn’t it nice when you wake up and your auth is broken, despite you not changing anything? Isn’t it fun to debug a third-party integration? Anyways, I stumbled on an annoying gRPC error: Given the backend didn’t log the request, I blamed the middle man, aka envoy. Indeed, ...

SSH supports FIDO2/U2F since 8.2, so you can generate a FIDO2 key with a trivial ssh-keygen -t ecdsa-sk. However, neither the default SSH in Windows 10 nor macOS is new enough, so I’ll guide you through the smartcard way (aka gpg ssh).

Setting up the key

To make sure you’re starting with a clean state, reset the OpenPGP applet on your yubikey. Beware, this will wipe the existing keys forever (but if you have some keys in there I suppose you don’t even need this guide).