Prometheus Vector Operations Are Not Associative
I was working on a PromQL query at work to find disk usage on Kubernetes control plane nodes - the first step was to identify those nodes:
Read More...Posted on 2025-03-29
Restic Snapshot Groups, AKA "Why isn't restic forgetting my snapshots?"
I use restic to manage backups on my machines, and not too long ago I ran into some surprising behavior that I thought I'd share.
Read More...Posted on 2025-03-23
Use GOTOOLCHAIN to run older versions of the Go toolchain
I've been working through Mastering Distributed Tracing to learn more about (you guessed it) tracing. The book is from 2019, and while I've been assured that its contents are still sufficiently up-to-date, the Go-based code examples are a bit dated; namely, they use dep for dependency management, rather than the new(er) built-in module system. dep seems like a fine tool that had its time, but seeing as I'm used to Go modules, and I don't really want to learn an obsolete tool just for some book exercises, I created a Go module for the exercises and started converting the Gopkg.toml/Gopkg.lock files by hand.
Posted on 2025-02-16
Beware staleness when creating recursive recording rules in Prometheus
At work, we have a Prometheus metric from our build system named autobuild_build_timestamp that exports the timestamp for the latest build for a given application and Git branch. The process that exports this is stateless, so if it restarts or a new version of it is deployed, we're out of luck if we want to do something like build a dashboard to show the latest builds. One thing we can do is use a range query, but we can also make use of Prometheus' recording rules to provide a nice abstraction to users who care about this data, and to forgo a bunch of redundant recalculations of the same data.
Posted on 2024-10-30
Why is my Lambda function getting retried when getting synchronously called from the command line?
I recently ran into a surprising behavior when calling a Lambda function from the command line that I thought I'd share.
Read More...Posted on 2024-09-13
Older Entries