Monitoring systemd user manager with node exporter

I use systemd and its timers to run various tasks on my machines on a regular basis, and since that's all happening in the background, I want to know if those jobs fail. Now, it's better to have a job-specific alert rule that checks that the state of the world is desirable (for example - instead of checking "did the e-mail backup job succeed?", check "is the most recent file in the e-mail backup destination less than 24 hours old?"), but it's nice to have the general "did the service succeed?" check as a backstop for when a specific alert is hard to set up, or when I just haven't gotten around to it yet.

Read More...
Posted on 2024-04-07

My trick for "hot reloading"

A lot of the software I use embeds a scripting language for easy extension - this is great, but much of it doesn't provide facilties for "hot reloading"; that is, iterating on an extension without needing to restart the application and blow away its current state.

Read More...
Posted on 2024-03-17

Use caution when using $rate_interval along with increase()

A few months ago, I was pairing with my coworker Graham on a Grafana dashboard, and we ran into a pitfall I thought it'd be nice to share. We were trying to create a dashboard that would show how many changes happened to a database table between points on the graph, and we were getting some mysterious results!

Read More...
Posted on 2024-02-28

With SQLite, INSERT OR IGNORE Is Often Not What You Want

tl;dr - INSERT OR IGNORE will silently fail if any constraint is violated (even a NOT NULL on a column type!). If you want to express "insert this row unless it already exists", use ON CONFLICT instead.

Read More...
Posted on 2024-02-04

Which fields can you use with Kubernetes field selectors?

Field selectors are a handy filter you can provide kubectl get via the --field-selector option to pare down the list of resources you get back from the server. The docs mention that supported fields vary from resource to resource, but sadly don't mention which resources support which fields. I did a little bit of poking around, and - as far as I can tell - this isn't documented anywhere! So, seeing as reading source code is one of my Maslow's hammers, I broke out that particular hammer and got to reading.

Read More...
Posted on 2022-01-24