Hello, I'm Gerhard

Hello, I’m Gerhard

Welcome to my blog. I started it in 2023 and now I’m picking it up again, so I’ve rewritten this post.

I work as a consultant, mostly on DevSecOps and platform engineering. But observability is where I started and it’s still the thing I like most: monitoring, alerting, logging and dashboards.

The tools I use the most are Zabbix, Prometheus, Thanos, AlertManager, ElasticSearch, Kibana and Grafana. And for the infrastructure around it: Terraform, Ansible, Kubernetes and Argo CD.

Why open source?

For me it’s about openness and transparency. You can see how it works, you can change it and if something is missing, you can build it yourself. And it lets you build monitoring that is cost-effective without giving up on quality.

Some examples from a previous assignment.

Logs with the full ELK stack

We had Zabbix for the metrics, but we also wanted the logs. So on a number of AIX machines we built a Logstash shipper that sent the logs to a Redis instance. On the other side, Logstash indexers picked the logs up from Redis and added extra fields or renamed fields. After that, the logs went to ElasticSearch and Kibana. And they also went to Zabbix, where I made trapper items so we could alert on warnings and errors in the log messages.

Metrics with Thanos

We had multiple AWS environments. Every environment had its own Prometheus, and each of them used remote-write to send its metrics to Thanos. We started with Thanos on AWS ECS and later moved it to one Kubernetes environment, as the central place for all metrics. The store gateway, the compactor, thanos-query, everything just ran fine.

We also used Thanos as the datasource in Grafana. Why? Because during a priority 1 incident everybody opens Grafana at the same time. Prometheus got overloaded and we could lose metrics. And that is exactly the moment you don’t want that. With Thanos answering the queries, Prometheus could keep doing its job.

AlertManager

I ran AlertManager as a separate component, because that made it easy to change the routes and receivers. We kept it simple: two severity levels. A message went to Slack or to PagerDuty, depending on the time of day.

My homelab

I used to have a homelab, an old HP DL360 G8 with 32 vCPU’s, 96 GB of RAM and 4TB of storage, running Proxmox. I used it to try things before I used them for real: containers, Kubernetes clusters, everything. That machine has been switched off for a while now.

Nowadays I use Terraform and Ansible to set up my VPS, together with Kubernetes and Argo CD. Everything is code, so I can rebuild it whenever I want.

What can you expect?

Things I run into at work and at home. Like the exporters I wrote in Go, and how I solved problems that weren’t solved out of the box. I’ll try to keep it short, and I’ll tell you what didn’t work too.