arrow-down coffee engineering consultancy development remote-management support linkedin twitter youtube email phone gitlab github

Writing Icinga2 configuration with modularity in mind

Submitted by oliver on October 08, 2018

This is a quick post on icinga2 configuration file style. But it’s really about modularity in text-file based computer systems.

In programming circles, I often hear about modularity. In operations circles, you hear less about it, but I think it is just as important. Not just for the robustness or availability of a service, but also the understandability and clarity of infrastructure.

Let’s take what I consider a pretty typical icinga2 configuration directory:

/etc/icinga2/conf.d/db.conf
/etc/icinga2/conf.d/app.conf
/etc/icinga2/conf.d/sys.conf
/etc/icinga2/conf.d/groups.conf

We have four configuration files. db.conf contains all the database checks. app.conf contains all the checks for the application servers. sys.conf contains “everything else”; stuff like disk space and ssh connection checks. groups.conf contains rules to put everything into groups so that it’s easy to see what belongs where.

When you look at these files, what can you see about the environment being monitored? It’s difficult to tell. You can see that there are some application servers, and they probably use some database(s) to store data.

Icinga2 doesn’t care what files contain what configuration. It all gets “compiled” into a big thing at runtime. This means you can structure your configuration however you like.

Let’s imagine that there are three applications running on this infrastructure. We have invoicer, scheduler and searcher. Another way to structure the configuration files is like so:

/etc/icinga2/conf.d/invoicer.conf
/etc/icinga2/conf.d/scheduler.conf
/etc/icinga2/conf.d/searcher.conf
/etc/icinga2/conf.d/sys.conf

Now when we look at the files, we quickly get an idea of what is being monitored. The files become a lot more human-readable, too. Just by opening the file we can get an idea of what each environment consists of. Let’s take invoicer.conf as an example.

object Host "app1.example.com" { ... }
object Host "app2.example.com" { ... }
object Host "db1.example.com" { ... }
object Host "db2.example.com" { ... }
object HostGroup "invoicer" { ... }
apply Service "invoicer_health" { ... }
apply Service "invoicer_generate_invoice" { ... }

It reads from top-to-bottom and left-to-right, like we expect when we read almost everything (in English, at least).

The configuration becomes more modular. You can be confident that when you are reading the monitoring configuration for invoicer.conf, you are seeing almost everything that is being monitored for that service, and nothing else. If the application needs to be checked from another icinga server, all you will need to do is the basic operation of moving a file, rather than doing a big grep(1) session through a bunch of files and extracting bits and pieces from everywhere. Moving to another monitoring system also becomes a lot less risky; read through the file and you can have confidence that you haven’t missed anything hidden in some other weird file somewhere.