Disclaimer: I played with few nagios, icinga and icinga2 servers on small clusters. But i am very new to automation, so there is some part of speculation in what follows.
I guess that provided the docker image, we will easily deploy a basic icinga on each VM. The most tricky part seems possible via the non-interactive way aforementioned (to be tested) or via the interactive way (well known) using the expect ansible module.
Here follows what I would like to do for the configuration part of an automated monitoring setup.
Today, let’s speak about the icinga configuration philosophy. In what follows, « service » means « check » (it’s nagios/icinga nomenclature).
Most deployments I seen are clearly inspired from a « nagios » pattern. It is what is called the « Object logic style » here. By doing this, you roughly define one service for each check you will enable. Adding a new check means copy/pasting a service, which is not terribly aesthetic (think about how does looks like the conf for one hundred of http vhosts). Moreover it is not very clever: think about monitoring a http host. Maybe you would like to monitor (at least) an answer on :80, another on :443, and also the validity of the certificate. Doing this « à la nagios » means that you will copy/paste 3 times the same-but-not-exactly service (again, think about how does looks like the conf for one hundred vhosts).
Automation will help us to manage this kind of stuff. But I would not be so proud to do so. 
Icinga2 offer a much more flexible and clever way to instantiate our checks, in a way much more descriptive; thus more concise, more aesthetic, more clever and terribly lovely. This is the « apply logic style ». You can describe every kind of behavior, the way you want, using a quite complete language (including list and associative array), as an host attribute.
So directly at host level you can add any attribute you like, e.g. a list of hardware block devices, a list of mounted volumes, a list of vhosts and some associated useful attributes, a list of process you would like to check and their associated limits, a list of git repos to be checked, etc.
Those attributes provided, you can define any generic service you like which will take into account these attributes, the best way possible. Since it it generic code, you don’t have any need to template this part of code. To give an idea of the simplicity of the icinga2 language, here follows how you generically check all the certificates of all the vhosts whoch are declared using TLS:
apply Service "Check TLS certificate " for (http_vhost => config in host.vars.http_vhosts) {
import "generic-service"
check_command = "http"
vars.http_address = config.http_vhost
command_endpoint = " ... "
vars.http_certificate = 21
vars.http_sni = true
vars += config
assign where config.http_ssl == true
}
with an host which simply contains this kind of declaration:
vars.http_vhosts["Forum"] = {
http_vhost = "forum.securedrop.org"
http_ssl = true
}
Well, fortunately the description may add a list of some attended strings at some given uri, and any other parameter of your choice, provided that you write the code which will exploit it.
I made this kind of configuration for my last (manual) deployment of monitoring stuff. And it really a pleasure to write it and to maintain it. With a very concise configuration at host level, and a little bit of easy generic code, we were able to check for vhosts, dns zones consistency, dns views consistency, attended processes, attended vhosts, attended output IPs, git repos, mails queues, services banners (ssh, smtp, etc.), upgrades, running kernels, mailname consistency, volumes, databases, etc. See e.g. here for an overview of the readability of a fully monitored host object : https://admin.chapril.org/doku.php?id=admin:procedures:ajout-d-une-machine#ajout_de_l_objet_a_la_configuration
Next time, I will discuss the way I would like to write it in ansible paybooks.
