Creating Nagios Config with puppet

Let's have a look at the classic way of managing Nagios configuration with exported Puppet resources.
Here is a good article about the topic Show archive.org snapshot written by Eric Holzbach.

First we export the checks on the nodes we want to monitor:

Copy@@nagios_service { “check_unicorns-${::hostname}”:
  host_name           => $::hostname,
  service_description => ‘Unicorn Status’,
  check_command       => ‘check_unicorn-${::hostname}“,
}

Then we realize the resources on the Nagios server

CopyNagios_service <<| |>>

Now it is quite easy to create new monitoring checks with Puppet. With the ease of creating monitoring checks the number will grow fast. In our case we have got round about 17000 services on 400 hosts. With the large number the puppet run on the Nagios server will become quite slow. In the end it took 10 minutes to complete.

Make it fast

Since the resource realization of nagios_service seems not to scale well, we could use some code to write the service check configuration. Since we known that a larger number of files doesn't scale well we go for a single file. The file is rendered with an eRuby template.

First we pull the information from the PuppetDB. Restricting the information to parameters and title attributes, makes the query faster.

Copy  $service_checks = puppetdb_query(
    "resources [
      parameters,
      title
    ]{
      exported=true
      and type = 'Nagios_service'
      and parameters.ensure != 'absent'
      order by certname, title
    }"
  )

Then we join the queried services with our template

Copy  
  file { '/etc/naemon/conf.d/services.cfg':
    ensure  => stdlib::ensure(true, 'file'),
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template('naemon/node_service.erb'),
  }

Iterating the service_checks, then iterating and joining the parameters with their values generates the Nagios configuration. Sorting the parameters will ensure equal results across multiple puppetruns. Filtering out parameters specific to puppet is required for valid Nagios configuration.

Copy# HEADER: This file was autogenerated
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
<%
    @service_checks.each do |service|
-%>
# Naemon::Node::Service['<%= service["title"] %>']
define service {
<%   service["parameters"].sort.each do |parameter, value|
        # do not add puppet parameters to nagios config
        next if ['tag', 'ensure', 'require', 'target', 'mode'].include?(parameter) -%>
  <%=  parameter.ljust(35,' ') %><%= value %>
<%-   end -%>
}

<% end %>

Conclusion

Since there is only one file the overhead for file system operations are down to the minimum. The file is written as a whole, which saves us from the performance penalty of seeking strings in a large file. Nagios is happy to use this huge config file.

In the end the puppet run on the Nagios server takes 65 seconds.

References

• Eric Holzbach - Automating Nagios with Puppet and Puppetdb Show archive.org snapshot
• Ben Hartshorne - Puppet Perfomance Investigation Show archive.org snapshot
• Pieter Barrezeele - Puppet and Nagios Show archive.org snapshot