Sensu - Host Masquerading


Note: This is now included in Sensu core in the form of JIT clients

A key part of monitoring infrastructure involves having the ability to monitor things that we can’t necessarily install a monitoring client on: switches and other network devices, external services and websites, etc..

In Nagios it’s pretty common to group active checks under virtual hosts that don’t really exist to create logical sets of checks. Sensu doesn’t yet have this ability.

There has been some discussion about the possibility of adding a masquerade feature and changing event data to drop the client info requirement in order to be able to craft event data with a custom source address. In the latter issue Kyle Anderson proposes a solution which was at one point implemented but then later reverted.

I applied Kyles patch to my Sensu server.rb and configured a set of checks with the :source attribute. My check data then contained a modified source and my handlers sent messages with the modified event data. Great! Unfortunately though, the new event data wasn’t accessible through the API. I emailed Kyle for advice and he kindly created this issue.

In order for clients to be visible in the Uchiwa frontend we need to fix sensu-api. After looking at the API code and trying a few things I eventually decided to try simply duplicating the original client data in Redis.

Duplicating the client data works well since it will get updated each time an event is processed. Each event includes a timestamp property that sensu-server uses to calculate the keepalive for each server. What this means is that our masqueraded host behaves exactly like a real host and all functionality in the sensu-api (and as a result, the Uchiwa frontend) behaves as expected.