Distribute Sensu checks using git

Since my environment at home does not yet use any kind of automation for configuration and orchestration like the one that you can get using puppet, chef etc. I needed another way to distribute my “local” Sensu checks to all the machines.

So after thinking about the problem a bit, I decided to make use of my in-house GitLab server for hosting the Sensu checks configurations.

The first thing to do was to create the repository in GitLab, remember this repository either has to be public or have ssh keys configured. (We need to have password-less git pull)

Now that I had the repository ready I created a folder on all the machines /etc/sensu/conf.d/checks where the repository was checked out into.

One the main machine I created the following structure under /etc/sensu/conf.d/checks/:

available
- base
- appl
- centos
- proc
enabled
cron

The names under available can be anything, they just indicate the family where the checks belong.

Then the check files where created under e.g. available/base. Below is an example of a disk usage check script. Note: all the check scripts that exists under available is named.json.tmpl instead of just .json this because all nodes that later will receive the checks should not perform them.

{
  "checks": {
    "disk-usage": {
      "command": "check-disk-usage.rb -w :::disk.warning|70::: -c :::disk.critical|85::: :::disk.extra|:::",
      "subscribers": [
        "production"
      ],
      "interval": 600,
      "environment": ":::environment|production:::",
      "standalone": true
    }
  }
}

More information about checks can be found here Sensu checks

In order to make it easier on a node to list, enable, disable checks that have been retrieved from GitLab, three scripts where created:

#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: enable_check type check-name"
echo "   e.g. enable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if template is avaliable
if [ ! -f "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" ]
then
echo "Unable to find template ${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl"
exit 1
fi

# Check if check already is enabled
if [ -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Check ${2} is already enabled"
exit 1
fi

# Check if the directory for the selected type exists under enabled, otherwise create it
if [ ! -d "${SCRIPTPATH}/enabled/${1}" ]
then
mkdir "${SCRIPTPATH}/enabled/${1}"
chmod 755 "${SCRIPTPATH}/enabled/${1}"
fi

# Link the template (.json.tmpl) to a check file (.json) under enabled
ln -s "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now enabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: disable_check type check-name"
echo "   e.g. disable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if check is enabled
if [ ! -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Unable to find template ${SCRIPTPATH}/enable/${1}/check-${2}.json"
exit 1
fi

# Remove the link
rm "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now disabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash

# Define the types directories, i.e. the directories under available
CHECK_DIRS=(
  'base'
  'centos'
  'appl'
  'proc'
)

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Print header
echo " TYPE      | CHECK-NAME"
echo " ----------|-----------------------------"

for type in "${CHECK_DIRS[@]}"
do
    pushd "${SCRIPTPATH}/available/${type}/" > /dev/null
    shopt -s nullglob
    files=(*)
    popd > /dev/null
        for file in "${files[@]}"
        do
            enabled=""
            check=`echo "$file" | sed 's/check-\(.*\)\.json\.tmpl/\1/g'`
            if [ -d "${SCRIPTPATH}/enabled/${type}" ]
            then
                if [ -f "${SCRIPTPATH}/enabled/${type}/check-${check}.json" ]
                then
                    enabled="*"
                fi
            fi
            echo " $type   | ${check} ${enabled}"
        done
    echo ""
done

To use these we can run commands like:

$ ./check_list
$ ./check_enable base disk-usage
$ ./check_disable base disk-usage

Now all that is needed is a mechanism to retrieve new or updated checks on all machines.

To do that cron is used, so under scripts the script the fetches new and updated checks and a crontab configuration is created.

#!/usr/bin/env bash

# Quick-check before we allow bad things to happen
if [ -z "${BASH_VERSINFO}" ]; then
  echo "ERROR: You must execute this script with BASH"
  exit 255
fi

# Config
DIR="/etc/sensu/conf.d/checks"

usage() {
        echo "Usage: $(basename $0) [-hqS]"
        echo "    -h This help"
        echo "    -q Quiet mode. No stdout, only stderr and exit codes"
        echo "    -S Silent mode. No text output, only exit codes"
        echo
        exit 0
}

# Parse commandline
ALLARGS="$@"
set -- $(getopt hqS: -- "$@")
while true;
do
	case "$1" in
                (-h) usage;;
                (-q) QUIET=yes;;
                (-S) SILENT=yes;;
                (--) ;;
                (-*) echo "Error: unrecognized option $1" 1>&2; exit 1;;
                (*)  break;;
	esac
	shift
done

# send all stdout to /dev/null
if [ "${QUIET}" = "yes" ] || [ "${SILENT}" = "yes" ]; then
        exec 1> /dev/null
fi

# send all stdout and stderr to /dev/null
if [ "${SILENT}" = "yes" ]; then
        exec 2> /dev/null
fi

cd "${DIR}"

# Check if new version of the configuration is available, if not abort
git remote update
UPDATES=`git log HEAD..origin/master --oneline`
if [ "1" == "1${UPDATES}" ]
then
	echo "No updates available, aborting"
	exit 0
fi

# Git retrieve files
git fetch --all && git reset --hard origin/master
0 * * * * root /etc/sensu/conf.d/checks/scripts/pull_sensu_checks

Add a .gitignore file so that the files in enabled folder is not committed

/enabled/*
!enabled/.gitkeep
!enabled/base
!enabled/centos
!enabled/appl
/enabled/base/*
!enabled/base/.gitkeep
/enabled/centos/*
!enabled/centos/.gitkeep
/enabled/appl/*
!enabled/appl/.gitkeep

All the code and setup is done, commit and push…

So now we can enable the cron script so it executes every hour

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

On every other node clone the repository into /etc/sensu/conf.d/checks/ and enable the cron script:

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

All this work only enable the distribution and update of the checks, the actual enabling has to be done on each node (preferably using the scripts above)
 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s