CentOS 7: Display DHCP assigned IP addresses console prompt

In my network I depend on (statically) DHCP assigned IP-addresses for my servers and sometimes the DNS and DHCP might be somehow off-sync.

So in order to easily access a server that either got an invalid or unknown IP-address, I modified the standard console prompt to display the actual IP-addresses for that server.

This can easily be done by editing the /etc/issue file

\S
Kernel \r on an \m

Hostname: \n
Domain: \O
Users: \U
IP Addresses:
    Public: \4{eth0} Backbone: \4{eth1}

The switches above are replaced as follows:
* \S – OS/Distribution release
* \r – Kernel version
* \m – Architecture
* \n – The node name
* \O – The domain name
* \U – # Logged in users
* \4{ifc} – The IP-address of ifc

More switches can be found by issuing the command

$ man agetty

So whenever a server becomes “missing” I just need to open the virtualisation environment and look at the login prompt.

Advertisements

Distribute Sensu checks using git

Since my environment at home does not yet use any kind of automation for configuration and orchestration like the one that you can get using puppet, chef etc. I needed another way to distribute my “local” Sensu checks to all the machines.

So after thinking about the problem a bit, I decided to make use of my in-house GitLab server for hosting the Sensu checks configurations.

The first thing to do was to create the repository in GitLab, remember this repository either has to be public or have ssh keys configured. (We need to have password-less git pull)

Now that I had the repository ready I created a folder on all the machines /etc/sensu/conf.d/checks where the repository was checked out into.

One the main machine I created the following structure under /etc/sensu/conf.d/checks/:

available
- base
- appl
- centos
- proc
enabled
cron

The names under available can be anything, they just indicate the family where the checks belong.

Then the check files where created under e.g. available/base. Below is an example of a disk usage check script. Note: all the check scripts that exists under available is named.json.tmpl instead of just .json this because all nodes that later will receive the checks should not perform them.

{
  "checks": {
    "disk-usage": {
      "command": "check-disk-usage.rb -w :::disk.warning|70::: -c :::disk.critical|85::: :::disk.extra|:::",
      "subscribers": [
        "production"
      ],
      "interval": 600,
      "environment": ":::environment|production:::",
      "standalone": true
    }
  }
}

More information about checks can be found here Sensu checks

In order to make it easier on a node to list, enable, disable checks that have been retrieved from GitLab, three scripts where created:

#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: enable_check type check-name"
echo "   e.g. enable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if template is avaliable
if [ ! -f "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" ]
then
echo "Unable to find template ${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl"
exit 1
fi

# Check if check already is enabled
if [ -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Check ${2} is already enabled"
exit 1
fi

# Check if the directory for the selected type exists under enabled, otherwise create it
if [ ! -d "${SCRIPTPATH}/enabled/${1}" ]
then
mkdir "${SCRIPTPATH}/enabled/${1}"
chmod 755 "${SCRIPTPATH}/enabled/${1}"
fi

# Link the template (.json.tmpl) to a check file (.json) under enabled
ln -s "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now enabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: disable_check type check-name"
echo "   e.g. disable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if check is enabled
if [ ! -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Unable to find template ${SCRIPTPATH}/enable/${1}/check-${2}.json"
exit 1
fi

# Remove the link
rm "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now disabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash

# Define the types directories, i.e. the directories under available
CHECK_DIRS=(
  'base'
  'centos'
  'appl'
  'proc'
)

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Print header
echo " TYPE      | CHECK-NAME"
echo " ----------|-----------------------------"

for type in "${CHECK_DIRS[@]}"
do
    pushd "${SCRIPTPATH}/available/${type}/" > /dev/null
    shopt -s nullglob
    files=(*)
    popd > /dev/null
        for file in "${files[@]}"
        do
            enabled=""
            check=`echo "$file" | sed 's/check-\(.*\)\.json\.tmpl/\1/g'`
            if [ -d "${SCRIPTPATH}/enabled/${type}" ]
            then
                if [ -f "${SCRIPTPATH}/enabled/${type}/check-${check}.json" ]
                then
                    enabled="*"
                fi
            fi
            echo " $type   | ${check} ${enabled}"
        done
    echo ""
done

To use these we can run commands like:

$ ./check_list
$ ./check_enable base disk-usage
$ ./check_disable base disk-usage

Now all that is needed is a mechanism to retrieve new or updated checks on all machines.

To do that cron is used, so under scripts the script the fetches new and updated checks and a crontab configuration is created.

#!/usr/bin/env bash

# Quick-check before we allow bad things to happen
if [ -z "${BASH_VERSINFO}" ]; then
  echo "ERROR: You must execute this script with BASH"
  exit 255
fi

# Config
DIR="/etc/sensu/conf.d/checks"

usage() {
        echo "Usage: $(basename $0) [-hqS]"
        echo "    -h This help"
        echo "    -q Quiet mode. No stdout, only stderr and exit codes"
        echo "    -S Silent mode. No text output, only exit codes"
        echo
        exit 0
}

# Parse commandline
ALLARGS="$@"
set -- $(getopt hqS: -- "$@")
while true;
do
	case "$1" in
                (-h) usage;;
                (-q) QUIET=yes;;
                (-S) SILENT=yes;;
                (--) ;;
                (-*) echo "Error: unrecognized option $1" 1>&2; exit 1;;
                (*)  break;;
	esac
	shift
done

# send all stdout to /dev/null
if [ "${QUIET}" = "yes" ] || [ "${SILENT}" = "yes" ]; then
        exec 1> /dev/null
fi

# send all stdout and stderr to /dev/null
if [ "${SILENT}" = "yes" ]; then
        exec 2> /dev/null
fi

cd "${DIR}"

# Check if new version of the configuration is available, if not abort
git remote update
UPDATES=`git log HEAD..origin/master --oneline`
if [ "1" == "1${UPDATES}" ]
then
	echo "No updates available, aborting"
	exit 0
fi

# Git retrieve files
git fetch --all && git reset --hard origin/master
0 * * * * root /etc/sensu/conf.d/checks/scripts/pull_sensu_checks

Add a .gitignore file so that the files in enabled folder is not committed

/enabled/*
!enabled/.gitkeep
!enabled/base
!enabled/centos
!enabled/appl
/enabled/base/*
!enabled/base/.gitkeep
/enabled/centos/*
!enabled/centos/.gitkeep
/enabled/appl/*
!enabled/appl/.gitkeep

All the code and setup is done, commit and push…

So now we can enable the cron script so it executes every hour

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

On every other node clone the repository into /etc/sensu/conf.d/checks/ and enable the cron script:

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

All this work only enable the distribution and update of the checks, the actual enabling has to be done on each node (preferably using the scripts above)
 

Execute one command on all hosts

I have a couple of machines in my data center and sometimes it can be useful to run the same command on all the hosts.

For that purpose, I created a simple bash script that loops over a list of machine names and executes a command using ssh.

host1.example.com
host2.example.com
#!/usr/bin/env bash
KEY_SWITCH="-i ~/.ssh/custom_id_rsa"
USER="root"

function usage() {
        echo "Usage: exec_command.sh list \"command\""
        echo "     list      List type (linux, centos, debian)"
        echo "     command   Command to be executed on all hosts in the list"
}

if [ -z "$1" ]
then
        usage
        exit 1
fi
list_file="list_$1"

if [ ! -f "$list_file" ]
then
        echo "Error: Unable to find list file $list_file"
        echo
        usage
        exit 1
fi

echo "Using list file: $list_file"

if [ -z "$2" ]
then
        echo "Error: Command is not provided"
        echo
        usage
        exit 1
fi
command="$2"

IFS=$'\n' read -d '' -r -a hosts < $list_file
for host in "${hosts[@]}"
do
        echo "Executing command on host $host"
        echo
        ssh -q ${KEY_SWITCH} ${USER}@${host} "${command}"
        echo
        echo "Done executing on host $host"
done

As you can see the list is just a file named list_LISTNAME that contains one hostname per line.

I am using a custom key file for ssh to connect to my hosts, if that is not used change KEY_SWITCH value to "".

When the script is available and executable I can now run commands like:

$ ./execute_command.sh centos "yum update -y"
$ ./execute_command.sh debian "apt-get update && apt-get upgrade -y"

CentOS 7: Use a live VM as template

When the need arises to create a new VM I normally clone a dedicated base machine with the common tools and configurations needed to get a new machine up and running quickly.

So every time I encounter a new application that I need in multiple future machines I install that application on the dedicated base machine.

Since the base machine is live, it gets log files, history, passwords etc., which should not be carried over to the new machine.

For this purpose, the following script has been created. The script cleans up the most common things, then unconfigures the system

#!/usr/bin/env bash

# Configuration of network interface names
NETWORK_NAME_PUBLIC="public"
NETWORK_NAME_BACKBONE="backbone"

# Make sure the user really wants to run this
read -r -p "This script will clear all local data and prepare the machine for templating. Remember to create a snapshot before running this script. Are you sure you want to continue (y/N)? " response
echo
if [[ ! $response =~ [Yy]$ ]]
then
exit 1;
fi

# Reset the hostname
echo "* Change to generic hostname"
hostnamectl set-hostname localhost.localdomain

# Reset the machine id
echo "* Resetting machine-id"
> /etc/machine-id

# Clean up temporary files created in the live base machine
echo "* Removing roots temporary files"
rm -f /etc/ssh/ssh_host_*
rm -f /root/.ssh/known_hosts*
rm -f /root/.ssh/id_*
rm -f /root/.ssh/config
rm -f /root/anaconda-ks.cfg
rm -f /root/.bash_history
rm -rf /root/.config
rm -rf /root/.cache
rm -f /root/.viminfo
unset HISTFILE

# Remove mac-addresses
echo "* Removing MAC addresses from network configurations"
sed -i '/.*HWADDR=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
sed -i '/.*UUID=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
sed -i '/.*DEVICE=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
echo "DEVICE=eth0" >> /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}

sed -i '/.*HWADDR=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
sed -i '/.*UUID=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
sed -i '/.*DEVICE=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
echo "DEVICE=eth1" >> /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}

# Clear logs
echo "* Clear logs"
rm -f /var/log/boot.log
rm -f /var/log/btmp
rm -f /var/log/cron
rm -f /var/log/dmesg
rm -f /var/log/dmesg.old
rm -f /var/log/grubby
rm -f /var/log/lastlog
rm -f /var/log/maillog
rm -f /var/log/messages
rm -f /var/log/secure
rm -f /var/log/spooler
rm -f /var/log/tallylog
rm -f /var/log/wpa_supplicant.log
rm -f /var/log/wtmp
rm -f /var/log/yum.log
rm -f /var/log/audit/audit.log
rm -f /var/log/qemu-ga/*
rm -f /var/log/tuned/tuned.log

# Force a disk check on next boot
echo "* Enable fsck on next boot"
touch /forcefsck

# Run sys-unconfig
read -r -p "System is prepared, run sys-unconfig (y/N)? " response
echo
if [[ ! $response =~ [Yy]$ ]]
then
exit 1;
fi

sys-unconfig

Use this script on a cloned machine, not the live dedicated base machine since it will remove all history and reset some configurations.

Note: My machines have 2 network interfaces and their names are configured at the top in NETWORK_NAME_PUBLIC, NETWORK_NAME_BACKBONE

Smarter bash history

When using bash the up/down arrows are useful to browse through the recent commands.

But since the wanted command may be a bit up in the history stack it can be a bit tedious to find the right command.

For example, I know that I had a long nice cat + grep + awk line earlier it might require many up-arrow presses or a history | grep cat…

Instead I added small script file to /etc/profile.d/complete_history.sh

if [ ! -z "$PS1" ]; then
bind '"\e[A": history-search-backward'
bind '"\e[B": history-search-forward'
fi

You probably need to make it executable…

Now after starting a new bash instance I can write:

cat [up-arrow]

This will only give me the history entries starting with cat …

If you started the line with the wrong keyword use CTRL+C to break out…

CentOS 7 Automated reboot after kernel update

When using applications like yum-cron to update the installed packages automatically there will be times when the kernel has been updated.

Instead of checking every machine if a reboot is needed I use the following cron script to automate the reboot.

The following cron script is place in /etc/cron.daily/reboot_on_kernel_update

#!/usr/bin/env bash

EMAIL_RECIPIENT="test@example.com"
LATEST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

if [ "$LATEST_KERNEL" != "$CURRENT_KERNEL" ]
then
    MAIL_BODY="The kernel has been updated from $CURRENT_KERNEL to $LATEST_KERNEL on $HOSTNAME. Because of this a reboot has been issued. // Kernel update monitor"
    mail -s "$HOSTNAME: Kernel update monitor" $EMAIL_RECIPIENT <<< $MAIL_BODY
    reboot
fi

Remember to make the script executable…