Execute one command on all hosts

I have a couple of machines in my data center and sometimes it can be useful to run the same command on all the hosts.

For that purpose, I created a simple bash script that loops over a list of machine names and executes a command using ssh.

host1.example.com
host2.example.com
#!/usr/bin/env bash
KEY_SWITCH="-i ~/.ssh/custom_id_rsa"
USER="root"

function usage() {
        echo "Usage: exec_command.sh list \"command\""
        echo "     list      List type (linux, centos, debian)"
        echo "     command   Command to be executed on all hosts in the list"
}

if [ -z "$1" ]
then
        usage
        exit 1
fi
list_file="list_$1"

if [ ! -f "$list_file" ]
then
        echo "Error: Unable to find list file $list_file"
        echo
        usage
        exit 1
fi

echo "Using list file: $list_file"

if [ -z "$2" ]
then
        echo "Error: Command is not provided"
        echo
        usage
        exit 1
fi
command="$2"

IFS=$'\n' read -d '' -r -a hosts < $list_file
for host in "${hosts[@]}"
do
        echo "Executing command on host $host"
        echo
        ssh -q ${KEY_SWITCH} ${USER}@${host} "${command}"
        echo
        echo "Done executing on host $host"
done

As you can see the list is just a file named list_LISTNAME that contains one hostname per line.

I am using a custom key file for ssh to connect to my hosts, if that is not used change KEY_SWITCH value to "".

When the script is available and executable I can now run commands like:

$ ./execute_command.sh centos "yum update -y"
$ ./execute_command.sh debian "apt-get update && apt-get upgrade -y"

CentOS 7: Use a live VM as template

When the need arises to create a new VM I normally clone a dedicated base machine with the common tools and configurations needed to get a new machine up and running quickly.

So every time I encounter a new application that I need in multiple future machines I install that application on the dedicated base machine.

Since the base machine is live, it gets log files, history, passwords etc., which should not be carried over to the new machine.

For this purpose, the following script has been created. The script cleans up the most common things, then unconfigures the system

#!/usr/bin/env bash

# Configuration of network interface names
NETWORK_NAME_PUBLIC="public"
NETWORK_NAME_BACKBONE="backbone"

# Make sure the user really wants to run this
read -r -p "This script will clear all local data and prepare the machine for templating. Remember to create a snapshot before running this script. Are you sure you want to continue (y/N)? " response
echo
if [[ ! $response =~ [Yy]$ ]]
then
exit 1;
fi

# Reset the hostname
echo "* Change to generic hostname"
hostnamectl set-hostname localhost.localdomain

# Reset the machine id
echo "* Resetting machine-id"
> /etc/machine-id

# Clean up temporary files created in the live base machine
echo "* Removing roots temporary files"
rm -f /etc/ssh/ssh_host_*
rm -f /root/.ssh/known_hosts*
rm -f /root/.ssh/id_*
rm -f /root/.ssh/config
rm -f /root/anaconda-ks.cfg
rm -f /root/.bash_history
rm -rf /root/.config
rm -rf /root/.cache
rm -f /root/.viminfo
unset HISTFILE

# Remove mac-addresses
echo "* Removing MAC addresses from network configurations"
sed -i '/.*HWADDR=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
sed -i '/.*UUID=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
sed -i '/.*DEVICE=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
echo "DEVICE=eth0" >> /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}

sed -i '/.*HWADDR=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
sed -i '/.*UUID=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
sed -i '/.*DEVICE=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
echo "DEVICE=eth1" >> /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}

# Clear logs
echo "* Clear logs"
rm -f /var/log/boot.log
rm -f /var/log/btmp
rm -f /var/log/cron
rm -f /var/log/dmesg
rm -f /var/log/dmesg.old
rm -f /var/log/grubby
rm -f /var/log/lastlog
rm -f /var/log/maillog
rm -f /var/log/messages
rm -f /var/log/secure
rm -f /var/log/spooler
rm -f /var/log/tallylog
rm -f /var/log/wpa_supplicant.log
rm -f /var/log/wtmp
rm -f /var/log/yum.log
rm -f /var/log/audit/audit.log
rm -f /var/log/qemu-ga/*
rm -f /var/log/tuned/tuned.log

# Force a disk check on next boot
echo "* Enable fsck on next boot"
touch /forcefsck

# Run sys-unconfig
read -r -p "System is prepared, run sys-unconfig (y/N)? " response
echo
if [[ ! $response =~ [Yy]$ ]]
then
exit 1;
fi

sys-unconfig

Use this script on a cloned machine, not the live dedicated base machine since it will remove all history and reset some configurations.

Note: My machines have 2 network interfaces and their names are configured at the top in NETWORK_NAME_PUBLIC, NETWORK_NAME_BACKBONE

Smarter bash history

When using bash the up/down arrows are useful to browse through the recent commands.

But since the wanted command may be a bit up in the history stack it can be a bit tedious to find the right command.

For example, I know that I had a long nice cat + grep + awk line earlier it might require many up-arrow presses or a history | grep cat…

Instead I added small script file to /etc/profile.d/complete_history.sh

if [ ! -z "$PS1" ]; then
bind '"\e[A": history-search-backward'
bind '"\e[B": history-search-forward'
fi

You probably need to make it executable…

Now after starting a new bash instance I can write:

cat [up-arrow]

This will only give me the history entries starting with cat …

If you started the line with the wrong keyword use CTRL+C to break out…

CentOS 7 Automated reboot after kernel update

When using applications like yum-cron to update the installed packages automatically there will be times when the kernel has been updated.

Instead of checking every machine if a reboot is needed I use the following cron script to automate the reboot.

The following cron script is place in /etc/cron.daily/reboot_on_kernel_update

#!/usr/bin/env bash

EMAIL_RECIPIENT="test@example.com"
LATEST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

if [ "$LATEST_KERNEL" != "$CURRENT_KERNEL" ]
then
    MAIL_BODY="The kernel has been updated from $CURRENT_KERNEL to $LATEST_KERNEL on $HOSTNAME. Because of this a reboot has been issued. // Kernel update monitor"
    mail -s "$HOSTNAME: Kernel update monitor" $EMAIL_RECIPIENT <<< $MAIL_BODY
    reboot
fi

Remember to make the script executable…