Proxmox resize linux guest partition

I sometimes come across that my partitions are too small in a VM Guest. And I can never remember the exact steps involved when resizing them. This post explains the steps needed to resize the root partition (can be any partition) of a VM.

The machines I am working with are CentOS 7 machines and they are using LVM.

I am going to be using fdisk to resize the disk instead of gparted or similar.

This is what we are starting with (df -h)

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  7.5G  2.2G  5.3G  30% /

Since I don’t need 100% uptime I can shutdown the VM to be resized (can probably be done live see https://pve.proxmox.com/wiki/Resize_disks)

Resize the disk in proxmox:

  1. Locate the VM to resize in the Proxmox web interface
  2. Select Hardware
  3. Find the Hard Disk to resize in the list of hardware
  4. Select Resize disk and enter the increment of disk size (in this example I selected 1 GB)

Note: Any snapshots on the machine needs to be removed before resize is allowed
Note 2: If the disk image is located on a nfs share, it will probably not work

Now we can boot up the machine again…

If we try the above df -h we can see that nothing has changed, this is because the partitions are the same, lets change that…

Using fdisk -l I get a list of all disks in the machine, here I can see that /dev/vda contains the LVM partition (/dev/vda2) that I want to resize

Disk /dev/vda: 11.8 GB, 11811160064 bytes, 23068672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b205b

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    20930559     9952256   8e  Linux LVM

So lets edit the partition map (here is the part that I find scary since we are going to delete the partition…)

# fdisk /dev/vda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p

Disk /dev/vda: 11.8 GB, 11811160064 bytes, 23068672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b205b

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    20930559     9952256   8e  Linux LVM

Command (m for help): d
Partition number (1,2, default 2): 2
Partition 2 is deleted

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p):
Using default response p
Partition number (2-4, default 2):
First sector (1026048-23068671, default 1026048):
Using default value 1026048
Last sector, +sectors or +size{K,M,G} (1026048-23068671, default 23068671):
Using default value 23068671
Partition 2 of type Linux and of size 10.5 GiB is set

Command (m for help): t
Partition number (1,2, default 2):
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): p

Disk /dev/vda: 11.8 GB, 11811160064 bytes, 23068672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b205b

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    23068671    11021312   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

So what did we do:

  1. Entering the command fdisk /dev/vda on the command line allows us to edit /dev/vda
  2. Entering the p command lists the current partition table
  3. Entering d and then 2 allows us to delete the partition /dev/vda2
  4. Now we can create the new partition by entering n and using default values by pressing ENTER
  5. Then we change the type of the partition from Linux to Linux LVM by entering command t and the code 8e
  6. After that we print the new partition table using p
  7. And finally we write the partition table to disk using w

Now in order for the system to notice the change there are probably several ways to do this, but the easiest is to reboot. So let’s do that now…

Next it is time to resize the lvm, for reference, this is how it looks before.

Physical volume

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vda2
  VG Name               centos
  PV Size               9.49 GiB / not usable 3.00 MiB
  ...
  Free PE               1
  Allocated PE          2428

Volume Group

# vgdisplay
  --- Volume group ---
  VG Name               centos
  ...
  VG Size               9.49 GiB
  ...
  Total PE              2429
  Alloc PE / Size       2428 / 9.48 GiB
  Free  PE / Size       1 / 4.00 MiB

Logical volume

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  ...
  LV Size                7.49 GiB

  --- Logical volume ---
  LV Path                /dev/centos/swap
  ...

So start by resizing the Physical volume:

# pvresize /dev/vda2
  Physical volume "/dev/vda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Running a pvdisplay show that the physical volume has changed from

PV Size               9.49 GiB / not usable 3.00 MiB
...
Free PE               1

To

PV Size               10.51 GiB / not usable 2.00 MiB
...
Free PE               262

We can also see that the volume group (vgdisplay) now has increased FreePE.

Let’s extend the Logical volume

# lvresize --extents +100%FREE --resizefs /dev/centos/root
  Size of logical volume centos/root changed from 7.49 GiB (1917 extents) to 8.49 GiB (2173 extents).
  Logical volume centos/root successfully resized.

Here I entered the path to the logical volume /dev/centos/root which can be found in the output of the lvdisplay command on LV Path.

Now the resize is complete and the new output from the previous commands are

Physical volume

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vda2
  VG Name               centos
  PV Size               10.51 GiB / not usable 2.00 MiB
  ...
  Free PE               0
  Allocated PE          2690

Volume Group

# vgdisplay
  --- Volume group ---
  VG Name               centos
  ...
  VG Size               10.51 GiB
  ...
  Total PE              2690
  Alloc PE / Size       2690 / 10.51 GiB
  Free  PE / Size       0 / 0   

Logical volume

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  ...
  LV Size                8.51 GiB

  --- Logical volume ---
  LV Path                /dev/centos/swap
  ...

Disk space usage

# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  8.6G  2.2G  6.4G  26% /
Advertisements

Proxmox scheduled snapshot creation

This post is not finished, more text will be provided soon.

The following script creates a snapshot of all running VMs on a Proxmox node, it might be suitable for scheduling by cron

#!/usr/bin/env bash
#
# Date: 2017-05-31
# Version: 1.0
# Author: Stellan Nordenbro <stellan.nordenbro@gmail.com>
#
# The MIT License (MIT)
#
# Copyright (c) 2017 Stellan Nordenbro
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#

# Quick-check before we allow bad things to happen
if [ -z "${BASH_VERSINFO}" ]; then
  echo "ERROR: You must execute this script with BASH"
  exit 255
fi

# Go to the correct folder
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $DIR

# The usage text
usage() { 
  echo "Usage: `basename $0` [-hdqS]" 1>&2
  echo
  echo "Optional parameters:" 1>&2
  echo " -h             Displays this message" 1>&2
  echo " -d             Dry-run: no commands will be executed" 1>&2
  echo " -q             Quiet: output to stdout will be hidden" 1>&2
  echo " -S             Silent: output to both stdout and stderr will be hidden" 1>&2
  echo
  exit 1; 
}

# Parse commandline
ALLARGS="$@"
set -- $(getopt hdqS -- "$@")
while true;
do
    case "$1" in
                (-h) usage;;
		(-d) DRY_RUN=yes;;
                (-q) QUIET=yes;;
                (-S) SILENT=yes;;
                (--) ;;
                (-*) echo "Error: unrecognized option $1" 1>&2; exit 1;;
                (*)  break;;
    esac
    shift
done
 
# send all stdout to /dev/null
if [ "${QUIET}" = "yes" ] || [ "${SILENT}" = "yes" ]; then
        exec 1> /dev/null
fi
 
# send all stdout and stderr to /dev/null
if [ "${SILENT}" = "yes" ]; then
        exec 2> /dev/null
fi

################################################################################
# FUNCTIONS
################################################################################
function GetVMName() {
	if [ $# -ne 1 ]; then
		echo "ERROR: ${FUNCNAME} expected 1 parameters: Usage ${FUNCNAME} VM_ID"
		exit 1
	fi
	
	value=$(qm config $1 | grep "name" | awk '{ print $2}')
	echo "$value"
}

function CreateVMSnapshot() {
	if [ $# -ne 1 ]; then
		echo "ERROR: ${FUNCNAME} expected 1 parameters: Usage ${FUNCNAME} VM_ID"
		exit 1
	fi
	
	VM_ID="$1"
	VM_NAME=$(GetVMName ${VM_ID})
	SNAPSHOT_NAME="AS_$(date +"%Y%m%d_%H%M")"
	SNAPSHOT_DESCRIPTION="Automated snapshot by script"
	
	echo "* Creating snapshot ${SNAPSHOT_NAME} on virtual machine ${VM_NAME} with id ${VM_ID}..."
	if [ "${DRY_RUN}" = "yes" ]; then
        	echo "   DRY-RUN:   qm snapshot ${VM_ID} \"${SNAPSHOT_NAME}\" -description \"${SNAPSHOT_DESCRIPTION}\" -vmstate 0"
	else
        	qm snapshot ${VM_ID} "${SNAPSHOT_NAME}" -description "${SNAPSHOT_DESCRIPTION}" -vmstate 0
        	snapshot_count=$(qm listsnapshot ${VM_ID} | awk '{print $1}' | grep "${SNAPSHOT_NAME}" | wc -l)
        	if [ $snapshot_count != 1 ]; then
        	        echo "ERROR: Snaphot creation failed on virtual machine ${VM_NAME} with id ${VM_ID}"
        	        exit 1
        	fi
	fi
	echo -e "* Done creating snapshot ${SNAPSHOT_NAME} on virtual machine ${VM_NAME} with id ${VM_ID}.\n\n"
}

################################################################################
# CONFIGURATIOM
################################################################################
HOSTNAME=$(hostname)

echo "Creating scheduled snapshots of virtual machines on $HOSTNAME"
echo -e "-------------------------------------------------------\n\n"

################################################################################
# SNAPSHOT
################################################################################

echo "* Retrieving running VMs"
RUNNING_VMS=($(qm list | grep -v VMID | grep "running" | awk '{print $1}'))
echo "* Done."

echo "* Creating snapshot for running VMs"
for VM_ID in "${RUNNING_VMS[@]}"; do
	CreateVMSnapshot "$VM_ID"
done
echo "* Done creating VM snapshots"

echo "Script execution done."

If anyone else have use for this script then go ahead, no guarantees of course, use it on your own risk…

Proxmox automated conversion to template

This post is not finished, more text will be provided soon.

The following script takes a virtual machine with the purpose to act as a base machine for new machines and creates a Proxmox template from it by:

  • Creating a snapshot
  • Deleting prevously created template if it exists
  • Cloning the machine
  • Connecting to the machine using temporary ssh key
  • Using previously published oem-config-prepare script on the cloned machine (with the added removal of the temporary ssh key)
  • Converting the cloned virtual machine to a template

Most configurable options are located in the configurations section

#!/usr/bin/env bash
#
# Date: 2017-05-31
# Version: 1.0
# Author: Stellan Nordenbro <stellan.nordenbro@gmail.com>
#
# The MIT License (MIT)
#
# Copyright (c) 2017 Stellan Nordenbro
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#

# Quick-check before we allow bad things to happen
if [ -z "${BASH_VERSINFO}" ]; then
  echo "ERROR: You must execute this script with BASH"
  exit 255
fi

# Go to the correct folder
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $DIR

# The usage text
usage() {
  echo "Usage: `basename $0` [-hdqS]" 1>&2
  echo
  echo "Optional parameters:" 1>&2
  echo " -h             Displays this message" 1>&2
  echo " -d             Dry-run: no commands will be executed" 1>&2
  echo " -q             Quiet: output to stdout will be hidden" 1>&2
  echo " -S             Silent: output to both stdout and stderr will be hidden" 1>&2
  echo
  exit 1;
}

# Parse commandline
ALLARGS="$@"
set -- $(getopt hdqS -- "$@")
while true;
do
    case "$1" in
                (-h) usage;;
                (-d) DRY_RUN=yes;;
                (-q) QUIET=yes;;
                (-S) SILENT=yes;;
                (--) ;;
                (-*) echo "Error: unrecognized option $1" 1>&2; exit 1;;
                (*)  break;;
    esac
    shift
done

# send all stdout to /dev/null
if [ "${QUIET}" = "yes" ] || [ "${SILENT}" = "yes" ]; then
        exec 1> /dev/null
fi

# send all stdout and stderr to /dev/null
if [ "${SILENT}" = "yes" ]; then
        exec 2> /dev/null
fi

##############################################
# FUNCTIONS
#############################################
function GetVMOption() {
	if [ $# -ne 2 ]; then
                echo "ERROR: ${FUNCNAME} expected 2 parameters: Usage ${FUNCNAME} VM_ID OPTION"
                exit 1
	fi

	value=$(qm config $1 | grep $2 | awk '{ print $2}')
	echo "$value"
}

function SetVMOption() {
	if [ $# -ne 3 ]; then
		echo "ERROR: ${FUNCNAME} expected 2 parameters: Usage ${FUNCNAME} VM_ID OPTION VALUE"
		exit 1
	fi

	if [ "${DRY_RUN}" = "yes" ]; then
		echo "   DRY-RUN:   qm set $1 -${2} $3"
	else
		qm set $1 -${2} $3
	fi
}

#############################################
# CONFIGURATION
#############################################
HOSTNAME=$(hostname)
VM_ID=10000
VM_NAME=$(GetVMOption ${VM_ID} name)
SNAPSHOT_NAME="A$(date +"%Y%m%d_%H%M")"
SNAPSHOT_DESCRIPTION="Automated snapshot by template creation script"
TEMPLATE_ID=10001
TEMPLATE_NAME="template-${VM_NAME}"
TEMPLATE_POOL="Templates"
TEMPLATE_STORAGE="shared-templates"
TEMPLATE_DESCRIPTION="Template created $(date +"%Y-%m-%d %T")"
SSH_KEY="~/.ssh/temp_id_rsa"

echo "Creating template of base virtual machines on $HOSTNAME"
echo -e "-------------------------------------------------------nn"

#############################################
# SNAPSHOT
#############################################
echo "* Creating snapshot ${SNAPSHOT_NAME} on virtual machine ${VM_NAME} with id ${VM_ID}..."
if [ "${DRY_RUN}" = "yes" ]; then
	echo "   DRY-RUN:   qm snapshot ${VM_ID} "${SNAPSHOT_NAME}" -description "${SNAPSHOT_DESCRIPTION}" -vmstate 0"
else
	qm snapshot ${VM_ID} "${SNAPSHOT_NAME}" -description "${SNAPSHOT_DESCRIPTION}" -vmstate 0
	snapshot_count=$(qm listsnapshot ${VM_ID} | awk '{print $1}' | grep "${SNAPSHOT_NAME}" | wc -l)
	if [ $snapshot_count != 1 ]; then
		echo "ERROR: Snaphot creation failed on virtual machine ${VM_NAME} with id ${VM_ID}"
		exit 1
	fi
fi
echo -e "* Done creating snapshot ${SNAPSHOT_NAME}.nn"

#############################################
# EXISTING TEMPLATE CHECK
#############################################
echo "* Checking for existing template vms..."
if [ "${DRY_RUN}" = "yes" ]; then
	echo "   DRY-RUN:   qm list | grep -v "VMID" | grep "${TEMPLATE_NAME}" | awk '{print $1}' | grep ${TEMPLATE_ID} | wc -l"
fi
echo -e "* Done checking for existing tempalte vms.nn"

#############################################
# EXISTING TEMPLATE DELETE
#############################################
vm_count=$(qm list | grep -v "VMID" | grep "${TEMPLATE_NAME}" | awk '{print $1}' | grep ${TEMPLATE_ID} | wc -l)
vm_id_count=$(qm list | grep -v "VMID" | awk '{print $1}' | grep ${TEMPLATE_ID} | wc -l)
if [ $vm_id_count != $vm_count ]; then
	echo "ERROR: There is a virtual machine with id ${TEMPLATE_ID} but probably not then name ${TEMPLATE_NAME}, aborting." | tee /dev/stderr
	exit 1
fi

if [ $vm_count = 1 ]; then
	echo "* Deleting old template ${TEMPLATE_NAME} with id ${TEMPLATE_ID} after clearing protection"
	if [ "${DRY_RUN}" = "yes" ]; then
		SetVMOption ${TEMPLATE_ID} protection 0
		echo "   DRY-RUN:   qm destroy ${TEMPLATE_ID}"
	else
		SetVMOption ${TEMPLATE_ID} protection 0
		qm destroy ${TEMPLATE_ID}
		if [ $(qm list | grep -v "VMID" | awk '{print $1}' | grep ${TEMPLATE_ID} | wc -l) = 1 ]; then
			echo "ERROR: Unable to delete VM ID: ${TEMPLATE_ID}, aborting." | tee /dev/stderr
			exit 1
		fi
	fi
	echo -e "* Done deleting existing template.nn"
fi

#############################################
# CLONE VM TO TEMPLATE
#############################################
echo "* Creating template clone ${TEMPLATE_NAME} from snapshot ${SNAPSHOT_NAME}"
if [ "${DRY_RUN}" = "yes" ]; then
	echo "   DRY-RUN:   qm clone ${VM_ID} ${TEMPLATE_ID} -description "${TEMPLATE_DESCRIPTION}" -format qcow2 -full 1 -name "${TEMPLATE_NAME}" -pool "${TEMPLATE_POOL}" -snapname "${SNAPSHOT_NAME}" -storage "${TEMPLATE_STORAGE}""
else
	qm clone ${VM_ID} ${TEMPLATE_ID} -description "${TEMPLATE_DESCRIPTION}" -format qcow2 -full 1 -name "${TEMPLATE_NAME}" -pool "${TEMPLATE_POOL}" -snapname "${SNAPSHOT_NAME}" -storage "${TEMPLATE_STORAGE}"
	sleep 10
	if [ $(qm list | grep -v "VMID" | grep "${TEMPLATE_NAME}" | awk '{print $1}' | grep ${TEMPLATE_ID} | wc -l) != 1 ]; then
		echo "ERROR: Unable to clone VM ID: ${VM_ID} to TEMPLATE ID: ${TEMPLATE_ID}, aborting." | tee /dev/stderr
		exit 1
	fi
fi
echo -e "* Done creating template clone ${TEMPLATE_NAME}.nn"

#############################################
# TEMPLATE PREPARATION
#############################################
echo "* Preparing template ${TEMPLATE_NAME}"
if [ "${DRY_RUN}" = "yes" ]; then
	echo "   DRY-RUN:   qm start ${TEMPLATE_ID}"
	echo "   DRY_RUN:   sleep 30"
	echo "   DRY-RUN:   ip=$(qm agent ${TEMPLATE_ID} network-get-interfaces | grep ip-address | grep -o "192.168.[0-9]*.[0-9]*" | head -n 1)"
	echo "   DRY_RUN:   ssh -4 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -i ~/.ssh/base_id_rsa root@${ip} "/sbin/oem-config-prepare -f""
else
	qm start ${TEMPLATE_ID}
	sleep 30
	ip=$(qm agent ${TEMPLATE_ID} network-get-interfaces | grep ip-address | grep -o "192.168.[0-9]*.[0-9]*" | head -n 1)
	count=0
	while [[ -z $ip ]] && [[ $count -lt 600 ]]
	do
		count=$((count+1))
		echo -n "."
		sleep 1
		ip=$(qm agent ${TEMPLATE_ID} network-get-interfaces | grep ip-address | grep -o "192.168.[0-9]*.[0-9]*" | head -n 1)
	done

	if [ -z $ip ]; then
		echo "ERROR: The machine does not seem to have an IP, maybe boot failed, aborting." | tee /dev/stderr
		exit 1
	fi

	ssh -4 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -i ${SSH_KEY} root@${ip} "/sbin/oem-config-prepare -f"
fi
echo -e "* Done preparing template ${TEMPLATE_NAME}.nn"

#############################################
# CONVERT TEMPLATE
#############################################
echo "* Converting virtual machine ${TEMPLATE_NAME} to template"
if [ "${DRY_RUN}" = "yes" ]; then
	echo "   DRY-RUN:   qm wait ${TEMPLATE_ID}"
	echo "   DRY-RUN:   qm template ${TEMPLATE_ID}"
else
	qm wait ${TEMPLATE_ID}
	qm template ${TEMPLATE_ID}
fi
echo -e "* Done converting virtual machine ${TEMPLATE_NAME} to template.nn"

echo "Script execution done."

If anyone else have use for this script then go ahead, no guarantees of course, use it on your own risk…

CentOS 7 Disable IPv6

To disable IPv6 on CentOS 7 run the following commands:

The following will disable IPv6 on a running system

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

To permanently disable IPv6 run

echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf

CentOS 7: Display DHCP assigned IP addresses console prompt

In my network I depend on (statically) DHCP assigned IP-addresses for my servers and sometimes the DNS and DHCP might be somehow off-sync.

So in order to easily access a server that either got an invalid or unknown IP-address, I modified the standard console prompt to display the actual IP-addresses for that server.

This can easily be done by editing the /etc/issue file

\S
Kernel \r on an \m

Hostname: \n
Domain: \O
Users: \U
IP Addresses:
    Public: \4{eth0} Backbone: \4{eth1}

The switches above are replaced as follows:
* \S – OS/Distribution release
* \r – Kernel version
* \m – Architecture
* \n – The node name
* \O – The domain name
* \U – # Logged in users
* \4{ifc} – The IP-address of ifc

More switches can be found by issuing the command

$ man agetty

So whenever a server becomes “missing” I just need to open the virtualisation environment and look at the login prompt.

Distribute Sensu checks using git

Since my environment at home does not yet use any kind of automation for configuration and orchestration like the one that you can get using puppet, chef etc. I needed another way to distribute my “local” Sensu checks to all the machines.

So after thinking about the problem a bit, I decided to make use of my in-house GitLab server for hosting the Sensu checks configurations.

The first thing to do was to create the repository in GitLab, remember this repository either has to be public or have ssh keys configured. (We need to have password-less git pull)

Now that I had the repository ready I created a folder on all the machines /etc/sensu/conf.d/checks where the repository was checked out into.

One the main machine I created the following structure under /etc/sensu/conf.d/checks/:

available
- base
- appl
- centos
- proc
enabled
cron

The names under available can be anything, they just indicate the family where the checks belong.

Then the check files where created under e.g. available/base. Below is an example of a disk usage check script. Note: all the check scripts that exists under available is named.json.tmpl instead of just .json this because all nodes that later will receive the checks should not perform them.

{
  "checks": {
    "disk-usage": {
      "command": "check-disk-usage.rb -w :::disk.warning|70::: -c :::disk.critical|85::: :::disk.extra|:::",
      "subscribers": [
        "production"
      ],
      "interval": 600,
      "environment": ":::environment|production:::",
      "standalone": true
    }
  }
}

More information about checks can be found here Sensu checks

In order to make it easier on a node to list, enable, disable checks that have been retrieved from GitLab, three scripts where created:

#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: enable_check type check-name"
echo "   e.g. enable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if template is avaliable
if [ ! -f "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" ]
then
echo "Unable to find template ${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl"
exit 1
fi

# Check if check already is enabled
if [ -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Check ${2} is already enabled"
exit 1
fi

# Check if the directory for the selected type exists under enabled, otherwise create it
if [ ! -d "${SCRIPTPATH}/enabled/${1}" ]
then
mkdir "${SCRIPTPATH}/enabled/${1}"
chmod 755 "${SCRIPTPATH}/enabled/${1}"
fi

# Link the template (.json.tmpl) to a check file (.json) under enabled
ln -s "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now enabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: disable_check type check-name"
echo "   e.g. disable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if check is enabled
if [ ! -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Unable to find template ${SCRIPTPATH}/enable/${1}/check-${2}.json"
exit 1
fi

# Remove the link
rm "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now disabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash

# Define the types directories, i.e. the directories under available
CHECK_DIRS=(
  'base'
  'centos'
  'appl'
  'proc'
)

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Print header
echo " TYPE      | CHECK-NAME"
echo " ----------|-----------------------------"

for type in "${CHECK_DIRS[@]}"
do
    pushd "${SCRIPTPATH}/available/${type}/" > /dev/null
    shopt -s nullglob
    files=(*)
    popd > /dev/null
        for file in "${files[@]}"
        do
            enabled=""
            check=`echo "$file" | sed 's/check-\(.*\)\.json\.tmpl/\1/g'`
            if [ -d "${SCRIPTPATH}/enabled/${type}" ]
            then
                if [ -f "${SCRIPTPATH}/enabled/${type}/check-${check}.json" ]
                then
                    enabled="*"
                fi
            fi
            echo " $type   | ${check} ${enabled}"
        done
    echo ""
done

To use these we can run commands like:

$ ./check_list
$ ./check_enable base disk-usage
$ ./check_disable base disk-usage

Now all that is needed is a mechanism to retrieve new or updated checks on all machines.

To do that cron is used, so under scripts the script the fetches new and updated checks and a crontab configuration is created.

#!/usr/bin/env bash

# Quick-check before we allow bad things to happen
if [ -z "${BASH_VERSINFO}" ]; then
  echo "ERROR: You must execute this script with BASH"
  exit 255
fi

# Config
DIR="/etc/sensu/conf.d/checks"

usage() {
        echo "Usage: $(basename $0) [-hqS]"
        echo "    -h This help"
        echo "    -q Quiet mode. No stdout, only stderr and exit codes"
        echo "    -S Silent mode. No text output, only exit codes"
        echo
        exit 0
}

# Parse commandline
ALLARGS="$@"
set -- $(getopt hqS: -- "$@")
while true;
do
	case "$1" in
                (-h) usage;;
                (-q) QUIET=yes;;
                (-S) SILENT=yes;;
                (--) ;;
                (-*) echo "Error: unrecognized option $1" 1>&2; exit 1;;
                (*)  break;;
	esac
	shift
done

# send all stdout to /dev/null
if [ "${QUIET}" = "yes" ] || [ "${SILENT}" = "yes" ]; then
        exec 1> /dev/null
fi

# send all stdout and stderr to /dev/null
if [ "${SILENT}" = "yes" ]; then
        exec 2> /dev/null
fi

cd "${DIR}"

# Check if new version of the configuration is available, if not abort
git remote update
UPDATES=`git log HEAD..origin/master --oneline`
if [ "1" == "1${UPDATES}" ]
then
	echo "No updates available, aborting"
	exit 0
fi

# Git retrieve files
git fetch --all && git reset --hard origin/master
0 * * * * root /etc/sensu/conf.d/checks/scripts/pull_sensu_checks

Add a .gitignore file so that the files in enabled folder is not committed

/enabled/*
!enabled/.gitkeep
!enabled/base
!enabled/centos
!enabled/appl
/enabled/base/*
!enabled/base/.gitkeep
/enabled/centos/*
!enabled/centos/.gitkeep
/enabled/appl/*
!enabled/appl/.gitkeep

All the code and setup is done, commit and push…

So now we can enable the cron script so it executes every hour

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

On every other node clone the repository into /etc/sensu/conf.d/checks/ and enable the cron script:

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

All this work only enable the distribution and update of the checks, the actual enabling has to be done on each node (preferably using the scripts above)
 

Execute one command on all hosts

I have a couple of machines in my data center and sometimes it can be useful to run the same command on all the hosts.

For that purpose, I created a simple bash script that loops over a list of machine names and executes a command using ssh.

host1.example.com
host2.example.com
#!/usr/bin/env bash
KEY_SWITCH="-i ~/.ssh/custom_id_rsa"
USER="root"

function usage() {
        echo "Usage: exec_command.sh list \"command\""
        echo "     list      List type (linux, centos, debian)"
        echo "     command   Command to be executed on all hosts in the list"
}

if [ -z "$1" ]
then
        usage
        exit 1
fi
list_file="list_$1"

if [ ! -f "$list_file" ]
then
        echo "Error: Unable to find list file $list_file"
        echo
        usage
        exit 1
fi

echo "Using list file: $list_file"

if [ -z "$2" ]
then
        echo "Error: Command is not provided"
        echo
        usage
        exit 1
fi
command="$2"

IFS=$'\n' read -d '' -r -a hosts < $list_file
for host in "${hosts[@]}"
do
        echo "Executing command on host $host"
        echo
        ssh -q ${KEY_SWITCH} ${USER}@${host} "${command}"
        echo
        echo "Done executing on host $host"
done

As you can see the list is just a file named list_LISTNAME that contains one hostname per line.

I am using a custom key file for ssh to connect to my hosts, if that is not used change KEY_SWITCH value to "".

When the script is available and executable I can now run commands like:

$ ./execute_command.sh centos "yum update -y"
$ ./execute_command.sh debian "apt-get update && apt-get upgrade -y"