Proxmox resize linux guest partition

I sometimes come across that my partitions are too small in a VM Guest. And I can never remember the exact steps involved when resizing them. This post explains the steps needed to resize the root partition (can be any partition) of a VM.

The machines I am working with are CentOS 7 machines and they are using LVM.

I am going to be using fdisk to resize the disk instead of gparted or similar.

This is what we are starting with (df -h)

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  7.5G  2.2G  5.3G  30% /

Since I don’t need 100% uptime I can shutdown the VM to be resized (can probably be done live see https://pve.proxmox.com/wiki/Resize_disks)

Resize the disk in proxmox:

  1. Locate the VM to resize in the Proxmox web interface
  2. Select Hardware
  3. Find the Hard Disk to resize in the list of hardware
  4. Select Resize disk and enter the increment of disk size (in this example I selected 1 GB)

Note: Any snapshots on the machine needs to be removed before resize is allowed
Note 2: If the disk image is located on a nfs share, it will probably not work

Now we can boot up the machine again…

If we try the above df -h we can see that nothing has changed, this is because the partitions are the same, lets change that…

Using fdisk -l I get a list of all disks in the machine, here I can see that /dev/vda contains the LVM partition (/dev/vda2) that I want to resize

Disk /dev/vda: 11.8 GB, 11811160064 bytes, 23068672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b205b

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    20930559     9952256   8e  Linux LVM

So lets edit the partition map (here is the part that I find scary since we are going to delete the partition…)

# fdisk /dev/vda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p

Disk /dev/vda: 11.8 GB, 11811160064 bytes, 23068672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b205b

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    20930559     9952256   8e  Linux LVM

Command (m for help): d
Partition number (1,2, default 2): 2
Partition 2 is deleted

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p):
Using default response p
Partition number (2-4, default 2):
First sector (1026048-23068671, default 1026048):
Using default value 1026048
Last sector, +sectors or +size{K,M,G} (1026048-23068671, default 23068671):
Using default value 23068671
Partition 2 of type Linux and of size 10.5 GiB is set

Command (m for help): t
Partition number (1,2, default 2):
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): p

Disk /dev/vda: 11.8 GB, 11811160064 bytes, 23068672 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b205b

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    23068671    11021312   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

So what did we do:

  1. Entering the command fdisk /dev/vda on the command line allows us to edit /dev/vda
  2. Entering the p command lists the current partition table
  3. Entering d and then 2 allows us to delete the partition /dev/vda2
  4. Now we can create the new partition by entering n and using default values by pressing ENTER
  5. Then we change the type of the partition from Linux to Linux LVM by entering command t and the code 8e
  6. After that we print the new partition table using p
  7. And finally we write the partition table to disk using w

Now in order for the system to notice the change there are probably several ways to do this, but the easiest is to reboot. So let’s do that now…

Next it is time to resize the lvm, for reference, this is how it looks before.

Physical volume

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vda2
  VG Name               centos
  PV Size               9.49 GiB / not usable 3.00 MiB
  ...
  Free PE               1
  Allocated PE          2428

Volume Group

# vgdisplay
  --- Volume group ---
  VG Name               centos
  ...
  VG Size               9.49 GiB
  ...
  Total PE              2429
  Alloc PE / Size       2428 / 9.48 GiB
  Free  PE / Size       1 / 4.00 MiB

Logical volume

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  ...
  LV Size                7.49 GiB

  --- Logical volume ---
  LV Path                /dev/centos/swap
  ...

So start by resizing the Physical volume:

# pvresize /dev/vda2
  Physical volume "/dev/vda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Running a pvdisplay show that the physical volume has changed from

PV Size               9.49 GiB / not usable 3.00 MiB
...
Free PE               1

To

PV Size               10.51 GiB / not usable 2.00 MiB
...
Free PE               262

We can also see that the volume group (vgdisplay) now has increased FreePE.

Let’s extend the Logical volume

# lvresize --extents +100%FREE --resizefs /dev/centos/root
  Size of logical volume centos/root changed from 7.49 GiB (1917 extents) to 8.49 GiB (2173 extents).
  Logical volume centos/root successfully resized.

Here I entered the path to the logical volume /dev/centos/root which can be found in the output of the lvdisplay command on LV Path.

Now the resize is complete and the new output from the previous commands are

Physical volume

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vda2
  VG Name               centos
  PV Size               10.51 GiB / not usable 2.00 MiB
  ...
  Free PE               0
  Allocated PE          2690

Volume Group

# vgdisplay
  --- Volume group ---
  VG Name               centos
  ...
  VG Size               10.51 GiB
  ...
  Total PE              2690
  Alloc PE / Size       2690 / 10.51 GiB
  Free  PE / Size       0 / 0   

Logical volume

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  ...
  LV Size                8.51 GiB

  --- Logical volume ---
  LV Path                /dev/centos/swap
  ...

Disk space usage

# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  8.6G  2.2G  6.4G  26% /
Advertisements

CentOS 7 Disable IPv6

To disable IPv6 on CentOS 7 run the following commands:

The following will disable IPv6 on a running system

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

To permanently disable IPv6 run

echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf

CentOS 7: Display DHCP assigned IP addresses console prompt

In my network I depend on (statically) DHCP assigned IP-addresses for my servers and sometimes the DNS and DHCP might be somehow off-sync.

So in order to easily access a server that either got an invalid or unknown IP-address, I modified the standard console prompt to display the actual IP-addresses for that server.

This can easily be done by editing the /etc/issue file

\S
Kernel \r on an \m

Hostname: \n
Domain: \O
Users: \U
IP Addresses:
    Public: \4{eth0} Backbone: \4{eth1}

The switches above are replaced as follows:
* \S – OS/Distribution release
* \r – Kernel version
* \m – Architecture
* \n – The node name
* \O – The domain name
* \U – # Logged in users
* \4{ifc} – The IP-address of ifc

More switches can be found by issuing the command

$ man agetty

So whenever a server becomes “missing” I just need to open the virtualisation environment and look at the login prompt.

Distribute Sensu checks using git

Since my environment at home does not yet use any kind of automation for configuration and orchestration like the one that you can get using puppet, chef etc. I needed another way to distribute my “local” Sensu checks to all the machines.

So after thinking about the problem a bit, I decided to make use of my in-house GitLab server for hosting the Sensu checks configurations.

The first thing to do was to create the repository in GitLab, remember this repository either has to be public or have ssh keys configured. (We need to have password-less git pull)

Now that I had the repository ready I created a folder on all the machines /etc/sensu/conf.d/checks where the repository was checked out into.

One the main machine I created the following structure under /etc/sensu/conf.d/checks/:

available
- base
- appl
- centos
- proc
enabled
cron

The names under available can be anything, they just indicate the family where the checks belong.

Then the check files where created under e.g. available/base. Below is an example of a disk usage check script. Note: all the check scripts that exists under available is named.json.tmpl instead of just .json this because all nodes that later will receive the checks should not perform them.

{
  "checks": {
    "disk-usage": {
      "command": "check-disk-usage.rb -w :::disk.warning|70::: -c :::disk.critical|85::: :::disk.extra|:::",
      "subscribers": [
        "production"
      ],
      "interval": 600,
      "environment": ":::environment|production:::",
      "standalone": true
    }
  }
}

More information about checks can be found here Sensu checks

In order to make it easier on a node to list, enable, disable checks that have been retrieved from GitLab, three scripts where created:

#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: enable_check type check-name"
echo "   e.g. enable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if template is avaliable
if [ ! -f "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" ]
then
echo "Unable to find template ${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl"
exit 1
fi

# Check if check already is enabled
if [ -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Check ${2} is already enabled"
exit 1
fi

# Check if the directory for the selected type exists under enabled, otherwise create it
if [ ! -d "${SCRIPTPATH}/enabled/${1}" ]
then
mkdir "${SCRIPTPATH}/enabled/${1}"
chmod 755 "${SCRIPTPATH}/enabled/${1}"
fi

# Link the template (.json.tmpl) to a check file (.json) under enabled
ln -s "${SCRIPTPATH}/available/${1}/check-${2}.json.tmpl" "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now enabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash
# Parameter check
if [ "$#" -ne 2 ]; then
echo "Usage: disable_check type check-name"
echo "   e.g. disable_check base disk-usage"
exit 1
fi

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Check if check is enabled
if [ ! -f "${SCRIPTPATH}/enabled/${1}/check-${2}.json" ]
then
echo "Unable to find template ${SCRIPTPATH}/enable/${1}/check-${2}.json"
exit 1
fi

# Remove the link
rm "${SCRIPTPATH}/enabled/${1}/check-${2}.json"

echo "Check ${2} is now disabled, run: systemctl restart sensu-client.service"
#!/usr/bin/env bash

# Define the types directories, i.e. the directories under available
CHECK_DIRS=(
  'base'
  'centos'
  'appl'
  'proc'
)

# Retrieve current dir
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd -P`
popd > /dev/null

# Print header
echo " TYPE      | CHECK-NAME"
echo " ----------|-----------------------------"

for type in "${CHECK_DIRS[@]}"
do
    pushd "${SCRIPTPATH}/available/${type}/" > /dev/null
    shopt -s nullglob
    files=(*)
    popd > /dev/null
        for file in "${files[@]}"
        do
            enabled=""
            check=`echo "$file" | sed 's/check-\(.*\)\.json\.tmpl/\1/g'`
            if [ -d "${SCRIPTPATH}/enabled/${type}" ]
            then
                if [ -f "${SCRIPTPATH}/enabled/${type}/check-${check}.json" ]
                then
                    enabled="*"
                fi
            fi
            echo " $type   | ${check} ${enabled}"
        done
    echo ""
done

To use these we can run commands like:

$ ./check_list
$ ./check_enable base disk-usage
$ ./check_disable base disk-usage

Now all that is needed is a mechanism to retrieve new or updated checks on all machines.

To do that cron is used, so under scripts the script the fetches new and updated checks and a crontab configuration is created.

#!/usr/bin/env bash

# Quick-check before we allow bad things to happen
if [ -z "${BASH_VERSINFO}" ]; then
  echo "ERROR: You must execute this script with BASH"
  exit 255
fi

# Config
DIR="/etc/sensu/conf.d/checks"

usage() {
        echo "Usage: $(basename $0) [-hqS]"
        echo "    -h This help"
        echo "    -q Quiet mode. No stdout, only stderr and exit codes"
        echo "    -S Silent mode. No text output, only exit codes"
        echo
        exit 0
}

# Parse commandline
ALLARGS="$@"
set -- $(getopt hqS: -- "$@")
while true;
do
	case "$1" in
                (-h) usage;;
                (-q) QUIET=yes;;
                (-S) SILENT=yes;;
                (--) ;;
                (-*) echo "Error: unrecognized option $1" 1>&2; exit 1;;
                (*)  break;;
	esac
	shift
done

# send all stdout to /dev/null
if [ "${QUIET}" = "yes" ] || [ "${SILENT}" = "yes" ]; then
        exec 1> /dev/null
fi

# send all stdout and stderr to /dev/null
if [ "${SILENT}" = "yes" ]; then
        exec 2> /dev/null
fi

cd "${DIR}"

# Check if new version of the configuration is available, if not abort
git remote update
UPDATES=`git log HEAD..origin/master --oneline`
if [ "1" == "1${UPDATES}" ]
then
	echo "No updates available, aborting"
	exit 0
fi

# Git retrieve files
git fetch --all && git reset --hard origin/master
0 * * * * root /etc/sensu/conf.d/checks/scripts/pull_sensu_checks

Add a .gitignore file so that the files in enabled folder is not committed

/enabled/*
!enabled/.gitkeep
!enabled/base
!enabled/centos
!enabled/appl
/enabled/base/*
!enabled/base/.gitkeep
/enabled/centos/*
!enabled/centos/.gitkeep
/enabled/appl/*
!enabled/appl/.gitkeep

All the code and setup is done, commit and push…

So now we can enable the cron script so it executes every hour

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

On every other node clone the repository into /etc/sensu/conf.d/checks/ and enable the cron script:

$ sudo ln -s /etc/sensu/conf.d/checks/scripts/pull_sensu_checks_cron /etc/cron.d/pull_sensu_checks

All this work only enable the distribution and update of the checks, the actual enabling has to be done on each node (preferably using the scripts above)
 

CentOS 7: Use a live VM as template

When the need arises to create a new VM I normally clone a dedicated base machine with the common tools and configurations needed to get a new machine up and running quickly.

So every time I encounter a new application that I need in multiple future machines I install that application on the dedicated base machine.

Since the base machine is live, it gets log files, history, passwords etc., which should not be carried over to the new machine.

For this purpose, the following script has been created. The script cleans up the most common things, then unconfigures the system

#!/usr/bin/env bash

# Configuration of network interface names
NETWORK_NAME_PUBLIC="public"
NETWORK_NAME_BACKBONE="backbone"

# Make sure the user really wants to run this
read -r -p "This script will clear all local data and prepare the machine for templating. Remember to create a snapshot before running this script. Are you sure you want to continue (y/N)? " response
echo
if [[ ! $response =~ [Yy]$ ]]
then
exit 1;
fi

# Reset the hostname
echo "* Change to generic hostname"
hostnamectl set-hostname localhost.localdomain

# Reset the machine id
echo "* Resetting machine-id"
> /etc/machine-id

# Clean up temporary files created in the live base machine
echo "* Removing roots temporary files"
rm -f /etc/ssh/ssh_host_*
rm -f /root/.ssh/known_hosts*
rm -f /root/.ssh/id_*
rm -f /root/.ssh/config
rm -f /root/anaconda-ks.cfg
rm -f /root/.bash_history
rm -rf /root/.config
rm -rf /root/.cache
rm -f /root/.viminfo
unset HISTFILE

# Remove mac-addresses
echo "* Removing MAC addresses from network configurations"
sed -i '/.*HWADDR=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
sed -i '/.*UUID=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
sed -i '/.*DEVICE=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}
echo "DEVICE=eth0" >> /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_PUBLIC}

sed -i '/.*HWADDR=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
sed -i '/.*UUID=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
sed -i '/.*DEVICE=.*$/d' /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}
echo "DEVICE=eth1" >> /etc/sysconfig/network-scripts/ifcfg-${NETWORK_NAME_BACKBONE}

# Clear logs
echo "* Clear logs"
rm -f /var/log/boot.log
rm -f /var/log/btmp
rm -f /var/log/cron
rm -f /var/log/dmesg
rm -f /var/log/dmesg.old
rm -f /var/log/grubby
rm -f /var/log/lastlog
rm -f /var/log/maillog
rm -f /var/log/messages
rm -f /var/log/secure
rm -f /var/log/spooler
rm -f /var/log/tallylog
rm -f /var/log/wpa_supplicant.log
rm -f /var/log/wtmp
rm -f /var/log/yum.log
rm -f /var/log/audit/audit.log
rm -f /var/log/qemu-ga/*
rm -f /var/log/tuned/tuned.log

# Force a disk check on next boot
echo "* Enable fsck on next boot"
touch /forcefsck

# Run sys-unconfig
read -r -p "System is prepared, run sys-unconfig (y/N)? " response
echo
if [[ ! $response =~ [Yy]$ ]]
then
exit 1;
fi

sys-unconfig

Use this script on a cloned machine, not the live dedicated base machine since it will remove all history and reset some configurations.

Note: My machines have 2 network interfaces and their names are configured at the top in NETWORK_NAME_PUBLIC, NETWORK_NAME_BACKBONE

Smarter bash history

When using bash the up/down arrows are useful to browse through the recent commands.

But since the wanted command may be a bit up in the history stack it can be a bit tedious to find the right command.

For example, I know that I had a long nice cat + grep + awk line earlier it might require many up-arrow presses or a history | grep cat…

Instead I added small script file to /etc/profile.d/complete_history.sh

if [ ! -z "$PS1" ]; then
bind '"\e[A": history-search-backward'
bind '"\e[B": history-search-forward'
fi

You probably need to make it executable…

Now after starting a new bash instance I can write:

cat [up-arrow]

This will only give me the history entries starting with cat …

If you started the line with the wrong keyword use CTRL+C to break out…

CentOS 7 Automated reboot after kernel update

When using applications like yum-cron to update the installed packages automatically there will be times when the kernel has been updated.

Instead of checking every machine if a reboot is needed I use the following cron script to automate the reboot.

The following cron script is place in /etc/cron.daily/reboot_on_kernel_update

#!/usr/bin/env bash

EMAIL_RECIPIENT="test@example.com"
LATEST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

if [ "$LATEST_KERNEL" != "$CURRENT_KERNEL" ]
then
    MAIL_BODY="The kernel has been updated from $CURRENT_KERNEL to $LATEST_KERNEL on $HOSTNAME. Because of this a reboot has been issued. // Kernel update monitor"
    mail -s "$HOSTNAME: Kernel update monitor" $EMAIL_RECIPIENT <<< $MAIL_BODY
    reboot
fi

Remember to make the script executable…