The old computerhok is running for almost 5 years now (Ubuntu 8.04, both hypervisor and guests), time for something new.
We went live on Sunday 2013-02-24!
We will no longer use OpenVZ, it is too restrictive in choice of guest OS'es.
We intend to use VirtualBox as hypervisor, and Ubuntu 12.04 LTS as both host and guest OS.
After doing some experiments I decided to leave Virtualbox:
it is poorly documented (only "click here, click that", no good reference of all command options)
you cannot resize harddisks (anymore ?)
unstable, vboxdrv driver modules disappearing for unknown reasons
root@apollo:~/Downloads# VBoxHeadless --startvm Ubuntu1
Oracle VM VirtualBox Headless Interface 4.2.6
(C) 2008-2012 Oracle Corporation
All rights reserved.
VRDE server is listening on port 13389.
VBoxManage snapshot Ubuntu1 take snapshot01 --description "first snapshot, almost vanilla ubuntu"
This commands ends fine, but the VM hangs, even no response from external pings anymore.
VRDP just shows a black screen.
Looks like it is paused (even if not specified on the snapshot subcommand):
So try to resume it:
root@apollo:# VBoxManage controlvm Ubuntu1 resume
root@apollo:# VBoxManage showvminfo Ubuntu1 --details|grep -i state
State: running (since 2013-01-27T15:40:09.024000000)
root@apollo:#
}}}
And yes it is running again.
Now clone this snapshot to a second machine:
root@apollo:~# VBoxManage clonevm Ubuntu1 --snapshot snapshot01 --options keepdisknames --name Ubuntu2 --register
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Machine has been successfully cloned as "Ubuntu2"
root@apollo:~#
(This takes a few minutes).
But then we have :
root@apollo:~# VBoxManage list vms
"Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019}
"Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1}
root@apollo:~#
Change the vrdeport (should not be the same as the first machine): VBoxManage modifyvm Ubuntu2 --vrdeport 13390
And fire up the thing:
root@apollo:~# VBoxHeadless --startvm Ubuntu2
Oracle VM VirtualBox Headless Interface 4.2.6
(C) 2008-2012 Oracle Corporation
All rights reserved.
VRDE server is listening on port 13390.
The first boot takes a bit longer, because of an fsck that runs because it thinks the fs was not cleanly unmounted.
First login through the VRDP console and change hostname with hostname ubuntu2 and editing /etc/hostname.
I also note that network device eth0 does not come up.
dmesg shows that udev has renamed eth0 to eth1, that sounds familiar: remove /etc/udev/rules.d/70-persistent-net.rules
listing HD's and cloning a harddisk:
root@apollo:~# vb list hdds
UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Parent UUID: base
Format: VDI
Location: /root/VirtualBox VMs/Ubuntu1/disk1.vdi
State: locked read
Type: normal
Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)]
UUID: 88a8d294-cb4c-4881-b42b-286b25b3ab3e
Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Format: VDI
Location: /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi
State: locked write
Type: normal
Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019)
UUID: fb289305-27fc-4344-bfb2-75942cbb1252
Parent UUID: base
Format: VDI
Location: /root/VirtualBox VMs/Ubuntu2/disk1.vdi
State: locked write
Type: normal
Usage: Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1)
root@apollo:~# vb clonehd fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 /tmp/cloned.from.ubuntu1
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'VDI'. UUID: 8a06d36c-13f0-4275-abe7-ebc64bff59a4
root@apollo:~# vb list hdds
UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Parent UUID: base
Format: VDI
Location: /root/VirtualBox VMs/Ubuntu1/disk1.vdi
State: locked read
Type: normal
Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)]
UUID: 88a8d294-cb4c-4881-b42b-286b25b3ab3e
Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Format: VDI
Location: /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi
State: locked write
Type: normal
Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019)
UUID: fb289305-27fc-4344-bfb2-75942cbb1252
Parent UUID: base
Format: VDI
Location: /root/VirtualBox VMs/Ubuntu2/disk1.vdi
State: locked write
Type: normal
Usage: Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1)
UUID: 8a06d36c-13f0-4275-abe7-ebc64bff59a4
Parent UUID: base
Format: VDI
Location: /tmp/cloned.from.ubuntu1
State: created
Type: normal
Now, you cannot just copy these files over :
root@apollo:~/VirtualBox VMs/Ubuntu2# cat /tmp/cloned.from.ubuntu1 > disk1.vdi
root@apollo:~/VirtualBox VMs/Ubuntu2# vb list vms
"Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019}
"Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1}
root@apollo:~/VirtualBox VMs/Ubuntu2# vb startvm Ubuntu2
Waiting for VM "Ubuntu2" to power on...
VBoxManage: error: The virtual machine 'Ubuntu2' has terminated unexpectedly during startup with exit code 0
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Machine, interface IMachine
root@apollo:~/VirtualBox VMs/Ubuntu2#
So now "detach" this disk again by "attaching none" :
I created the following ~root/bin/lxc-rsync script :
#!/bin/bash
#
# rsync on lxc with another
# args: <src lxc> <tgt lxc>
#
NUMARGS=$#
if [ $NUMARGS -ne 2 ]; then
echo "Usage: lxc-rsync <src lxc> <tgt lxc>"
exit 8
fi
SRCLXC=$1
TGTLXC=$2
SRCDIR=/var/lib/lxc/${SRCLXC}/rootfs
TGTDIR=/var/lib/lxc/${TGTLXC}/rootfs
if [ -d $SRCDIR -a -d $TGTDIR ]; then
echo "rsyncing from $SRCLXC to $TGTLXC"
cd $SRCDIR || exit 8
# save the old IP address
OLDIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`
rsync --exclude "tmp" --exclude "dev" --exclude "media" --exclude "mnt" --exclude "proc" --exclude "sys" --exclude "var/run" --verbose --recursive --links --perms --acls --times --owner --group --one-file-system --delete . $TGTDIR
# patching hostname and IP address
echo "patching /etc/hostname, /etc/hosts and /etc/network/interfaces ==> $TGTLXC / $OLDIP"
sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hostname
sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hosts
WRONGIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`
sed --in-place s/${WRONGIP}/${OLDIP}/g ${TGTDIR}/etc/network/interfaces
else
echo "either $SRCDIR or $TGTDIR does not exist"
exit 8
fi
But this rsync does not properly clone, for example /run/zabbix /var/run/zabbix fails, and mysqld does not start.
Therefore I created lxc-copy that uses fsarchiver to do the heavy lifting.
The following script (takes longer but) works better :
#!/bin/bash
#
# copy an lxc (filesystem) to another one
# args: <src lxc> (must bu either cn1 or cn4, they will get copied to resp. cn2 and cn3)
#
# funtions first
logDie()
{
msg=$1
echo "$msg"
exit 8
}
# m a i n l i n e
#
NUMARGS=$#
if [ $NUMARGS -ne 1 ]; then
echo "Usage: lxc-copy <src lxc> (must bu either cn1 or cn4, they will get copied to resp. cn2 and cn3)"
exit 8
fi
SRCLXC=$1
if [ $SRCLXC != "cn1" -a $SRCLXC != "cn4" ]; then
echo "argument must be either cn1 or cn4"
exit 8
fi
TGTLXC=cn2
if [ $SRCLXC = "cn4" ]; then
TGTLXC=cn3
fi
echo "cloning from $SRCLXC to $TGTLXC"
mount |grep "/var/lib/lxc/${TGTLXC}" > /dev/null || logDie "/var/lib/lxc/${TGTLXC} is not mounted"
TGTDIR=/var/lib/lxc/${TGTLXC}/rootfs
OLDIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{print $NF}'`
OLDUUID=`cat /etc/fstab|grep /${TGTLXC} | awk '{ print $1}'|awk -F= '{print $NF}'`
# save old metadata of target lxc
cd /var/lib/lxc/${TGTLXC} && tar -cf /tmp/old-metadata-of-TGTLXC.tar config fstab rootfs.hold || logDie "saving old metadata of ${TGTLXC} failed"
cd -
MNTPOINT=/var/lib/lxc/${TGTLXC}
echo "umounting ${MNTPOINT}"
umount ${MNTPOINT} || logDie "umount failed for ${MNTPOINT}"
#
# check if container fs has been mounted
df /mnt/container/ | grep /dev/mapper/vg0-container > /dev/null
RC=$?
if [ $RC -eq 1 ]; then
mount /dev/mapper/vg0-container /mnt/container
echo "container fs mounted"
fi
echo "dumping source filesystem"
TAPEFILE=/mnt/container/savedfs.fsa
fsarchiver -A savefs ${TAPEFILE} /dev/mapper/vg0-lvol.${SRCLXC} || logDie "fsarchive savefs failed"
echo "restoring filesystem"
fsarchiver restfs ${TAPEFILE} id=0,dest=/dev/mapper/vg0-lvol.${TGTLXC} || logDie "fsarchive restfs failed"
echo "restoring old UUID ${OLDUUID}"
tune2fs /dev/mapper/vg0-lvol.${TGTLXC} -U ${OLDUUID}
echo "mounting back again"
mount -a || logDie "mount -a failed"
echo "restoring old metadata"
cd /var/lib/lxc/${TGTLXC} && tar -xf /tmp/old-metadata-of-TGTLXC.tar || logDie "restoring old metadata failed"
echo "patching /etc/hostname, /etc/hosts and /etc/network/interfaces ==> $TGTLXC / $OLDIP"
sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hostname || logDie "patching /etc/hostname failed"
sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hosts || logDie "patching /etc/hosts failed"
WRONGIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`
sed --in-place s/${WRONGIP}/${OLDIP}/g ${TGTDIR}/etc/network/interfaces || logDie "patching /etc/network/interfaces failed"
rm -f ${TAPEFILE} /tmp/old-metadata-of-TGTLXC.tar
umount /mnt/container
Computerhok 2013#
Table of Contents
Intro#
The old computerhok is running for almost 5 years now (Ubuntu 8.04, both hypervisor and guests), time for something new. We went live on Sunday 2013-02-24!Functions to be ported from old to new#
VirtualBox#
We will no longer use OpenVZ, it is too restrictive in choice of guest OS'es.
We intend to use VirtualBox as hypervisor, and Ubuntu 12.04 LTS as both host and guest OS.
I stumbled upon lxc
, so I started figuring out what that offers, see here my experiments with lxc.
Work in progress / install log#
Ubuntu 12.04.1 LTS#
kernel recompile#
VirtualBox installation#
VBoxManage syntax
VirtualBox cloning/snapshotting#
Take a snapshot of a running VM:
This commands ends fine, but the VM hangs, even no response from external pings anymore. VRDP just shows a black screen.
So try to resume it: root@apollo:# VBoxManage controlvm Ubuntu1 resume root@apollo:# VBoxManage showvminfo Ubuntu1 --details|grep -i state State: running (since 2013-01-27T15:40:09.024000000) root@apollo:# }}} And yes it is running again.Looks like it is paused (even if not specified on the snapshot subcommand):
Now clone this snapshot to a second machine:
(This takes a few minutes).But then we have :
root@apollo:~# VBoxManage list vms "Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019} "Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1} root@apollo:~#Change the vrdeport (should not be the same as the first machine):
VBoxManage modifyvm Ubuntu2 --vrdeport 13390
And fire up the thing:
The first boot takes a bit longer, because of an fsck that runs because it thinks the fs was not cleanly unmounted.
First login through the VRDP console and change hostname with hostname ubuntu2 and editing /etc/hostname.
I also note that network device eth0 does not come up. dmesg shows that udev has renamed eth0 to eth1, that sounds familiar: remove /etc/udev/rules.d/70-persistent-net.rules
root@apollo:~# vb list hdds UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 Parent UUID: base Format: VDI Location: /root/VirtualBox VMs/Ubuntu1/disk1.vdi State: locked read Type: normal Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)] UUID: 88a8d294-cb4c-4881-b42b-286b25b3ab3e Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 Format: VDI Location: /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi State: locked write Type: normal Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) UUID: fb289305-27fc-4344-bfb2-75942cbb1252 Parent UUID: base Format: VDI Location: /root/VirtualBox VMs/Ubuntu2/disk1.vdi State: locked write Type: normal Usage: Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1) root@apollo:~# vb clonehd fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 /tmp/cloned.from.ubuntu1 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Clone hard disk created in format 'VDI'. UUID: 8a06d36c-13f0-4275-abe7-ebc64bff59a4 root@apollo:~# vb list hdds UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 Parent UUID: base Format: VDI Location: /root/VirtualBox VMs/Ubuntu1/disk1.vdi State: locked read Type: normal Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)] UUID: 88a8d294-cb4c-4881-b42b-286b25b3ab3e Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 Format: VDI Location: /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi State: locked write Type: normal Usage: Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) UUID: fb289305-27fc-4344-bfb2-75942cbb1252 Parent UUID: base Format: VDI Location: /root/VirtualBox VMs/Ubuntu2/disk1.vdi State: locked write Type: normal Usage: Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1) UUID: 8a06d36c-13f0-4275-abe7-ebc64bff59a4 Parent UUID: base Format: VDI Location: /tmp/cloned.from.ubuntu1 State: created Type: normalNow, you cannot just copy these files over :root@apollo:~/VirtualBox VMs/Ubuntu2# cat /tmp/cloned.from.ubuntu1 > disk1.vdi root@apollo:~/VirtualBox VMs/Ubuntu2# vb list vms "Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019} "Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1} root@apollo:~/VirtualBox VMs/Ubuntu2# vb startvm Ubuntu2 Waiting for VM "Ubuntu2" to power on... VBoxManage: error: The virtual machine 'Ubuntu2' has terminated unexpectedly during startup with exit code 0 VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Machine, interface IMachine root@apollo:~/VirtualBox VMs/Ubuntu2#So now "detach" this disk again by "attaching none" :
#iface eth0 inet dhcp iface eth0 inet static address 10.0.0.155 netmask 255.255.255.0 network 10.0.0.0 broadcast 10.0.0.255 gateway 10.0.0.138 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 213.197.28.3 213.197.30.28 dns-search computerhok.nlExtra installed packages#
Extra installed packages for the Host#
Extra installed packages for the container(s)#
Install/migrate Gallery#
Now before copying over photo's from the old site, we first have to increase the filesystem space.
Forget about gallery3#
After a couple of hours trying/googling I gave up on migrating from gallery2 to gallery3.
zabbix 2.0 experiment#
cn4 cloned for this purpose
firewall configurations#
apollo#
Stuck into /etc/rc.local:
cn1#
Stuck into /etc/rc.local:
Miscellaneous#
lxc-rsync#
To be able to quickly "copy/clone" lxc's (while having their own filesystem/lv already) :
I created the following ~root/bin/lxc-rsync script :
#!/bin/bash # # rsync on lxc with another # args: <src lxc> <tgt lxc> # NUMARGS=$# if [ $NUMARGS -ne 2 ]; then echo "Usage: lxc-rsync <src lxc> <tgt lxc>" exit 8 fi SRCLXC=$1 TGTLXC=$2 SRCDIR=/var/lib/lxc/${SRCLXC}/rootfs TGTDIR=/var/lib/lxc/${TGTLXC}/rootfs if [ -d $SRCDIR -a -d $TGTDIR ]; then echo "rsyncing from $SRCLXC to $TGTLXC" cd $SRCDIR || exit 8 # save the old IP address OLDIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'` rsync --exclude "tmp" --exclude "dev" --exclude "media" --exclude "mnt" --exclude "proc" --exclude "sys" --exclude "var/run" --verbose --recursive --links --perms --acls --times --owner --group --one-file-system --delete . $TGTDIR # patching hostname and IP address echo "patching /etc/hostname, /etc/hosts and /etc/network/interfaces ==> $TGTLXC / $OLDIP" sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hostname sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hosts WRONGIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'` sed --in-place s/${WRONGIP}/${OLDIP}/g ${TGTDIR}/etc/network/interfaces else echo "either $SRCDIR or $TGTDIR does not exist" exit 8 filxc-copy#
The following script (takes longer but) works better :
#!/bin/bash # # copy an lxc (filesystem) to another one # args: <src lxc> (must bu either cn1 or cn4, they will get copied to resp. cn2 and cn3) # # funtions first logDie() { msg=$1 echo "$msg" exit 8 } # m a i n l i n e # NUMARGS=$# if [ $NUMARGS -ne 1 ]; then echo "Usage: lxc-copy <src lxc> (must bu either cn1 or cn4, they will get copied to resp. cn2 and cn3)" exit 8 fi SRCLXC=$1 if [ $SRCLXC != "cn1" -a $SRCLXC != "cn4" ]; then echo "argument must be either cn1 or cn4" exit 8 fi TGTLXC=cn2 if [ $SRCLXC = "cn4" ]; then TGTLXC=cn3 fi echo "cloning from $SRCLXC to $TGTLXC" mount |grep "/var/lib/lxc/${TGTLXC}" > /dev/null || logDie "/var/lib/lxc/${TGTLXC} is not mounted" TGTDIR=/var/lib/lxc/${TGTLXC}/rootfs OLDIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{print $NF}'` OLDUUID=`cat /etc/fstab|grep /${TGTLXC} | awk '{ print $1}'|awk -F= '{print $NF}'` # save old metadata of target lxc cd /var/lib/lxc/${TGTLXC} && tar -cf /tmp/old-metadata-of-TGTLXC.tar config fstab rootfs.hold || logDie "saving old metadata of ${TGTLXC} failed" cd - MNTPOINT=/var/lib/lxc/${TGTLXC} echo "umounting ${MNTPOINT}" umount ${MNTPOINT} || logDie "umount failed for ${MNTPOINT}" # # check if container fs has been mounted df /mnt/container/ | grep /dev/mapper/vg0-container > /dev/null RC=$? if [ $RC -eq 1 ]; then mount /dev/mapper/vg0-container /mnt/container echo "container fs mounted" fi echo "dumping source filesystem" TAPEFILE=/mnt/container/savedfs.fsa fsarchiver -A savefs ${TAPEFILE} /dev/mapper/vg0-lvol.${SRCLXC} || logDie "fsarchive savefs failed" echo "restoring filesystem" fsarchiver restfs ${TAPEFILE} id=0,dest=/dev/mapper/vg0-lvol.${TGTLXC} || logDie "fsarchive restfs failed" echo "restoring old UUID ${OLDUUID}" tune2fs /dev/mapper/vg0-lvol.${TGTLXC} -U ${OLDUUID} echo "mounting back again" mount -a || logDie "mount -a failed" echo "restoring old metadata" cd /var/lib/lxc/${TGTLXC} && tar -xf /tmp/old-metadata-of-TGTLXC.tar || logDie "restoring old metadata failed" echo "patching /etc/hostname, /etc/hosts and /etc/network/interfaces ==> $TGTLXC / $OLDIP" sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hostname || logDie "patching /etc/hostname failed" sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hosts || logDie "patching /etc/hosts failed" WRONGIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'` sed --in-place s/${WRONGIP}/${OLDIP}/g ${TGTDIR}/etc/network/interfaces || logDie "patching /etc/network/interfaces failed" rm -f ${TAPEFILE} /tmp/old-metadata-of-TGTLXC.tar umount /mnt/container