Computerhok 2013
Back to current versionRestore this version

Computerhok 2013#

Intro#

The old computerhok is running for almost 5 years now (Ubuntu 8.04, both hypervisor and guests), time for something new. We went live on Sunday 2013-02-24!

Functions to be ported from old to new#

VirtualBox#

We will no longer use OpenVZ, it is too restrictive in choice of guest OS'es.
We intend to use VirtualBox as hypervisor, and Ubuntu 12.04 LTS as both host and guest OS.

After doing some experiments I decided to leave Virtualbox:
  • it is poorly documented (only "click here, click that", no good reference of all command options)
  • you cannot resize harddisks (anymore ?)
  • unstable, vboxdrv driver modules disappearing for unknown reasons

I stumbled upon lxc, so I started figuring out what that offers, see here my experiments with lxc.

Work in progress / install log#

Ubuntu 12.04.1 LTS#

kernel recompile#

VirtualBox installation#

VBoxManage syntax

root@apollo:~/Downloads# sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.2.6-82870.vbox-extpack 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully installed "Oracle VM VirtualBox Extension Pack".
root@apollo:~/Downloads# 
root@apollo:~# VBoxManage createvm --name Ubuntu1 --ostype Linux --register 
Virtual machine 'Ubuntu1' is created and registered.
UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019
Settings file: '/root/VirtualBox VMs/Ubuntu1/Ubuntu1.vbox'
root@apollo:~#
VBoxManage modifyvm Ubuntu1 --memory=512 --hwvirtex=off --hwvirtexexcl=off --vtxvpid=off --boot1=dvd --boot2=disk --boot3=none --nic1=bridged --bridgeadapter1=wlan0 --nicpromisc1=allow-all --vrde=on --vrdeport=13389
root@apollo:~/src# du -cms linux-source-3.2.0/
5172	linux-source-3.2.0/
5172	total
root@apollo:~/src# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       9.2G  6.6G  2.2G  76% /
root@apollo:~/src# rm -rf linux-source-3.2.0/
root@apollo:~/src# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       9.2G  1.6G  7.2G  18% /
VBoxManage storagectl Ubuntu1 --name='IDE Controller' --add=ide --controller=PIIX4 --bootable=on
VBoxManage storagectl Ubuntu1 --name='SATA Controller' --add=sata --controller=IntelAhci --bootable=on
root@apollo:~/VirtualBox VMs/Ubuntu1# VBoxManage showvminfo Ubuntu1
Name:            Ubuntu1
Groups:          /
Guest OS:        Other Linux
UUID:            1ed5e417-1eca-4a60-b6d4-5c8f40e44019
Config file:     /root/VirtualBox VMs/Ubuntu1/Ubuntu1.vbox
Snapshot folder: /root/VirtualBox VMs/Ubuntu1/Snapshots
Log folder:      /root/VirtualBox VMs/Ubuntu1/Logs
Hardware UUID:   1ed5e417-1eca-4a60-b6d4-5c8f40e44019
Memory size:     512MB
Page Fusion:     off
VRAM size:       8MB
CPU exec cap:    100%
HPET:            off
Chipset:         piix3
Firmware:        BIOS
Number of CPUs:  1
Synthetic Cpu:   off
CPUID overrides: None
Boot menu mode:  message and menu
Boot Device (1): DVD
Boot Device (2): HardDisk
Boot Device (3): Not Assigned
Boot Device (4): Not Assigned
ACPI:            on
IOAPIC:          off
PAE:             off
Time offset:     0ms
RTC:             local time
Hardw. virt.ext: off
Hardw. virt.ext exclusive: off
Nested Paging:   on
Large Pages:     off
VT-x VPID:       off
State:           powered off (since 2013-01-26T17:41:58.000000000)
Monitor count:   1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address: 
Teleporter Password: 
Tracing Enabled: off
Allow Tracing to Access VM: off
Tracing Configuration: 
Autostart Enabled: off
Autostart Delay: 0
Storage Controller Name (0):            IDE Controller
Storage Controller Type (0):            PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  2
Storage Controller Port Count (0):      2
Storage Controller Bootable (0):        on
Storage Controller Name (1):            SATA Controller
Storage Controller Type (1):            IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1):  30
Storage Controller Port Count (1):      30
Storage Controller Bootable (1):        on
NIC 1:           MAC: 0800276CD273, Attachment: Bridged Interface 'wlan0', Cable connected: on, Trace: off (file: none), Type: Am79C973, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
Pointing Device: PS/2 Mouse
Keyboard Device: PS/2 Keyboard
UART 1:          disabled
UART 2:          disabled
LPT 1:           disabled
LPT 2:           disabled
Audio:           disabled
Clipboard Mode:  disabled
Drag'n'drop Mode:  disabled
VRDE:            enabled (Address 0.0.0.0, Ports 13389, MultiConn: off, ReuseSingleConn: off, Authentication type: null)
Video redirection: disabled
USB:             disabled
EHCI:            disabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Bandwidth groups:  <none>

Shared folders:  <none>

VRDE Connection:    not active
Clients so far:     0

Guest:

Configured memory balloon size:      0 MB
VBoxManage storageattach Ubuntu1 --storagectl='IDE Controller' --port=1 --device=1 --type=dvddrive --medium=/tmp/ff/ubuntu-12.04.1-server-i386.iso
VBoxManage createhd --filename '/root/VirtualBox VMs/Ubuntu1/disk1.vdi' --size=15000
VBoxManage storageattach Ubuntu1 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium='/root/VirtualBox VMs/Ubuntu1/disk1.vdi'
root@apollo:~/Downloads# VBoxHeadless --startvm Ubuntu1 
Oracle VM VirtualBox Headless Interface 4.2.6
(C) 2008-2012 Oracle Corporation
All rights reserved.

VRDE server is listening on port 13389.

VirtualBox cloning/snapshotting#

Take a snapshot of a running VM:

VBoxManage snapshot Ubuntu1 take snapshot01 --description "first snapshot, almost vanilla ubuntu" 

This commands ends fine, but the VM hangs, even no response from external pings anymore. VRDP just shows a black screen.
Looks like it is paused (even if not specified on the snapshot subcommand):

root@apollo:~# VBoxManage showvminfo Ubuntu1 --details|grep -i state
State:           paused (since 2013-01-27T15:23:36.808000000)
root@apollo:~# 
So try to resume it: root@apollo:# VBoxManage controlvm Ubuntu1 resume root@apollo:# VBoxManage showvminfo Ubuntu1 --details|grep -i state State: running (since 2013-01-27T15:40:09.024000000) root@apollo:# }}} And yes it is running again.

Now clone this snapshot to a second machine:

root@apollo:~# VBoxManage clonevm Ubuntu1 --snapshot snapshot01 --options keepdisknames --name Ubuntu2 --register
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Machine has been successfully cloned as "Ubuntu2"
root@apollo:~# 
(This takes a few minutes).
But then we have :
root@apollo:~# VBoxManage list vms
"Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019}
"Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1}
root@apollo:~# 

Change the vrdeport (should not be the same as the first machine):
VBoxManage modifyvm Ubuntu2 --vrdeport 13390

And fire up the thing:

root@apollo:~# VBoxHeadless --startvm Ubuntu2 
Oracle VM VirtualBox Headless Interface 4.2.6
(C) 2008-2012 Oracle Corporation
All rights reserved.

VRDE server is listening on port 13390.

The first boot takes a bit longer, because of an fsck that runs because it thinks the fs was not cleanly unmounted.
First login through the VRDP console and change hostname with hostname ubuntu2 and editing /etc/hostname.
I also note that network device eth0 does not come up. dmesg shows that udev has renamed eth0 to eth1, that sounds familiar: remove /etc/udev/rules.d/70-persistent-net.rules

root@apollo:~# vb list hdds
UUID:        fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/disk1.vdi
State:       locked read
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)]

UUID:        88a8d294-cb4c-4881-b42b-286b25b3ab3e
Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019)

UUID:        fb289305-27fc-4344-bfb2-75942cbb1252
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu2/disk1.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1)

root@apollo:~# vb clonehd fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 /tmp/cloned.from.ubuntu1
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'VDI'. UUID: 8a06d36c-13f0-4275-abe7-ebc64bff59a4
root@apollo:~# vb list hdds
UUID:        fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/disk1.vdi
State:       locked read
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)]

UUID:        88a8d294-cb4c-4881-b42b-286b25b3ab3e
Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019)

UUID:        fb289305-27fc-4344-bfb2-75942cbb1252
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu2/disk1.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1)

UUID:        8a06d36c-13f0-4275-abe7-ebc64bff59a4
Parent UUID: base
Format:      VDI
Location:    /tmp/cloned.from.ubuntu1
State:       created
Type:        normal
Now, you cannot just copy these files over :
root@apollo:~/VirtualBox VMs/Ubuntu2# cat /tmp/cloned.from.ubuntu1 > disk1.vdi 
root@apollo:~/VirtualBox VMs/Ubuntu2# vb list vms
"Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019}
"Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1}

root@apollo:~/VirtualBox VMs/Ubuntu2# vb startvm Ubuntu2
Waiting for VM "Ubuntu2" to power on...
VBoxManage: error: The virtual machine 'Ubuntu2' has terminated unexpectedly during startup with exit code 0
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Machine, interface IMachine
root@apollo:~/VirtualBox VMs/Ubuntu2# 

So now "detach" this disk again by "attaching none" :

VBoxManage storageattach Ubuntu1 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium none
VBoxManage storageattach Ubuntu2 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium=none
VBoxManage closemedium disk fb289305-27fc-4344-bfb2-75942cbb1252 --delete
VBoxManage storageattach Ubuntu2 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium='/root/VirtualBox VMs/Ubuntu2/disk1.vdi'
VBoxManage modifyvm Ubuntu2 --boot1 disk
#iface eth0 inet dhcp
iface eth0 inet static
        address 10.0.0.155
        netmask 255.255.255.0
        network 10.0.0.0
        broadcast 10.0.0.255
        gateway 10.0.0.138
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 213.197.28.3 213.197.30.28
        dns-search computerhok.nl

Extra installed packages#

Extra installed packages for the Host#

Extra installed packages for the container(s)#

sudo sh -c 'echo "deb http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu precise main" >> /etc/apt/sources.list.d/mariadb.list' 

Install/migrate Gallery#

MariaDB [(none)]> create user 'gallery3'@'localhost' identified by "gallery3pw";
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database gallery3;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all privileges on gallery3.* to gallery3;
Query OK, 0 rows affected (0.00 sec)
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4F4EA0AAE5267A6C
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CBCB082A1BB943DB
apt-get update

Now before copying over photo's from the old site, we first have to increase the filesystem space.

Forget about gallery3#

After a couple of hours trying/googling I gave up on migrating from gallery2 to gallery3.

zabbix 2.0 experiment#

cn4 cloned for this purpose

firewall configurations#

apollo#

Stuck into /etc/rc.local:

/sbin/iptables -F
/sbin/iptables -F -t nat

/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A INPUT -p icmp -j ACCEPT
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -i lxcbr0 -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.0.0/24 -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.3.1/24 -j ACCEPT
/sbin/iptables -A INPUT -s 140.211.11.9 -j ACCEPT
/sbin/iptables -A INPUT -s 145.72.98.1 -j ACCEPT
#/sbin/iptables -A INPUT -j LOG
/sbin/iptables -A INPUT -j DROP

/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1122 -j DNAT --to 10.0.3.11:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1125 -j DNAT --to 10.0.3.11:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1180 -j DNAT --to 10.0.3.11:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 11443 -j DNAT --to 10.0.3.11:443
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 11808 -j DNAT --to 10.0.3.11:8080
#
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1222 -j DNAT --to 10.0.3.12:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1225 -j DNAT --to 10.0.3.12:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1280 -j DNAT --to 10.0.3.12:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 12443 -j DNAT --to 10.0.3.12:443
#
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1322 -j DNAT --to 10.0.3.13:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1325 -j DNAT --to 10.0.3.13:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1380 -j DNAT --to 10.0.3.13:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 13443 -j DNAT --to 10.0.3.13:443
#
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1422 -j DNAT --to 10.0.3.14:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1425 -j DNAT --to 10.0.3.14:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1480 -j DNAT --to 10.0.3.14:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 14443 -j DNAT --to 10.0.3.14:443

cn1#

Stuck into /etc/rc.local:

/sbin/iptables -F

/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A INPUT -p icmp -j ACCEPT
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.0.0/24 -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.3.0/24 -j ACCEPT
/sbin/iptables -A INPUT -s 140.211.11.9 -j ACCEPT
/sbin/iptables -A INPUT -s 145.72.98.1 -j ACCEPT
/sbin/iptables -A INPUT -s 178.18.80.48 -j ACCEPT
/sbin/iptables -A INPUT --match multiport -p tcp --dports 80,443 -j ACCEPT
/sbin/iptables -A INPUT --match multiport -p udp --dports 80,443 -j ACCEPT
#/sbin/iptables -A INPUT -j LOG
/sbin/iptables -A INPUT -j DROP

Miscellaneous#

lxc-rsync#

To be able to quickly "copy/clone" lxc's (while having their own filesystem/lv already) :

root@apollo:~/bin# df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/sda7                 9.2G  5.3G  3.5G  61% /
udev                      971M   12K  971M   1% /dev
tmpfs                     389M  408K  389M   1% /run
none                      5.0M     0  5.0M   0% /run/lock
none                      972M     0  972M   0% /run/shm
cgroup                    972M     0  972M   0% /sys/fs/cgroup
/dev/mapper/vg0-lvol.cn1   20G  7.4G   12G  40% /var/lib/lxc/cn1
/dev/mapper/vg0-lvol.cn4  5.0G  2.0G  2.7G  43% /var/lib/lxc/cn4
/dev/mapper/vg0-lvol.cn2   20G  6.7G   13G  36% /var/lib/lxc/cn2
/dev/mapper/vg0-lvol.cn3  3.0G   69M  2.8G   3% /var/lib/lxc/cn3

I created the following ~root/bin/lxc-rsync script :

#!/bin/bash
#
# rsync on lxc with another
# args: <src lxc> <tgt lxc>
#
NUMARGS=$#
if [ $NUMARGS -ne 2 ]; then
  echo "Usage: lxc-rsync <src lxc> <tgt lxc>"
  exit 8
fi
SRCLXC=$1
TGTLXC=$2
SRCDIR=/var/lib/lxc/${SRCLXC}/rootfs
TGTDIR=/var/lib/lxc/${TGTLXC}/rootfs
if [ -d $SRCDIR -a -d $TGTDIR ]; then
  echo "rsyncing from $SRCLXC to $TGTLXC"
  cd $SRCDIR || exit 8
  # save the old IP address
  OLDIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`

  rsync --exclude "tmp" --exclude "dev" --exclude "media" --exclude "mnt" --exclude "proc" --exclude "sys" --exclude "var/run" --verbose --recursive --links --perms --acls --times --owner --group --one-file-system --delete . $TGTDIR

  #  patching hostname and IP address
  echo "patching /etc/hostname, /etc/hosts and /etc/network/interfaces ==> $TGTLXC / $OLDIP"
  sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hostname
  sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hosts
  WRONGIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`
  sed --in-place s/${WRONGIP}/${OLDIP}/g ${TGTDIR}/etc/network/interfaces
else
  echo "either $SRCDIR or $TGTDIR does not exist"
  exit 8
fi
But this rsync does not properly clone, for example /run/zabbix /var/run/zabbix fails, and mysqld does not start. Therefore I created lxc-copy that uses fsarchiver to do the heavy lifting.

lxc-copy#

The following script (takes longer but) works better :


fail2ban#

TODO yet