!!!  Computerhok 2013

[{TableOfContents }]

!! Intro
The old computerhok is running for almost 5 years now (Ubuntu 8.04, both hypervisor and guests), time for something new.

%%warning We went live on Sunday 2013-02-24! %%
!! Functions to be ported from old to new

* Gallery 
* JSPWiki, including apache frontend
* home for geocaching ''hof'' images
* nagios (on a separate guest or on hypervisor?)
* fail2ban like solution (see current {{~~/bin/cron/blockHost.sh}})
* backuphost for krm2DB
* a backup hardware node on remote location, and __easy__ backup
* dhfds
* [http://www.esthercreations.nl] can retire
* webmin still necessary ?
* mail server (not open relay!, see /etc/postfix/main.cf)
* [uptime script|http://www.computerhok.nl/cgi-bin/user/uptime.cgi] 
* OpenVZ logo should go from homepage

!! VirtualBox

We will no longer use OpenVZ, it is too restrictive in choice of guest OS'es.\\
We intend to use [VirtualBox|VirtualBox-4.1] as hypervisor, and Ubuntu 12.04 LTS as both host and guest OS.

%%warning
After doing some experiments I decided to leave Virtualbox:
* it is poorly documented (only "click here, click that", no good reference of all command options)
* you cannot resize harddisks (anymore ?)
* unstable, vboxdrv driver modules disappearing for unknown reasons

I stumbled upon __[lxc|http://lxc.sourceforge.net/]__, so I started figuring out what that offers, see here [my experiments with lxc|lxc].
%%
!! Work in progress / install log

! Ubuntu 12.04.1 LTS

* Installed Ubuntu 12.04.1 LTS from USB stick.
** LVM setup and 10GB root partitiion
** temporary wlan0 (wireless) as primary interface
** IP address 10.0.0.150
** only SSH server
** Locales: en_US.UTF-8 and nl_NL.UTF-8 (first one default)

! kernel recompile
* recompile kernel because of [DSDT issue|Installatie Ubuntu#KernelCompile]

! VirtualBox installation

 __[VBoxManage syntax]__
 
 
* install packages
** add {{deb http://download.virtualbox.org/virtualbox/debian precise contrib}}
** add the apt key : [sudo apt-key add oracle_vbox.asc|http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc] 
** {{apt-get install virtualbox-4.2}}
* install extension pack
{{{
root@apollo:~/Downloads# sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.2.6-82870.vbox-extpack 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully installed "Oracle VM VirtualBox Extension Pack".
root@apollo:~/Downloads# 
}}}
* create test VM: \\
{{{root@apollo:~# VBoxManage createvm --name Ubuntu1 --ostype Linux --register 
Virtual machine 'Ubuntu1' is created and registered.
UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019
Settings file: '/root/VirtualBox VMs/Ubuntu1/Ubuntu1.vbox'
root@apollo:~#
}}}
* modify attributes (more memory, VT off, network bridged instead of NAT:
{{{
VBoxManage modifyvm Ubuntu1 --memory=512 --hwvirtex=off --hwvirtexexcl=off --vtxvpid=off --boot1=dvd --boot2=disk --boot3=none --nic1=bridged --bridgeadapter1=wlan0 --nicpromisc1=allow-all --vrde=on --vrdeport=13389
}}}
* Deleted the kernel source again to free up some disk space:
{{{
root@apollo:~/src# du -cms linux-source-3.2.0/
5172	linux-source-3.2.0/
5172	total
root@apollo:~/src# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       9.2G  6.6G  2.2G  76% /
root@apollo:~/src# rm -rf linux-source-3.2.0/
root@apollo:~/src# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       9.2G  1.6G  7.2G  18% /
}}}
* Add CD and disk controller:
{{{
VBoxManage storagectl Ubuntu1 --name='IDE Controller' --add=ide --controller=PIIX4 --bootable=on
VBoxManage storagectl Ubuntu1 --name='SATA Controller' --add=sata --controller=IntelAhci --bootable=on
}}}
* show me how you look now:
{{{
root@apollo:~/VirtualBox VMs/Ubuntu1# VBoxManage showvminfo Ubuntu1
Name:            Ubuntu1
Groups:          /
Guest OS:        Other Linux
UUID:            1ed5e417-1eca-4a60-b6d4-5c8f40e44019
Config file:     /root/VirtualBox VMs/Ubuntu1/Ubuntu1.vbox
Snapshot folder: /root/VirtualBox VMs/Ubuntu1/Snapshots
Log folder:      /root/VirtualBox VMs/Ubuntu1/Logs
Hardware UUID:   1ed5e417-1eca-4a60-b6d4-5c8f40e44019
Memory size:     512MB
Page Fusion:     off
VRAM size:       8MB
CPU exec cap:    100%
HPET:            off
Chipset:         piix3
Firmware:        BIOS
Number of CPUs:  1
Synthetic Cpu:   off
CPUID overrides: None
Boot menu mode:  message and menu
Boot Device (1): DVD
Boot Device (2): HardDisk
Boot Device (3): Not Assigned
Boot Device (4): Not Assigned
ACPI:            on
IOAPIC:          off
PAE:             off
Time offset:     0ms
RTC:             local time
Hardw. virt.ext: off
Hardw. virt.ext exclusive: off
Nested Paging:   on
Large Pages:     off
VT-x VPID:       off
State:           powered off (since 2013-01-26T17:41:58.000000000)
Monitor count:   1
3D Acceleration: off
2D Video Acceleration: off
Teleporter Enabled: off
Teleporter Port: 0
Teleporter Address: 
Teleporter Password: 
Tracing Enabled: off
Allow Tracing to Access VM: off
Tracing Configuration: 
Autostart Enabled: off
Autostart Delay: 0
Storage Controller Name (0):            IDE Controller
Storage Controller Type (0):            PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  2
Storage Controller Port Count (0):      2
Storage Controller Bootable (0):        on
Storage Controller Name (1):            SATA Controller
Storage Controller Type (1):            IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1):  30
Storage Controller Port Count (1):      30
Storage Controller Bootable (1):        on
NIC 1:           MAC: 0800276CD273, Attachment: Bridged Interface 'wlan0', Cable connected: on, Trace: off (file: none), Type: Am79C973, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
Pointing Device: PS/2 Mouse
Keyboard Device: PS/2 Keyboard
UART 1:          disabled
UART 2:          disabled
LPT 1:           disabled
LPT 2:           disabled
Audio:           disabled
Clipboard Mode:  disabled
Drag'n'drop Mode:  disabled
VRDE:            enabled (Address 0.0.0.0, Ports 13389, MultiConn: off, ReuseSingleConn: off, Authentication type: null)
Video redirection: disabled
USB:             disabled
EHCI:            disabled

USB Device Filters:

<none>

Available remote USB devices:

<none>

Currently Attached USB Devices:

<none>

Bandwidth groups:  <none>

Shared folders:  <none>

VRDE Connection:    not active
Clients so far:     0

Guest:

Configured memory balloon size:      0 MB
}}}
* attach the CD image
{{{
VBoxManage storageattach Ubuntu1 --storagectl='IDE Controller' --port=1 --device=1 --type=dvddrive --medium=/tmp/ff/ubuntu-12.04.1-server-i386.iso
}}}
* create HD:
{{{
VBoxManage createhd --filename '/root/VirtualBox VMs/Ubuntu1/disk1.vdi' --size=15000
}}}
* attach HD:
{{{
VBoxManage storageattach Ubuntu1 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium='/root/VirtualBox VMs/Ubuntu1/disk1.vdi'
}}}
* fire up the VM :
{{{
root@apollo:~/Downloads# VBoxHeadless --startvm Ubuntu1 
Oracle VM VirtualBox Headless Interface 4.2.6
(C) 2008-2012 Oracle Corporation
All rights reserved.

VRDE server is listening on port 13389.
}}}

! VirtualBox cloning/snapshotting

Take a snapshot of a running VM:
{{{
VBoxManage snapshot Ubuntu1 take snapshot01 --description "first snapshot, almost vanilla ubuntu" 
}}}

This commands ends fine, but the VM hangs, even no response from external pings anymore.
VRDP just shows a black screen.\\
Looks like it is paused (even if not specified on the snapshot subcommand):
{{{
root@apollo:~# VBoxManage showvminfo Ubuntu1 --details|grep -i state
State:           paused (since 2013-01-27T15:23:36.808000000)
root@apollo:~# 
}}}
So try to resume it:
root@apollo:~# VBoxManage controlvm Ubuntu1 resume
root@apollo:~# VBoxManage showvminfo Ubuntu1 --details|grep -i state
State:           running (since 2013-01-27T15:40:09.024000000)
root@apollo:~# 
}}}
And yes it is running again.

Now clone this snapshot to a second machine:
{{{
root@apollo:~# VBoxManage clonevm Ubuntu1 --snapshot snapshot01 --options keepdisknames --name Ubuntu2 --register
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Machine has been successfully cloned as "Ubuntu2"
root@apollo:~# 
}}}
(This takes a few minutes).\\
But then we have :
{{{
root@apollo:~# VBoxManage list vms
"Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019}
"Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1}
root@apollo:~# 
}}}

Change the vrdeport (should not be the same as the first machine):\\
{{VBoxManage modifyvm Ubuntu2 --vrdeport 13390}}

And fire up the thing:
{{{
root@apollo:~# VBoxHeadless --startvm Ubuntu2 
Oracle VM VirtualBox Headless Interface 4.2.6
(C) 2008-2012 Oracle Corporation
All rights reserved.

VRDE server is listening on port 13390.
}}}

The first boot takes a bit longer, because of an fsck that runs because it thinks the fs was not cleanly unmounted.\\
First login through the VRDP console and change hostname with {{hostname ubuntu2}} and editing /etc/hostname.\\
I also note that network device eth0 does not come up.  
dmesg shows that udev has renamed eth0 to eth1, that sounds familiar: remove {{ /etc/udev/rules.d/70-persistent-net.rules}}

* listing HD's and cloning a harddisk:
{{{
root@apollo:~# vb list hdds
UUID:        fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/disk1.vdi
State:       locked read
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)]

UUID:        88a8d294-cb4c-4881-b42b-286b25b3ab3e
Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019)

UUID:        fb289305-27fc-4344-bfb2-75942cbb1252
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu2/disk1.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1)

root@apollo:~# vb clonehd fc263d6a-18d4-4bda-a987-d8ffdc2a11b3 /tmp/cloned.from.ubuntu1
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'VDI'. UUID: 8a06d36c-13f0-4275-abe7-ebc64bff59a4
root@apollo:~# vb list hdds
UUID:        fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/disk1.vdi
State:       locked read
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019) [snapshot01 (UUID: 36426ba8-c819-45a3-87e5-e0d599cfc309)]

UUID:        88a8d294-cb4c-4881-b42b-286b25b3ab3e
Parent UUID: fc263d6a-18d4-4bda-a987-d8ffdc2a11b3
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu1/Snapshots/{88a8d294-cb4c-4881-b42b-286b25b3ab3e}.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu1 (UUID: 1ed5e417-1eca-4a60-b6d4-5c8f40e44019)

UUID:        fb289305-27fc-4344-bfb2-75942cbb1252
Parent UUID: base
Format:      VDI
Location:    /root/VirtualBox VMs/Ubuntu2/disk1.vdi
State:       locked write
Type:        normal
Usage:       Ubuntu2 (UUID: e82b1ffb-21be-45e1-b467-c84cf5fda1a1)

UUID:        8a06d36c-13f0-4275-abe7-ebc64bff59a4
Parent UUID: base
Format:      VDI
Location:    /tmp/cloned.from.ubuntu1
State:       created
Type:        normal
}}}
Now, you cannot just copy these files over :

{{{
root@apollo:~/VirtualBox VMs/Ubuntu2# cat /tmp/cloned.from.ubuntu1 > disk1.vdi 
root@apollo:~/VirtualBox VMs/Ubuntu2# vb list vms
"Ubuntu1" {1ed5e417-1eca-4a60-b6d4-5c8f40e44019}
"Ubuntu2" {e82b1ffb-21be-45e1-b467-c84cf5fda1a1}

root@apollo:~/VirtualBox VMs/Ubuntu2# vb startvm Ubuntu2
Waiting for VM "Ubuntu2" to power on...
VBoxManage: error: The virtual machine 'Ubuntu2' has terminated unexpectedly during startup with exit code 0
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Machine, interface IMachine
root@apollo:~/VirtualBox VMs/Ubuntu2# 
}}}

So now "detach" this disk again by "attaching none" :
{{{
VBoxManage storageattach Ubuntu1 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium none
}}}
* also detach from Ubuntu2 and delete the disk :
{{{
VBoxManage storageattach Ubuntu2 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium=none
VBoxManage closemedium disk fb289305-27fc-4344-bfb2-75942cbb1252 --delete
}}}
* and attach the snapshotted disk to Ubuntu2 (I first renamed/moved the cloned hdd), and make it bootable:
{{{
VBoxManage storageattach Ubuntu2 --storagectl='SATA Controller' --port=1 --device=0 --type=hdd --medium='/root/VirtualBox VMs/Ubuntu2/disk1.vdi'
VBoxManage modifyvm Ubuntu2 --boot1 disk
}}}

* After booting the machine:
** remove the {{/etc/udev/rules.d/70...net...}} file. 
** update the {{/etc/network/interfaces}} to make it a static IP adres:
{{{
#iface eth0 inet dhcp
iface eth0 inet static
        address 10.0.0.155
        netmask 255.255.255.0
        network 10.0.0.0
        broadcast 10.0.0.255
        gateway 10.0.0.138
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 213.197.28.3 213.197.30.28
        dns-search computerhok.nl
}}}
** change the {{/etc/hostname}}

!! Extra installed packages

! Extra installed packages for the Host
* See [DSDT issue|Installatie Ubuntu#KernelCompile]
* vim htop openjdk-7-jdk wget mailutils lsof uuid fsarchiver
! Extra installed packages for the container(s)

* htop apache2 aptitude unzip locate fping postfix telnet mailutils lsof
{{{
sudo sh -c 'echo "deb http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu precise main" >> /etc/apt/sources.list.d/mariadb.list' 
}}}
* mariadb-server mariadb-client
* imagemagick
* php5-imagick
* man-db
!! Install/migrate Gallery

* Download the Gallery 3 zip
* unpack to /var/www/gallery3
* {{mkdir /var/www/gallery3/var}}
* {{chown www-data.www-data /var/www/gallery3}}
* install mariadb (root pw  see keepass)
*
{{{
MariaDB [(none)]> create user 'gallery3'@'localhost' identified by "gallery3pw";
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database gallery3;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all privileges on gallery3.* to gallery3;
Query OK, 0 rows affected (0.00 sec)
}}}
* install php stuff:
* add {{deb http://ppa.launchpad.net/ondrej/php5/ubuntu precise main}} to /etc/apt/sources.list
{{{
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4F4EA0AAE5267A6C
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CBCB082A1BB943DB
apt-get update
}}}
* apt-get install php5-common libapache2-mod-php5 php5-mysql php5-gd

Now before copying over photo's from the old site, we first have to increase the filesystem space.
* shut down container
* {{lvcreate --size=20G --name=lvol.cn1 vg0}}
* {{cd /var/lib/lxc }}
* {{mv cn1 cn1.ff && mkdir cn1}}
* {{mkfs.ext4 /dev/vg0/lvol.cn1}}
* update {{/etc/fstab}} and issue {{mount -a}}
* {{cd /var/lib/lxc && cp -pR cn1.ff/* cn1}}

! Forget about gallery3

After a couple of hours trying/googling I gave up on migrating from gallery2 to gallery3.

!! zabbix 2.0 experiment

cn4 cloned for this purpose

* installed mariadb
* downloaded and untarred zabbix-server 2.0
* group and user zabbix created
* installed mariadb, including the devel package {{libmariadbclient-dev}}
* follow the default installation procedure from the zabbix website
** create user zabbix
** create database zabbix, dbuser zabbix/zabbixpw
** loaded the tables
** modified {{/etc/php5/apache2/php.ini}}
** created dir {{/var/log/zabbix}}
** modified {{/usr/local/etc/zabbix*}}
** (copied from ./misc dir) to /etc/init : upstart config files (adding ''setuid zabbix'')

!! firewall configurations

! apollo

Stuck into /etc/rc.local:
%%small
{{{
/sbin/iptables -F
/sbin/iptables -F -t nat

/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A INPUT -p icmp -j ACCEPT
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -i lxcbr0 -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.0.0/24 -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.3.1/24 -j ACCEPT
/sbin/iptables -A INPUT -s 140.211.11.9 -j ACCEPT
/sbin/iptables -A INPUT -s 145.72.98.1 -j ACCEPT
#/sbin/iptables -A INPUT -j LOG
/sbin/iptables -A INPUT -j DROP

/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1122 -j DNAT --to 10.0.3.11:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1125 -j DNAT --to 10.0.3.11:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1180 -j DNAT --to 10.0.3.11:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 11443 -j DNAT --to 10.0.3.11:443
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 11808 -j DNAT --to 10.0.3.11:8080
#
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1222 -j DNAT --to 10.0.3.12:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1225 -j DNAT --to 10.0.3.12:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1280 -j DNAT --to 10.0.3.12:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 12443 -j DNAT --to 10.0.3.12:443
#
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1322 -j DNAT --to 10.0.3.13:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1325 -j DNAT --to 10.0.3.13:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1380 -j DNAT --to 10.0.3.13:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 13443 -j DNAT --to 10.0.3.13:443
#
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1422 -j DNAT --to 10.0.3.14:22
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1425 -j DNAT --to 10.0.3.14:25
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 1480 -j DNAT --to 10.0.3.14:80
/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 14443 -j DNAT --to 10.0.3.14:443

}}}
%%

! cn1

Stuck into /etc/rc.local:
%%small
{{{
/sbin/iptables -F

/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A INPUT -p icmp -j ACCEPT
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.0.0/24 -j ACCEPT
/sbin/iptables -A INPUT -s 10.0.3.0/24 -j ACCEPT
/sbin/iptables -A INPUT -s 140.211.11.9 -j ACCEPT
/sbin/iptables -A INPUT -s 145.72.98.1 -j ACCEPT
/sbin/iptables -A INPUT -s 178.18.80.48 -j ACCEPT
/sbin/iptables -A INPUT --match multiport -p tcp --dports 80,443 -j ACCEPT
/sbin/iptables -A INPUT --match multiport -p udp --dports 80,443 -j ACCEPT
#/sbin/iptables -A INPUT -j LOG
/sbin/iptables -A INPUT -j DROP
}}}
%%

!! Miscellaneous

* set timezone on containers, put ''Europe/Amsterdam'' to ''/etc/timezone'' and run ''dpkg-reconfigure --frontend noninteractive tzdata''.
* zabbix MySQL support, adjust ''/usr/local/etc/zabbix_agentd.conf.d/userparameter_mysql.conf'' and create ~~zabbix/.my.cnf with user/pw in it

! lxc-rsync

To be able to quickly "copy/clone" lxc's (while having their own filesystem/lv already) :
%%small
{{{
root@apollo:~/bin# df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/sda7                 9.2G  5.3G  3.5G  61% /
udev                      971M   12K  971M   1% /dev
tmpfs                     389M  408K  389M   1% /run
none                      5.0M     0  5.0M   0% /run/lock
none                      972M     0  972M   0% /run/shm
cgroup                    972M     0  972M   0% /sys/fs/cgroup
/dev/mapper/vg0-lvol.cn1   20G  7.4G   12G  40% /var/lib/lxc/cn1
/dev/mapper/vg0-lvol.cn4  5.0G  2.0G  2.7G  43% /var/lib/lxc/cn4
/dev/mapper/vg0-lvol.cn2   20G  6.7G   13G  36% /var/lib/lxc/cn2
/dev/mapper/vg0-lvol.cn3  3.0G   69M  2.8G   3% /var/lib/lxc/cn3
}}}
%%

I created the following ~root/bin/lxc-rsync script :
%%small
%%prettify
{{{
#!/bin/bash
#
# rsync on lxc with another
# args: <src lxc> <tgt lxc>
#
NUMARGS=$#
if [ $NUMARGS -ne 2 ]; then
  echo "Usage: lxc-rsync <src lxc> <tgt lxc>"
  exit 8
fi
SRCLXC=$1
TGTLXC=$2
SRCDIR=/var/lib/lxc/${SRCLXC}/rootfs
TGTDIR=/var/lib/lxc/${TGTLXC}/rootfs
if [ -d $SRCDIR -a -d $TGTDIR ]; then
  echo "rsyncing from $SRCLXC to $TGTLXC"
  cd $SRCDIR || exit 8
  # save the old IP address
  OLDIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`

  rsync --exclude "tmp" --exclude "dev" --exclude "media" --exclude "mnt" --exclude "proc" --exclude "sys" --exclude "var/run" --verbose --recursive --links --perms --acls --times --owner --group --one-file-system --delete . $TGTDIR

  #  patching hostname and IP address
  echo "patching /etc/hostname, /etc/hosts and /etc/network/interfaces ==> $TGTLXC / $OLDIP"
  sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hostname
  sed --in-place s/${SRCLXC}/${TGTLXC}/g ${TGTDIR}/etc/hosts
  WRONGIP=`grep address ${TGTDIR}/etc/network/interfaces | awk '{ print $NF}'`
  sed --in-place s/${WRONGIP}/${OLDIP}/g ${TGTDIR}/etc/network/interfaces
else
  echo "either $SRCDIR or $TGTDIR does not exist"
  exit 8
fi
}}}
%%
%%

%%warning But this rsync does not properly clone, for example /run/zabbix /var/run/zabbix fails, and mysqld does not start. %%
Therefore I created lxc-copy that uses __fsarchiver__ to do the heavy lifting.

! lxc-copy

The following script (takes longer but) works better :

%%prettify
{{{
}}}
%% 
! fail2ban

%%warning TODO yet %%