Xen 3.0, nx, and FC4

We recently upgraded our server to Fedora Core 4 (FC4) which neatly supports Xen. I think Xen is a much cleaner solution for our remote access service than jail -- furthermore I couldn't use the jail configuration I had for Redhat so I just embarked on the new Xen architecture.

The latest 3.0 rpms from Xen-Source worked right out of the box -- there are just two things you need to change:

The entry in /etc/grub.conf looks like:

title Xen (2.6.12.6-xen3_2.1)
root (hd0,0)
kernel /xen-3.gz com1=115200,8n1
module /vmlinuz-2.6.12.6-xen3_2.1_fc4 root=/dev/VolGroup00/LogVol00
ro maxcpus=1 console=tty1 console=ttyS0,115200n8
module /initrd-2.6.12.6-xen3_2.1_fc4.img

Note the fc4 -- the documentation just forgets them and nothing works.

The second thing you need to do is replace the /etc/xen/scripts/network-bridge with the following:
#!/bin/sh -x
#============================================================================
# Default Xen network start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates a bridge (default xen-br0), adds a device
# (default eth0) to it, copies the IP addresses from the device
# to the bridge and adjusts the routes accordingly.
#
# If all goes well, this should ensure that networking stays up.
# However, some configurations are upset by this, especially
# NFS roots. If the bridged setup does not meet your needs,
# configure a different script, for example using routing instead.
#
# Usage:
#
# network (start|stop|status) {VAR=VAL}*
#
# Vars:
#
# bridge The bridge to use (default xen-br0).
# netdev The interface to add to the bridge (default eth0).
# antispoof Whether to use iptables to prevent spoofing (default yes).
#
# start:
# Creates the bridge and enslaves netdev to it.
# Copies the IP addresses from netdev to the bridge.
# Deletes the routes to netdev and adds them on bridge.
#
# stop:
# Removes netdev from the bridge.
# Deletes the routes to bridge and adds them to netdev.
#
# status:
# Print ifconfig for netdev and bridge.
# Print routes.
#
#============================================================================

# Exit if anything goes wrong.
set -e

# First arg is the operation.
OP=$1
shift

# Pull variables in args in to environment.
for arg ; do export "${arg}" ; done

bridge=${bridge:-xenbr0}
netdev=${netdev:-eth0}
antispoof=${antispoof:-no}

echo "*network $OP bridge=$bridge netdev=$netdev antispoof=$antispoof" >&2

# Usage: transfer_addrs src dst
# Copy all IP addresses (including aliases) from device $src to device $dst.
transfer_addrs () {
local src=$1
local dst=$2
# Don't bother if $dst already has IP addresses.
if ip addr show dev ${dst} | egrep -q '^ *inet ' ; then
return
fi
# Address lines start with 'inet' and have the device in them.
# Replace 'inet' with 'ip addr add' and change the device name $src
# to 'dev $src'.
ip addr show dev ${src} | egrep '^ *inet ' | sed -e "
s/inet/ip addr add/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+/[0-9]\+\)@\1@
s/${src}/dev ${dst}/
" | sh -e
# Remove automatic routes on destionation device
ip route list | sed -ne "
/dev ${dst}\( \|$\)/ {
s/^/ip route del /
p
}" | sh -e
}

# Usage: del_addrs src
del_addrs () {
local src=$1
ip addr show dev ${src} | egrep '^ *inet ' | sed -e "
s/inet/ip addr del/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${src}/
" | sh -e
}

# Usage: transfer_routes src dst
# Get all IP routes to device $src, delete them, and
# add the same routes to device $dst.
# The original routes have to be deleted, otherwise adding them
# for $dst fails (duplicate routes).
transfer_routes () {
local src=$1
local dst=$2
# List all routes and grep the ones with $src in.
# Stick 'ip route del' on the front to delete.
# Change $src to $dst and use 'ip route add' to add.
ip route list | sed -ne "
/dev ${src}\( \|$\)/ {
h
s/^/ip route del /
P
g
s/${src}/${dst}/
s/^/ip route add /
P
d
}" | sh -e
}

# Usage: create_bridge bridge
create_bridge () {
local bridge=$1

# Don't create the bridge if it already exists.
if ! brctl show | grep -q ${bridge} ; then
brctl addbr ${bridge}
brctl stp ${bridge} off
brctl setfd ${bridge} 0
fi
ifconfig ${bridge} up
}

# Usage: add_to_bridge bridge dev
add_to_bridge () {
local bridge=$1
local dev=$2
# Don't add $dev to $bridge if it's already on a bridge.
if ! brctl show | grep -q ${dev} ; then
brctl addif ${bridge} ${dev}
fi
}

# Usage: antispoofing dev bridge
# Set the default forwarding policy for $dev to drop.
# Allow forwarding to the bridge.
antispoofing () {
local dev=$1
local bridge=$2

iptables -P FORWARD DROP
iptables -A FORWARD -m physdev --physdev-in ${dev} -j ACCEPT
}

# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
local dev=$1
local bridge=$2

echo '============================================================'
ifconfig ${dev}
ifconfig ${bridge}
echo ' '
ip route list
echo ' '
route -n
echo '============================================================'
}

op_start () {
if [ "${bridge}" == "null" ] ; then
return
fi

create_bridge ${bridge}

if ifconfig 2>/dev/null | grep -q veth0 ; then
return
fi

if ifconfig veth0 2>/dev/null | grep -q veth0 ; then
mac=`ifconfig ${netdev} | grep HWadd | sed -e 's/.*\(..:..:..:..:..:..\).*/\1/'`
if ! ifdown ${netdev} ; then
# if ifup didn't work, see if we have an ip= on cmd line
if egrep 'ip=[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:' /proc/cmdline ;
then
kip=`sed -e 's!.*ip=\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\):.*!\1!' /proc/cmdline`
kmask=`sed -e 's!.*ip=[^:]*:[^:]*:[^:]*:\([^:]*\):.*!\1!' /proc/cmdline`
kgate=`sed -e 's!.*ip=[^:]*:[^:]*:\([^:]*\):.*!\1!' /proc/cmdline`
ifconfig ${netdev} 0.0.0.0 down
fi
fi
ip link set ${netdev} name p${netdev}
ip link set veth0 name ${netdev}
ifconfig p${netdev} 0.0.0.0 -arp down
ifconfig p${netdev} hw ether fe:ff:ff:ff:ff:ff
ifconfig ${netdev} hw ether ${mac}
add_to_bridge ${bridge} vif0.0
add_to_bridge ${bridge} p${netdev}
ip link set ${bridge} up
ip link set vif0.0 up
ip link set p${netdev} up
if ! ifup ${netdev} ; then
if [ ${kip} ] ; then
# use the addresses we grocked from /proc/cmdline
ifconfig ${netdev} ${kip}
[ ${kmask} ] && ifconfig ${netdev} netmask ${kmask}
ifconfig ${netdev} up
[ ${kgate} ] && ip route add default via ${kgate}
fi
fi
else
# old style without veth0
transfer_addrs ${netdev} ${bridge}
transfer_routes ${netdev} ${bridge}
fi

if [ ${antispoof} == 'yes' ] ; then
antispoofing ${netdev} ${bridge}
fi
}

op_stop () {
if [ "${bridge}" == "null" ] ; then
return
fi

brctl delif ${bridge} ${netdev}

if ifconfig veth0 2>/dev/null | grep -q veth0 ; then
brctl delif ${bridge} vif0.0
ifconfig vif0.0 down
mac=`ifconfig veth0 | grep HWadd | sed -e 's/.*\(..:..:..:..:..:..\).*/\1/'`
ifconfig ${netdev} down
ifconfig ${netdev} hw ether ${mac}
ifconfig ${netdev} arp up
transfer_addrs veth0 ${netdev}
transfer_routes veth0 ${netdev}
del_addrs veth0
ifconfig veth0 -arp down
ifconfig veth0 hw ether 00:00:00:00:00:00
else
transfer_routes ${bridge} ${netdev}
fi
}

case ${OP} in
start)
op_start
;;

stop)
op_stop
;;

status)
show_status ${netdev} ${bridge}
;;

*)
echo 'Unknown command: ' ${OP} >&2
echo 'Valid commands are: start, stop, status' >&2
exit 1
esac


For some voodoo reason you have to execute /etc/ini.d/xend restart to get the network set up.

To configure a DomU follow the description on the Fedora Wiki -- be aware that you will need probably at least 3 or 4 GigaByte to do something useful. Once your DomU works (you will need another "public" IP-adress) you can add nx like already described in this blog.