How to Build an 802.1q LACP Trunk for Xen in CentOS 5.5

Overview

In this how-to we’re going to install CentOS 5.5 and prep it to become a virtual machine server. Before proceeding PLEASE read our Xen software page so you understand how the VM and network topology works. This how-to was built combining various methods publicly already available. If you find some copy and pasted text just remember: a good sysadmin is a lazy sysadmin.

Prerequisites

LACP bonded 802.1q trunk with necessary VLAN’s assigned

Installation

Install CentOS 5.5, during the package selection screen make sure to enable the Virtualization option.

Configuration

First let’s make sure that xen is running properly. Log into the GUI of the server, if you do not have a GUI then log into an ssh session as root with X forwarding on. Run virt-manager from the console. You should see “dom0” running, if you double click on it you’ll be able to see the statistics for this domain and how much resources it is consuming. If you cannot see this or if virt-manager does not find a dom0 to connect to you have done something wrong. I cannot help you.

1. Physical Networking Devices
What we first need to do is setup our networking devices.
Edit some files in /etc/sysconfig/network-scripts. Remember that bond0 is the physical interface that the trunk is connected to therefore it does not need an IP address.
# ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
TYPE=BOND
BONDING_OPTS="mode=4"


#ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
TYPE=Ethernet
HWADDR:(MAC-ADDRESS OF ETH0)

#ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
TYPE=Ethernet
HWADDR:(MAC-ADDRESS OF ETH1F)

Then for every VLAN that you are passing in your trunk create a file as follows. Please replace the ‘N’ with the VLAN ID number

[root@xentest jeunson]# more /etc/sysconfig/network-scripts/ifcfg-bond0.N
DEVICE=bond0.N
BOOTPROTO=none
#IPADDR=192.168.100.51
#NETMASK=255.255.255.0
ONBOOT=yes
VLAN=yes
TYPE=BOND


Note how the IP address is commented out on this VLAN interface. You can assign an IP address to the VLAN interface that will be the management interface for the hardware.

Finally don’t forget to add the following line to /etc/modprobe.conf

alias bond0 bonding


Now that you’ve created the VLANs and assigned a management interface we edit the file /etc/sysconfig/network

[root@xentest jeunson]# more /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=xentest.example.com
GATEWAY=192.168.101.254
GATEWAYDEV=bond0.101


Now restart the networking with /etc/init.d/network restart It should restart with out any errors

Shutting down interface bond0.100: Removed VLAN -:bond0.100:-
[ OK ]
Shutting down interface bond0.101: Removed VLAN -:bond.101:-
[ OK ]
Shutting down interface bond: [ OK ]
Shutting down loopback interface: [ OK ]
Disabling IPv4 packet forwarding: net.ipv4.ip_forward = 0 [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface bond0.100: Added VLAN with VID == 100 to IF -:bond0:-
[ OK ]
Bringing up interface bond0.101: Added VLAN with VID == 101 to IF -:bond0:-
[ OK ]


2. Xen Networking Scripts
First modify /etc/xen/xend-config.asp and change the netdev device from eth0 to bond0. You’ll probably have to reboot the computer.

A.Modify the network-bridge script

Now that we’re done that we need to configure Xen to tell it what networking interfaces to use. First make a backup of the xen network-bridge script and edit as follows.
cp /etc/xen/scripts/network-bridge /etc/xen/scripts/network-bridge.orig
Change around line 78 from
[ -f "/sys/class/net/$1/bonding/slaves" ]
to
[ -f "/sys/class/net/$1/bonding/slaves" ] || [ -f /proc/net/vlan/$1 ]

B. Modfiy xend-config.sxp
The second change involves modifying the xend configuration file to use a new network initialization script.
[root@xentest scripts]# cd /etc/xen
[root@xentest scripts]# cp -p xend-config.sxp xend-config.sxp-dist
Change the network-script configuration directive to the new script name – such as network-multi-bridge. After the modification:
[root@xentest xen]# diff xend-config.sxp-dist xend-config.sxp
91c91
< (network-script network-bridge) --- > (network-script network-multi-bridge)
C. Create network-multi-bridge script
This script uses the two VLAN interfaces previously defined.

[root@xentest scripts]# cd /etc/xen/scripts
[root@xentest scripts]# more network-multi-bridge
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=bond0 bridge="xbr_trunk"

/sbin/ifup eth0.100
“$dir/network-bridge” “$@” vifnum=1 bridge=”xbr_v100″ netdev=”bond0.100″
/sbin/ifup eth0.101
“$dir/network-bridge” “$@” vifnum=2 bridge=”xbr_v101″ netdev=”bond0.101″

At this point you should reboot your server to make sure that everything comes up cleanly. Use the logs to debug problems and look out for which type of bond your computer connects to the switch as.