So i have spent a good part of yesterday afternoon and today morning setting up all this. Disclaimer: i do not pretend to know anything about network security. I don’t. With that said, the setup is as follows:

I will call the machines main, firewall, public and the rest network will be referred to as “internal”. The three machines here are all running debian sid.

The xen networking setup is as follows: there are 3 bridges:

/etc/xen/scripts/network-bridge start bridge=xenbrI netdev=eth1 vifnum=1
/etc/xen/scripts/network-bridge start bridge=xenbrE netdev=eth0 vifnum=0
brctl addbr xenbrD
ip link set xenbrD up

I put (vif-script vif-bridge) into /etc/xen/xend-config.sxp and comment out network-script completely (i run the equivalent of the above from initscripts).

Here, xenbrI is for the internal network — it has internal interface of the box attached to it (the other end of that is attached to physical switch), as well as eth1 of main and eth1 of firewall.

xenbrE is the external network: there is some trickery involved in setting this one up: it has eth0 of main attached to it, then eth0 of firewall and peth0 which is the physical interface connected to cable modem. I had to set the mac address of eth0 on firewall to be the same as mac address of my physical ethernet card attached to cable modem. I am not sure if this is only needed because my dear ISP only allows access to the lease from my mac address or there are some deeper ethernet routing problems otherwise. You can try if you have more liberal provider.

Finally, xenbrD is the demilitarized zone switch — this one is purely virtual — no physical interfaces attached. It will be used by public to talk to firewall.

As you can see, the firewall domU has 3 virtual ethernet devices, one attached to each of the bridges. It uses dhcp on its eth0 (attached to xenbrE) to get a lease from provider. In my setup, it also gets a lease for eth1 from my internal dhcpd. That’s pretty optional though. The “vif” directive of xen config for firewall looks roughly like this: vif = [ 'mac=my-external-mac,bridge=xenbrE', 'mac=make-up-something,bridge=xenbrI', 'mac=make-up-something-else,bridge=xenbrD' ]

Before bringing firewall up, you probably want to shut down eth0 on main. You don’t want them competing for the ISP-assigned lease. When it is up, the internet routing looks like this: packets from cable modem hit eth0 on main, pass to xenbrE and from there to eth0 on firewall, not bouncing into the tcp/ip stack of main (note: the bridge actually passes packets through iptables… however, neither xenbrE nor eth0 on main have IP address so there is no way i can think of they could hit anything in main… the only problem is that you need to allow FORWARDing of traffic to/from the respective physical interfaces… i currently have empty (accept-all) iptables on main so i am not affected). The firewall picks them on eth0 and “does things”.

I have packets from internal network routed through main to eth1 on firewall, but for most setups i guess you can just set firewall’s internal IP address (the one on eth1) as gateway.

After a bit of playing with different firewall software, i have ended up installing shorewall on firewall. I set up 3 zones in /etc/shorewall/zones, one for each interface (the fw zone concerns the firewall itself):

fw firewall
int ipv4 # internal
dmz ipv4 # dmz
ext ipv4 # external

And the accompanying /etc/shorewall/interfaces:

#ZONE   IFACE   BCAST   OPT
ext     eth0    detect  dhcp
int     eth1    detect  dhcp
dmz     eth2    detect

The policies are under /etc/shorewall/policy:

#SRC    DEST         POLICY          LOG             LIMIT:BURST
ext     fw           DROP            info
ext     int          DROP            info
int     int          NONE            info
int     ext          ACCEPT
dmz     ext          ACCEPT
int     dmz          ACCEPT
all     fw           ACCEPT
fw      all          ACCEPT
all     all          REJECT          info

Since my ISP only gives me single IP address, i need to masquerade outgoing connections (/etc/shorewall/masq, <internal-network> and <dmz-network> could be eg. 192.168.1.0/24 and 192.168.2.0/24 — whatever you are using):

eth0    <internal-network>
eth0    <dmz-network>

I have some rules under /etc/shorewall/rules, but most these are not a concern of anyone i guess — they are pretty specific to my setup. If you need to set up some, shorewall has a great manual and a good set of macros to make this easy. What you will probably find useful is how to do DNAT (destination network address translation — the “public” there is the virtual machine i mentioned at the start):

DNAT    ext     dmz:<ip of "public"> tcp    80  -
DNAT    ext     dmz:<ip of "public"> tcp    ftp -

Obviously, you need to enable shorewall startup. Don’t forget to set FORWARDING=On in shorewall.conf — unless you enable it by other means, it won’t work without :).

If you have set up everything correctly, you should be able to start your shorewall: invoke-rc.d shorewall start. Both main and your internal network should be able to ping out into the wild internet. Yay.

So now you only need to bring up your public and set up services. The “vif” of the xen config for public looks like vif = [ 'mac=make-up-some,bridge=xenbrD' ]. I use static IP allocation in the DMZ and i put it in a different IP subnet than internal network. If you have set up the routes right, your public host should be able to ping out. There is one additional catch with networking in domUs. If your network can ping but nothing much else, try to run ethtool -K eth0 tx off — this turns off checksum checking or somesuch which makes the interface drop packets for no apparent reason. Google to find out more. When you have done this, you should be able to get to your services on public from outside, assuming you set up the right DNAT rules on firewall.

I think that’s more than enough for a blog now. I will continue to write down my experiences with either selinux or oprofile, when i find time for that. If people are interested in a more detailed account/guide on setting up individual Xen hosts, i could serve, just let me know.

I better go doing something productive now :-).