KVM & BRCTL in Linux – bringing VLANs to the guests

I recently had the opportunity to setup a KVM machine running on a Centos machine. Tools such as Virtual Machine Manager (VMM) help immensely in the provisioning and administration of virtual machine guests within KVM, and bring the KVM solution on par with other solutions such as Xen, VMWare and VirtualBox.

Networking is one of my main tasks when setting up such an environment, and my particular scenario presented a bit of a challenge. Most articles regarding KVM and VLANs deal with having the guests in “access” mode – that is they can only access a single VLAN. In my particular scenario (a virtual guest hosting a pfsense install) I needed to preserve the VLAN tagging across the virtual bridge, in other words, having the guest in “trunking” mode, making it vlan-aware.

This article explores my understanding of the whole setup above using linux’s inbuilt uml-tools, a.k.a. brctl. I will go over the “normal” access mode that is presented in most articles, as well as a way to put the guests in trunking mode, which is very much less well documented on the web. When reading through the article pay particular attention to the order of the individual components that make up the solution, as when troubleshooting this will be invaluable (hence my many colorful, if inept, diagrams in this article.

I found some very good documentation on several sites, though it took a while to find what I needed, the following was of particular interest:

http://nickapedia.com/2011/11/28/now-for-something-completely-different-ubuntu-11-10-kvm-vlan-trunking/

I will be using CentOS rather than Ubuntu in my article. However, do make note of the following settings:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
These lines disable any filtering of frames on the bridge devices.

The above are found under /proc/net/bridge in Centos

Starting Point.

Our starting point will be simply bridging all physical and virtual interfaces together. Throughout this article we will be discussing 4 main components:

– The physical network interface (eth0)

– Two virtual network interfaces, attached to two virtual guests (vNIC1 and vNIC2)

– A virtual bridge (BR0)

We would be aiming for something along the lines of:

brctl_starting

At this stage, the objective is to get eth0, vNIC1 and vNIC2 talking together. I wont focus on creating the actual KVM guests, plenty of documentation elsewhere on how to do that. We’ll keep our focus on the bridge, BR0. Creating the bridge is simple as is adding the member interfaces:

brctl add br0

brctl addif br0 eth0

brctl addif br0 vNIC1

brctl addif br0 vNIC2

With this setup, all three member interfaces can communicate together, like so:

brctl_simple

Adding VLANs to the mix – the usual guest access mode

Before continuing further, from this stage onwards make sure to have the 8021q module loaded in the kernel. At this stage, this is what we’re end up with:

brctl_access_guests

It’s a busy diagram so I’ll take some of your time to explain it. the physical interface, eth0, will be subdivided into vlan subinterfaces using the vconfig command. Tagged and untagged (native) traffic will enter on eth0. If a subinterface is defined for a particular vlan, that traffic is stripped of it’s vlan header, and presented untagged to the bridge. Any virtual machines which should have access to this vlan should be members of this bridge. So to achieve the above we first define the two eth0 vlan subinterfaces, for vlan 1 (eth0.1) and vlan 5 (eth0.5):

vconfig add eth0 1

vconfig add th0 5

ifconfig eth0.1 up

ifconfig eth0.5 up

You can check that the vlans have been correctly setup by running the command:

cat /proc/net/vlan/conf

Next we have the two virtual bridges, one for vlan1 traffic (BR01) and the other for vlan5 (BR05):

brctl add br01

brctl add br05

We then combine place the appropriate interfaces into the correct bridge:

brctl addif br01 eth0.1

brctl addif br01 vNIC1

brctl addif br05 eth0.5

brctl addif br05 vNIC2

After bringing up the interface, you should have KVM GUEST 1 as a member of vlan 1, and KVM GUEST 2 as a member of vlan 2

Homerun: going to guest trunk mode

Important, the method presented above, and the method about to be presented are mutually exclusive…. you must choose between one or the other. Having them both will give unexpected results (as I learnt the hard way). See the troubleshooting section below for some details.

We’ll be aiming for something like this:

brctl_tunking_guests

So, to explain the above: Tagged traffic comes through on eth0, which does not have any vlan subinterfaces defined as before. Traffic hits the virtual bridge, which is divided into vlan subinterfaces using vconfig. Basically, the vlan subinterfaces are defined directly on the bridge rather than on eth0.

The difference is that when subinterfaces are defined  on eth0, as noted previously Linux will strip the vlan tag, but when defined on the bridge, the vlan tags are kept. The vNICs are both members of the bridge, with the result that the tagged traffic is presented directly to them, with the VLAN tagging intact Smile Notice that if the bridge does not have a particular vlan subinterface (eg vlan 8 in my diagram above – there is no corresponding br0.8) that vlan traffic will be dropped.

To achieve the above first define the bridge:

brctl add br0

then we define the vlan subinterfaces on the bridge:

vconfig add br0 1

vconfig add br0 5

ifconfig br0.1 up

ifconfig br0.5 up

and we finally tie it all together

brctl add if br0 vNIC1

brctl add if br0 vNIC2

And now KVM GUEST 1 and 2 will be able to see tagged traffic from vlan 1 and 5. Of course the guest must be able to terminate the vlans themselves as they are now vlan aware.

Troubleshooting notes

While doing the above, if you run into problems, keep in mind the following:

1. Make sure all interfaces are up. Running ifconfig should show all interfaces, bridges and subinterfaces as “UP”

2. Traffic flow. Refer to the above diagrams. Though I only show the incoming traffic path (from the physical world to the VMs) in my diagrams, the outbound traffic flow (from the VMs to the physical world) is exactly the same, but in reverse.

This is essential so as to know where to run packet captures and where to know what is going wrong.

To give a real example of what happened in my case… I had defined all the above as in the second scenario presented above, but neglected to remove all the eth0 vlan subinterfaces that I had left there as a result of testing the first scenario. This resulted in tagged traffic leaving the VM, hitting the bridge correctly, and being sent out the physical world through eth0 correctly. But on it’s return, since the traffic was tagged and the kernel had eth0.1 defined, it was attempting to use those subinterfaces, and removing the vlan tags. Removing the eth0 subinterfaces meant that the tagged traffic could hit the bridge directly and the tagging was kept intact

3. TCPDUMP… when dumping vlan traffic remember that by default tcpdump will only monitor untagged traffic. To monitor tagged traffic from, say, vlan 5, you’d need:

tcpdump –i eth0 vlan 5

23 responses

  1. this is really2 good. any implication on perf on the two later options?

    1. Till now, there havent been any performance impacts at all. The two methods have the same performance

      1. I tried it on my setup. No performance imposed on it. But noticed one thing. We had 10GE connection, split two 3 VLAN and each VLAN a bridge. Using iPerf noticed that each VLAN got 3Gbps each. Trying to check why it happened actually.

  2. Great walk-through, just what I was struggling with. However in this scenario if eth0 loses connectivity you lose all that’s built on it. Have you tried this with bonded interfaces for physical redundancy?

    1. That is a really good question! You are absolutely right, but no I havent tested this yet. I will whenever I have some lab time, will keep you posted

      1. So I got scenario 2 working on bonded interfaces by just replacing “eth0” with “bond0”. However it does not work with mode=balance-alb or mode=balance-tlb. I am using mode=1 (active/passive) and it works great. This would probably work too with LACP (mode=802.3ad) but since I’m using 10 gig-E I don’t need 20 Gbps I just need redundancy so mode=1 is fine. I never did try your last scenario since my VMs don’t require tagging to be presented to the VM, only the platform they run on does. Thanks for the post!

      2. Excellent, thank you for carrying out the tests!

  3. Great tutorial, that was exactly what i was looking for. Seems bringing a vlan trunk into kvm guests isn’t documented very well. Thanks for your effort to share your wisdom with us😉

  4. Hi

    Aren’t you missing a line like

    brctl addif br0 eth0 ?

    Without that I don’t see how the packets will pass from the ethernet inteface to the bridge br0

    cheers
    PF

    1. Hi PF,

      That command is there, its between the first and second diagram. Yes, without it you are right, there would be no physical interface members in thebrjdge so no traffic would be able to flow into/out of your network

      Thanks for reading and commenting!

      Dave

      1. Hi Dave

        I thought they were distinct diagrams and configuration sets

        I have a quick question for you.
        Let’ say we have just this
        brctl add br0
        brctl addif br0 eth0
        brctl addif br0 vNIC1

        and the vNIC guest generates tagged traffic (two vlans)
        What woud you expect to see on eth0? Tagged or untagged ?

        What would be the benefits or adding the two vlans to eth0 ?

        thank
        PF

      2. Without defining the vlans anywhere in the config your results may vary. Some systems like pfsense automatically bridge vlan IDs even if they are not defined anywhere. Other systems like XenServer will accept incoming VLAN tags, but strip them in outgoing packets.

        In the context of KVM your use case is not something I have tested, but my best guess is that barring any driver problems, your outgoing traffic might remain tagged, but your incoming traffic may get dropped, or vlan tags may get stripped, giving problems to the virtual guest.

        In my case, I had to define the vlans on the bridge in order for the traffic to remain tagged in both incoming (from physical network to guest) and outgoing directions (vice versa)

        If you do test this and get some results please share, it would be very good to see the behaviour under different use cases

      3. Hi Dave

        after some testing I have come to the conclusion that it works like this:

        -initialy any bridge is not vlan aware (it is strictly 802.1d)
        -any ethernet interface can be considered a bridge if put in promiscuous mode (this is what happens when an interface is enslaved to a bridge)

        When a bridge and an real/virtual interface are connected by enslaving the interface to the bridge they pass frames to each other blindly, ignoring any any VID marking that might come from outside via the real interface.

        If you sniff them each will show the same, tagged or untagged frames, whatever is injected.

        Let’s say we have this: eth0, br0 and vlan 6 and 7 tagged traffic injected via eth0

        If you create br0.7 and you sniff this interface you will see untagged VLAN7 traffic
        If you sniff Br0 using VLAN 7 as filter you will obtain the same results
        If you sniff Br0 with no filtering you will see VLAN6 and VLAN7 ….AND…. the sniffer will show you 802.1q encapsulation

        So basically from my testing it resulted that your config above would work without creating the two br0.1 and br0.5 interfaces.

        Think of any subinterface created above as a way to extract unlabelled traffic (if there is any) .
        these interface are to use directly with other apps or virtual devices BUT not to enslave them to another bridge

        Does it make sense ?
        PF

  5. This is wrong “BUT not to enslave them to another bridge” ..I just tested that :-))
    I injected tagged trffic into br0, I created br0.6 and br07 and I enslaved br0.6 to a test bridge
    Sniffig that test bridge is showing me unlabelled vlan 6 traffic.

    More testing needs to be done but I am running out of time.

  6. The above statement should have been written “you can not enslave virtual bridge to another virtual bridge.

  7. […] servers. i have used seem this article that describes what i have tried to achieve (option 1). http://blog.davidvassallo.me/2012/05…to-the-guests/ i have setup: bond for eth0 and eth1 created VLANs on the bond – bond0.10, bond0.100 then […]

  8. Thanks very much for this article, I read and re-read it about 5 times (as well as many others!) in my mission to get (different) trunked VLANs to some VMs and plain untagged traffic to others.

    I threw a question up on ServerFault in the hope someone would know (but to no avail), but I figured it out in the end.

    For anyone else interested, here’s the link http://serverfault.com/questions/543434/multiple-different-vlan-trunks-to-kvm-guests-linux/543682#543682

  9. […] a surprisingly large amount of searching, I came across this article, which details the first way, and then another — the way to trunk vlans to KVM guests. […]

  10. Hi,
    I am setting up from scratch a new cluster at work (linux-ha, libvirt, centos6.5). The firewalls are virtualized, as well as the VMs used, and I’ve encountered a quite specific issue : my VMs cannot ping their gateways (hosted on the firewall) ! I’ve banged my head against the wall trying to understand what was happening, and your article was really helpful on this issue. What I was trying to achieve was exactly what you tell people not to do :
    “Important, the method presented above, and the method about to be presented are mutually exclusive…. you must choose between one or the other. Having them both will give unexpected results (as I learnt the hard way).”

    I’ve solved my problem by using openvswitch. I downloaded the sources and compiled it, found some tutorial on the www, and it works perfectly !

    This soft (openvswitch) could also be an elegant solution for Ben’s problem (above my post).

    Anyway, thank’s for sharing your knowledge on this post !

    Cheers,

  11. Won’t this all be lost on reboot? How is this done via config files?

  12. In my blogpost I described a similar bridge configuration. I used libvirt to manage the bridging.
    http://alesnosek.com/blog/2015/09/07/bridging-vlan-trunk-to-the-guest/

    Based on my testing I think the following steps in your description are actually not needed:
    vconfig add br0 1
    vconfig add br0 5
    ifconfig br0.1 up
    ifconfig br0.5 up

    This should be sufficient to make the bridging work.
    brctl add br0
    brctl add if br0 vNIC1
    brctl add if br0 vNIC2

    1. I used the script for a good long time, then just switched to OpenVSwitch. Simpler, faster and more powerful.

  13. […] I’ve been reading (http://blog.davidvassallo.me/2012/05/05/kvm-brctl-in-linux-bringing-vlans-to-the-guests/), and it seems like tagging on physical interfaces (em1.3) causes linux to strip the tag prior to […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 317 other followers

%d bloggers like this: