Horms

PCI Virtualisation Terminology

Fri, 12 Dec 2008 17:35:11 +1100

My last post was titled "Building qemu-dm for HVM VT-d PCI Pass-Through", a piece about building a component of Xen. But what does all this terminology mean?

Firstly, a very brief overview of the Xen architecture. The three main components of a system running Xen are the hypervisor, privileged domains and unprivileged domains. The hypervisor is run when the system first boots and controls access to the hardware. Privileged domains essentially provide both drivers and a management interface to xen. Usually there is one privileged domain, called dom0. They hypervisor by itself is not a functional system and generally the hypervisor + dom0 is required to boot the system. Unprivileged domains are the guest OSes on the system, often referred to as domU. As the name suggests they generally have to access hardware via the hypervisor - which may delegate the task to a privileged domain.

Xen supports two methods of domain (guest) virtualisation. In the beginning there were paravirtualised (PV) domains. In paravirtualised domains portions of the guest OS that execute privileged instructions are replaced by hypercalls. The hypercall asks the hypervisor to perform the relevant action. At the time that Xen was originally developed this was the only type of virtualisation that was possible on x86 and ia64.

With the advent of Intel's VT-x for x86 and VT-i for ia64, and AMD's AMD-V it has became possible to run unmodified operating systems as guests. This is called full-virtualisation or hardare virtual machine (HVM). It has obvious advantages when the guest operating sytem can't be modified.

There are several techniques used to give unprivileged domains access to PCI devices. For PV domains the common approach is to split the driver into front-end and back-end portions. The front-end driver sits in the unprivileged domain and the back-end driver sits in a privileged domain. The front-end driver communicates with the back-end driver. And the back-end driver communicates with the hardware. The back-end driver may multiplex IO from multiple front-end drivers. And thus a PCI device may be shared between domains.

Another approach used with PV domains is PCI pass-through. This basically involves giving a PCI device exclusively to an unprivileged domain. The advantage of this is a potential performance increase, as I/O no longer has to go through a back-end driver sitting in a privileged domain. The disadvantages of this approach are:

The PCI device in question can no longer be shared between domains
Hotplug needs to be used to achieve live migration - not all devices are supported
As the owner of a PCI card can potentially DMA main memory, passing through a PCI device to an unprivileged domain effectively makes the domain privileged

Systems that have Intel VT-d have IOMMU capabilities. The IOMMU constrains the area to with a device passed-through to a guest may DMA. Which removes the security problem relating to passing-through PCI devices to unprivileged guests. Xen allows pass-through to paravirtualised to optionally use the IOMMU if present.

For fully virtualised domains devices are commonly emulated. This is done by running the device emulation code from qemu as qemu-dm in a privileged domain. In order to improve I/O performance in fully virtualised domains it is possible to use PV drivers. These essentially provide front-end drivers to fully virtualised domains. And with the advent of Intel's VT-d it is possible to pass-through PCI devices to fully virtualised domains.

Building qemu-dm for HVM VT-d PCI Pass-Through

Fri, 12 Dec 2008 16:58:52 +1100

If you are building the qemu-dm, device models for xen, from http://xenbits.xensource.com/git-http/qemu-xen-unstable.git or the tools/ioemu-dir/ directory that is created when building http://xenbits.xensource.com/xen-unstable.hg (due to the wonders of the xen build system, they are the same thing). And you are doing this on Debian. And you would like PCI pass-through to work for HVM domains. Then make sure you have the libpci-dev package installed before building. Otherwise qemu-dm will be built without pass-through support. If you install libpci-dev after building then run make clean before rebuilding, to ensure that all files are rebuilt correctly - they won't be otherwise.

If you do this then the instructions in the Xen VT-d How-To should work.

Otherwise you will end up the following nasty message, which just cost me more time than I care to admit, in /var/log/xen/xend.log

				ERROR (XendDomainInfo:581) Device model didn't tell the vslots for PCI device

On a semi-related note, if you wish to use VT-d PCI pass-through in paravirtualised domains then pass iommu=pv as an argument to xen when booting. Otherwise non-VT-d pass-through will be used for paravirtualised domains. This is independent whether qemu-dm was compiled with libpci-dev present or not.

Sculpture by the Sea

Sun, 19 Oct 2008 21:32:53 +1100

Today was a beautiful pre-summer day in Sydney and Chiz and I went to see Sculpture by the Sea, which is held on the coastal walk between Bondi and Tamarama Beaches.

photos

Perisher Ski Trip

Mon, 13 Oct 2008 21:56:15 +1100

Some photos from our ski trip to Perisher in August. more...

Struan and Genny's Wedding

Sun, 05 Oct 2008 13:20:59 +1100

Yesterday Chiz and I had the pleasure to gather together with Struan and Genny's friends and family for their wedding in the Royal Botanical Gardens in Sydney. It was a wet morning but the weather cleared just in time for the event. I took a few snaps of the occasion. more...

I also have some snaps of the Skydiving that took place in Picton two weeks ago as part of Struan's bucks party. more...

Xen Netfront/Netback Transmit Flow-Control

Thu, 25 Sep 2008 21:45:19 +1000

Flow-control is implemented from netfront all the way to the end of life for a packet, typically transmit in the destination interface.

Introduction

This post explains briefly how flow-control is implemented in netfront/netback. The paths examined are for the case where dom0 bridges traffic from netback to the destination interface. The same logic should also hold for packets from netback that are routed by dom0.

Netback/Netfont is used to provide networking to paravirtualised domain Us. Thus, much of this discussion is not applicable to fully-virtualised domains, also known as HVM domains.

The code reference is from the linux-2.6.18-xen tree on xenbits.xensource.com, around the time of the Xen 3.3 release.

This analysis is based upon the notion of domU sending a stream of packets as fast as it can. There is no congestion control, no packet filtering, and the CPU(s) of domU are fast enough to processes packets as fast as the physical interface can send them without dropping any packets. Furthermore there are no return packets. In short, the notion is of UDP packets being blindly sent by domU.

The Netfront/Netback Ring-Buffer

Packets are received by netback from netfront using a ring-buffer that is implemented as a grant table. Typically there are 256 slots in this buffer. And once it becomes full netfront can no longer send packets.

The relevant netfront code in the reference netfront implementation is in drivers/xen/netfront/netfront.c:network_start_xmit().

				        if (!netfront_tx_slot_available(np))
						netif_stop_queue(dev);

For each packet transmitted by netfront to netback, one ring-buffer slot is used for header information and one additional slot is used per fragment. So in the common case of a packet with one fragment (that is, an unfragmented packet), two slots will be used.

Assuming that the number of packets to be sent is significantly greater than the number of slots in ring-buffer, the rate at which netfront can send packets will be the same as the rate at which netback frees slots in the ring-buffer. So the ring-buffer provides a mechanism for netback to enforce flow-control on netfront.

Strictly speaking the above logic applies to fragments not packets. But as it is common to have unfragmented packets it seems more comfortable to talk in terms of packets.

Packet processing in Netback

Once all fragments for a packet have been received by netback from netfront via the ring-buffer it packs them into an sk_buff, the structure that the Linux networking code uses to represent packets. This work is done by drivers/xen/netback/netback.c:net_tx_action().

At this point the slot used in the ring-buffer for header information is queued to be released for reused by netfront. This is done by calling drivers/xen/netback/netback.c:netif_idx_release(). The release queue is processed by calling net_tx_action_dealloc() the next time that net_tx_action() is called.

The ring-buffer slot associated with each fragment contains a page which in turn contains the data for the fragment. This page is associated with a fragment within the sk_bufs created for the packet. This work is done by drivers/xen/netback/netback.c:netbk_fill_frags() which is called by net_tx_action(). The critical line is:

					frag->page = virt_to_page(idx_to_kaddr(pending_idx));

Once all fragments have been added to the sk_buff it is passed to the Linux networking core using net/core/dev.c:netif_rx(). At this point the packet has been received by the netback device driver.

Usually netif_rx() will simply move the sk_buff onto a received queue and schedule processing of the queue if it is not scheduled already. In any case, sk_buff will end up being destroyed, either after bing transmitted or being dropped - for example because of packet filtering rules.

Notification of the Completion of Packet Processing

At this point a packet is in the Linux networking core in in the form of an sk_buff. The page that was associated with each fragment of each sk_buff in netback by netbk_fill_frags() have a particular property that is of relevance to flow-control. These pages are members of drivers/xen/netback/netback.c:mmap_pages[] which is set up in drivers/xen/netback/netback.c:netback_init(). Each member of mmap_pages[] is marked as foreign and has drivers/xen/netback/netback.c:netif_page_release() set as its destructor using include/linux/page-flags.h:SetPageForeign().

					SetPageForeign(page, netif_page_release);

When an sk_buff is freed, the destructor is called for each page that has both been associated with a fragment within the sk_buff and marked as foreign using SetPageForeign(). This is done in mm/page_alloc.c:free_hot_cold_page().

				#ifdef CONFIG_XEN
					if (PageForeign(page)) {
						PageForeignDestructor(page);
						return;
					}
				#endif

sk_buffs are freed using kfree_skb(), the call path from there to free_hot_cold_page() is:

    net/core/skbuf.c:kfree_skb()
⟶ net/core/skbuf.c:__kfree_skb()
⟶ net/core/skbuf.c:kfree_skbmem()
  ⟶ net/core/skbuf.c:skb_release_data()
   ⟶ mm/swap.c:put_page()
    ⟶ mm/swap.c:__page_cache_release()
     ⟶ mm/page_alloc.c:free_hot_page()
      ⟶ mm/page_alloc.c:free_hot_cold_page()

Thus when an sk_buff is freed, for any reason, netif_page_release() will be called. It simply calls netif_idx_release(), which as discussed above queues a ring-buffer slot up to be released for reuse by netfront. This freeing will typically take place in the physical NIC's driver, when the packet is transmitted on the physical interface.

Netfront to Destination Interface Flow-Control

As sk_buffs are generally freed as packets are transmitted on the physical interface, the rate at which slots are freed relates to the rate at which packets are sent. Thus the rate at which netfront can send packets is effectively the rate at which the destination interface, usually a physical ethernet interface, can sent packets, assuming the dom0 CPU can keep up.

The following diagram shows the usage cycle of ring-buffer slots. They are consumed by domU. Slots used for meta-data are added to the free list immediately by Netback. Slots used for fragments are added to the free list once the skb to which they are associated is freed. Slot buffers that are placed on the free list become available for reuse in the ring-buffer by DomU.

It is worth noting that if packets are dropped, for instance by packet filtering, before they make it to the destination device, then the ring-buffer slot will be freed. This should allow for faster transmission by netfilter. That is, packets that get dropped don't take as long to transmit and thus netback can transmit at a faster rate.

Just Practicing

Mon, 18 Aug 2008 16:19:55 +1000

The Army has been practicing running two helicopters up Pitt St. this afternoon. So far they have done about half a dozen laps, down from Circular Quay to about Town Hall where they do some maneuver that I can't quite see out of my window before looping back North over the Domain. They seem to take about a 15-30 minute rest between each lap. Not sure where they go during this time. Perhaps up to Palm Beach. Or perhaps they drop in for scones with the PM at Kirribilli House. In any case, I'm glad that they are keeping themselves busy and giving the office workers something to peer at our of their windows.

Browser Crud

Mon, 18 Aug 2008 12:12:27 +1000

Am I the only person who invokes firefox -P profile and expects it to work even if firefox is already running?

Word Games

Wed, 06 Aug 2008 09:56:52 +1000

Sometimes when you live in a foreign country interesting things happen. For Chiz, yesterday was one of those days.

Before, during and after dinner she was complaining of not feeling very well due to a sore stomache. She was wondering if perhaps she was coming down with a cold - there is at least one going around. On the short walk home from dinner she asked me what the sasuage like things below ones stomache are called, as they hurt too. I let her know that intestines was the word that she was after.

When we got home she showed me the elite new tea that she had been drinking on recommendation from a friend. Four or five cups since she started the day before...

kexec-tools 2.0.0

Sat, 19 Jul 2008 11:04:38 +1000

Some time back in 2006 I sifted through many mailing list posts and pulled together all the patches to kexec-tools that were floating about. I subsequently put these in a git tree and released them as kexec-tools-testing.

As time went on, kexec-tools-testing ended up having releases about every 3 months or so, and became the focus for development of kexec-tools.

It has become a bit silly to call this code kexec-tools-testing, as it is the main development branch. The name only seems to serve to confuse people and perhaps scare some people off.

Along with the latest release of the code I have renamed the tree kexec-tools. So don't be confused any longer!

More about how to get kexec-tools here