Archive for the ‘virtualization’ Category

How I run Oracle VM 2.2 guests with custom network configuration

August 15th, 2012 No comments

Recently I was given three virtual machines running Oracle Enterprise Linux 5 and Oracle 11gR2 RAC on Oracle VM 2.2.1, copied straight from /OVS/running_pool/. I had to get these machines up and running at my lab environment, but I found hard to setup the network. I’ve spent half day in debugging without success, but finally found a workaround, which I’ll explain here.

Just a little technical notes – Oracle VM (xen) has three main setup configurations within /etc/xen/xend-config.sxp:

Bridge Networking – this configuration is configured by default and it’s simplest to configure. Using this type of networking means that the VM guest should have IP from the same network as the VM host. Another thing is that the VM guest could take advantage of DHCP, if any. The following lines should be uncommented in /etc/xen/xend-config.sxp:
(network-script network-bridge)
(vif-script vif-bridge)

Routed Networking with NAT – this configuration is most common where a private LAN must be used, for example you have a VM host running  on your notebook and you can’t get another IP from corporate or lab network. For this you have to setup private LAN and NAT the VM guests so they can access the rest of the network. The following lines should be uncommented in /etc/xen/xend-config.sxp:
(network-script network-nat)
(vif-script vif-nat)

Two-way Routed Network – this configuration requires more manual steps, but offers greater flexibility. This one is exactly the same at the second one, except the fact that VM guests are exposed on the external network. For example when VM guest make connection to external machine, its original IP is seen. The following lines should be uncommented in /etc/xen/xend-config.sxp:
(network-script network-route)
(vif-script vif-route)

Typically only one of the above can be used at one time and selection and choice depends on the network setup. For second and third configurations to work, a “route” must be added to the Default Gateway. For example if my Oracle VM host has an IP address, then on the default gateway ( a route has to be added to explicitly route all connection requests to my VM guests through my VM host. Something like that:
route add -net netmask gw

Now back to the case itself. Each of the RAC nodes had two NICs – one for the public connections and one for the private, which is used by GI an RAC. The public network was 10.0.1.X and private 192.168.1.X. What I wanted was to run the VM guests at my lab and access them directly with IP addresses from the lab network, which was 192.168.143.X. As we know the default network configuration is to use bridged networking so I went with this one. Having the vm guests config files all I had to do was to change the first address of every guest:

vif = [‘mac=00:16:3e:22:0d:04, ip=, bridge=xenbr0’, ‘mac=00:16:3e:22:0d:14, ip=’,]

vif = [‘mac=00:16:3e:22:0d:04, ip=, bridge=xenbr0’, ‘mac=00:16:3e:22:0d:14, ip=’,]

This turned to be real nightmare, I’ve spent half a day looking why my VM gusts doesn’t have access to the lab network. They had access to VM host, but not to the outside world. Maybe because I’m running Oracle VM on top of VMWare, but finally I  gave up this configuration.

Thus I had to use one of the other two network configurations – Routed Networking with NAT OR Two-way Routed Network. Either case I didn’t have access to the default gateway and would not be able to put static route to my VM guests.

Here is how I solved this – to run three nodes RAC on Oracle VM Server 2.2.1, keep their original network configuration and access them with IP address from my lab network (192.168.143.X). I’ve put logical IP’s of the VM guests on the VM host using ip (ifconfig could also be used) and then using iptables change packet destination to the VM guests themselves (10.0.1.X).

1. Change Oracle VM configuration to Two-way Routed Network, comment the lines for default bridge configuration and remove comments for routed networking:
(network-script network-route)
(vif-script vif-route)

2. Configure VM host itself for forwarding:
echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp
iptables -t nat -A POSTROUTING -s -j MASQUERADE

3. Set network alias with the IP address that you want to use for the VM guests:
ip addr add dev eth0:1
ip addr add dev eth0:2
ip addr add dev eth0:3

4. Create iptables rules in PREROUTING chain that will redirect the request to VM guests original IPs once it receive it on the lab network IP:
iptables -t nat -A PREROUTING -d -i eth0 -j DNAT –to-destination
iptables -t nat -A PREROUTING -d -i eth0 -j DNAT –to-destination
iptables -t nat -A PREROUTING -d -i eth0 -j DNAT –to-destination

5. Just untar the VM guest in /OVS/running_pool/

[root@ovm22 running_pool]# ls -al /OVS/running_pool/dbnode1/
total 26358330
drwxr-xr-x 2 root root        3896 Aug  6 17:27 .
drwxrwxrwx 6 root root        3896 Aug  3 11:18 ..
-rw-r–r– 1 root root  2294367596 May 16  17:27 swap.img
-rw-r–r– 1 root root  4589434792 May 16  17:27 system.img
-rw-r–r– 1 root root 20107128360 May 16  17:27 u01.img
-rw-r–r– 1 root root         436 Aug 6 11:20 vm.cfg

6. Run the guest:
xm create /OVS/running_pool/dbnode1/vm.cfg

Now I have a three node RAC, nodes have their original public IPs and I can access them using my lab network IPs. The mapping is like this:

Request to –> the IP address is up on the VM host –> on the VM host iptables takes action –> packet destination IP address is changed to –> static route is already in place at VM host routing packet to the vif interface of the VM guest.

Now I can access my dbnode1 ( directly with its lab network IP


Categories: linux, oracle, virtualization Tags: ,

No ethernet adapters with OEL6 under VMware ESX 4.1

May 13th, 2011 3 comments

I was playing around with Oracle Database 11.2 on OEL6.0 guest machine running on VMware ESX 4.1 and after installation I had no ethernet adapters. I didn’t notice that before, because I always set the network to E1000 and now it was VMXNET3. Here is the description of the networks from the VMware manual:

E1000: An emulated version of the Intel 82545EM Gigabit Ethernet NIC. A driver for this NIC is not included with all guest operating systems. Typically Linux versions 2.4.19 and later, Windows XP Professional x64 Edition and later, and Windows Server 2003 (32-bit) and later include the E1000 driver.

VMXNET 3: The VMXNET 3 adapter is the next generation of a paravirtualized NIC designed for performance, and is not related to VMXNET or VMXNET 2. It offers all the features available in VMXNET 2, and adds several new features like multiqueue support (also known as Receive Side Scaling in Windows), IPv6 offloads, and MSI/MSI-X interrupt delivery. VMXNET 3 is supported only for virtual machines version 7 and later, with a limited set of guest operating systems:

* 32 and 64bit versions of Microsoft Windows XP, 2003, 2003 R2, 2008,and 2008 R2.
* 32 and 64bit versions of Red Hat Enterprise Linux 5.0 and later
* 32 and 64bit versions of SUSE Linux Enterprise Server 10 and later
* 32 and 64bit versions of Asianux 3 and later
* 32 and 64bit versions of Debian 4
* 32 and 64bit versions of Ubuntu 7.04 and later
* 32 and 64bit versions of Sun Solaris 10 U4 and later

So it turns out that it should be supported by Oracle Enterprise Linux and I need to install the vmware tools.

Here is how to do it with default installation of OEL6:
– Connect the OEL installation media to the guest

– mount the media:
mount /dev/cdrom /mnt

– change directory to installation packages and install the following packages:
cd /mnt/Packages

rpm -ivh gcc-4.4.4-13.el6.x86_64.rpm cloog-ppl-0.15.7-1.2.el6.x86_64.rpm glibc-devel-2.12-1.7.el6.x86_64.rpm cpp-4.4.4-13.el6.x86_64.rpm ppl-0.10.2-11.el6.x86_64.rpm mpfr-2.4.1-6.el6.x86_64.rpm glibc-headers-2.12-1.7.el6.x86_64.rpm kernel-uek-headers-2.6.32-100.28.5.el6.x86_64.rpm kernel-uek-devel-2.6.32-100.28.5.el6.x86_64.rpm

* Because OEL6 ships by default with Oracle Enterprise Linux Kernel, kernel-uek-devel should be installed for proper compilation of the vmware tools.

– Umount the installation media and install vmware tools:
umount /mnt
VM -> Guest -> Install/Upgrade Vmware tools

– Mount the vmware tools media and copy the packages:
mount /dev/cdrom /mnt
cp /mnt/VMwareTools-8.3.2-257589.tar.gz /tmp
umount /mnt
cd /tmp
tar xfz VMwareTools-8.3.2-257589.tar.gz
cd vmware-tools-distrib

Answer by default to all the questions, if all the packages are installed the compilation should finish in a minute and network interfaces should become available.

After susccefful installation the following could be seen in dmesg:

VMware PVSCSI driver – version
VMware memory control driver initialized
vmmemctl: started kernel thread pid=14962

VMware vmxnet3 virtual NIC driver – version
eth0: NIC Link is Up 10000 Mbps


For OEL 5.6 the following packages has to be installed, rest is the same:
rpm -ivh gcc-4.1.2-50.el5.x86_64.rpm glibc-headers-2.5-58.x86_64.rpm glibc-devel-2.5-58.x86_64.rpm kernel-uek-headers-2.6.32-100.26.2.el5.x86_64.rpm kernel-uek-devel-2.6.32-100.26.2.el5.x86_64.rpm



Categories: linux, virtualization Tags:

Shared disk support for VirtualBox

August 9th, 2010 2 comments

I’m very happy to announce that VirtualBox now supports shared disks. Finally we can attach one disk to several virtual machines and run Oracle RAC and other clusters. As Oracle promised, this feature is released with the next maintenance patch (thanks!).

There is a new image write mode which is called shareable and this options is now available for the commands createhd and modifyhd of VBoxManage. To create new shared image use the command VBoxManage createhd with type shareble, creating shared disk from the GUI is not possible. To mark an existing image as a shared use the command VBoxManage modifyhd with type shareable.

Something important is that only fixed size disks are supported. If the disk is dynamic you will encounter the following error if you try to modify the image:
ERROR: Cannot change type for medium ‘/home/vm/ora11g_shared.vdi’ to ‘Shareable’ since it is a dynamic medium storage unit

There is other minor issue, if the image is already attached to two virtual machines the command modifyhd will also fail:
ERROR: Cannot change the type of medium ‘/home/vm/ora11g_shared.vdi’ because it is attached to 2 virtual machines

And finally, YES it works, I have tested it already!

sve@host:~$ VBoxManage showhdinfo /home/vm/ora11g_shared.vdi
Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.

UUID:                     7521f059-1196-4d68-a1a6-cf0082fb446a
Accessible:               yes
Logical size:             2048 MBytes
Current size on disk:     2048 MBytes
Type:                     shareable
Storage format:           VDI
In use by VMs:            labs1 (UUID: 25475ff4-70bc-4e2e-aa38-d8fae289273e)
                          labs2 (UUID: e4441f4c-1ef9-42e0-8e54-d2aec2c6cf4f)
Location:                 /home/vm/ora11g_shared.vdi

Regards and happy migration 😉

Categories: oracle, virtualization Tags: , , ,

Oracle will bring back VirtualBox shared disk capability

July 1st, 2010 No comments

During the questions section of the last webinar Introducing Oracle VM VirtualBox 3.2 Oracle said that they received a complains from a lot of customers using VirtualBox regarding the installation of Oracle RAC. This requires a shared disk drive to be accessed by the nodes (VMs) of the cluster simultaneously, but this cannot be achieved directly. There is a workaround by using iSCSI, but this is not the point.

Achim Hasenmueller from VirtualBox engineering team said that they plan to deliver this capability very soon with the next maintenance release and not to wait for the major update. I was surprised to hear that they used to have this feature working, but during one of the major changes to the storage stack they have lost it. I was not able to find this one at the changelogs, but by accident I found the announcement of this limitation at debian bug report log:

From: "VirtualBox" <>
Subject: Re: [VirtualBox] #1188: Please support to share a disk image
 between two guests
Date: Wed, 08 Apr 2009 15:24:49 -0000

#1188: Please support to share a disk image between two guests
Reporter:  bzed              |        Owner:
    Type:  enhancement       |       Status:  closed
Priority:  minor             |    Component:  VM control
 Version:  VirtualBox 1.5.4  |   Resolution:  wontfix
Keywords:                    |        Guest:  other
    Host:  other             |
Changes (by frank):

  * status:  new => closed
  * resolution:  => wontfix


 Starting with 2.1.0, a disk image can be attached to two VMs at the same
 time, but only one of these two VMs can be powered on at the same time.
 Klaus already explained why we wouldn't implement sharing an image between
 running VMs. Closing

I’ve been using VirtualBox for an year now, but recently I decided to install Oracle RAC. Like most of the ex-vmware users I’ve just created a new disk and added it to two virtual machines. The first one started normaly, but when I tryed to start the second one I got the following error:

Component: Machine
Interface: IMachine {6d9212cb-a5c0-48b7-bbc1-3fa2ba2ee6d2}

It turns out that VirtualBox will not allow more than one running VM to use a VDI file. The solution I found most useful is to setup a third server (or VM) with Openfiler iSCSI host. Then VirtualBox can transparently present iSCSI disk to a virtual machine as a virtual hard disk. The guest operating system will not see any difference between a virtual disk image (VDI file) and an iSCSI target. To achieve this, VirtualBox has an integrated iSCSI initiator.


Categories: oracle, virtualization Tags: , ,