Configuring Exalytics Machine for Virtualization

This post describes the tasks to perform when configuring Exalytics Machines for virtualization. The tasks include deploying Oracle Virtual Machine Server (Oracle VM Server) on the Exalytics Machine and configuring Oracle VM Server with Oracle Virtual Machine Manager (Oracle VM Manager)

Environment:
  • Use exalytics box as a vm-server
  • Use another machine as vm-manager to manage server-pools specified above.
  • Create virtual machines on top of vm-server using vm-manager
  • Client PC is used to coordinate among VM Server and VM Manager using ILOM (Integrated lights out manager).
Prerequisites for Configuring Exalytics Machine for Virtualization

The following prerequisites must be met before you can virtualize an Exalytics Machine:
  • The Exalytics Machine has been racked and mounted in a data center.
  • Oracle Integrated Lights Out Manager (ILOM) 3.1.2.24 or later, is configured to be accessible from the network using the web interface.
  • All networking information such as host name, IP address, subnet, gateway, and DNS are available.
  • Spare, unassigned IP addresses are available for the server pool and for each virtual machine deployed on the Exalytics Machine.
  • A Linux environment is available to install Oracle VM Manager.
  • A database server is available to install Oracle VM Manager schemas.
  • For information about hardware and software specifications, see Oracle VM Installation and Upgrade Guide for Release 3.2.1.
  • You have access to the internet from a computer (not necessarily the Exalytics Machine) to download required software.
  • A RHEL 5.8 Setup Image to be installed on each VM.

    1. Base Setup

  • Log-in to client PC, preferably collocated geographically within Exalytics environment, in our case, it was a windows server.
  • Install VNC Viewer, Putty and WinSCP on this client PC to access the Linux Box.

    1. Troubleshooting ILOM Remote Console



  1. ======== Oracle Exalytics HW ======== 
    Issue : Firewall rule is blocking port 7578 for ILOM video redirection: 
    ——— 
    Solution: 
    ————- 
    Check with responsible network administrator, that in either network direction between the ILOM Service Processor and the used remote client system for ILOM redirection, the needed network ports are available and not blocked. 

    The Oracle ILOM 3.1.X Remote Console uses the following TCP/IP communication ports by default: 

    Port: 5120 for CD redirection. 
    Port: 5123 for floppy redirection. 
    Port: 5556 for user authentication redirection. 
    Port: 7578 for video redirection. 
    Port: 7579 for SPARC server redirection only. 

    For detailed information regarding requirements for using the Oracle ILOM Remote Console, please review the following URL: 

    Oracle Integrated Lights Out Manager (ILOM) 3.1 Documentation Collection 
    Requirements for Using the Oracle ILOM Remote Console 

    http://docs.oracle.com/cd/E24707_01/html/E24528/z40000191582334.html 

    => ILOM Java remote console returns “Connection failed: Check the destination address and confirm that the KVMS service is enabled and running a supported version. ( Doc ID 1674985.1 ) 


  1. ======= Exadata Hardware ======= 

    The remote console application uses the following TCP ports: 
    therefore I would request you to check and confirm whether all the below ports are open in order to perform the Storage Redirection 

    TABLE 8-2 Remote Console Ports and Interfaces 
    Port Interface Application 
    443 TCP HTTPS 
    5120 TCP Remote CD 
    5121 TCP Remote keyboard and mouse 
    5123 TCP Remote Diskette 
    6577 TCP CURI (API) – TCP and SSL 
    TCP 7578 Video Data 
    UDP 161 SNMP V3 Access 
    UDP 3072 Trap Out (outgoing only) 

    Note – If the ILOM is configured to use HTTP, it uses TCP port 80. 

    For more details please refer to this MOS Doc : ILOM JavaRconsole through a NAT firewall ( Doc ID 1006926.1 ) 

  1. Configuring Exalytics Machine for Virtualization

    1. Preface

This section will guide through the installation steps that will cover the path from original state of exalytics to the final proposed state.
As per the scenario, we are installing
  • Oracle VM Server on Exalytics Server.
  • Oracle VM Manager  on  separate Linux Box.
Now, for fresh installation of Oracle VM Server 3.2.7, it is recommended to first install and configure Oracle VM Server 3.0.3 before upgrading to Oracle VM Server 3.2.7.
Installation steps for Oracle VM Server 3.0.3 is given in subsection 3.2.-3.5. Installation requires the setup image to be downloaded on the client PC and through redirection, this setup image will be mounted on target system i.e. exalytics server.
Installation of Oracle VM Manager 3.2.7 will follow the installation of Oracle VM Server 3.0.3. Now for installation of Oracle VM Manager 3.2.7, we use another machine. Installation steps are given in subsection 3.7.
After successful installation of Oracle VM Manager 3.2.7, discovery of Oracle VM Servers will follow. For discovery (see subsection 3.8.), we use the OVM console URL of OVM Manager  from the client PC.
Once discovered, it’s time to upgrade the OVM Server 3.0.3 to OVM Server 3.2.7, please go through subsection 3.6 for this.
Once all these steps are done, it’s hardly the hardware configuration we are left with. Now software virtualization tasks will take place. All the steps from this onwards, will be done from Oracle VM Manager console URL.
For each VM (Virtual Machine), that we have to create, a physical address (aka MAC address) will be required, for that goto subsection 3.9 (creating Virtual Network Interface Controllers).
Now, the discovered Oracle VM Server will be assigned under some Server Pool which contains a collection of OVM Servers. In our scenario, we just have one. (see subsection 3.10.)
Exalytics Repository is then created, which will host all resources, such as virtual machines, templates for creating virtual machines, virtual machine assemblies, and so on. (see subsection 3.11).
Now finally, creation of individual virtual machines will follow, which includes all the steps from creation of virtual machine in OVM Manager to successfully installing RHEL 5.8 image in that (see subsection 3.12-3.13).
Once the virtual machine is created and started from OVM Manager, it’s time to fresh install Linux image in the allocated virtual disk associated with the virtual machine. Please refer the RHEL 5.8. OS installation steps from internet.

    1. Downloading required softwares:

Download and save the Oracle Exalytics Base Image 2.0.1.1.0 for Exalytics Oracle VM x86-64 (Part Number V34491-01), by performing the following steps:
  1. Navigate to the Oracle Software Delivery Cloud web site at the following link:
  1. In the Select a Product Pack list, select Oracle Business Intelligence.
  2. In the Platform list, select Linux X86-64.
  3. Click Go.
  4. Select Oracle Exalytics Software Media Pack for Linux x86-64 Release: 1.0.0.4.0.
  5. Click Continue.
  6. Download Oracle Exalytics Base Image 2.0.1.1.0 for Exalytics Oracle VM x86-64 Part Number: V34491-01.
  7. The base image contains the Oracle VM Server 3.0.3.
  8. Save it to a local directory.


    1. Oracle VM Server Installation

  1. On the Exalytics Machine, connect and log on to Oracle Integrated Lights Out Manager (ILOM).
  2. In ILOM, select Remote Control, then Redirection, then Storage Redirection, and then Launch Service. The ILOM remote console is displayed.
  3. From the menu bar, select Devices and then CD-ROM Image to link the Oracle VM Server image as a remote virtual CD-ROM in ILOM.
  4. Perform the following action to restart the machine:
  • In ILOM, select ILOM, then Host Management, and then Power Cycle.
  1. In ILOM, select Host Management, then Host Control, and then Next Boot Device.
The setting for the Next Boot Device is displayed.
  1. Perform the following steps to move CD-ROM to the top of the BOOT menu:
  • In the Next Boot Device list, select CD-ROM.
  • Click Save.
  1. In ILOM, select ILOM, then Host Management, and then Power Cycle.
  2. The Exalytics Machine restarts and displays the Oracle VM Server screen.
  3. Press Enter to install Oracle VM Server.
  4. The Oracle VM Server is installed on the Exalytics Machine. After Oracle VM Server is installed, the Exalytics Machine restarts automatically.
  5. From the menu bar, select Devices, and then deselect CD-ROM Image.
  6. From the menu bar, select Devices, and then Save as host defaults.
  7. In ILOM, select ILOM, then Host Management, and then Power Cycle.
  8. The Exalytics Machine is ready for configuration.

Log on to Oracle VM Server, and enter the following command to inspect the RAID configuration:
/opt/MegaRAID/MegaCli/MegaCli64 -LDinfo -LALL –aALL

    1. Configuring the Network on Oracle VM Server

Before configuring the network, ensure that the Exalytics network configuration is complete. After the Exalytics Machine restarts, perform the following actions:
  1. Enter “I” to access the interactive mode.
  2. At the prompt, enter the following information:
    1. Enter 3 for Full Rack and confirm your entry.
    2. Enter 1 for Node Index.
    3. Enter an IP address for the InfiniBand interface. If the InfiniBand is not configured, enter a network configuration that does not conflict with existing networks. For example, enter 192.168.74.200.
    4. Enter a netmask address for the InfiniBand interface. If the InfiniBand is not configured, enter a network configuration that does not conflict with existing networks. For example, enter 255.255.255.0.
    5. Enter the system IP address for the ethernet bond0.
    6. Enter the bond0 netmask and gateway addresses.
    7. Enter the host name.
    8. Enter the domain.
    9. The Exalytics Machine restarts automatically.

    1. Troubleshooting Network on Oracle VM Server

See  3.7.1. Troubleshooting Network on Oracle VM Server

    1. Upgrading Oracle VM Server 3.0.3 (Base Image 2.0.1.1) to Oracle VM Server 3.2.7

Before you upgrade Oracle VM Server, ensure you installed Oracle VM Manager
Note: If you are configuring Exalytics Machine for virtualization for the first time, you must first install Oracle VM Server 3.0.3 and then upgrade to Oracle VM Server 3.2.7.

    1. Installing and Configuring Oracle VM Manager

If you are a new customer, you can install Oracle VM Manager 3.2.7 directly.

    1. Hardware Requirements


    1. Installing Oracle VM Manager 3.2.7

Oracle recommends that you install Oracle VM Manager on a separate machine than the Exalytics Machine, preferably collocated geographically within Oracle VM Server.
  1. Start up the operating system on the computer on which you want to install Oracle VM Manager.
  2. Download Oracle VM Manager 3.2.7 (Part Number V42727-01) from the Oracle Software Delivery Cloud web site at the following link:
  1. Insert and mount the Oracle VM Manager CD.
# mkdir mount-point
# mount -o loop OracleVM-Manager-version.iso mount-point
  1. As the root user, start the Oracle VM Manager installation script:
# ./runInstaller.sh
  1. Follow the prompts to install Oracle VM Manager.

    1. Discovering Oracle VM Servers

Adding Oracle VM Servers to your Oracle VM Manager environment is known as discovering Oracle VM Servers.

  1. Connect to Oracle VM Manager by entering the following address in a Web browser.
  1. Select the Servers and VMs tab.
  2. In the left pane, right-click Server Pools, and then select Discover Servers.
  3. Enter the following information for the server:
    1. Oracle VM Agent Port. The default port number is 8899.
    2. Oracle VM Agent Password. The default password is “oracle”.
    3. IP Address of OVM Server.
  4. Click OK.

Oracle VM Server and its details, including number of processors, IP address, processor speed, RAM, networking ports and configuration are discovered and added to the Unassigned Servers folder.

    1. Troubleshooting Network on Oracle VM Server

  1. https://support.oracle.com/epmos/adf/images/t.gif Problem Summary
    —————————————————
    Mapping of Oracle VM-Server with VM-Manager

    Problem Description
    —————————————————
    Unable to discover servers from Oracle VM-Manager
    —————————————————————————————————————–
    Job Construction Phase
    ———————-
    Job ID: 1405323971930

    begin()
    Appended operation ‘Discover Manager Server Discover’ to object ‘OVM Foundry : Discover Manager’.
    commit()
    Completed Step: COMMIT

    Objects and Operations
    ———————-
    Object (IN_USE): [DiscoverManager] OVM Foundry : Discover Manager
    Operation: Discover Manager Server Discover

    Job Running Phase at 2014-07-14 03:46:11,930
    ———————————————-
    Job Participants: []


    Actioner
    ——–
    03:46:14,319: Starting operation ‘Discover Manager Server Discover’ on object ‘OVM Foundry : Discover Manager’
    Setting Context to model only in job with id=1405323971930
    Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: get_api_version to server: 192.168.74.17 failed. OVMAPI_4004E Server Failed Command: get_api_version , Status: org.apache.xmlrpc.XmlRpcException: I/O error while communicating with HTTP server: No route to host [Mon Jul 14 03:46:17 EDT 2014] [Mon Jul 14 03:46:17 EDT 2014]
    at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:513)
    at com.oracle.ovm.mgr.action.ActionEngine.sendUndispatchedServerCommand(ActionEngine.java:400)
    at com.oracle.ovm.mgr.action.ServerAction.getSupportedApiVersions(ServerAction.java:314)
    at com.oracle.ovm.mgr.discover.DiscoverEngine.getServerApiVersions(DiscoverEngine.java:446)
    at com.oracle.ovm.mgr.discover.DiscoverEngine.discoverNewServer(DiscoverEngine.java:286)
    at com.oracle.ovm.mgr.discover.DiscoverEngine.discoverServer(DiscoverEngine.java:203)
    at com.oracle.ovm.mgr.op.manager.DiscoverManagerServerDiscover.action(DiscoverManagerServerDiscover.java:48)
    at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)
  2. Solutions
    1. Ping result from OVM manger to OVM server(192.168.74.17)
    2. Result of the following command from OVM manager:
      # nc -nv   8899
    3. Result of the following command from the OVS server:
      # nc -nv 7002
    4. Sosreport of the OVS server which you are trying to add.
      Vmpinfo( Doc ID 1364933.1 )
    5. Get bridge connection
#brctl show
    1. Worked for me: To check the status of the link (on which Ethernet port cable is attached)
# ethtool eth0
    1. For me it was on eth2 so replicated ifcfg-eth0 file with ifcfg-eth2 in /etc/sysconfig/network-scripts/ and edited hwaddr and device name with eth2 one.
    2. Start network services: #service network restart/reload
    3. Shutdown bond1 interface: #ifdown bond1

    1. Creating Virtual Network Interface Controllers

Define MAC Addresses for each VM.
  1. Log on to Oracle VM Manager.
  2. Select the Servers and VMs tab.
  3. In the left pane, right-click Server Pools, and then select Create VNICs.
  4. Click Next to retrieve the next available MAC addresses.
  5. Select the number of addresses that you want to create, and click Create.
  6. Click Close.

    1. Creating Server Pools and Adding a Server

A server pool consists of at least one Oracle VM Server.


  1. Click the Servers and VMs tab.
  2. Click Create Server Pool in the toolbar to start the Create Server Pool wizard.
  3. Fill in all the details to create a new server pool.

    1. Creating an Exalytics Repository

A repository stores Oracle VM resources, such as virtual machines, templates for creating virtual machines, virtual machine assemblies, and so on.
  1. Log on to Oracle VM Manager.
  2. Select the Repositories tab.
  3. In the toolbar, click Create New Repository to display the Create a Data Repository dialog.
  4. Perform the following actions:
    1. In the Repository Name field, enter the name for the repository.
    2. For Repository Location, select Physical Disk and click Search.
  1. From the drop-down list, select a local storage array and select the smaller of the two storage disks.
  2. Click OK.
  3. Select the server pool for the repository.
  4. Click Next.
  1. Select and move Oracle VM Server from the Servers field to the Present to Server(s) field and click Finish.

    1. Creating Virtual Machines

For creating the new virtual machines follow the given steps.
  1. Under OVM Manager, in Servers and VM tab select Create Virtual Machine.
  2. Select Create a new VM in popup wizard and click next.
  3. Give the appropriate details for the VM you want to create.
  4. Add a unassigned VNIC under Set up Networks tab.
  5. Under Arrange Disks, create a new virtual disk that will host the operating system and also select the RHEL OS setup cd-rom in that and click next.
  6. Arrange cd-rom at top under boot options so that your VM will start cd-rom instead of virtual disk.

    1. Configuring and Starting a Virtual Machine

Once you started a particular virtual machine in OVM manager, connect though it via Launch Console or launch serial console option. If its not working for you there is a workaround to get through this.

    1. Troubleshooting display for OVM Guest

For more inf Doc ID 1548825.1)
This method is actually based upon the tunneling of vnc services though putty.
  1. Login to vm server using ssh or ILOM remote console.
  2. Start the Guest Virtual Machine from OVM Manager
  3. In VM Server, enter the following command:
# xm list
The Domain-0 and ID number for the virtual machine are displayed.
  1. Execute the following command:
xm list -l | grep 59
This will list the VNC port number upon which the Guest/Virtual Machine is running.
  1. Now we have to tunnel this port via putty.

    1. Tunneling VNC Port via Putty

  1. Start Putty client.
  2. Give the IP Address of the OVM Server.

  1. Now in category goto Connection>>SSH>>Tunnels

    1. Give details as:
      1. Source Port:
      2. Destination: localhost:

    1. Click Add the add the details to Forwarded ports list.

  1. Click Open
  2. Give username/password for OVM Server.
  3. Leave the Putty session open and start vnc client (tightvnc)
c:Program Files (x86)Javajdkbinjava -jar c:ilomtightvnc-jviewer.jar
  1. Specify server as localhost: and click Connect

  1. Guest OS screen will display. Go though the RHEL installation steps to install the OS on respective VM.

1 thought on “Configuring Exalytics Machine for Virtualization”

Leave a Comment

Your email address will not be published. Required fields are marked *