RAC Attack - Oracle Cluster Database at Home/RAC Attack 12c/Print Book

From Wikibooks, open books for an open world
Jump to: navigation, search
RAC Attack - Oracle Cluster Database 12c at Home

Introduction[edit]

Overview[edit]

RAC Attack is a free curriculum and platform for hands-on learning labs related to Oracle RAC (cluster database), motivated by the belief that the best way to learn RAC is through lots of hands-on experience. The original contributors were Jeremy Schneider, Dan Norris and Parto Jalili. This curriculum has been used since 2008 by organizers of events, by instructors in classes and by individuals at home. Its goal is to help students learn about Oracle RAC cluster databases through guided examples.

RAC Attack differs in depth from other tutorials currently available.

  • Every keystroke and mouse click is carefully documented here.
  • The process is covered from the very beginning to the very end - from the very first installation of the Virtual Hypervisor on your laptop to various experiments on your running cluster database... with everything in between.
  • The labs in the main workbook have been tested thoroughly and repeatedly.

To learn about upcoming RAC Attack events or to organize one yourself, visit the Events page. You can use the shortcut http://racattack.org/events to access this page at any time.

12c Overview[edit]

The 12c version of RAC Attack was written collaboratively by many authors all around the world. A full list of contributors is available by clicking the "history" tab on any wiki page or at the end of the print book. Note that Seth Miller's contributions are undercounted; he wrote almost all of the original instructions up to the first node config but many of his initial edits were on a set of draft pages whose content was copied here. Ludovico Caldara and Björn Rost also made extraordinary contributions to the book as reflected in the contributor lists.

Additionally, credit goes to the many volunteer testers who reported issues with the first draft of instructions. Many of their names can be seen in the mailing list archives at http://racattack.org/list during August and September of 2013.

And most importantly, we can't give enough credit to the entire Oracle Openworld 2013 team. Especially Yury Velikanov who not only led the organization of officers and assignments but kept the energy and excitement level at stratospheric levels! Without the excitement of presenting at OpenWorld, we'd never have maintained such great momentum for finishing the first 12c revision so quickly!

Architecture[edit]

To better understand the RAC Installation, this picture illustrates the architecture that is implemented when following the book.

RAC Attack 12c architecture

IP Addresses[edit]

In order to install a fully functional RAC, the following IP addresses are required:

  • 2 public IPs, one for each node, for the primary OS network interface
  • 2 public IPs, one for each node, for the Virtual IP
  • 3 public IPs, one for each SCAN listener
  • 2 private IPs, one for each node, for the cluster private interconnect

In the book, the public addresses belong to the network 192.178.78.0/24, and the private addresses belong to the network 172.16.100.0/24.


Technical choices[edit]

The book aims to provide instructions as simple as possible to get a basic RAC installation on your laptop. There are many, many advanced topologies and topics that are not covered here. If you are curious about technical possibilities, just ask a volunteer, he/she will be glad to explain you something more.


Hardware Requirements[edit]

This handbook will walk you through the process of creating a two-node Oracle RAC cluster on your own laptop or desktop computer.

Hardware Minimum Requirements[edit]

Most modern laptop and desktop computers should be powerful enough to run a two-node virtual RAC cluster. In a nutshell, these are the recommended minimums:

  • Modern CPU (most of laptops produced after 2011 should be ok)
  • 8Gb memory
  • 40Gb of free disk space
    • 9Gb Software Staging
    • 29Gb - 2VMs + 2 ASM disks
  • Windows 64bit (XP, Vista or 7) (Linux & Mac have been tested as well and differences to the Windows instructions are included in the book)


Software Components[edit]

Before starting you need to know what software will be installed. If attending an event, would be a good idea to download the software in advance to your laptop in order to avoid the download during the labs. The copyrighted software is not distributable so the volunteers will not be able to give you all the required software components. But organizers may have set up a proxy server to speed up downloads or provide at least the free software.

Windows 7 64 bit[edit]

This book covers Windows 7 64 bit as host even if all operating systems that can run VirtualBox 64bit can be used. 64 bits are mandatory since Oracle 12c for Linux 32bit is not available.


VirtualBox[edit]

This book uses VirtualBox as many Oracle specialists consider it as a mature and free virtualization solution, fully compatible with Oracle Software: https://www.virtualbox.org/. The VirtualBox versions from 4.2.12 up to 4.2.18 have been tested successfully with OEL6 and Oracle 12c.

Putty[edit]

One of the preferred SSH clients for Windows. You can download it here:

http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe


Vnc Viewer[edit]

The servers will be installed without X server, so you'll need VNC to get the graphics:

http://www.realvnc.com/download/get/1295/

Oracle Software[edit]

Oracle Enterprise Linux 6.4[edit]

Can be downloaded from http://edelivery.oracle.com/linux/.

  1. Login to: http://edelivery.oracle.com/linux with your Oracle Account
  2. On the Media Pack Search page, select:

    Product pack: Oracle Linux

    Platform: x86 64 bit

    Click GO

    Linux Media Pack Search


  3. Click on: Oracle Linux Release 6 Update 4 Media Pack for x86_64 (64 bit)

    Linux Media Pack Search Results


  4. Click on the download button at the first line:

    Linux Media Pack Download


  5. Once the download of the file V37084-01.iso is finished, rename it to Oracle_Linux_6_64.iso to distinguish it easily from other Oracle files.


Oracle Grid Infrastructure and Database 12c (12.1.0.1)[edit]

Can be downloaded from http://edelivery.oracle.com/.

  1. Login to: http://edelivery.oracle.com/ with your Oracle Account
  2. On the Media Pack Search page, select:

    Product pack: Oracle Database

    Platform: Linux x86-64

    Click GO

    Database Media Pack Search


  3. Click on: Oracle Database 12c Release 1 (12.1.0.1.0) Media Pack for Linux x86-64

    Database Media Pack Search Results


  4. Click on the download button for the following four packs:
    • Oracle Database 12c Release 1 (12.1.0.1.0) for Linux x86-64 (Part 1 of 2)
    • Oracle Database 12c Release 1 (12.1.0.1.0) for Linux x86-64 (Part 2 of 2)
    • Oracle Database 12c Release 1 Grid Infrastructure (12.1.0.1.0) for Linux x86-64 (Part 1 of 2)
    • Oracle Database 12c Release 1 Grid Infrastructure (12.1.0.1.0) for Linux x86-64 (Part 2 of 2)

    Database Media Pack Download

  5. The following files are downloaded:
    • V38501-01_1of2.zip
    • V38501-01_2of2.zip
    • V38500-01_1of2.zip
    • V38500-01_2of2.zip

    After uncompressing them, there will be two folders: "grid" and "database" containing the full installation.

    Prepare Host[edit]

    VirtualBox Setup[edit]

    1. Run Oracle VM VirtualBox 4.2.16 Setup.
    2. On the welcome page, click Next:

      Vbox install welcome


    3. Leave the default component selection, click Next:

      Vbox install component selection


    4. Click Next:

      Vbox install create shortcuts


    5. Unless you have pending downloads or batches over the network, click Yes:

      Vbox install network reset


    6. Click Install:

      Vbox install ready to install


    7. The installation is completed. Click Finish to open VirtualBox:

      Vbox install completed



    VirtualBox Network Configuration[edit]

    1. Run VirtualBox Manager.
    2. From the main screen, choose File > Preferences > Network.

      Network adapter preferences

    3. Double click on VirtualBox Host-Only Ethernet Adapter. Update the settings as shown and click Ok twice.

      IPv4 Address: 192.168.78.1

      IPv4 Network Mask: 255.255.255.0

      Network adapter 1 address


    Install Linux[edit]

    Create VirtualBox VM[edit]

    1. On your computer, create a new folder racattack12c on a location of your choice that will contain all the virtual disks.

      At the end of the installation the files will look like this:

      RA-Windows_7-Create Directory



    2. In VirtualBox, from the main screen, click the New icon in the upper left hand corner.

      VirtualBox Manager Home



    3. Type in collabn1 for the Name of the VM. Choose Linux for the Type and Oracle (64 bit) for the Version and click Next.

      RA-vbox 4214-create vm-name and os



    4. Type 3072 in the Size field and click Next.

      RA-vbox 4214-create vm-vm memory



    5. Leave Create a virtual hard drive now selected and click Create.

      RA-vbox 4214-create vm-create primary hard disk



    6. Leave VDI (VirtualBox Disk Image) selected and click Next.

      RA-vbox 4214-create vm-primary hard disk type



    7. Leave Dynamically allocated selected and click Next.

      RA-vbox 4214-create vm-storage on drive



    8. Type in the full path or choose through the browse icon the Location for the disk file to be created.

      It's better to use the common folder racattack12c previously created for all the virtual disks.

      The file should be named collabn1.vdi.

      Type in 30.00 GB in the Size field and click Create.


      RA-vbox 4214-create vm-file location and size





    VirtualBox VM Settings[edit]

    1. From the main screen, select the virtual machine collabn1 and click the Settings icon in the upper left hand corner.

      RA-vbox 4214-create vm-finished vm



    2. Open the USB sub-menu. Uncheck the Enable USB Controller check box.

      RA-vbox 4214-create vm-usb settings



    3. Open the Storage sub-menu. Delete the Empty disk under the IDE Controller.

      RA-vbox 4214-create vm-delete empty ide



    4. Click on Remove to confirm.

      RA-vbox 4214-create vm-delete ide


    5. Click on the Add CD/DVD Device.

      RA-vbox 4214-create vm-add iso



    6. Click on the Choose disk.

      VirtualBox New VM Settings New IDE



    7. Locate and open the Oracle_Linux_6_64.iso file.

      RA-vbox 4214-create vm-oracle linux 6 iso added



    8. Open the Network sub-menu. Under the Adapter 1 tab, change the Attached to: dropdown to Host-only Adapter.

      VirtualBox_Settings_Network_Adapter_1




    9. Choose the Adapter 2 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to Internal Network and type the name rac-priv in the Name field.

      VirtualBox_New_VM_Settings_Network_Adapter_2



    10. Choose the Adapter 3 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to NAT.

      VirtualBox_New_VM_Settings_Network_Adapter_3



    11. Select the Shared Folders tab. Click Add Shared Folder.

      Enter the path where you have downloaded the Oracle installation media and enter a name for your folder.

      Check Auto-mount and click OK.

      VirtualBox_New_VM_Settings_Shared folder


    12. Click on Ok to save the modifications.


    13. Review the summary of the new virtual machine.



    OS Installation[edit]

    1. Click on Start to boot the virtual machine. If asked to select a start-up disk, choose Oracle_Linux_6_64.iso and click Start again.

      RA-Oracle_Linux_6_64bit-Install_OS-select boot iso



    2. The Oracle Linux install screen will automatically come up. Hit enter to begin the install.

      RA-Oracle_Linux_6_64bit-Install_OS-boot menu



    3. Skip the CD testing by pressing the right arrow key and press enter.

      RA-Oracle_Linux_6_64bit-Install_OS-media test



    4. From this point on you can use the mouse to navigate. To unlock the mouse and keyboard from the VM, hit the right Ctrl key. Click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-splash start



    5. Leave English (English) selected and click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-choose language



    6. Leave U.S. English selected and click Next (unless you have another preferred keyboard layout).

      RA-Oracle_Linux_6_64bit-Install_OS-choose keyboard



    7. Leave Basic Storage Devices selected and click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-install device



    8. Click Yes, discard any data for the warning box stating The storage device below may contain data.

      RA-Oracle_Linux_6_64bit-Install_OS-storage device warning



    9. In the hostname field, type collabn1.racattack.
      • Click Configure Network in the bottom left corner.
      • Highlight System eth0 and click the Edit... button.
      • Click the Connect automatically checkbox.
      • Click the IPv4 Settings tab.
      • Click the Method: dropdown and select Manual.
      • Click Add and type 192.168.78.51 for the address.
      • Leave Netmask at 24.
      • In the DNS servers: box, type 192.168.78.51, 192.168.78.52.
      • In the Search domains: box, type racattack.
      • Click the Apply button.

      RA-Oracle_Linux_6_64bit-Install_OS-eth0 ipv4 settings



    10. Highlight System eth1 and click the Edit... button.
      • Click the Connect automatically checkbox.
      • Click the IPv4 Settings tab.
      • Click the Method: dropdown and select Manual.
      • Click Add and type 172.16.100.51.
      • Change Netmask to 24.
      • Click the Apply button.

      RA-Oracle_Linux_6_64bit-Install_OS-eth1 ipv4 settings



    11. Highlight System eth2 and click the Edit... button.
      • Click the Connect automatically checkbox.
      • Click the IPv4 Settings tab.
      • Click the Method: dropdown and select Automatic (DHCP) addresses only.
      • Click the Apply button.
      • Click Close to close the network configuration menu and click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-eth2 ipv4 settings



    12. Leave the timezone settings as they are and click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-time zone



    13. Enter the Root Password as racattack and click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-root password



    14. Click Use Anyway when warned about the weak password.

      RA-Oracle_Linux_6_64bit-Install_OS-root password confirm



    15. Click Use All Space for the installation type and click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-use all disk space



    16. Click Write changes to disk when warned about Writing storage configuration to disk.

      RA-Oracle_Linux_6_64bit-Install_OS-write changes to disk confirm




    17. Leave Basic Server as the default installation type. Click Next.

      RA-Oracle_Linux_6_64bit-Install_OS-default installation type




    18. The operating system will take a while to install.

      RA-Oracle_Linux_6_64bit-Install_OS-install progressing



    19. When the installation is complete, click Reboot.

      RA-Oracle_Linux_6_64bit-Install_OS-complete congrats



    20. The reboot no longer shows messages, just a progress bar.
      • You can see the progress by hitting F8 while the server is booting.
      • The system should boot to a command line login prompt.

      RA-Oracle_Linux_6_64bit-Install_OS-post install reboot progress



    Linux Post Installation[edit]

    1. To connect to the new created virtual machine, create a connection in Putty for the new VM called collabn1 with the IP Address of 192.168.78.51.


      • Optionally, create another connection for collabn2 with the IP Address of 192.168.78.52 that will be created later as a clone of collabn1.

      Putty connection screen

    2. Open the collabn1 connection by clicking Open.
      • Click Yes on the Security Alert dialog box.

      Putty security alert

    3. Enter root at the login as: prompt and racattack at the password: prompt.

      Putty connected

    4. Turn off and disable the firewall IPTables.


      [root@collabn1 ~]# service iptables stop
      iptables: Flushing firewall rules:                         [  OK  ]
      iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
      iptables: Unloading modules:                               [  OK  ]
      
      [root@collabn1 ~]# chkconfig iptables off
      
      [root@collabn1 ~]# chkconfig --list iptables
      iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off
      
    5. Disable SELinux. Open the config file and change the SELINUX variable from enforcing to disabled.
      [root@collabn1 ~]# vim /etc/selinux/config
      
       # This file controls the state of SELinux on the system.
       # SELINUX= can take one of these three values:
       #     enforcing - SELinux security policy is enforced.
       #     permissive - SELinux prints warnings instead of enforcing.
       #     disabled - No SELinux policy is loaded.
       SELINUX=disabled
       # SELINUXTYPE= can take one of these two values:
       #     targeted - Targeted processes are protected,
       #     mls - Multi Level Security protection.
       SELINUXTYPE=targeted
      
    6. Verify that all the network interfaces are up.
      [root@collabn1 ~]# ip l
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
          link/ether 08:00:27:4f:8a:0b brd ff:ff:ff:ff:ff:ff
      3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
          link/ether 08:00:27:99:7e:95 brd ff:ff:ff:ff:ff:ff
      4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
          link/ether 08:00:27:e2:63:7f brd ff:ff:ff:ff:ff:ff
      


    7. Add the DVD as repository.

      Go to the VBOX menu => Storage => Add the DVD device to the server Device -> CD/DVD Devices -> Oracle_Linux_6_4.iso

      Run the following commands:

      mkdir /media/cdrom
      mount /dev/cdrom /media/cdrom/
      cd /etc/yum.repos.d
      mkdir /tmp/old.yum.repos.d
      mv * /tmp/old.yum.repos.d
      
      cat <<EOF > ol64.repo
      [OL64]
      name=Oracle Linux 6.4 x86_64
      baseurl=file:///media//cdrom
      gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
      gpgcheck=1
      enabled=1
      EOF
      
    8. Install the database preinstall package oracle-rdbms-server-11gR2-preinstall.
      • This package installs all the package requirements for Oracle 11g (but will work also for 12c), configures kernel parameters, creates oracle user and groups and sets the user limits.
      [root@collabn1 ~]# yum install -y oracle-rdbms-server-11gR2-preinstall
      ....
      Transaction Summary
      ========================================================================
      Install      21 Package(s)
      Upgrade       2 Package(s)
      
      Total download size: 43 M
      
      ....
      
      Installed:
        oracle-rdbms-server-11gR2-preinstall.x86_64 0:1.0-8.el6
      
      ....
      
      Complete!
      
    9. Install additional RPMs that will be used to install and administer the servers.
      [root@collabn1 ~]# rpm -ivh /media/cdrom/Packages/kernel-uek-devel-$(uname -r).rpm
      ....
      [root@collabn1 ~]# yum install -y tigervnc-server.x86_64 xclock man parted.x86_64 unzip.x86_64 xterm lsof bind  xorg-x11-twm
      Install      12 Package(s)
      .... 
      Total download size: 22 M
      ....
      Complete!
      
    10. Uninstall NTP.
      [root@collabn1 ~]# yum remove -y ntp
      ....
      Removed:
       ntp.x86_64 0:4.2.4p8-3.el6
      ....
      Complete!
      
    11. Unmount the DVD before continuing with the next steps.
      [root@collabn1 ~]# umount /media/cdrom
      


      Install VirtualBox Addons[edit]

      1. Make the VirtualBox guest additions available to the OS by clicking Devices->Install Guest Additions.

        RA-Oracle_Linux_6_64bit-Configure_OS-install guest additions

      2. Mount the virtual CD-Rom created by VirtualBox.
        [root@collabn1 ~]# mount /dev/cdrom /media/cdrom
         mount: block device /dev/sr0 is write-protected, mounting read-only
        
      3. Install the VirtualBox Guest Additions. The error related to the step Installing the Window System drivers is ok to ignore, it will not appear on newest VBox versions.
        [root@collabn1 ~]# sh /media/cdrom/VBoxLinuxAdditions.run
         Verifying archive integrity... All good.
         Uncompressing VirtualBox 4.2.14 Guest Additions for Linux............
         VirtualBox Guest Additions installer
         Removing installed version 4.2.14 of VirtualBox Guest Additions...
         Copying additional installer modules ...
         Installing additional modules ...
         Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
         Building the VirtualBox Guest Additions kernel modules
         The headers for the current running kernel were not found. If the following
         module compilation fails then this could be the reason.
         The missing package can be probably installed with
         yum install kernel-uek-devel-2.6.39-400.17.1.el6uek.x86_64
         
         Building the main Guest Additions module                   [  OK  ]
         Building the shared folder support module                  [  OK  ]
         Building the OpenGL support module                         [  OK  ]
         Doing non-kernel setup of the Guest Additions              [  OK  ]
         You should restart your guest to make sure the new modules are actually used
         
         Installing the Window System drivers                       [FAILED]
         (Could not find the X.Org or XFree86 Window System.)
        
      4. Dismount the cdrom.
        [root@collabn1 ~]# umount /media/cdrom
        


      Configure Bind DNS[edit]

      1. Enable BIND DNS to start at boot time.
        [root@collabn1 ~]# chkconfig named on
        
      2. Change named directory permissions.
        [root@collabn1 ~]# touch /var/named/racattack
        [root@collabn1 ~]# chmod 664 /var/named/racattack
        [root@collabn1 ~]# chgrp named /var/named/racattack
        [root@collabn1 ~]# chmod g+w /var/named
        [root@collabn1 ~]# chmod g+w /var/named/racattack
        
      3. Backup the BIND configuration file.
        [root@collabn1 ~]#  cp /etc/named.conf /etc/named.conf.org
        
      4. Run the following command or edit the /etc/named.conf file to change the named configuration manually.
        sed -i -e 's/listen-on .*/listen-on port 53 { 192.168.78.51; };/' \
        -e 's/allow-query .*/allow-query     { 192.168.78.0\/24; localhost; };\n        allow-transfer  { 192.168.78.0\/24; };/' \
        -e '$azone "racattack" {\n  type master;\n  file "racattack";\n};\n\nzone "in-addr.arpa" {\n  type master;\n  file "in-addr.arpa";\n};' \
        /etc/named.conf
        
        • In bold the lines that have been modified from the default.
        options {
               listen-on port 53 { 192.168.78.51; };
               listen-on-v6 port 53 { ::1; };
               directory       "/var/named";
               dump-file       "/var/named/data/cache_dump.db";
               statistics-file "/var/named/data/named_stats.txt";
               memstatistics-file "/var/named/data/named_mem_stats.txt";
               allow-query     { 192.168.78.0/24; localhost; };
               allow-transfer  { 192.168.78.0/24; };
               recursion yes;
        
               dnssec-enable yes;
               dnssec-validation yes;
               dnssec-lookaside auto;
        
               /* Path to ISC DLV key */
               bindkeys-file "/etc/named.iscdlv.key";
        
               managed-keys-directory "/var/named/dynamic";
        };
        
        logging {
               channel default_debug {
                       file "data/named.run";
                       severity dynamic;
               };
        };
        
        zone "." IN {
               type hint;
               file "named.ca";
        };
        
        include "/etc/named.rfc1912.zones";
        include "/etc/named.root.key";
        
        zone "racattack" {
         type master;
         file "racattack";
        };
        
        zone "in-addr.arpa" {
         type master;
         file "in-addr.arpa";
        };
        
      5. Create the zone file for the racattack domain on collabn1 by running the following command:

        (Copy & Paste the whole box)

        echo '$TTL 3H
        @       IN SOA  collabn1        hostmaster      (
                                                101   ; serial
                                                1D      ; refresh
                                                1H      ; retry
                                                1W      ; expire
                                                3H )    ; minimum
                        NS      collabn1
                        NS      collabn2
        localhost       A       127.0.0.1
        collabn1        A       192.168.78.51
        collabn1-vip    A       192.168.78.61
        collabn1-priv   A       172.16.100.51
        collabn2        A       192.168.78.52
        collabn2-vip    A       192.168.78.62
        collabn2-priv   A       172.16.100.52
        collabn-cluster-scan     A       192.168.78.251
        collabn-cluster-scan     A       192.168.78.252
        collabn-cluster-scan     A       192.168.78.253' \
        > /var/named/racattack
        
      6. Create the reverse zone file on collabn1.

        (Copy & Paste the whole box)


        echo '$TTL 3H
        @       IN SOA  collabn1.racattack.        hostmaster.racattack.      (
                                                101   ; serial
                                                1D      ; refresh
                                                1H      ; retry
                                                1W      ; expire
                                                3H )    ; minimum
                        NS      collabn1.racattack.
                        NS      collabn2.racattack. 
        
        51.78.168.192   PTR     collabn1.racattack.
        61.78.168.192   PTR     collabn1-vip.racattack.
        51.100.16.172   PTR     collabn1-priv.racattack.
        52.78.168.192   PTR     collabn2.racattack.
        62.78.168.192   PTR     collabn2-vip.racattack.
        52.100.16.172   PTR     collabn2-priv.racattack.
        251.78.168.192  PTR     collabn-cluster-scan.racattack.
        252.78.168.192  PTR     collabn-cluster-scan.racattack.
        253.78.168.192  PTR     collabn-cluster-scan.racattack.' \
        > /var/named/in-addr.arpa
        
      7. Generate the rndc.key file.
        [root@collabn1 ~]# rndc-confgen -a -r /dev/urandom
         wrote key file "/etc/rndc.key"
        
        chgrp named /etc/rndc.key
        chmod g+r /etc/rndc.key
        
      8. Restart the named service.
        [root@collabn1 ~]# service named restart
         Stopping named:                                            [  OK  ]
         Starting named:                                            [  OK  ]
        
      9. Check that the parameter PEERDNS is set to no in /etc/sysconfig/network-scripts/ifcfg-eth2 to prevent the resolv.conf from being overwritten by the dhcp client:
        DEVICE=eth2
        TYPE=Ethernet
        UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
        ONBOOT=yes
        NM_CONTROLLED=yes
        BOOTPROTO=dhcp
        HWADDR=xx:xx:xx:xx:xx
        DEFROUTE=yes
        PEERDNS=no
        PEERROUTES=yes
        IPV4_FAILURE_FATAL=yes
        IPV6INIT=no
        NAME="System eth2"
        USERCTL=no
        

        note: I (Yury) found that the following two should be set to NO to => DEFROUTE=no, PEERROUTES=no

      10. If it was set to yes previously, restart the network and verify that the file /etc/resolv.conf contains now the correct nameservers:
        [root@collabn1 ~]# service network restart
        Shutting down interface eth0:                              [  OK  ]
        Shutting down interface eth1:                              [  OK  ]
        Shutting down interface eth2:                              [  OK  ]
        Shutting down loopback interface:                          [  OK  ]
        Bringing up loopback interface:                            [  OK  ]
        Bringing up interface eth0:                                [  OK  ]
        Bringing up interface eth1:                                [  OK  ]
        Bringing up interface eth2:
        Determining IP information for eth2... done.
                                                                  [  OK  ]
        
      11. /etc/resolv.conf should contain:
        [root@collabn1 ~]#  cat /etc/resolv.conf
        ; generated by /sbin/dhclient-script
        nameserver 192.168.78.51
        nameserver 192.168.78.52
        search racattack
        
      12. Check that the master DNS on collabn1 is working.
        [root@collabn1 ~]# nslookup collabn-cluster-scan.racattack
        Server:         192.168.78.51
        Address:        192.168.78.51#53
        
        Name:   collabn-cluster-scan.racattack
        Address: 192.168.78.251
        Name:   collabn-cluster-scan.racattack
        Address: 192.168.78.252
        Name:   collabn-cluster-scan.racattack
        Address: 192.168.78.253
        


      Prepare Linux for Oracle[edit]

      1. Modify the pam.d login file to use limits.
        sed -i -e '/session    required     pam_selinux.so open/i\
        session    required     \/lib64\/security\/pam_limits.so\
        session    required     pam_limits.so' /etc/pam.d/login
        
      2. Change the password for the oracle user to racattack.
        [root@collabn1 ~]# passwd oracle
         Changing password for user oracle.
         New password:
         BAD PASSWORD: it is based on a dictionary word
         Retype new password:
         passwd: all authentication tokens updated successfully.
        
      3. Create and change the ownership of the directories for Clusterware and the Database installations.
        [root@collabn1 ~]# mkdir -p /u01/app
        [root@collabn1 ~]# chown oracle:oinstall /u01/app
        
      4. Modify the Oracle user so that it belongs to the vboxsf group.
        [root@collabn1 ~]# usermod -G oinstall,dba,vboxsf oracle
        [root@collabn1 ~]# id oracle
         uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf)
        


      Create Cluster[edit]

      Create VirtualBox Shared Storage[edit]

      1. Shutdown the server:
        [root@collabn1 ~]# shutdown -h now
        
        Broadcast message from root@collabn1.racattack
               (/dev/pts/0) at 9:06 ...
        
        The system is going down for halt NOW!
        
      2. In the VirtualBox Manager, select the machine collabn1, click Settings -> Storage.

        Select Controller SATA and click on the Add Hard Disk button:

        Virtualbox 4: Create Shared Storage - Add Hard Disk


        Click on Create New Disk:

        Virtualbox 4: Create Shared Storage - Select New Disk


      3. Leave VDI (VirtualBox Disk Image) selected and click Next.

        RA-vbox 4214-create vm-primary hard disk type


      4. For the shared storage, select Fixed size and click Next.

        Virtualbox 4: Create Shared Storage - Fixed Size


      5. Change the location of the file to the folder racattack12 previously created.

        Enter asm1.vdi as disk name.

        Enter 5.00Gb as disk size.

        Click Create.

        Virtualbox 4: Create Shared Storage - Disk Name and Size

      6. Now the disk is physically created with a size of 5Gb, so it can take more time to complete.

        Virtualbox 4: Create Shared Storage - Creation in progress

      7. Repeat the steps to create three more disks:
        • asm2.vdi (5.00GB)
        • asm3.vdi (5.00GB)
        • asm4.vdi (5.00GB)
      8. Finally, four virtual disks should have been created for asm.

        Click OK.

        Virtualbox 4: Create Shared Storage - 4 disks created

      9. Make all the asm devices sharable: from VirtualBox Manager, click File -> Virtual Media Manager.

        Virtualbox 4: Create Shared Storage - virtual media manager

      10. For each asm devices: right-click -> Modify:

        Virtualbox 4: Create Shared Storage - virtual media manager modify

      11. Select Shareable and click OK. Do it for the four asm devices.

        Virtualbox 4: Create Shared Storage - virtual media manager shareable

      12. Start the machine collabn1.

        You should see now four more disks named /dev/sd*:

        [root@collabn1 ~]# ls -l /dev/sd*
        brw-rw----. 1 root disk 8,  0 Jul 28 10:16 /dev/sda
        brw-rw----. 1 root disk 8,  1 Jul 28 10:16 /dev/sda1
        brw-rw----. 1 root disk 8,  2 Jul 28 10:16 /dev/sda2
        brw-rw----. 1 root disk 8, 16 Jul 28 10:16 /dev/sdb
        brw-rw----. 1 root disk 8, 32 Jul 28 10:16 /dev/sdc
        brw-rw----. 1 root disk 8, 48 Jul 28 10:16 /dev/sdd
        brw-rw----. 1 root disk 8, 64 Jul 28 10:16 /dev/sde
        


      Configure Storage Persistent Naming[edit]

      1. Once the new disks are visible by the server, add a primary partition on each of them with fdisk.
        [root@collabn1 ~]# fdisk /dev/sdb
        Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
        Building a new DOS disklabel with disk identifier 0x97cc9f57.
        Changes will remain in memory only, until you decide to write them.
        After that, of course, the previous content won't be recoverable.
        
        Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
        
        WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
                switch off the mode (command 'c') and change display units to
                sectors (command 'u').
        
        Command (m for help): n
        Command action
          e   extended
          p   primary partition (1-4)
        p
        Partition number (1-4): 1
        First cylinder (1-652, default 1): <enter>
        Using default value 1
        Last cylinder, +cylinders or +size{K,M,G} (1-652, default 652): <enter>
        Using default value 652
        
        Command (m for help): w
        The partition table has been altered!
        
        Calling ioctl() to re-read partition table.
        Syncing disks.
        
      2. Repeat the same step for sdc, sdd and sde.

        When finished, all disks have at least a partition:

        [root@collabn1 ~]# ls -l /dev/sd?1
        brw-rw----. 1 root disk 8,  1 Jul 28 10:16 /dev/sda1
        brw-rw----. 1 root disk 8, 17 Jul 28 10:23 /dev/sdb1
        brw-rw----. 1 root disk 8, 33 Jul 28 10:27 /dev/sdc1
        brw-rw----. 1 root disk 8, 49 Jul 28 10:27 /dev/sdd1
        brw-rw----. 1 root disk 8, 65 Jul 28 10:27 /dev/sde1
        
      3. Configure the option -g for the scsi_id command to expect an UUID from the shared devices.


        [root@collabn1 ~]# echo "options=-g" > /etc/scsi_id.config
        
      4. Prepare the file /etc/udev/rules.d/99-oracle-asmdevices.rules by running the following script:

        http://en.wikibooks.org/w/index.php?title=RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c/Configure_Storage_Persistent_Naming&stable=0


        i=1
        cmd="/sbin/scsi_id -g -u -d"
        for disk in sdb sdc sdd sde ; do 
                 cat <<EOF >> /etc/udev/rules.d/99-oracle-asmdevices.rules
        KERNEL=="sd?1", BUS=="scsi", PROGRAM=="$cmd /dev/\$parent", \
         RESULT=="`$cmd /dev/$disk`", NAME="asm-disk$i", OWNER="oracle", GROUP="dba", MODE="0660"
        EOF
                 i=$(($i+1)) 
        done
        
      5. Once completed, the file should like the following (except for the scsi UUID).

        Note: The lines beginning with "KERNEL" should not be splitted.

        [root@collabn1 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
         KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB2216c54a-825b4598", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
         KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB83dd4d7f-d052e72c", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
         KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB636da22f-87dc2f88", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
         KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB86aaf297-60dcda74", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
        
      6. Reload the udev rules and restart udev:
        [root@collabn1 ~]# /sbin/partprobe /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
        [root@collabn1 ~]# /sbin/udevadm test /block/sdb/sdb1
        [root@collabn1 ~]# /sbin/udevadm test /block/sdc/sdc1
        [root@collabn1 ~]# /sbin/udevadm test /block/sdd/sdd1
        [root@collabn1 ~]# /sbin/udevadm test /block/sde/sde1
        [root@collabn1 ~]# /sbin/udevadm control --reload-rules
        [root@collabn1 ~]# /sbin/start_udev
        Starting udev:               [OK]
        
      7. The new disks are ready for use:
        [root@collabn1 ~]# ls -l /dev/asm*
         brw-rw----. 1 oracle dba 8, 17 Jul 30 16:17 /dev/asm-disk1
         brw-rw----. 1 oracle dba 8, 33 Jul 30 16:17 /dev/asm-disk2
         brw-rw----. 1 oracle dba 8, 49 Jul 30 16:17 /dev/asm-disk3
         brw-rw----. 1 oracle dba 8, 65 Jul 30 16:17 /dev/asm-disk4
        


      Clone VirtualBox VM[edit]

      1. Shutdown the VM.
        [root@collabn1 ~]# shutdown -h now
        
         Broadcast message from root@collabn1.racattack
                 (/dev/pts/0) at 8:42 ...
        
         The system is going down for halt NOW!
        
      2. Clone the disk collabn1.vdi: from VirtualBox Manager, click File -> Virtual Media Manager.


        Virtualbox 4: Virtual box manager

      3. Right-click on the disk collabn1.vdi and click Copy...


        Virtualbox 4: Clone Disk - Virtual Media Manager copy

      4. Leave the file collabn1.vdi, click Next.

        Virtualbox 4: Clone Disk - Virtual Media Manager HD to copy

      5. Leave VDI as file type. Click Next.

        Virtualbox 4: Clone Disk - HD type

      6. Leave Dinamically allocated. Click Next.

        Virtualbox 4: Clone Disk - allocation type

      7. Type in the full path or choose through the browse icon the Location for the disk file to be created.

        Again, it's better to use the common folder racattack12c previously created for all the virtual disks.

        The file should be named collabn2.vdi.

        Click Copy.

        Virtualbox 4: Clone Disk - new disk name

      8. Click Close to exit from the Virtual Media Manager.


      Create Second VirtualBox VM[edit]

      1. In VirtualBox, click New icon in the upper left corner to create the new VM.
      2. Type in collabn2 for the Name of the VM. Choose Linux for the Type and Oracle (64 bit) for the Version and click Next.

        Virtualbox 4: Second VM Name and OS

      3. Type 3072 in the Size field and click Next.

        RA-vbox 4214-create vm-vm memory

      4. Select Use an existing virtual hard drive file and use the folder icon to locate the file collabn2.vdi previously created.

        Click Create to complete the creation of the second VM.

        Virtualbox 4: Create second vm - Use existing HD

      5. There are now two virtual machines; collabn2 being a clone of collabn1.

        From the main screen, select the virtual machine collabn2 and click the Settings icon in the upper left hand corner.

        RA-Oracle_Linux_6_64bit-Configure_OS-finished cloning

      6. Open the USB sub-menu. Uncheck the Enable USB Controller check box.

        RA-vbox 4214-create vm-usb settings

      7. Open the Network sub-menu. Under the Adapter 1 tab, change the Attached to: dropdown to Host-only Adapter.

        VirtualBox_Settings_Network_Adapter_1

      8. Choose the Adapter 2 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to Internal Network and type the name rac-priv in the Name field.

        VirtualBox_New_VM_Settings_Network_Adapter_2

      9. Choose the Adapter 3 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to NAT.

        VirtualBox_New_VM_Settings_Network_Adapter_3

      10. Select the Shared Folders tab. Click Add Shared Folder.

        Enter the path where you've downloaded the Oracle installation media and enter a name for your folder.

        Check Auto-mount and click OK.

        VirtualBox_New_VM_Settings_Shared folder

      11. Open the Storage sub-menu.

        Click on Controller SATA and then on Add Disk:

        VirtualBox New VM Settings - Add shared storage

      12. Click on Choose existing disk:

        VirtualBox New VM Settings - Choose existing disk

      13. Select the disk asm1.vdi and click Open.

        VirtualBox New VM Settings - Choose existing disk ASM

      14. Repeat the operation to add all remaining asm disks.

        VirtualBox New VM Settings - Storage added

      15. Click on Ok to save the modifications.
      16. Review the summary of the new virtual machine.

        Vbox 4: Create Second VM - complete VM settings

      17. Start the collabn2 VM.


      Configure Second Linux VM[edit]

      1. The network won't be available yet because we reinitialized the network adapters so the following changes will need to be done in the VM. Log in as root/racattack. Type the command system-config-network. Device configuration is already highlighted. Press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network change

      2. eth0 (eth0) - Ethernet is already highlighted. Press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network save devices

      3. Go down to the Static IP line. Change the IP to 192.168.78.52. Go down to Ok and press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network eth0

      4. Select eth1 (eth1) - Ethernet and press Enter. Go down to the Static IP line. Change the IP to 172.16.100.52. Go down to Ok and press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network eth1

      5. Tab down to Save and press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network save devices

      6. Arrow down to DNS configuration and press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone DNS configuration

      7. In the Hostname field, change to collabn2.racattack. Tab down to Ok and press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network dns change

      8. Tab down to Save&Quit and press Enter.

        RA-Oracle_Linux_6_64bit-Configure_OS-post clone network save quit

      9. Check that the parameter PEERDNS is set to no in /etc/sysconfig/network-scripts/ifcfg-eth2 to prevent the resolv.conf from being overwritten by the dhcp client:
        DEVICE=eth2
        TYPE=Ethernet
        UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
        ONBOOT=yes
        NM_CONTROLLED=yes
        BOOTPROTO=dhcp
        HWADDR=xx:xx:xx:xx:xx
        DEFROUTE=yes
        PEERDNS=no
        PEERROUTES=yes
        IPV4_FAILURE_FATAL=yes
        IPV6INIT=no
        NAME="System eth2"
        USERCTL=no
        
      10. Remove the udev network rules file. It will be regenerated on the next reboot with the new MAC addresses.
        [root@collabn2 ~]# rm -f /etc/udev/rules.d/70-persistent-net.rules
        
      11. Remove the HWADDR and UUID lines in the network adapter configuration files.


        [root@collabn2 ~]# sed -i -e '/HWADDR/d' -e '/UUID/d' /etc/sysconfig/network-scripts/ifcfg-eth[0-2]
        
      12. Reboot the collabn2 VM by typing reboot.

        Start the collabn1 VM as well and start Putty sessions for both.

        RA-Oracle_Linux_6_64bit-Configure_OS-collabn1 reboot

      13. Verify that all the network interfaces are up.
        [root@collabn2 ~]# ip l
        1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
           link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
           link/ether 08:00:27:c9:39:d3 brd ff:ff:ff:ff:ff:ff
        3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
           link/ether 08:00:27:33:bd:a8 brd ff:ff:ff:ff:ff:ff
        4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
           link/ether 08:00:27:5f:61:29 brd ff:ff:ff:ff:ff:ff
        
      14. Verify that all the ASM devices are visible.
        [root@collabn2 ~]# ls -l /dev/asm*
        brw-rw----. 1 oracle dba 8, 17 Aug  5 15:40 /dev/asm-disk1
        brw-rw----. 1 oracle dba 8, 33 Aug  5 15:40 /dev/asm-disk2
        brw-rw----. 1 oracle dba 8, 49 Aug  5 15:40 /dev/asm-disk3
        brw-rw----. 1 oracle dba 8, 65 Aug  5 15:40 /dev/asm-disk4
        


      Complete DNS Setup[edit]

      1. Open a session on collabn2.
      2. Stop the DNS service.
        [root@collabn2 ~]# service named stop
        Stopping named: .                                          [  OK  ]
        
      3. Remove the actual DNS files.
        [root@collabn2 ~]# rm -f /var/named/racattack /var/named/in-addr.arpa
        
      4. Modify the file /etc/named.conf by using the following command:

        http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c/Complete_DNS_Setup

        sed -i -e 's/listen-on .*/listen-on port 53 { 192.168.78.52; };/' \
        -e 's/type master;/type slave;\n masters  {192.168.78.51; };/' \
        /etc/named.conf
        

        so that at the end it looks like this one:

        options {
               listen-on port 53 { 192.168.78.52; };
               listen-on-v6 port 53 { ::1; };
               directory       "/var/named";
               dump-file       "/var/named/data/cache_dump.db";
               statistics-file "/var/named/data/named_stats.txt";
               memstatistics-file "/var/named/data/named_mem_stats.txt";
               allow-query     { 192.168.78.0/24; localhost; };
               allow-transfer  { 192.168.78.0/24; };
               recursion yes;
        
               dnssec-enable yes;
               dnssec-validation yes;
               dnssec-lookaside auto;
        
               /* Path to ISC DLV key */
               bindkeys-file "/etc/named.iscdlv.key";
        
               managed-keys-directory "/var/named/dynamic";
        };
        
        logging {
               channel default_debug {
                       file "data/named.run";
                       severity dynamic;
               };
        };
        
        zone "." IN {
               type hint;
               file "named.ca";
        };
        
        include "/etc/named.rfc1912.zones";
        include "/etc/named.root.key";
        
        zone "racattack" {
         type slave;
         masters  { 192.168.78.51; };
         file "racattack";
        };
        
        zone "in-addr.arpa" {
         type slave;
         masters  { 192.168.78.51; };
         file "in-addr.arpa";
        };
        
      5. Start the named service.
        [root@collabn2 ~]# service named start
         Starting named:                                            [  OK  ]
        
      6. Check that both the master on collabn1 and slave on collabn2 DNS servers are working.

        Run all the following statements to verify the correct configuration of both DNS servers:

        http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c/Linux_Post_Installation_No_Internet

        [root@collabn2 ~]# dig @collabn1 collabn1.racattack
        [root@collabn2 ~]# dig @collabn1 collabn2.racattack
        [root@collabn2 ~]# dig @collabn1 collabn1-vip.racattack
        [root@collabn2 ~]# dig @collabn1 collabn2-vip.racattack
        [root@collabn2 ~]# dig @collabn1 collabn1-priv.racattack
        [root@collabn2 ~]# dig @collabn1 collabn2-priv.racattack
        [root@collabn2 ~]# dig @collabn1 collabn-cluster-scan.racattack
        
        [root@collabn2 ~]# dig @collabn2 collabn1.racattack
        [root@collabn2 ~]# dig @collabn2 collabn2.racattack
        [root@collabn2 ~]# dig @collabn2 collabn1-vip.racattack
        [root@collabn2 ~]# dig @collabn2 collabn2-vip.racattack
        [root@collabn2 ~]# dig @collabn2 collabn1-priv.racattack
        [root@collabn2 ~]# dig @collabn2 collabn2-priv.racattack
        [root@collabn2 ~]# dig @collabn2 collabn-cluster-scan.racattack
        


      VNC Server Setup[edit]

      1. Configure VNC Server with the oracle account (passwords won't be displayed):
        [root@collabn1 ~]# su - oracle
        [oracle@collabn1 ~]$ vncserver :1
        
        You will require a password to access your desktops.
        
        Password: racattack
        Verify: racattack
        xauth:  creating new authority file /home/oracle/.Xauthority
        
        New 'collabn1.racattack:1 (oracle)' desktop is collabn1.racattack:1
        
        Creating default startup script /home/oracle/.vnc/xstartup
        Starting applications specified in /home/oracle/.vnc/xstartup
        Log file is /home/oracle/.vnc/collabn1.racattack:1.log
        
      2. Open a vncviewer on your laptop and fill the ip address of collabn1 followed by :1.

        RA-VNCViewer-Connect form

      3. Enter the password racattack when prompted:

        RA-VNCViewer-Password form

      4. The graphical interface is ready for the Grid Infrastructure Installation.

        RA-VNCViewer-Connected


      Install Grid Infrastructure[edit]

      Prepare for GI Install[edit]

      1. Verify that the VirtualBox Shared folder containing the installation media is mounted correctly and that Oracle has access to it:
        [root@collabn1 ~]# su - oracle
        [oracle@collabn1 ~]$ df -k
        Filesystem           1K-blocks      Used Available Use% Mounted on
        /dev/mapper/vg_collabn1-lv_root
                              27407400   2798080  23217092  11% /
        tmpfs                  1544640         0   1544640   0% /dev/shm
        /dev/sda1               495844     55641    414603  12% /boot
        12cR1                976728060 863308936 113419124  89% /media/sf_12cR1
        [oracle@collabn1 ~]$ ls -l /media/sf_12cR1
        total 6295549
        -rwxrwx---. 1 root vboxsf 1361028723 Jun 27 16:21 linuxamd64_12c_database_1of2.zip
        -rwxrwx---. 1 root vboxsf 1116527103 Jun 27 16:12 linuxamd64_12c_database_2of2.zip
        -rwxrwx---. 1 root vboxsf 1750478910 Jun 27 16:27 linuxamd64_12c_grid_1of2.zip
        -rwxrwx---. 1 root vboxsf  201673595 Jun 27 15:23 linuxamd64_12c_grid_2of2.zip
        
      2. You can decide to unzip it from your laptop (using Windows native functionalities, WinZip, 7-Zip or other tools or directly from the server.
        [oracle@collabn1 ~]$ cd /media/sf_12cR1
        [oracle@collabn1 sf_12cR1]$ unzip linuxamd64_12c_database_1of2.zip 
        Archive:  linuxamd64_12c_database_1of2.zip
           creating: database/                OK
           inflating: database/welcome.html    OK
           creating: database/response/       OK
           inflating: database/response/db_install.rsp   OK
           inflating: database/response/netca.rsp   OK
           inflating: database/response/dbca.rsp   OK
           creating: database/sshsetup/       OK
           inflating: database/sshsetup/sshUserSetup.sh   OK
           creating: database/rpm/            OK
           inflating: database/rpm/cvuqdisk-1.0.9-1.rpm   OK
           [...]
        [oracle@collabn1 sf_12cR1]$ unzip linuxamd64_12c_database_2of2.zip
           [...]
        [oracle@collabn1 sf_12cR1]$ unzip linuxamd64_12c_grid_1of2.zip
           [...]
        [oracle@collabn1 sf_12cR1]$ unzip linuxamd64_12c_grid_2of2.zip
           [...]
        
      3. big>Once the package decompression is completed, you'll end up with two directories, grid and database containing the installation files.
        [oracle@collabn1 sf_12cR1]$ ls -l
        total 6295549
        drwxrwx---. 1 root vboxsf          0 Jul  3 06:38 database
        drwxrwx---. 1 root vboxsf          0 Jun 10 08:15 grid
        -rwxrwx---. 1 root vboxsf 1361028723 Jun 27 16:21 linuxamd64_12c_database_1of2.zip
        -rwxrwx---. 1 root vboxsf 1116527103 Jun 27 16:12 linuxamd64_12c_database_2of2.zip
        -rwxrwx---. 1 root vboxsf 1750478910 Jun 27 16:27 linuxamd64_12c_grid_1of2.zip
        -rwxrwx---. 1 root vboxsf  201673595 Jun 27 15:23 linuxamd64_12c_grid_2of2.zip
        
      4. Install the cvuqdisk package as root:
        [root@collabn1 ~]# rpm -Uvh /media/sf_12cR1/grid/rpm/cvuqdisk-1.0.9-1.rpm
        Preparing...                ########################################### [100%]
        Using default group oinstall to install package
          1:cvuqdisk               ########################################### [100%]
        [root@collabn1 ~]#
        
      5. Do the same on node collabn2:
        [root@collabn2 ~]# rpm -Uvh /media/sf_12cR1/grid/rpm/cvuqdisk-1.0.9-1.rpm
        Preparing...                ########################################### [100%]
        Using default group oinstall to install package
          1:cvuqdisk               ########################################### [100%]
        [root@collabn2 ~]#
        


      Install Grid Infrastructure[edit]

      1. From the VNC remote session, run the installation of Grid Infrastructure as oracle:
        [oracle@collabn1 ~]$ /media/sf_12cR1/grid/runInstaller
        Starting Oracle Universal Installer...
        
        Checking Temp space: must be greater than 120 MB.   Actual 22670 MB    Passed
        Checking swap space: must be greater than 150 MB.   Actual 3023 MB    Passed
        Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
        Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-08-06_04-25-00PM. Please wait ...[oracle@collabn1 ~]$ 
        [oracle@collabn1 ~]$ 
        
      2. The Installation starts displaying a Splash Screen:

        RA-Oracle_GI_12101-Install-Splash

      3. Leave Skip software updates and click Next:

        RA-Oracle_GI_12101-Install-Software Updates

      4. Leave Install and Configure Oracle Grid Infrastructure for a Cluster and click Next:

        RA-Oracle_GI_12101-Install-Installation option

      5. Leave Configure a Standard Cluster and click Next:

        RA-Oracle_GI_12101-Install-Cluster Type

      6. Leave Typical Installation and click Next:

        RA-Oracle_GI_12101-Install-Installation type

      7. In the SCAN Name box, enter collabn-cluster-scan.racattack

        In the central box, only the first node is displayed. Click Add... and enter the network names of the second node:

        • Public Hostname: collabn2.racattack
        • Virtual Hostname: collabn2-vip.racattack

        RA-Oracle_GI_12101-Install-Cluster configuration

      8. Click SSH connectivity...

        RA-Oracle_GI_12101-Install-Cluster configuration done

      9. Enter racattack as the password of oracle user and click Setup.

        RA-Oracle_GI_12101-Install-SSH connectivity

      10. The setup of SSH equivalency is initiated.

        RA-Oracle_GI_12101-Install-SSH connectivity progress

      11. Click OK when it's completed.

        RA-Oracle_GI_12101-Install-SSH connectivity OK

      12. Click on Identify Network Interfaces...

        Check that the correct addresses are set:

        • eth0 is marked as Public
        • eth1 is marked as Private
        • eth2 is marked as Do Not Use

        Click OK to close the Identify Network Interfaces window.

        Finally, click Next

        RA-Oracle_GI_12101-Install-Identify Network Interfaces

      13. Some prerequisite validation is done at this step. It can require some minutes to complete.

        RA-Oracle_GI_12101-Install-Installer validation

      14. Since the SCAN Name chosen is too long (more than 15 chars.) Oracle proposes a different name as the Cluster Name. Click Yes.

        RA-Oracle_GI_12101-Install-Confirm Cluster Name

      15. Leave the default Oracle Base and Software Location.

        In the Cluster Registry Storage Type combo box, select Oracle Automatic Storage Management

        Enter racattack as the SYSASM password.

        Select dba as OSASM group.

        Click Next.

        RA-Oracle_GI_12101-Install-Install Locations

      16. racattack is a weak password and Oracle raise an alert. Click Yes to continue:

        RA-Oracle_GI_12101-Install-Sysasm weak password

      17. Leave DATA as Disk Group Name.

        Select External redundancy (it is safe to do this in a RAC Attack event!).

        The Candidate Disks list is empty. Click on Change Discovery Path...

        Enter /dev/asm* in the Disk Discovery Path text field.

        Click Ok.

        RA-Oracle_GI_12101-Install-Create ASM Diskgroup

      18. Check two disks that will be immediately allocated to the new disk group.

        Click Next

        RA-Oracle_GI_12101-Install-Create ASM Diskgroup disks

      19. Leave the default Inventory Directory. Click Next.

        RA-Oracle_GI_12101-Install-Create Inventory

      20. Check Automatically run configuration script and then Use "root" user credentials.

        Enter racattack as the password of root and click Next.

        RA-Oracle_GI_12101-Install-Root script execution credentials

      21. It takes some time to complete the prerequisite checks.

        RA-Oracle_GI_12101-Install-Prerequisite Check progress

      22. If everything is correct, you will notice two checks that have failed:
        • Physical Memory
        • Device Checks for ASM

        Check "Ignore All" and click Next.

        RA-Oracle_GI_12101-Install-Prerequisite checks

      23. Click Yes to confirm that you want to skip some prerequisites.

        RA-Oracle_GI_12101-Install-Prerequisite checks continue

      24. The summary page appears. Optionally, click Save Response File and choose a location to have the response file saved for this installation session.

        RA-Oracle_GI_12101-Install-Save response file

      25. Review the summary and click Install to start the installation.

        RA-Oracle_GI_12101-Install-Summary

      26. The installation starts. It may take more than an hour, depending on your laptop hardware.

        RA-Oracle_GI_12101-Install-Install product progress

      27. After a while, the installation asks a confirmation before running some scripts as root. Click Yes to continue.

        RA-Oracle_GI_12101-Install-Confirm root execution

      28. The installation continues, again, it may take longtime to complete. Once it's finished, click on Close to exit.

        RA-Oracle_GI_12101-Install-Completed


      Check Cluster Status After GI Install[edit]

      1. Once your Grid Infrastructure installation is finished, you can get the status of the cluster components:
        [oracle@collabn1 ~]$ . oraenv
        ORACLE_SID = [oracle] ? +ASM1
        [oracle@collabn1 ~]$ crsctl stat res -t
        --------------------------------------------------------------------------------
        Name           Target  State        Server                   State details
        --------------------------------------------------------------------------------
        Local Resources
        --------------------------------------------------------------------------------
        ora.DATA.dg
                       ONLINE  ONLINE       collabn1                 STABLE
                       ONLINE  ONLINE       collabn2                 STABLE
        ora.LISTENER.lsnr
                       ONLINE  ONLINE       collabn1                 STABLE
                       ONLINE  ONLINE       collabn2                 STABLE
        ora.asm
                       ONLINE  ONLINE       collabn1                 Started,STABLE
                       ONLINE  ONLINE       collabn2                 Started,STABLE
        ora.net1.network
                       ONLINE  ONLINE       collabn1                 STABLE
                       ONLINE  ONLINE       collabn2                 STABLE
        ora.ons
                       ONLINE  ONLINE       collabn1                 STABLE
                       ONLINE  ONLINE       collabn2                 STABLE
        --------------------------------------------------------------------------------
        Cluster Resources
        --------------------------------------------------------------------------------
        ora.LISTENER_SCAN1.lsnr
              1        ONLINE  ONLINE       collabn2                 STABLE
        ora.LISTENER_SCAN2.lsnr
              1        ONLINE  ONLINE       collabn1                 STABLE
        ora.LISTENER_SCAN3.lsnr
              1        ONLINE  ONLINE       collabn1                 STABLE
        ora.collabn1.vip
              1        ONLINE  ONLINE       collabn1                 STABLE
        ora.collabn2.vip
              1        ONLINE  ONLINE       collabn2                 STABLE
        ora.cvu
              1        ONLINE  ONLINE       collabn1                 STABLE
        ora.oc4j
              1        OFFLINE OFFLINE                               STABLE
        ora.scan1.vip
              1        ONLINE  ONLINE       collabn2                 STABLE
        ora.scan2.vip
              1        ONLINE  ONLINE       collabn1                 STABLE
        ora.scan3.vip
              1        ONLINE  ONLINE       collabn1                 STABLE
        --------------------------------------------------------------------------------
        
      2. Optional step: in order to increase the resistance of your nodes to the huge latency of a Virtualbox environment, you can increase the timeout of CRS before it causes a fencing (restart) of the node.

        You'll need to stop the second node while applying the configuration to the first node.

        [oracle@collabn1 ~]$ ssh collabn2
        [oracle@collabn2 ~]$ su -
        Password:
        [root@collabn2 ~]# . oraenv
        ORACLE_SID = [root] ? +ASM2
        The Oracle base has been set to /u01/app/oracle
        [root@collabn2 ~]# crsctl stop crs
        CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'collabn2'
        CRS-2673: Attempting to stop 'ora.crsd' on 'collabn2'
        CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'collabn2'
        CRS-2673: Attempting to stop 'ora.DATA.dg' on 'collabn2'
        ...
        CRS-2677: Stop of 'ora.gipcd' on 'collabn2' succeeded
        CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'collabn2' has completed
        CRS-4133: Oracle High Availability Services has been stopped.
        [root@collabn2 ~]# exit
        logout
        [oracle@collabn2 ~]$ exit
        logout
        Connection to collabn2 closed.
        [oracle@collabn1 ~]$ su -
        Password:
        [root@collabn1 ~]# . oraenv
        ORACLE_SID = [root] ? +ASM1
        The Oracle base has been set to /u01/app/oracle
        [root@collabn1 ~]#  crsctl get css misscount
        CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
        [root@collabn1 ~]# crsctl set css misscount 90
        CRS-4684: Successful set of parameter misscount to 90 for Cluster Synchronization Services.
        [root@collabn1 ~]#  crsctl get css disktimeout
        CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
        [root@collabn1 ~]# crsctl set css disktimeout 600
        CRS-4684: Successful set of parameter disktimeout to 600 for Cluster Synchronization Services.
        [root@collabn1 ~]# ssh collabn2
        root@collabn2's password:
        Last login: Tue Aug  6 16:19:56 2013 from 192.168.78.51
        [root@collabn2 ~]# . oraenv
        ORACLE_SID = [root] ? +ASM2
        The Oracle base has been set to /u01/app/oracle
        [root@collabn2 ~]# crsctl start crs
        CRS-4123: Oracle High Availability Services has been started.
        

        The start command returns the prompt in few seconds. However it can take minutes before the whole stack is started entirely.



      ASM Configuration[edit]

      1. From a VNC session, run the ASM Configuration Assistant:
        [oracle@collabn1 ~] . oraenv
        ORACLE_SID = [oracle] ? +ASM1
        The Oracle base has been set to /u01/app/oracle
        [oracle@collabn1 ~]$ asmca
        


      2. The asmca opens on the Disk Groups tab, showing the DATA diskgroup created by the installer. Click on Create.

        RA-Oracle_GI_12101-asmca-diskgroups


      3. Enter FRA as the disk group name.

        Choose external (none) redundancy.

        Check one disk to create a 5Gb diskgroup.

        Click OK.

        RA-Oracle_GI_12101-asmca-add_diskgroups


      4. The diskgroup is being created.

        RA-Oracle_GI_12101-asmca-Creating diskgroup


      5. The diskgroup is created.

        RA-Oracle_GI_12101-asmca-Diskgroup created


      6. Click on the Volumes tab, then on Create.

        EnterSHARED as the volume name.

        Choose the diskgroup DATA.

        The size will be irrelevant in our labs, so enter 1.

        Click OK.

        RA-Oracle_GI_12101-asmca-Creating volume


      7. The volume is created.

        RA-Oracle GI 12101-asmca-Volume created


      8. Select the ASM Cluster File Systems tab.

        Click Create.

        RA-Oracle_GI_12101-asmca-Filesystems


      9. Select Cluster File System.

        Mountpoint: /shared

        Check Automount.

        User Name: oracle

        Group Name: dba

        Volume: SHARED

        Click OK.

        RA-Oracle_GI_12101-asmca-Creating Filesystem


      10. To complete the filesystem creation, run this script as root.

        RA-Oracle_GI_12101-asmca-Creating Filesystem acfs script


        [root@collabn1 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/acfs_script.sh
        ACFS file system /shared is mounted on nodes collabn1
        ACFS file system /shared is mounted on nodes collabn2
        


      11. Finally, the filesystem is mounted on both nodes.


        [root@collabn1 ~]# df -k
        Filesystem           1K-blocks      Used Available Use% Mounted on
        /dev/mapper/vg_collabn1-lv_root
                              27407400   8778184  17236988  34% /
        tmpfs                  1544640    653576    891064  43% /dev/shm
        /dev/sda1               495844     55641    414603  12% /boot
        12cR1                976728060 863308972 113419088  89% /media/sf_12cR1
        /dev/asm/shared-336    1048576     80176    968400   8% /shared
        
        [root@collabn2 ~]# df -k
        Filesystem           1K-blocks      Used Available Use% Mounted on
        /dev/mapper/vg_collabn1-lv_root
                              27407400   8883544  17131628  35% /
        tmpfs                  1544640    653576    891064  43% /dev/shm
        /dev/sda1               495844     55641    414603  12% /boot
        /dev/asm/shared-336    1048576     80176    968400   8% /shared
        


      Install Database Software[edit]

      1. In a VNC session, run the installer as oracle user.
        [oracle@collabn1 ~]$ /media/sf_12cR1/database/runInstaller 
        Starting Oracle Universal Installer...
        
        Checking Temp space: must be greater than 500 MB.   Actual 16829 MB    Passed
        Checking swap space: must be greater than 150 MB.   Actual 3023 MB    Passed
        Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
        Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-08-08_06-27-48PM. Please wait ...
        


      2. The splash screen appears.

        RA-Oracle_RAC_12101-Install-Splash


      3. Uncheck I wish to receive security updates via My Oracle Support and click Next.

        RA-Oracle_RAC_12101-Install-Security_updates


      4. Click Yes to confirm that you don't want to be informed about security updates.

        RA-Oracle_RAC_12101-Install-Address not specified



      5. Select Skip software updates and click Next.

        RA-Oracle_RAC_12101-Install-Software_updates


      6. Select Install database software only and click Next.

        RA-Oracle_RAC_12101-Install-Installation option


      7. Select Oracle Real Application Clusters database installation and click Next.

        RA-Oracle_RAC_12101-Install-Grid Installation option


      8. Verify that all nodes are selected and click Next.

        RA-Oracle_RAC_12101-Install-Node selection


      9. Click Next to accept the default language selection.

        RA-Oracle_RAC_12101-Install-Product languages


      10. Select Enterprise Edition and click Next.

        RA-Oracle_RAC_12101-Install-Database edition



      11. If you have previously created the ACFS 1Gb filesystem, you'll need to specify another location now.

        Oracle Base: /u01/app/oracle

        Software location: /u01/app/oracle/product/12.1.0/dbhome_1

        Click Next.

        RA-Oracle_RAC_12101-Install-Installation location


      12. Leave dba for all groups except for the OSOPER group (leave it empty) and click Next.

        RA-Oracle_RAC_12101-Install-Operating system groups


      13. The installer checks all prerequisites.

        RA-Oracle_RAC_12101-Install-Prerequisite checks


      14. Review the installation summary, optionally save a response file, and click Install.

        RA-Oracle_RAC_12101-Install-Summary


      15. The installation can take half an hour on common laptop configurations.

        RA-Oracle_RAC_12101-Install-Installation progress


      16. right before completing, the installer ask to run a script on both nodes as root. Don't click OK.

        RA-Oracle_RAC_12101-Install-Execute root scripts


      17. Execute the script on both nodes as root.
        [root@collabn1 ~]# /u01/app/oracle/product/12.1.0/dbhome_1/root.sh
        Performing root user operation for Oracle 12c
        
        The following environment variables are set as:
            ORACLE_OWNER= oracle
            ORACLE_HOME=  /u01/app/oracle/product/12.1.0/dbhome_1
        
        Enter the full pathname of the local bin directory: [/usr/local/bin]:
        The contents of "dbhome" have not changed. No need to overwrite.
        The contents of "oraenv" have not changed. No need to overwrite.
        The contents of "coraenv" have not changed. No need to overwrite.
        
        Entries will be added to the /etc/oratab file as needed by
        Database Configuration Assistant when a database is created
        Finished running generic part of root script.
        Now product-specific root actions will be performed.
        [root@collabn1 ~]# ssh collabn2
        root@collabn2's password:
        Last login: Thu Aug  8 16:54:49 2013 from 192.168.78.1
        [root@collabn2 ~]# /u01/app/oracle/product/12.1.0/dbhome_1/root.sh
        Performing root user operation for Oracle 12c
        
        The following environment variables are set as:
           ORACLE_OWNER= oracle
           ORACLE_HOME=  /u01/app/oracle/product/12.1.0/dbhome_1
        
        Enter the full pathname of the local bin directory: [/usr/local/bin]:
        The contents of "dbhome" have not changed. No need to overwrite.
        The contents of "oraenv" have not changed. No need to overwrite.
        The contents of "coraenv" have not changed. No need to overwrite.
        
        Entries will be added to the /etc/oratab file as needed by
        Database Configuration Assistant when a database is created
        Finished running generic part of root script.
        Now product-specific root actions will be performed.
        


      18. Click OK on the previous window, then click Close to exit the Installer.

        RA-Oracle_RAC_12101-Install-Completed


      Create Database[edit]

      1. Run the Database Configuration Assistant:
        [oracle@collabn1 ~]$ /u01/app/oracle/product/12.1.0/dbhome_1/bin/dbca 
        


      2. The splash screen appears.

        RA-Oracle_RAC_12101-DBCA Splash


      3. Select Create Database and click Next.

        RA-Oracle_RAC_12101-DBCA Database Operation


      4. Select Create a database with default configuration.

        Global Database Name: RAC.racattack

        Storage Type: Automatic Storage Management (ASM)

        Database Files Location: +DATA

        Fast Recovery Area: +FRA

        Database Character Set: AL32UTF8 - Unicode UTF-8 Universal character set

        Administrative password: racattack

        Check Create As Container Database

        Pluggable Database Name: PDB (*)

        Click Next.

        RA-Oracle_RAC_12101-DBCA Creation mode


      5. A warning is shown because the password is weak. Click Yes to continue.


      6. The prerequisite checks may fail due to space requirements. It's safe to ignore by checking Ignore all.

        Click Next.

        RA-Oracle_RAC_12101-DBCA Prerequisite checks


      7. Review the summary page and click Finish.

        RA-Oracle_RAC_12101-DBCA Summary


      8. The database creation is finished. Click Exit, then Close.

        RA-Oracle_RAC_12101-DBCA Database Created