Install Oracle 11G Release 2 RAC (11.2) on Oracle Linux 5 (OEL5)

Article presents quick installation of Oracle Enterprise RAC 11.2.0.3 on Oracle Enterprise Linux 5 using Oracle Virtual Box.

Architecture

Software used:
Oracle Virtual Box – 64 bit
Oracle Enterprise Linux 5 – 64 bit
Oracle Database 11G Release 2(11.2.0.3) – 64 bit for Linux
Binaries 11.2.0.3
p10404530_112030_Linux-x86-64_1of7.zip - database software
p10404530_112030_Linux-x86-64_2of7.zip - database software
p10404530_112030_Linux-x86-64_3of7.zip - grid software
Two virtual machines are created:
yesbolisetty1.localdomain
yesbolisetty2.localdomain.com


# Public
192.168.0.135   yesbolisetty1.localdomain        yesbolisetty1
192.168.0.136   yesbolisetty2.localdomain        yesbolisetty2

# Private
192.168.1.135   yesbolisetty1-priv.localdomain   yesbolisetty1-priv
192.168.1.136   yesbolisetty2-priv.localdomain   yesbolisetty2-priv
# Virtual

192.168.0.137   yesbolisetty1-vip.localdomain    yesbolisetty1-vip
192.168.0.138   yesbolisetty2-vip.localdomain    yesbolisetty2-vip

# SCAN
192.168.0.139   yesbolisetty-scan.localdomain        yesbolisetty-scan
192.168.0.140   yesbolisetty-scan.localdomain        yesbolisetty-scan
192.168.0.141   yesbolisetty-scan.localdomain        yesbolisetty-scan

with single client access  name (SCAN) address
Each virtual machines consist of:
32G local disk (dynamic space allocation)
3G ram memory
3 network cards (2 dedicated just for RAC)
NOTE to save space yesbolisetty2.localdomain is linked clone of yesbolisetty1.localdomain
NOTE – to get connection to internet from virtual box be sure you are not using the same subnet as your wifi card
In my case all are different:
192.168.0.X is public – you can change it
192.168.1.X is private – you can change it
192.168.2.X is my internet – usually it is static for your network unless you change it on router – doesn’t make sense :)
Shared storage visible on each node:
20G shared virtual disk (fixed size)
Virtual Machine configuration
1. Start virtual box manager and click “New” button.
2. Enter virtual machine name “oel5_112_yesbolisetty1″, select type “Linux”, version “Oracle(64bit)”, set memory to 3096M. Click “Create” button to allocate storage for this machine.

3. Enter storage attributes: name, size 32G, type VDI, dynamically allocated then click “Create” button.

4. Virtual machine is ready click “Settings” to change some details.

5. Select “System” option, tab “Processor” in case you want to assign more processor to your virtual machine. I have 4 core machines so I assigned to yesbolisetty1.localdomain 2 CPU.
6. Select “Option” option, “Video” tab and assign 128MB of memory and enable 3D acceleration to improve graphics in your virtual machine.

7. Select “Network” option and select “Internal Network” for Adapter 1 and 2 and “NAT” for Adapter 3. Adapter 1 will be used as public RAC interface, Adapter 2 as private and Adapter 3 to connect to internet(it’s optional but I use it to have access to internet from my virtual machine).

8. On “Storage” tab assign iso file to “CD/DVD Drive”  to install Oracle Enterprise Linux 5 64 bit. Once it’s done click “OK” button.

9. Virtual box is ready to install so click “Start” button.

Install Oracle Enterprise Linux 5
Installation of OEL5 is described in details in following article: Install Oracle Linux 5 64-bit. The only difference is size of our disk, memory and number of network cards. Remember to assign hostname to “yesbolisetty1.localdomain” during installation and select “Active on Boot” for each network card.

We have 3 network interfaces with ip addresses from DHCP. For RAC installation we need static addresses for eth0 and eth1. You can assign static ip addresses by click “Edit” button or you can do it later using “Network manager”.
192.168.0.135/255.255.255.0 for eth0 (public)
192.168.1.135/255.255.255.0 for eth1 (private)
“Network Manager” is available from System->Administration->Network. You can apply here static ip address for eth0 and eth1 and activate both cards. Please remember to save configuration from network manager by selection from menu File->Save.
NOTE: eth2 is NAT interface so nothing has to be change. This card is used to communicate with Internet.
Shared storage
1. To add shared storage virtual machine yesbolisetty1.localdomain must be down. Once it’s down go to “Settings” select “Storage” then click on icon with plus on disk.
2. Click on “Create new disk” button.
3. Default option “VDI” is fine click “Next” button.
4. Select “Fixed size” then click “Next” button.
5. Enter storage name “Shared” and set size to 20G then click “Create” button.
6. Once created you should see new disk attached to our virtual machine. Click “OK” button.
8. From main menu you need to select File->Virtual Media Manager to change “Shared” disk attribute. Select “Shared.vdi” and click “Modify” button.
9. Select “Sharable” and click “OK” button. It will prepare this storage to be shareable between RAC instances. Please start virtual machine to create partitions on shared storage.
Host configuration
All commands are executed as user root.
Configure shared storage.Shared storage is visible as /dev/sdb device. Two partitions each 10G will be created using fdisk command. Both new partitions will be used for ASM.
--list devices

ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb
--add two partitions

fdisk /dev/sdb

The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2610, default 1):   
Using default value 1
Last cylinder or +size or +sizeM 
or +sizeK (1-2610, default 2610): +10240M

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (1247-2610, default 1247): 
Using default value 1247
Last cylinder or +size or +sizeM or +sizeK (1247-2610, default 2610): 
Using default value 2610

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
--list new partitions

fdisk -l /dev/sdb

Disk /dev/sdb: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1246    10008463+  83  Linux
/dev/sdb2            1247        2610    10956330   83  Linux
Add groups
--required groups
/usr/sbin/groupadd -g 501 oinstall
/usr/sbin/groupadd -g 502 dba
/usr/sbin/groupadd -g 503 oper
Add user oracle
/usr/sbin/useradd -u 502 -g oinstall -G dba oracle
Change password for user oracle
passwd oracle
Add kernel parameters to /etc/sysctl.conf
--kernel parameters for 11g installation
kernel.shmmni = 4096 
kernel.shmmax = 4398046511104
kernel.shmall = 1073741824
kernel.sem = 250 32000 100 128

fs.aio-max-nr = 1048576
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
Apply kernel parameters
/sbin/sysctl -p
Add following line for user oracle to set shell limits in file /etc/security/limits.conf
--shell limits for users oracle 11gR2
oracle   soft   nproc    131072
oracle   hard   nproc    131072
oracle   soft   nofile   131072
oracle   hard   nofile   131072
oracle   soft   core     unlimited
oracle   hard   core     unlimited
oracle   soft   memlock  50000000
oracle   hard   memlock  50000000
The “/etc/hosts” file must contain a fully qualified name for the server.
    
Enter following data to “/ect/hosts”
# Public
192.168.0.135   yesbolisetty1.localdomain        yesbolisetty1
192.168.0.136   yesbolisetty2.localdomain        yesbolisetty2
# Private
192.168.1.135   yesbolisetty1-priv.localdomain   yesbolisetty1-priv
192.168.1.136   yesbolisetty2-priv.localdomain   yesbolisetty2-priv
# Virtual
192.168.0.137   yesbolisetty1-vip.localdomain    yesbolisetty1-vip
192.168.0.138   yesbolisetty2-vip.localdomain    yesbolisetty2-vip
# SCAN
192.168.0.139   yesbolisetty-scan.localdomain        yesbolisetty-scan
192.168.0.140   yesbolisetty-scan.localdomain        yesbolisetty-scan
192.168.0.141   yesbolisetty-scan.localdomain        yesbolisetty-scan
Verify network response on yesbolisetty1.localdomain for th0 and eth1
[root@yesbolisetty1 ~]# ping yesbolisetty1 -c 1
PING yesbolisetty1.localdomain (192.168.0.50) 56(84) bytes of data.
64 bytes from yesbolisetty1.localdomain (192.168.0.50): 
icmp_seq=1 ttl=64 time=0.032 ms

--- yesbolisetty1.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.032/0.032/0.032/0.000 ms

[root@yesbolisetty1 ~]# ping yesbolisetty1-priv -c 1
PING yesbolisetty1-priv.dbaora.com (192.168.1.60) 56(84) bytes of data.
64 bytes from yesbolisetty1-priv.dbaora.com (192.168.1.60): 
icmp_seq=1 ttl=64 time=0.036 ms

--- yesbolisetty1-priv.dbaora.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.036/0.036/0.036/0.000 ms
Check which packages are installed and which are missing
rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE}(%{ARCH})\n' binutils \
compat-libstdc++-33 \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc \
libstdc++ \
libstdc++-devel \
make \
sysstat \
unixODBC \
unixODBC-devel
Install missing packages. It’s just example:
#directory with mounted Oracle Enterprise Linux 5 install disk

cd /Server/Packages

#install missed packages (example for package unixODBC*)

rpm -Uvh unixODBC*
Disable Secure Linux
To disable secure Linux edit the “/etc/selinux/config” file, making sure the SELINUX flag is set as follows. It requires REBOOT to be effective !!!
SELINUX=disabled
Disable Firewall
Firewall can be disabled in graphical mode or manually.
Option 1 – graphical mode
Run firewall graphical manager “System”->”Administration”->”Security Level and Firewall” and click “Disable” button and “Apply” button. As you can notice you can change SELinux on second tab as well.
Option 2 – manual mode
Run following command
service iptables stop
chkconfig iptables off
Change NTP
It’s required to disable NTP(Network Time Protocol) or modify its settings so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes.
Option 1 – disable NTP
service ntpd stop
Shutting down ntpd:                                        [  OK  ]
chkconfig ntpd off
mv /etc/ntp.conf /etc/ntp.conf.orig
rm /var/run/ntpd.pid
Option 2 – modify NTP
If you want to keep NTP, add the “-x” option in the “/etc/sysconfig/ntpd” file.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
and restart NTP
# service ntpd restart
Directories
Create ORACLE_BASE
mkdir -p /ora01/app/oracle
Create ORACLE_HOME for grid and database software
mkdir -p /ora01/app/oracle/product/11.2.0/db_1
mkdir -p /ora01/app/grid/product/11.2.0/grid

chown oracle:oinstall -R /ora01
chmod 775 /ora01/app/oracle
New profiles for user oracle
change oracle profile /home/oracle/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

alias genv='. /home/oracle/.bash_profile_grid;envo'
alias denv='. /home/oracle/.bash_profile_database;envo'

. /home/oracle/.bash_profile_database

envo
create new profile for user oracle /home/oracle/.bash_profile_grid. This profile will be used for grid software.
# Oracle Settings
export TMP=/tmp

export ORACLE_HOSTNAME=yesbolisetty1.localdomain
export ORACLE_UNQNAME=+ASM
export ORACLE_BASE=/ora01/app/oracle
export ORACLE_HOME=/ora01/app/grid/product/11.2.0/grid
export ORACLE_SID=+ASM1

PATH=/usr/sbin:$PATH:$ORACLE_HOME/bin

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib;
export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;

alias cdob='cd $ORACLE_BASE'
alias cdoh='cd $ORACLE_HOME'
alias tns='cd $ORACLE_HOME/network/admin'
alias envo='env | grep ORACLE'
create new profile for user oracle /home/oracle/.bash_profile_database.

This profile will be used for database software.
# Oracle Settings
export TMP=/tmp

export ORACLE_HOSTNAME=yesbolisetty1.localdomain
export ORACLE_UNQNAME=ORA11G
export ORACLE_BASE=/ora01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export ORACLE_SID=ORA11G1

PATH=/usr/sbin:$PATH:$ORACLE_HOME/bin

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib;
export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;

alias cdob='cd $ORACLE_BASE'
alias cdoh='cd $ORACLE_HOME'
alias tns='cd $ORACLE_HOME/network/admin'
alias envo='env | grep ORACLE'

umask 022
So next time you do su to oracle user you should have environment prepared for database software
[root@yesbolisetty1 ~]# su - oracle
ORACLE_UNQNAME=ORA11G
ORACLE_SID=ORA11G1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/oracle/product/11.2.0/db_1
Number of aliases are defined in profiles to quickly managed environment
genv – sets environment for grid software
alias genv=’. /home/oracle/.bash_profile_grid;envo’
[oracle@yesbolisetty1 ~]$ genv
ORACLE_UNQNAME=+ASM
ORACLE_SID=+ASM1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/grid/product/11.2.0/grid
denv – sets environment for database software
alias denv=’. /home/oracle/.bash_profile_database;envo’
[oracle@yesbolisetty1 ~]$ denv
ORACLE_UNQNAME=ORA11G
ORACLE_SID=ORA11G1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/oracle/product/11.2.0/db_1
cdob – change directory to $ORACLE_BASE
alias cdob=’cd $ORACLE_BASE’
cdoh – change directory to $ORACLE_HOME
alias cdoh=’cd $ORACLE_HOME’
tns – change directory to $ORACLE_HOME/network/admin
alias tns=’cd $ORACLE_HOME/network/admin’
envo – displays current environments settings and filters by ORACLE word
alias envo=’env | grep ORACLE’
[oracle@yesbolisetty1 ~]$ envo
ORACLE_UNQNAME=ORA11G
ORACLE_SID=ORA11G1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/oracle/product/11.2.0/db_1
Remember to modify owner of all profiles
chown oracle:oinstall /home/oracle/.bash*
chmod 750 oracle:oinstall /home/oracle/.bash*
As oracle user unzip database and grid software. It should create 2 directories:
database – database software
grid – grid software
--database software
p10404530_112030_Linux-x86-64_1of7.zip
p10404530_112030_Linux-x86-64_2of7.zip

--grid software
p10404530_112030_Linux-x86-64_3of7.zip

[oracle@yesbolisetty1]$ unzip p10404530_112030_Linux-x86-64_1of7.zip
[oracle@yesbolisetty1]$ unzip p10404530_112030_Linux-x86-64_2of7.zip
[oracle@yesbolisetty1]$ unzip p10404530_112030_Linux-x86-64_3of7.zip

[oracle@yesbolisetty1]ls
database grid
Install package cvuqdisk-1.0.9-1.rpm from grid software as user root
cd /rpm
[root@yesbolisetty1 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm
Preparing...                ############################## [100%]
Using default group oinstall to install package
   1:cvuqdisk               ############################## [100%]

Configure ASM devices
To configure ASMlib first you need to download ASMLib rpms from OTN. If you are using UEK kernel all required rpms are already installed.
oracleasm-support-2.1.8-1.el5.i386.rpm
oracleasmlib-2.0.4-1.el5.i386.rpm
oracleasm-[your-kernel-version].rpm
Install the packages using the following command
rpm -Uvh oracleasm*.rpm
Configure ASMlib
[root@yesbolisetty1 ~]# /usr/sbin/oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: 
Writing Oracle ASM library driver configuration: done
Load asm kernel module
[root@yesbolisetty1 ~]# /usr/sbin/oracleasm init
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
Add ASM disks
[root@yesbolisetty1 ~]# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@yesbolisetty1 ~]# /usr/sbin/oracleasm createdisk DISK2 /dev/sdb2
Writing disk header: done
Instantiating disk: done
Scan ASM disks
[root@yesbolisetty1 ~]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
List ASM disks
[root@yesbolisetty1 ~]# /usr/sbin/oracleasm listdisks
DISK1
DISK2
Cloning machine
New machine yesbolisetty2.localdomain is cloned from machine yesbolisetty1.localdomain using “linked clone” method. It allows to speed up process of cloning. it’s especially useful for development environments.
Full clone: In this mode all depending disk images are copied to the new VM folder. The clone can fully operate without the source VM.

Linked clone: In this mode new differencing disk images are created where the parent disk images are the source disk images. If you selected the current state of the source VM as clone point, a new snapshot will be created implicitly.
1. Before we create “linked clone” we need to remove shared storage. On Storage tab select Shared.vdi then click “Remove Attachment”.
2. Now you should see oel5_112_yesbolisetty1 without Shared storage.
3. In menu of VirtualBox select “Machine”-“Clone” option. Enter name for new machine oel5_112_yesbolisetty2 and click “Next” button.
4. Select “Linked clone” option and click “Clone” button.
5. You should see new machine oel5_112_yesbolisetty26. It’s time to attach Shared.vdi disk to both machine. Select first machine oel5_yesbolisetty112_yesbolisetty1 then click “Storage”. Select “Controller:SATA” and click last button “Add Hard Disk”.
7. Click “Choose existing disk”. You should see standard window to select a file. Select “Shared.vdi” to attach it to machine.
8. Once it’s done shared disk should be visible by machine oel5_112_yesbolisetty1. Click “OK” button to save your configuration.
9. You need to repeat the same steps 6,7 to attach shared disk to machine oel5_112_yesbolisetty2. Don’t be surprised that both machines are using the same disk “oel5_112_arc1.vdi”. New machine oel5_112_yesbolisetty2 is linked clone.
Network settings for second machine
Start second linked and use “Network Manager”  System->Administration->Networkto apply new static ip address for eth0 and eth1 and activate both cards. Please remember to save configuration from network manager by selection from menu File->Save.
192.168.0.135/255.255.255.0 for eth0 (public)
192.168.1.135/255.255.255.0 for eth1 (private)
Remember to change host name as well to
yesbolisetty2.localdomain
Verify network between nodes
It’s good time to run verify network on each node: yesbolisetty1.localdomain and yesbolisetty2.localdomain
hostname
ping yesbolisetty1 -c 1
ping yesbolisetty2 -c 1
ping yesbolisetty1-priv -c 1
ping yesbolisetty2-priv -c 1
test on yesbolisetty1.localdomain
 [root@yesbolisetty1 ~]# hostname
yesbolisetty1.localdomain
[root@yesbolisetty1 ~]# ping yesbolisetty1 -c 1
PING yesbolisetty1.localdomain (192.168.0.50) 56(84) bytes of data.
64 bytes from yesbolisetty1.localdomain (192.168.0.50): icmp_seq=1 ttl=64 time=0.035 ms

--- yesbolisetty1.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.035/0.035/0.035/0.000 ms
[root@yesbolisetty1 ~]# ping yesbolisetty2 -c 1
PING yesbolisetty2.localdomain (192.168.0.51) 56(84) bytes of data.
64 bytes from yesbolisetty2.localdomain (192.168.0.51): icmp_seq=1 ttl=64 time=0.318 ms

--- yesbolisetty2.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.318/0.318/0.318/0.000 ms
[root@yesbolisetty1 ~]# ping yesbolisetty1-priv -c 1
PING yesbolisetty1-priv.dbaora.com (192.168.1.60) 56(84) bytes of data.
64 bytes from yesbolisetty1-priv.dbaora.com (192.168.1.60): icmp_seq=1 ttl=64 time=0.025 ms

--- yesbolisetty1-priv.dbaora.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.025/0.025/0.025/0.000 ms
[root@yesbolisetty1 ~]# ping yesbolisetty2-priv -c 1
PING yesbolisetty2-priv.dbaora.com (192.168.1.61) 56(84) bytes of data.
64 bytes from yesbolisetty2-priv.dbaora.com (192.168.1.61): icmp_seq=1 ttl=64 time=0.352 ms

--- yesbolisetty2-priv.dbaora.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.352/0.352/0.352/0.000 ms
test on yesbolisetty2.localdomain
 [root@yesbolisetty2 ~]# hostname
yesbolisetty2.localdomain
[root@yesbolisetty2 ~]# ping yesbolisetty1 -c 1
PING yesbolisetty1.localdomain (192.168.0.50) 56(84) bytes of data.
64 bytes from yesbolisetty1.localdomain (192.168.0.50): icmp_seq=1 ttl=64 time=0.342 ms

--- yesbolisetty1.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.342/0.342/0.342/0.000 ms
[root@yesbolisetty2 ~]# ping yesbolisetty2 -c 1
PING yesbolisetty2.localdomain (192.168.0.51) 56(84) bytes of data.
64 bytes from yesbolisetty2.localdomain (192.168.0.51): icmp_seq=1 ttl=64 time=0.039 ms

--- yesbolisetty2.localdomain ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.039/0.039/0.039/0.000 ms
[root@yesbolisetty2 ~]# ping yesbolisetty1-priv -c 1
PING yesbolisetty1-priv.dbaora.com (192.168.1.60) 56(84) bytes of data.
64 bytes from yesbolisetty1-priv.dbaora.com (192.168.1.60): icmp_seq=1 ttl=64 time=0.312 ms

--- yesbolisetty1-priv.dbaora.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.312/0.312/0.312/0.000 ms
[root@yesbolisetty2 ~]# ping yesbolisetty2-priv -c 1
PING yesbolisetty2-priv.dbaora.com (192.168.1.61) 56(84) bytes of data.
64 bytes from yesbolisetty2-priv.dbaora.com (192.168.1.61): icmp_seq=1 ttl=64 time=0.055 ms

--- yesbolisetty2-priv.dbaora.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.055/0.055/0.055/0.000 ms
Grid software installation
Start grid software installation as user oracle, remember to run command xhost+ as root.
xhost +
access control disabled, clients can connect from any host
Set grid environment and run grid installation software
su - oracle

[oracle@yesbolisetty1 ~]$ genv
ORACLE_UNQNAME=+ASM
ORACLE_SID=+ASM1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/grid/product/11.2.0/grid

--run installation 
cd 
./runInstall
1. Select “Skip software updates” and click “Next” button.
2. Select “Install and Configure Oracle Grid Infrastructure for a Cluster” and click “Next” button.
3. Accept default “Typical installation” and click “Next” button
4. Enter SCAN Name “rac-scan” then click “Add” button to add second node yesbolisetty2.localdomain
5. Enter public and virtual host name for second node and click “OK” button

6.Click on “Identify network interfaces” button

7. Ensure eth0 is public and eth1 is private interface type and click “OK” button

8. It’s time to establish SSH connection between cluster nodes. Enter OS Password for user oracle and click “Setup” button. Once configuration is done click “Next” button



9. Select “Oracle Automatic Storage Management” and enter SYSASM user password and click “Next”

10. Click on “Change Discovery Path” button
11. Enter path for ASM devices “/dev/oracleasm/disk*”
12. You should see available devices. Select first one and click “next” button
13. Accept default directory for “Inventory Directory” and click “Next” button
14. Checks are executed to show any errors that need to be fixed before installation can proceed. Ignore “Device Checks for ASM” by check “Ignore All” checkbox and click “Next” button
15. Confirm you ignore checks click “Yes” button

16. It’s the last moment to go back and make modifications. Click “Install” to proceed with installation
17. It will take some time to install software so be patient

18. When prompted run configuration scripts on each node. Once it’s done click “OK” button
19. Installation should continue …

20. You should receive information about errors related to SCAN setup, assuming you are not using DNS. Just ignore it.
21. Click “Next” Button

22. Just click “Yes” button
23. Installation of grid software is completed so click “Close” button

ASM configuration assistant
So far we have created only one ASM group DATA. I want to show you how quickly you can add extra group to ASM instance.
As user oracle start ASMCA. Remeber to start xhost + as user root to enable run graphical tools from other users like oracle.
ASMCA – ASM configuration assistant is part of grid software so proper environment must be set
su - oracle

[oracle@yesbolisetty1 ~]$ genv
ORACLE_UNQNAME=+ASM
ORACLE_SID=+ASM1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/grid/product/11.2.0/grid

[oracle@yesbolisetty1 ~]$ asmca

1. ASMCA started click “Disk Groups” tab.
2. Click “Create” button to add new ASM diskgroup.

3. Enter group name BACKUP, check checkbox for “/dev/oracleasm/disks/DISK2″ device and click “OK” button.
4. You should see following screen. Click “OK”.

5. Now you should see we have 2 ASM groups “BACKUP” and “DATA”. Click “Exit” button.
Now you can verify in sqlplus you have 2 ASM groups.
[oracle@yesbolisetty1 ~]$ sqlplus

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jun 16 22:39:11 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Enter user-name: / as sysasm

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit
Production With the Automatic Storage Management option

SQL> select name from v$asm_diskgroup;

NAME
------------------------------
DATA
BACKUP

Database software installation
Let’s start with database software installation as user oracle. Set environment to database software
su - oracle
[oracle@yesbolisetty1 ~]$ denv
ORACLE_UNQNAME=ORA11G
ORACLE_SID=ORA11G1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/oracle/product/11.2.0/db_1

--run installation
cd
./runInstall
1. Uncheck checkbox “I wish to receive security updates via My Oracle Support” and click “Next” button.
2. Ignore this message and click “Yes”.

3. Select “Skip software updates” and click “Next” button.
4. Select “Create and configure a database” and click “Next” button.

5. Select “Server Class” and click “Next” button.
6. Make sure both nodes are selected, then click the “Next” button.

7. Select “Advanced install” and click “Next” button.

8. Select your languages and click “Next” button.
9. Accept default “Enterprise Edition” and click “Next” button. “Select Options” button enables to enable/disable extra options to install like:

§  Partitioning – default enabled
§  OLAP – default enabled
§  Label Security
§  Data Mining – default enabled
§  Database Vault
§  Real Application Testing – default enabled

10. It shows directories for database software installation. Just click “Next” button.
11. Accept default settings and click “Next” button.
12. Enter database name and click “Next” button.
13. On following tabs you can specify many options for your database like:
§  memory settings
§  database character sets
§  turn on extra new security features
§  install extra schemas
Click “Next” button to continue.
 14. Just click “Next” button.

15. Select “Oracle Automatic Storage Management” and enter password for user ASMSNMP. The user is used by OEM similar to DBSNMP. Click “Next” button.
16. Accept default settings “Do not enable automated backups” or specify details for your automated backup then click “Next” button.
17. Select “DATA” diskgroup to install our new database and click “Next” button.
18. Specify passwords for users: SYS, SYSTEM, SYSMAN, DBSNMP. You can specify the same password for all of them like in example below. Click “Next” button to continue.
19. Accept default OS groups and click “Next” button.
20. Prerequisites checks are executed to verify cluster settings.
21. Ignore error related to SCAN by check checkbox “Ignore All” and click “Next” button.
22. Click “Yes” button.
23. It’s the last moment to make changes before starting installation. Click “Install” button to continue.
24. Just wait installation in progress …
25. When software installation is completed DBCA is started to create database.
26. Summary window for our new database. Just click “OK” button. it’s worth to change passwords for our important users like SYS, SYSTEM etc “Password Management”.
27. As last step run root.sh script on both nodes. when it’s completed click “OK” button.
28. Installation of database software and new database ORA11G is completed. Click “Close” button.
Verify RAC installation
[root@yesbolisetty1 ~]# su - oracle
[oracle@yesbolisetty1 ~]$ genv
ORACLE_UNQNAME=+ASM
ORACLE_SID=+ASM1
ORACLE_BASE=/ora01/app/oracle
ORACLE_HOSTNAME=yesbolisetty1.localdomain
ORACLE_HOME=/ora01/app/grid/product/11.2.0/grid

[oracle@yesbolisetty1 ~]$ srvctl config database -d ORA11G
Database unique name: ORA11G
Database name: ORA11G
Oracle home: /ora01/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/ORA11G/spfileORA11G.ora
Domain: dbaora.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ORA11G
Database instances: ORA11G1,ORA11G2
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed

[oracle@yesbolisetty1 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): yesbolisetty2,yesbolisetty1

[oracle@yesbolisetty1 ~]$ srvctl status asm
ASM is running on yesbolisetty2,yesbolisetty1

[oracle@yesbolisetty1 ~]$ srvctl status database -d ORA11G
Instance ORA11G1 is running on node yesbolisetty1
Instance ORA11G2 is running on node yesbolisetty2

Status of your database can be visible in 

OEM https://yesbolisetty.localadmin:1158/em/
oracle@yesbolisetty1.localdomain:/home/oracle [ORA11G1] >srvctl status instance -d ORA11G -i ORA11G1
Instance ORA11G1 is running on node yesbolisetty1
oracle@yesbolisetty1.localdomain:/home/oracle [ORA11G1] >



oracle@yesbolisetty1.localdomain:/home/oracle [ORA11G1] >ps -ef | grep pmon
oracle    3845     1  0 16:26 ?        00:00:01 asm_pmon_+ASM1
oracle    4836     1  0 16:38 ?        00:00:01 ora_pmon_ORA11G1
oracle   14286  7768  0 20:11 pts/2    00:00:00 grep pmon
oracle@yesbolisetty1.localdomain:/home/oracle [ORA11G1] >



oracle@yesbolisetty2.localdomain:/home/oracle [ORA11G2] >srvctl status instance -d ORA11G -i ORA11G2
Instance ORA11G2 is running on node yesbolisetty2
oracle@yesbolisetty2.localdomain:/home/oracle [ORA11G2] >


oracle@yesbolisetty2.localdomain:/home/oracle [ORA11G2] >ps -ef |grep pmon
oracle    4114     1  0 16:57 ?        00:00:01 asm_pmon_+ASM2
oracle    5235     1  0 17:18 ?        00:00:01 ora_pmon_ORA11G2
oracle   23982 22296  0 20:13 pts/1    00:00:00 grep pmon
oracle@yesbolisetty2.localdomain:/home/oracle [ORA11G2] >


oracle@yesbolisetty1.localdomain:/home/oracle [+ASM1] >crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.BACKUP.dg  ora....up.type ONLINE    ONLINE    yesb...tty1
ora.DATA.dg    ora....up.type ONLINE    ONLINE    yesb...tty1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    yesb...tty1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    yesb...tty1
ora.asm        ora.asm.type   ONLINE    ONLINE    yesb...tty1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    yesb...tty1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    yesb...tty1
ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE
ora.ons        ora.ons.type   ONLINE    ONLINE    yesb...tty1
ora.ora11g.db  ora....se.type ONLINE    ONLINE    yesb...tty1
ora....ry.acfs ora....fs.type ONLINE    OFFLINE
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    yesb...tty1
ora....SM1.asm application    ONLINE    ONLINE    yesb...tty1
ora....Y1.lsnr application    ONLINE    ONLINE    yesb...tty1
ora....ty1.gsd application    OFFLINE   OFFLINE
ora....ty1.ons application    ONLINE    ONLINE    yesb...tty1
ora....ty1.vip ora....t1.type ONLINE    ONLINE    yesb...tty1
ora....SM2.asm application    ONLINE    ONLINE    yesb...tty2
ora....Y2.lsnr application    ONLINE    ONLINE    yesb...tty2
ora....ty2.gsd application    OFFLINE   OFFLINE
ora....ty2.ons application    ONLINE    ONLINE    yesb...tty2
ora....ty2.vip ora....t1.type ONLINE    ONLINE    yesb...tty2
oracle@yesbolisetty1.localdomain:/home/oracle [+ASM1] >crsctl check cluster -all
**************************************************************
yesbolisetty1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
yesbolisetty2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oracle@yesbolisetty1.localdomain:/home/oracle [+ASM1] >




oracle@yesbolisetty2.localdomain:/home/oracle [+ASM2] >crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.BACKUP.dg  ora....up.type ONLINE    ONLINE    yesb...tty1
ora.DATA.dg    ora....up.type ONLINE    ONLINE    yesb...tty1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    yesb...tty1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    yesb...tty1
ora.asm        ora.asm.type   ONLINE    ONLINE    yesb...tty1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    yesb...tty1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    yesb...tty1
ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE
ora.ons        ora.ons.type   ONLINE    ONLINE    yesb...tty1
ora.ora11g.db  ora....se.type ONLINE    ONLINE    yesb...tty1
ora....ry.acfs ora....fs.type ONLINE    OFFLINE
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    yesb...tty1
ora....SM1.asm application    ONLINE    ONLINE    yesb...tty1
ora....Y1.lsnr application    ONLINE    ONLINE    yesb...tty1
ora....ty1.gsd application    OFFLINE   OFFLINE
ora....ty1.ons application    ONLINE    ONLINE    yesb...tty1
ora....ty1.vip ora....t1.type ONLINE    ONLINE    yesb...tty1
ora....SM2.asm application    ONLINE    ONLINE    yesb...tty2
ora....Y2.lsnr application    ONLINE    ONLINE    yesb...tty2
ora....ty2.gsd application    OFFLINE   OFFLINE
ora....ty2.ons application    ONLINE    ONLINE    yesb...tty2
ora....ty2.vip ora....t1.type ONLINE    ONLINE    yesb...tty2
oracle@yesbolisetty2.localdomain:/home/oracle [+ASM2] >crsctl check cluster -all
**************************************************************
yesbolisetty1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
yesbolisetty2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oracle@yesbolisetty2.localdomain:/home/oracle [+ASM2] >

Have a fun  :)