Step-by-step guide on installing Oracle RAC 12.1.0.2 on RHEL7.3 (part I)

1. Hardware requirements

1) two or more PC Servers

At least 4 1G (or 10G) TCP network ports, configure 2 team devices (one for public network and one for heartbeat)
At least 2 dual port 8G (or 16G and higher) Fibre Channel cards to connect to shared SAN storage

My Server model is Inspur NF5280M4 2U.

2) IP address allocation

3 persistent IP addresses : one for each RAC node
3 VIP addresses: one for each RAC node
1 SCAN IP address: one for each RAC cluster
3 private IP addresses: one for each RAC node, all the addresses must be in the same VLAN with the same netmask
It is recommended that persistent IP, VIP and SCAN IP addresses are in the same VLAN for simpler network topology.

The ip addresses used during the installation are as follows:

199.31.222.5    soradbs05
199.31.222.7    soradbs06
199.31.222.9    soradbs07
10.10.50.1      soradbs05-priv
10.10.50.2      soradbs06-priv
10.10.50.3      soradbs07-priv
199.31.222.6    soradbs05-vip
199.31.222.8    soradbs06-vip
199.31.222.10   soradbs07-vip
199.31.222.11   soradbs-scanip

3) Storage configuration

Shared storage will be accessed through SAN network. Please check your storage vendor’s compatibility matrix before deploying RAC cluster.

At least two storage devices is recommended to eliminate single point of failure. In this case, redundancy settings of all the disk groups should be NORMAL in ASM. However, if you only have one storage, the redundancy setting should be EXTERNAL.

Three DGs is required: OCRDG(for OCR files), DATADG(for data files) and FLASHDG(for archivelogs).
OCRDG must contain at least three 5GB disks, all of which must reside in three separate storage devices.
The size of DATADG and FLASHDG should be considered according to your workload and capacity concerns.

My storage is IBM V5000.

Multipath must be properly configured before installing RAC.

2. Media preparation

OS : RHEL 7.3 x86_64
Oracle: Grid and database 12.1.0.2 with the latest Bundle Patch (12.1.0.2.170117 for this installation)

Oracle Database Enterprise Edition:
p21419221_121020_Linux-x86-64_1of10.zip
p21419221_121020_Linux-x86-64_2of10.zip

GI:
p21419221_121020_Linux-x86-64_5of10.zip
p21419221_121020_Linux-x86-64_6of10.zip

Client:
p21419221_121020_Linux-x86-64_7of10.zip

3. Operating System Configuration

3.1 File Systems

The following file systems are required:
/oraapp (used for software files), at least 50G
/home/oracle (used for database backups), at least 20G

lvcreate -L 20G -n home_oracle_lv VolGroup
mkfs.xfs /dev/VolGroup/home_oracle_lv
lvs
umask
mkdir /home/oracle
mount /dev/VolGroup/home_oracle_lv /home/oracle

lvcreate -L 50G -n oraapp_lv VolGroup 
mkfs.xfs /dev/VolGroup/oraapp_lv
mkdir /oraapp
mount /dev/VolGroup/oraapp_lv /oraapp

lvs
df –h

vi /etc/fstab, add the following lines:
/dev/mapper/VolGroup-home_oracle_lv /home/oracle              xfs     defaults        0 0
/dev/mapper/VolGroup-oraapp_lv /oraapp              xfs     defaults        0 0

mount –av # mount the file systems manually 

3.2 Create Groups

groupadd -g 3001 oinstall
groupadd -g 3002 dba
groupadd -g 3003 oper
groupadd -g 3016 asmadmin
groupadd -g 3017 asmdba
groupadd -g 3018 asmoper

3.3 Create Users

useradd -g oinstall -G "dba,oper,asmdba" -d /tmp/oracle -u 3001 oracle
useradd -g oinstall -G "asmadmin,asmdba,asmoper" -d /home/grid  -u 3016 grid

cd /tmp/oracle
cp -p .* /home/oracle
usermod -d /home/oracle oracle
chown -R oracle:oinstall /home/oracle
chown -R oracle:oinstall /oraapp

echo "Password" | passwd --stdin oracle
echo "Password" | passwd --stdin grid

3.4 Modify shell limits

vi /etc/security/limits.conf, add the following lines:

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 10240
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 10240
*  soft  memlock  unlimited
*  hard  memlock  unlimited

vi /etc/pam.d/login, add the following line:

session required pam_limits.so

vi /etc/profile, add the following lines :

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
  umask 022
fi

3.5 Configure Multipath devices

rpm -qa | grep device-mapper # verify device-mapper and device-mapper-multipath are installed
/sbin/mpathconf --enable # create file /etc/multipath.conf
service multipathd start
systemctl enable multipathd.service
systemctl list-unit-files | grep -i multipath 
systemctl stop multipathd.service

NOTE: you must consult your vendor’s official documents to configure multipath
The following is only an example of IBM V5000.

vi /etc/multipath.conf, add the following lines:

add the following in default section:

polling_interval 30

append the following lines at the end:

devices {
        device {
                vendor "IBM"
                product "2145"
                path_grouping_policy "group_by_prio"
#               path_selector "round-robin 0"
                path_selector "service-time 0" # Used by RedHat 7.x
                prio "alua"
                path_checker "tur"
                failback "immediate"
                no_path_retry 5
                rr_weight uniform
                rr_min_io_rq "1"
                dev_loss_tmo 120  
      }
}

Reference (Please check IBM SAN Volume Controller version first):
http://www.ibm.com/support/knowledgecenter/en/STPVGU_7.3.0/com.ibm.storage.svc.console.730.doc/svc_linux_settings.html
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/DM_Multipath/mpio_setup.html

systemctl start multipathd.service
systemctl is-active multipathd.service
chmod 644 /etc/multipath.conf

# scan the newly allocated LAN

dmesg |grep -i scsi | head -50

[   10.747368] scsi host5: qla2xxx
[   10.851268] scsi: waiting for bus probes to complete ...
[   10.948390] scsi host6: qla2xxx
[   11.149334] scsi host7: qla2xxx
[   11.352461] scsi host8: qla2xxx

# The above output means the host has 4 FC ports, i.e. host5-8

awk '{print FILENAME,$0}' /sys/class/scsi_host/host[5678]/link_state  #check FC port status

echo "- - -" > /sys/class/scsi_host/host5/scan
echo "- - -" > /sys/class/scsi_host/host7/scan

multipath
multipath -ll # check devices and paths

3.6 Partition the disks

fdisk -c /dev/mapper/mpatha
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x7a7dc79c.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: cylinders as display units are deprecated. Use command 'u' to
         change units to sectors.

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-10485759, default 2048): 2048
Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759): 
Using default value 10485759

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

In each case, the sequence of answers is “n”, “p”, “1”, “Return”, “Return” and “w”.

Repeat the above steps for each multipath device.

3.7 Configure udev rules

For Oracle Automatic Storage Manager (ASM) to use disks, it needs to be able to identify the devices consistently and for them to have the correct ownership and permissions. In Linux you can use ASMLib to manage these tasks, but it is seen as an additional layer of complexity and has never really gained any popularity. Instead, many people use the Linux device manager “udev” to perform these tasks.

# check DM_UUID information of each disk

udevadm info --query=all --name=/dev/mapper/mpatha | grep -i DM_UUID
E: DM_UUID=mpath-3600507680c80803a48000000000000bf
udevadm info --query=all --name=/dev/mapper/mpathb | grep -i DM_UUID
E: DM_UUID=mpath-3600507680c80803a48000000000000c0
udevadm info --query=all --name=/dev/mapper/mpathc | grep -i DM_UUID
E: DM_UUID=mpath-3600507680c80803a48000000000000c4
udevadm info --query=all --name=/dev/mapper/mpathd | grep -i DM_UUID
E: DM_UUID=mpath-3600507680c80803a48000000000000c5
udevadm info --query=all --name=/dev/mapper/mpathe | grep -i DM_UUID
E: DM_UUID=mpath-3600507680c80803a48000000000000c6

vi /etc/udev/rules.d/99-oracle-asmdevices.rules #This is a new file. Add the following lines

ACTION=="add|change", ENV{DM_UUID}=="mpath-3600507680c80803a48000000000000bf", SYMLINK+="oracleasm/asm-disk1", OWNER="grid", GROUP="oinstall", MODE="0660"
ACTION=="add|change", ENV{DM_UUID}=="mpath-3600507680c80803a48000000000000c0", SYMLINK+="oracleasm/asm-disk2", OWNER="grid", GROUP="oinstall", MODE="0660"
ACTION=="add|change", ENV{DM_UUID}=="mpath-3600507680c80803a48000000000000c4", SYMLINK+="oracleasm/asm-disk3", OWNER="grid", GROUP="oinstall", MODE="0660"
ACTION=="add|change", ENV{DM_UUID}=="mpath-3600507680c80803a48000000000000c5", SYMLINK+="oracleasm/asm-disk4", OWNER="grid", GROUP="oinstall", MODE="0660"
ACTION=="add|change", ENV{DM_UUID}=="mpath-3600507680c80803a48000000000000c6", SYMLINK+="oracleasm/asm-disk5", OWNER="grid", GROUP="oinstall", MODE="0660"

NOTE: Since RHEL7, multipath will be more aggressive and you may fail to get /dev/sd* devices if you intend to manage those devices in udev rules.

# Load updated partition tables

/sbin/partprobe /dev/mapper/mpatha1
/sbin/partprobe /dev/mapper/mpathb1
/sbin/partprobe /dev/mapper/mpathc1
/sbin/partprobe /dev/mapper/mpathd1
/sbin/partprobe /dev/mapper/mpathe1

# Load udev rules

/sbin/udevadm control --reload
/sbin/udevadm trigger --type=devices --action=change

ls -lL /dev/oracleasm/asm-disk*
brw-rw---- 1 grid oinstall 253, 4 Jan 22 14:20 /dev/oracleasm/asm-disk1
brw-rw---- 1 grid oinstall 253, 6 Jan 22 14:20 /dev/oracleasm/asm-disk2
brw-rw---- 1 grid oinstall 253, 3 Jan 22 14:20 /dev/oracleasm/asm-disk3
brw-rw---- 1 grid oinstall 253, 5 Jan 22 14:20 /dev/oracleasm/asm-disk4
brw-rw---- 1 grid oinstall 253, 2 Jan 22 14:20 /dev/oracleasm/asm-disk5

3.8 Edit /etc/hosts

Make sure file /etc/hosts is identical on each node.

199.31.222.5    soradbs05
199.31.222.7    soradbs06
199.31.222.9    soradbs07

10.10.50.1      soradbs05-priv
10.10.50.2      soradbs06-priv
10.10.50.3      soradbs07-priv

199.31.222.6    soradbs05-vip
199.31.222.8    soradbs06-vip
199.31.222.10   soradbs07-vip

199.31.222.11   soradbs-scanip

3.9 System parameters and services

vi /etc/sysctl.conf, add the following lines:

kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.shmall = 67108864  # maximum total shm size: 67108864 *4096 = 256G
kernel.shmmax =  137438953472 # maximum single shm segment size: 128G
kernel.shmmni = 4096

NOTE:
shmmax should be at least half the size of your physical memory.
shmall * PAGESIZE should be at least 80% of your physical memory.

# update the above settings

sysctl -p

# disable avahi-daemon, NetworkManager, selinux and firewall

systemctl list-unit-files | grep -i avahi-daemon 
systemctl list-unit-files | grep -i networkmanager
getenforce 
systemctl list-unit-files | grep -i firewalld

3.10 Configure NTP

According to BUG 20936562, RAC before 12.2 doesn’t support chrony, the new time service in RHEL7 and we still need NTP service.

cp /etc/ntp.conf /etc/ntp.conf.bak

vi /etc/ntp.conf, comment out the default NTP servers and add your own NTP servers

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 200.31.131.131 prefer
server 200.31.131.132

vi /etc/sysconfig/ntpd, add -x to the comand line options

# Command line options for ntpd
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"

systemctl disable chronyd.service
systemctl enable ntpd.service
systemctl start ntpd.service

systemctl list-unit-files | grep -iE "chrony|ntp"

ntpq –pn

       remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*200.31.131.131  .GPS.            1 u   12   64    3    1.043   -0.220   0.624
+200.31.131.132  .GPS.            1 u   12   64    3    0.992    7.795   0.412

ps -ef |grep ntp # check command options of ntpd

3.11 Create software directories

# as user root on each server

umask 022
mkdir -p /oraapp
chown -R grid:oinstall /oraapp
chmod -R 775 /oraapp

mkdir -p /oraapp/oracle
chown -R oracle.oinstall /oraapp/oracle
chmod -R 755 /oraapp/oracle

mkdir -p /oraapp/grid/gridbase
chown grid:oinstall  /oraapp/grid
chown -R grid.oinstall /oraapp/grid/gridbase
chmod -R 755 /oraapp/grid/gridbase

mkdir -p /oraapp/grid/gridhome
chown -R grid.oinstall /oraapp/grid/gridhome
chmod -R 755 /oraapp/grid/gridhome

3.12 Modify .bash_profiles

# as user grid on each server, please modify ORACLE_SID accordingly

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

# added for grid
export ORACLE_SID=+ASM1
export ORACLE_BASE=/oraapp/grid/gridbase
export ORACLE_HOME=/oraapp/grid/gridhome
export ORACLE_TERM=xterm
export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
export NLS_LANG="AMERICAN_AMERICA.AL32UTF8"
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=${PATH}:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$HOME/bin
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export TEMP=/tmp
export TMPDIR=/tmp

umask 022

# as user oracle on each server, please modify ORACLE_SID accordingly

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

#added for oracle
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_BASE=/oraapp/oracle
export ORACLE_HOME=$ORACLE_BASE/product/12.1.0.2/dbhome_1
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:/usr/sbin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
export NLS_LANG="AMERICAN_AMERICA.AL32UTF8"

3.13 Check installed rpms

rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils  \
compat-libcap1  \
compat-libstdc++  \
compat-libstdc++-33.i686  \
gcc  \
gcc-c++  \
glibc  \
glibc.i686  \
glibc-devel  \
glibc-devel.i686  \
ksh  \
libgcc  \
libgcc.i686  \
libstdc++  \
libstdc++.i686  \
libstdc++-devel  \
libstdc++-devel.i686  \
libaio  \
libaio.i686  \
libaio-devel  \
libaio-devel.i686  \
libXext  \
libXext.i686  \
libXtst  \
libXtst.i686  \
libX11  \
libX11.i686  \
libXau  \
libXau.i686  \
libxcb  \
libxcb.i686  \
libXi  \
libXi.i686  \
make  \
sysstat  \
unixODBC  \
unixODBC-devel  \
net-tools  \
smartmontools 

# you may need to install the following rpms

yum install binutils -y
yum install compat-libcap1 -y
#yum install compat-libstdc++ -y
#yum install compat-libstdc++-33.i686 -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libXext -y
yum install libXext.i686 -y
yum install libXtst -y
yum install libXtst.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel –y
yum install net-tools –y
yum install smartmontools -y

# These two rpms were not shipped by the default RHEL media and need to be installed separately

rpm -ivh compat-libstdc++-33-3.2.3-71.el7.x86_64.rpm
rpm -ivh compat-libstdc++-33-3.2.3-71.el7.i686.rpm

# Change directory to grid installation media and install cvuqdisk

rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]

3.14 Switch off NUMA

According to Doc ID 1962100.1:
It is observed there are hang issues in RHEL 7 with many CPU cores and more RAM, due NUMA was enabled. As a workaround it is recommended to turn off NUMA.

# as user root on each server

# Check default NUMA settings

grubby --default-kernel
/boot/vmlinuz-3.10.0-514.el7.x86_64

# Check default grub information

grubby --info /boot/vmlinuz-3.10.0-514.el7.x86_64
index=0
kernel=/boot/vmlinuz-3.10.0-514.el7.x86_64
args="ro crashkernel=auto rd.lvm.lv=VolGroup/root_lv rd.lvm.lv=VolGroup/swap_lv biosdevname=0 net.ifnames=0 rhgb quiet LANG=en_US.UTF-8"
root=/dev/mapper/VolGroup-root_lv
initrd=/boot/initramfs-3.10.0-514.el7.x86_64.img
title=Red Hat Enterprise Linux Server (3.10.0-514.el7.x86_64) 7.3 (Maipo)

# Add numa=off option

grubby --args=numa=off --update-kernel /boot/vmlinuz-3.10.0-514.el7.x86_64

# reboot the servers

sync
shutdown -r now

# verify numa was turned off

dmesg | grep -i numa
NUMA turned off

3.15 Check network settings

vi /etc/sysconfig/network, add the following line:

NOZEROCONF=yes

How to Prevent Bogus Entry 169.254.0.0/255.255.0.0 Automatically Added To Routing Table (Doc ID 1161144.1)

Also, you need to ping each persistent and private IP and check the response time.

to be continued…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s