Sunday, September 12, 2010

Step-By-Step Installation of 9.2.0.5 RAC on Linux

Purpose
This document will provide the reader with step-by-step instructions on how to install a cluster, install Oracle Real Application Clusters (RAC) (Version 9.2.0.5), and start a cluster database on Linux. For additional explanation or information on any of these steps, please see the references listed at the end of this document.
Disclaimer: If there are any errors or issues prior to step 2, please contact your Linux distributor.
The information contained here is as accurate as possible at the time of writing.
1. Configuring the Clusters Hardware<>
1.1 Minimal Hardware list / System Requirements
Please check the RAC/Linux certification matrix for information on currently supported hardware/software.
1.1.1 Hardware
  • Requirements:
    • Refer to the RAC/Linux certification matrix for information on supported configurations.  Ensure that the system has at least the following resources:
      - 400 MB in /tmp
      - 512 MB of Physical Memory (RAM)
      - Three times the amount of Physical Memory for Swap space (unless the system exceeds 1 GB of Physical Memory, where two times the amount of Physical Memory for Swap space is sufficient)
An example system disk layout is as follows:-
A sample system disk layout
Slice Contents Allocation (in Mbytes)
0 / 2000 or more
1 /boot 64
2 /tmp 1000
3 /usr 3000-7000 depending on operating system and packages installed
4 /var 512 (can be more if required)
5 swap Three times the amount of Physical Memory for Swap space (unless the system exceeds 1 GB of Physical Memory, where two times the amount of Physical Memory for Swap space is sufficient).
6 /home 2000 (can be more if required)
1.1.2 Software
  • For RAC on Linux support, consult the operating system vendor and see the RAC/Linux certification matrix.
  • Make sure you have make and rsh-server packages installed, check with: $rpm -q rsh-server make
    rsh-server-0.17-5
    make-3.79.1-8
    If these are not installed, use your favorite package manager to install them.
1.1.3 Patches
Consult with your operating system vendor to get on the latest patch version of the kernel.
1.2 Installing the Shared Disk Subsystem
This is highly dependent on the subsystem you have chosen. Please refer to your hardware documentation for installation and configuration instructions on Linux. Additional drivers and patches might be required. In this article we assume that the shared disk subsystem is correctly installed and that the shared disks are visible to all nodes in the cluster.
1.3 Configuring the Cluster Interconnect and Public Network Hardware
If not already installed, install host adapters in your cluster nodes. For the procedure on installing host adapters, see the documentation that shipped with your host adapters and node hardware.
Each system will have at least an IP address for the public network and one for the private cluster interconnect. For the public network, get the addresses from your network manager. For the private interconnect use 1.1.1.1 , 1.1.1.2 for the first and second node. Make sure to add all addresses in /etc/hosts.
[oracle@opcbrh1 oracle]$ more /etc/hosts
ex:
9.25.120.143    rac1        #Oracle 9i Rac node 1 – public network
9.25.120.143    rac2        #Oracle 9i Rac node 2 – public network
1.1.1.1              int-rac1   #Oracle 9i Rac node 1 – interconnect
1.1.1.2              int-rac2   #Oracle 9I Rac node 2 – interconnect
Use your favorite tool to configure these adapters. Make sure your public network is the primary (eth0).
Interprocess communication is an important issue for RAC since cache fusion transfers buffers between instances using this mechanism. Thus, networking parameters are important for RAC databases. The values in the following table are the recommended values. These are NOT the default on most distributions.
Parameter Meaning Value
/proc/sys/net/core/rmem_default The default setting in bytes of the socket receive buffer 262144
/proc/sys/net/core/rmem_max The maximum socket receive buffer size in bytes 262144
/proc/sys/net/core/wmem_default The default setting in bytes of the socket send buffer 262144
/proc/sys/net/core/wmem_max The maximum socket  send  buffer  size  in bytes 262144

You can see these settings with:
$ cat /proc/sys/net/core/rmem_default
Change them with:
$ echo  262144 >  /proc/sys/net/core/rmem_default
This will need to be done each time the system boots. Some distributions already have setup a method for this during boot. On Red Hat , this can be configured in /etc/sysctl.conf (like : net.core.rmem_default = 262144).



2. Creating a Cluster
On Linux, the cluster software required to run Real Application Clusters is included in the Oracle distribution.
The Oracle Cluster Manager (ORACM) installation process includes eight major tasks.
  1. UNIX pre-installation tasks.
  2. Configuring the shared disks
  3. Run the Oracle Universal Installer to install the 9.2.0.4 ORACM (Oracle Cluster Manager)
  4. Configure the hangcheck-timer.
  5. Install version 10.1.0.2 of the Oracle Universal Installer
  6. Run the 10.1.0.2 Oracle Universal Installer to patch the Oracle Cluster Manager (ORACM) to 9.2.0.5
  7. Modify the ORACM configuration files to utilize the hangcheck-timer.
  8. Start the ORACM (Oracle Cluster Manager)
2.1 UNIX Pre-installation tasks
These steps need to be performed on ALL nodes.
  • First, on each node, create the Oracle group.  Example:
# groupadd dba -g 501
  • Next, make the Oracle user’s home directory.  Example:
# mkdir -p /u01/home/oracle
  • On each node, create the Oracle user.  Make sure that the Oracle user is part of the dba group.  Example:
# useradd -c “Oracle Software Owner” -G dba -u 101 -m -d /u01/home/oracle -s /bin/csh oracle
  • On each node, Create a mount point for the Oracle software installation (at least 2.5 GB, typically /u01).  The oracle user should own this mount point and all of the directories below the mount point.  Example:
# mkdir /u01
# chown -R oracle.dba /u01
# chmod -R ug=rwx,o=rx /u01
  • Once this is done, test the permissions on each node to ensure that the oracle user can write to the new mount points.  Example:
# su – oracle
$ touch /u01/test
$ ls -l /u01/test
-rw-rw-r– 1 oracle dba 0 Aug 15 09:36 /u01/test
  • Depending on your Linux distribution, make sure inetd or xinetd is started on all nodes and that the ftp, telnet, shell and login (or rsh) services are enabled (see /etc/inetd.conf or /etc/xinetd.conf and /etc/xinetd.d).   Example:
# more /etc/xinetd.d/telnet
# default: on
# description: The telnet server serves telnet sessions; it uses \
# unencrypted username/password pairs for authentication.
service telnet
{
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/sbin/in.telnetd
log_on_failure += USERID
disable = no
}
In this example, disable should be set to ‘no’.
  • On the node from which you will run the Oracle Universal Installer, set up user equivalence by adding entries for all nodes in the cluster, including the local node, to the .rhosts file of the oracle account, or the /etc/hosts.equiv file.
Sample entries in /etc/hosts.equiv file:
rac1
rac2
int-rac1
int-rac2
  • As oracle user, check for user equivalence for the oracle account by performing a remote copy (rcp) to each node (public and private) in the cluster.  Example:
RAC1:
$ touch /u01/test
$ rcp /u01/test rac2:/u01/test1
$ rcp /u01/test int-rac2:/u01/test2
RAC2:
$ touch /u01/test
$ rcp /u01/test rac1:/u01/test1
$ rcp /u01/test int-rac1:/u01/test2
$ ls /u01/test*
/u01/test /u01/test1 /u01/test2
RAC1:
$ ls /u01/test*
/u01/test /u01/test1 /u01/test2
Note: If you are prompted for a password, you have not given the oracle account the same attributes on all nodes. You must correct this because the Oracle Universal Installer cannot use the rcp command to copy Oracle products to the remote node’s directories without user equivalence.
System Kernel Parameters
Verify operating system kernel parameters are set to appropriate levels:
Kernel Parameter Setting Purpose
SHMMAX 2147483648 Maximum allowable size of one shared memory segment.
SHMMIN 1 Minimum allowable size of a single shared memory segment.
SHMMNI 100 Maximum number of shared memory segments in the entire system.
SHMSEG 10 Maximum number of shared memory segments one process can attach.
SEMMNI 100 Maximum number of semaphore sets in the entire system.
SEMMSL 250 Minimum recommended value. SEMMSL should be 10 plus the largest PROCESSES parameter of any Oracle database on the system.
SEMMNS 1000 Maximum semaphores on the system. This setting is a minimum recommended value. SEMMNS should be set to the sum of the PROCESSES parameter for each Oracle database, add the largest one twice, plus add an additional 10 for each database.
SEMOPM 100 Maximum number of operations per semop call.
You will have to set the correct parameters during system startup, so include them in your startup script (startoracle_root.sh):
$ export SEMMSL=100
$ export SEMMNS=1000
$ export SEMOPM=100
$ export SEMMNI=100
$ echo  $SEMMSL $SEMMNS $SEMOPM $ SEMMNI > /proc/sys/kernel/sem
$ export SHMMAX=2147483648
$ echo $SHMMAX > /proc/sys/kernel/shmmax
Check these with:
$ cat /proc/sys/kernel/sem
$ cat /proc/sys/kernel/shmmax
You might want to increase the maximum number of file handles, include this in your startup script or use /etc/sysctl.conf :
$ echo 65536 > /proc/sys/fs/file-max
To allow your oracle processes to use these file handles, add the following to your oracle account login script (ex.: .profile)
$ ulimit -n 65536
Note: This will only allow you to set the soft limit as high as the hard limit. You might have to increase the hard limit on system level. This can be done by adding ulimit -Hn 65536 to /etc/initscript. You will have to reboot the system to make this active.  Sample /etc/initscript: ulimit -Hn 65536
eval exec “$4″
Establish Oracle environment variables: Set the following Oracle environment variables:
Environment Variable Suggested value
ORACLE_HOME eg /u01/app/oracle/product/920
ORACLE_TERM xterm
PATH /u01/app/oracle/product/9.2.0/bin: /usr/ccs/bin:/usr/bin/X11/:/usr/local/bin and any other items you require in your PATH
DISPLAY <ip-address>:0.0 (review Note:153960.1 for detailed information)
TMPDIR Set a temporary directory path for TMPDIR with at least 100 Mb of free space to which the OUI has write permission.
ORACLE_SID Set this to what you will call your database instance. This should be UNIQUE on each node.
It is best to save these in a .login or .profile file so that you do not have to set the environment every time you log in.
  • Create the directory /var/opt/oracle and set ownership to the oracle user.  Example:
$ mkdir /var/opt/oracle
$ chown oracle.dba /var/opt/oracle
  • Set the oracle user’s umask to “022″ in you “.profile” or “.login” file. Example:
$ umask 022
Note: There is a verification script InstallPrep.sh available which may be downloaded and run prior to the installation of Oracle Real Application Clusters. This script verifies that the system is configured correctly according to the Installation Guide. The output of the script will report any further tasks that need to be performed before successfully installing Oracle 9.x DataServer (RDBMS). This script performs the following verifications:-
  • ORACLE_HOME Directory Verification
  • UNIX User/umask Verification
  • UNIX Group Verification
  • Memory/Swap Verification
  • TMP Space Verification
  • Real Application Cluster Option Verification
  • Unix Kernel Verification
. ./InstallPrep.sh
You are currently logged on as oracle
Is oracle the unix user that will be installing Oracle Software? y or n
y
Enter the unix group that will be used during the installation
Default: dba
Enter the version of Oracle RDBMS you will be installing
Enter either : 901 OR 920 – Default: 920
920
The rdbms version being installed is 920
Enter Location where you will be installing Oracle
Default: /u01/app/oracle/product/oracle9i
/u01/app/oracle/product/9.2.0
Your Operating System is Linux
Gathering information… Please wait
JDK check is ignored for Linux since it is provided by Oracle
Checking unix user …
Checking unix umask …
umask test passed
Checking unix group …
Unix Group test passed
Checking Memory & Swap…
Memory test passed
/tmp test passed
Checking for a cluster…
Linux Cluster test section has not been implemented yet
No cluster warnings detected
Processing kernel parameters… Please wait
Running Kernel Parameter Report…
Check the report for Kernel parameter verification\n
Completed.
/tmp/Oracle_InstallPrep_Report has been generated
Please review this report and resolve all issues before attempting to
install the Oracle Database Software
Note: If you get an error like this:
InstallPrep.sh: line 45: syntax error near unexpected token `fi’
or
./InstallPrep.sh: Command not found.
Then you need to copy the script into a text file (it will not run if the file is in binary format).
2.2 Configuring the Shared Disks
For 9.2 Real Application Clusters on Linux, you can use either OCFS (Oracle Cluster Filesystem), RAW, or NFS (Redhat and Network Appliance Only) for storage of Oracle database files.
  • For more information on setting up OCFS for RAC on Linux, see the following MetaLink Note:
Note 220178.1 – Installing and setting up ocfs on Linux – Basic Guide
  • For more information on setting up RAW for RAC on Linux, see the following MetaLink Note:
Note 246205.1 – Configuring Raw Devices for Real Application Clusters on Linux
  • For more information on setting up NFS for RAC on Linux, see the following MetaLink Note (Steps 1-6):
Note 210889.1 – RAC Installation with a NetApp Filer in Red Hat Linux Environment

2.3 Run the Oracle Universal Installer to install the 9.2.0.4 ORACM (Oracle Cluster Manager)
These steps only need to be performed on the node that you are installing from (typically Node 1).
  • If you are using OCFS or NFS for your shared storage, pre-create the quorum file and srvm file. Example:
#  dd if=/dev/zero of=/ocfs/quorum.dbf bs=1M count=20
#  dd if=/dev/zero of=/ocfs/srvm.dbf bs=1M count=100
#  chown root:dba /ocfs/quorum.dbf
#  chmod 664 /ocfs/quorum.dbf
#  chown oracle:dba  /ocfs/srvm.dbf
#  chmod 664 /ocfs/srvm.dbf
  • Verify the Environment – Log off and log on as the oracle user to ensure all environment variables
    are set correctly. Use the following command to view them:     % env | more
Note: If you are on Redhat Advanced Server 3.0, you will need to temporarily use an older gcc for the install: mv gcc gcc3.2.3
mv g++ g++3.2.3
ln -s /usr/bin/gcc296 /usr/bin/gcc
ln -s /usr/bin/g++296 /usr/bin/g++

You will also need to apply patch 3006854 if on RHAS 3.0
  • Before attempting to run the Oracle Universal Installer, verify that you can successfully run the following command: % /usr/bin/X11/xclock
  • If this does not display a clock on your display screen, please review the following article:
Note 153960.1 FAQ: X Server testing and troubleshooting
  • Start the Oracle Universal Installer and install the RDBMS software – Follow these procedures to use the Oracle Universal Installer to install the Oracle Cluster Manager software. Oracle9i is supplied on multiple CD-ROM disks. During the installation process it is necessary to switch between the CD-ROMS. OUI will manage the switching between CDs. Use the following commands to start the installer:
    % cd /tmp
    % /cdrom/runInstaller
    Or cd to /stage/Disk1 and run ./runInstaller
    Respond to the installer prompts as shown below:
  • At the “Welcome Screen”, click Next.
If this is your first install on this machine:
  • If the “Inventory Location” screen appears, enter the inventory location then click OK.
  • If the “Unix Group Name” screen appears, enter the unix group name created in step 2.1 then click Next.
  • At this point you may be prompted to run /tmp/orainstRoot.sh.  Run this and click Continue.
  • At the “File Locations Screen”, verify the destination listed is your ORACLE_HOME directory.  Also enter a NAME to identify this ORACLE_HOME.  The NAME can be anything.
  • At the “Available Products Screen”, Check “Oracle Cluster Manager”. Click Next.
  • At the public node information screen, enter the public node names and click Next.
  • At the private node information screen, enter the interconnect node names. Click Next.
  • Enter the full name of the file or raw device you have created for the ORACM Quorum disk information. Click Next.
  • Press Install at the summary screen.
  • You will now briefly get a progress window followed by the end of installation screen. Click Exit and confirm by clicking Yes.
Note: Create the directory $ORACLE_HOME/oracm/log (as oracle) on the other nodes if it doesn’t exist.
2.4 Configure the hangcheck-timer
These steps need to be performed on ALL nodes.
Some kernel versions include the hangcheck-timer with the kernel.  You can check to see if your kernel contains the hangcheck-timer by running:
# /sbin/lsmod
Then you will see hangcheck-timer listed.  Also verify that hangcheck-timer is starting in your /etc/rc.local file (on Redhat) or /etc/init.d/boot.local (on United Linux).  If you see hangcheck-timer listed in lsmod and in the rc.local file or boot.local, you can skip to section 2.5.
If hangcheck-timer is not listed here and you are not using Redhat Advanced Server, see the following note for information on obtaining the hangcheck-timer:
Note 232355.1 – Hangcheck Timer FAQ
If you are on Redhat Advanced Server, you can either apply the latest errata version (> 12) or go to MetaLink – Patches:
Enter 2594820 in the Patch Number field.
Click Go.
Click Download.
Save the file p2594820_20_LINUX.zip to the local disk, such as /tmp.
Unzip the file. The output should be similar to the following:
inflating: hangcheck-timer-2.4.9-e.3-0.4.0-1.i686.rpm
inflating: hangcheck-timer-2.4.9-e.3-enterprise-0.4.0-1.i686.rpm
inflating: hangcheck-timer-2.4.9-e.3-smp-0.4.0-1.i686.rpm
inflating: README.TXT
Run the uname -a command to identify the RPM that corresponds to the kernel in use. This will show if the kernel is single CPU, smp, or enterprise.
The p2594820_20_LINUX.zip file contains four files. The following describes the files:
hangcheck-timer-2.4.9-e.3-0.4.0-1.i686.rpm is for single CPU machines
hangcheck-timer-2.4.9-e.3-enterprise=0.4.0-1.i686.rpm is for multi-processor machines with more than 4 GB of RAM
hangcheck-timer-2.4.9-e.3-smp-0.4.0-1.i686.rpm is for multi-processor machines with 4 GB of RAM or less
The three RPMs will work for the e3 kernels in Red Hat Advanced Server 2.1 gold and the e8 kernels in the latest Red Hat Advanced Server 2.1 errata release. These RPMs are for Red Hat Advanced Server 2.1 kernels only.
Transfer the relevant hangcheck-timer RPM to the /tmp directory of the Oracle Real Applications Cluster node.
Log in to the node as the root user.
Change to the /tmp directory.
Run the following command to install the module:
#rpm -ivh hangcheck-timer RPM name
If you have previously installed RAC on this cluster, remove or disable the mechanism that loads the softdog module at system start up, if that module is not used by other software on the node. This is necessary for subsequent steps in the installation process. This step may require log in as the root user.  One method for setting up previous versions of Oracle Real Applications Clusters involved loading the softdog module in the /etc/rc.local (on Redhat) or /etc/init.d/boot.local (on United Linux) file. If this method was used, then remove or comment out the following line in the file: /sbin/insmod softdog nowayout=0 soft_noboot=1 soft_margin=60
Append the following line to the /etc/rc.local file (on Redhat) or /etc/init.d/boot.local (on United Linux):
/sbin/insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
Load the hangcheck-timer kernel module using the following command as root user:
# /sbin/insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
Repeat the above steps on all Oracle Real Applications Clusters nodes where the kernel module needs to be installed.
Run dmesg after the module is loaded. Note the build number while running the command. The following is the relevant information output:
build 334adfa62c1a153a41bd68a787fbe0e9
The build number is required when making support calls.
2.5 Install Version 10.1.0.2 of the Oracle Universal Installer
These steps need to be performed on ALL nodes.
Download the 9.2.0.5 patchset from MetaLink – Patches:
Enter 3501955 in the Patch Number field.
Click Go.
Click Download.
  • Place the file in a patchset directory on the node you are installing from.  Example:
$ mkdir $ORACLE_HOME/9205
$ cp  p3501955_9205_LINUX.zip $ORACLE_HOME/9205
  • Unzip the file:
$ cd $ORACLE_HOME/9205
$ unzip p3501955_9205_LINUX.zip
Archive: p3501955_9205_LINUX
inflating: 9205_lnx32_release.cpio
inflating: README.html
inflating: ReleaseNote9205.pdf
  • Run CPIO against the file:
$ cpio -idmv < 9205_lnx32_release.cpio
Run the installer from the 9.2.0.5 staging location:
$ cd $ORACLE_HOME/9205/Disk1
$ ./runInstaller
Respond to the installer prompts as shown below:
  • At the “Welcome Screen”, click Next.
  • At the “File Locations Screen”, Change the $ORACLE_HOME name from the dropdown list to the 9.2 $ORACLE_HOME name.  ClickNext.
  • On the “Available Products Screen”, Check “Oracle Universal Installer 10.1.0.2. Click Next.
  • Press Install at the summary screen.
  • You will now briefly get a progress window followed by the end of installation screen. Click Exit and confirm by clicking Yes.
Remember to install the 10.1.0.2 Installer on ALL cluster nodes.  Note that you may need to ADD the 9.2 $ORACLE_HOME name on the “File Locations Screen” for other nodes.  It will ask if you want to specify a non-empty directory, say “Yes”.
2.6 Run the 10.1.0.2 Oracle Universal Installer to patch the Oracle Cluster Manager (ORACM) to 9.2.0.5
These steps only need to be performed on the node that you are installing from (typically Node 1).
The 10.1.0.2 OUI will use SSH (Secure Shell) if it is configured.  If it is not configured it will use RSH (Remote Shell).  If you have SSH configured on your cluster, test and make sure that you can SSH and SCP to all nodes of the cluster without being prompted.  If you do not have SSH configured, skip this step and run the installer from $ORACLE_BASE/oui/bin as noted below.
SSH Test:
As oracle user, check for user equivalence for the oracle account by performing a secure copy (scp) to each node (public and private) in the cluster.  Example:
RAC1:
$ touch /u01/sshtest
$ scp /u01/sshtest rac2:/u01/sshtest1
$ scp /u01/sshtest int-rac2:/u01/sshtest2
RAC2:
$ touch /u01/sshtest
$ scp /u01/sshtest rac1:/u01/sshtest1
$ scp /u01/sshtest int-rac1:/u01/sshtest2
$ ls /u01/sshtest*
/u01/sshtest /u01/sshtest1 /u01/sshtest2
RAC1:
$ ls /u01/sshtest*
/u01/sshtest /u01/sshtest1 /u01/sshtest2
Run the installer from the 9.2.0.5 oracm staging location:
$ cd $ORACLE_HOME/9205/Disk1/oracm
$ ./runInstaller
Respond to the installer prompts as shown below:
  • At the “Welcome Screen”, click Next.
  • At the “File Locations Screen”, make sure the source location is to the products.xml file in the 9.2.0.5 patchset location under Disk1/stage.  Also verify the destination listed is your ORACLE_HOME directory.  Change the $ORACLE_HOME name from the dropdown list to the 9.2 $ORACLE_HOME name.  Click Next.
  • At the “Available Products Screen”, Check “Oracle9iR2 Cluster Manager 9.2.0.5.0″. Click Next.
  • At the public node information screen, enter the public node names and click Next.
  • At the private node information screen, enter the interconnect node names. Click Next.
  • Click Install at the summary screen.
  • You will now briefly get a progress window followed by the end of installation screen. Click Exit and confirm by clicking Yes.
2.7 Modify the ORACM configuration files to utilize the hangcheck-timer.
These steps need to be performed on ALL nodes.
Modify the $ORACLE_HOME/oracm/admin/cmcfg.ora file:
Add the following line:
KernelModuleName=hangcheck-timer
Adjust the value of the MissCount line based on the sum of the hangcheck_tick and hangcheck_margin values. (> 210)
MissCount=210
Make sure that you can ping each of the names listed in the private and public node name sections from each node.  Example:
$ ping rac2
PING opcbrh2.us.oracle.com (138.1.137.46) from 138.1.137.45 : 56(84) bytes of data.
64 bytes from opcbrh2.us.oracle.com (138.1.137.46): icmp_seq=0 ttl=255 time=1.295 msec
64 bytes from opcbrh2.us.oracle.com (138.1.137.46): icmp_seq=1 ttl=255 time=154 usec
Verify that a valid CmDiskFile line exists in the following format:
CmDiskFile=file or raw device name
In the preceding command, the file or raw device must be valid. If a file is used but does not exist, then the file will be created if the base directory exists. If a raw device is used, then the raw device must exist and have the correct ownership and permissions.  Sample cmcfg.ora file:
ClusterName=Oracle Cluster Manager, version 9i
MissCount=210
PrivateNodeNames=int-rac1 int-rac2
PublicNodeNames=rac1 rac2
ServicePort=9998
CmDiskFile=/u04/quorum.dbf
KernelModuleName=hangcheck-timer
HostName=int-rac1
Note: The cmcfg.ora file should be the same on both nodes with the exception of the HostName parameter which should be set to the local (internal) hostname.
Make sure all of these changes have been made to all RAC nodes.  More information on ORACM parameters can be found in the following note:
Note 222746.1 – RAC Linux 9.2: Configuration of cmcfg.ora and ocmargs.ora
Note: At this point it would be a good idea to patch to the latest ORACM, especially if you have more than 2 nodes.  For more information see: Note 278156.1 – ORA-29740 or ORA-29702 After Applying 9.2.0.5 Patchset on RAC / Linux

2.8 Start the ORACM (Oracle Cluster Manager)
These steps need to be performed on ALL nodes.
Cd to the $ORACLE_HOME/oracm/bin directory, change to the root user, and start the ORACM.
$ cd $ORACLE_HOME/oracm/bin
$ su root
# ./ocmstart.sh
oracm </dev/null 2>&1 >/u01/app/oracle/product/9.2.0/oracm/log/cm.out &
Verify that ORACM is running with the following:
#  ps -ef | grep oracm
On RHEL 3.0, add the -m option:

#  ps -efm | grep oracm
You should see several oracm threads running.  Also verify that the ORACM version is the same on each node:
# cd $ORACLE_HOME/oracm/log
# head -1 cm.log
oracm, version[ 9.2.0.2.0.49 ] started {Fri May 14 09:22:28 2004 }



3.0 Installing RAC
The Real Application Clusters installation process includes four major tasks.
  1. Install 9.2.0.4 RAC.
  2. Patch the RAC Installation to 9.2.0.5.
  3. Start the GSD.
  4. Create and configure your database.
3.1 Install 9.2.0.4 RAC
These steps only need to be performed on the node that you are installing from (typically Node 1).
Note: Due to bug 3547724, temporarily create a symbolic link /oradata directory pointing to an oradata directory with space available as root prior to running the RAC install: # mkdir /u04/oradata
# chmod 777 /u04/oradata
# ln -s /u04/oradata /oradata
Install 9.2.0.4 RAC into your $ORACLE_HOME by running the installer from the 9.2.0.4 cd or your original stage location for the 9.2.0.4 install.
Use the following commands to start the installer:
% cd /tmp
% /cdrom/runInstaller
Or cd to /stage/Disk1 and run ./runInstaller
Respond to the installer prompts as shown below:
  • At the “Welcome Screen”, click Next.
  • At the “Cluster Node Selection Screen”, make sure that all RAC nodes are selected.
  • At the “File Locations Screen”, verify the destination listed is your ORACLE_HOME directory and that the source directory is pointing to the products.jar from the 9.2.0.4 cd or staging location.
  • At the “Available Products Screen”, check “Oracle 9i Database 9.2.0.4″. Click Next.
  • At the “Installation Types Screen”, check “Enterprise Edition” (or whichever option your prefer), click Next.
  • At the “Database Configuration Screen”, check “Software Only”.  Click Next.
  • At the “Shared Configuration File Name Screen”, enter the path of the CFS or NFS srvm file created at the beginning of step 2.3 or the raw device created for the shared configuration file.  Click Next.
  • Click Install at the summary screen.  Note that some of the items installed will say “9.2.0.1″ for the version, this is normal because only some items needed to be patched up to 9.2.0.4.
  • You will now get a progress window, run root.sh when prompted.
  • You will then see the end of installation screen. Click Exit and confirm by clicking Yes.
Note: You can now remove the /oradata symbolic link: # rm /oradata
3.2 Patch the RAC Installation to 9.2.0.5
These steps only need to be performed on the node that you are installing from.
Run the installer from the 9.2.0.5 staging location:
$ cd $ORACLE_HOME/9205/Disk1
$ ./runInstaller
Respond to the installer prompts as shown below:
  • At the “Welcome Screen”, click Next.
  • View the “Cluster Node Selection Screen”, click Next.
  • At the “File Locations Screen”, make sure the source location is to the products.xml file in the 9.2.0.5 patchset location under Disk1/stage.  Also verify the destination listed is your ORACLE_HOME directory.  Change the $ORACLE_HOME name from the dropdown list to the 9.2 $ORACLE_HOME name.  Click Next.
  • At the “Available Products Screen”, Check “Oracle9iR2 PatchSets 9.2.0.5.0″. Click Next.
  • Click Install at the summary screen.
  • You will now get a progress window, run root.sh when prompted.
  • You will then see the end of installation screen. Click Exit and confirm by clicking Yes.
3.3 Start the GSD (Global Service Daemon)
These steps need to be performed on ALL nodes.
Start the GSD on each node with:
% gsdctl start
Successfully started GSD on local node
Then check the status with:
% gsdctl stat
GSD is running on the local node
If the GSD does not stay up, try running ‘srvconfig -init -f’ from the OS prompt.  If you get a raw device exception error or PRKR-1064 error then see the following note to troubleshoot:
Note 212631.1 – Resolving PRKR-1064 in a RAC Environment
Note: After confirming that GSD starts, if you are on Redhat Advanced Server 3.0, restore gcc296: rm /usr/bin/gcc
mv /usr/bin/gcc3.2.3 /usr/bin/gcc
rm /usr/bin/g++
mv /usr/bin/g++3.2.3 /usr/bin/g++
3.4 Create a RAC Database using the Oracle Database Configuration Assistant
These steps only need to be performed on the node that you are installing from (typically Node 1).
The Oracle Database Configuration Assistant (DBCA) will create a database for you.  The DBCA creates your database using the optimal flexible architecture (OFA). This means the DBCA creates your database files, including the default server parameter file, using standard file naming and file placement practices. The primary phases of DBCA processing are:-
  • Verify that you correctly configured the shared disks for each tablespace (for non-cluster file system platforms)
  • Create the database
  • Configure the Oracle network services
  • Start the database instances and listeners
Oracle Corporation recommends that you use the DBCA to create your database. This is because the DBCA preconfigured databases optimize your environment to take advantage of Oracle9i features such as the server parameter file and automatic undo management. The DBCA also enables you to define arbitrary tablespaces as part of the database creation process. So even if you have datafile requirements that differ from those offered in one of the DBCA templates, use the DBCA. You can also execute user-specified scripts as part of the database creation process.
Note: Prior to running the DBCA it may be necessary to run the NETCA tool or to manually set up your network files. To run the NETCA tool execute the command netca from the $ORACLE_HOME/bin directory. This will configure the necessary listener names and protocol addresses, client naming methods, Net service names and Directory server usage.

If you are using OCFS or NFS, launch DBCA with the -datafileDestination option and point to the shared location where Oracle datafiles will be stored.  Example:
% cd $ORACLE_HOME/bin
% dbca -datafileDestination /ocfs/oradata
If you are using RAW, launch DBCA without the -datafileDestination option.  Example:
% cd $ORACLE_HOME/bin
% dbca
Respond to the DBCA prompts as shown below:
  • Choose Oracle Cluster Database option and select Next.
  • The Operations page is displayed. Choose the option Create a Database and click Next.
  • The Node Selection page appears. Select the nodes that you want to configure as part of the RAC database and click Next.
  • The Database Templates page is displayed. The templates other than New Database include datafiles. Choose New Database and then clickNext.  Note: The Show Details button provides information on the database template selected.
  • DBCA now displays the Database Identification page. Enter the Global Database Name and Oracle System Identifier (SID). The Global Database Name is typically of the form name.domain, for example mydb.us.oracle.com while the SID is used to uniquely identify an instance (DBCA should insert a suggested SID, equivalent to name1 where name was entered in the Database Name field). In the RAC case the SID specified will be used as a prefix for the instance number. For example, MYDB, would become MYDB1, MYDB2 for instance 1 and 2 respectively.
  • The Database Options page is displayed. Select the options you wish to configure and then choose Next. Note: If you did not choose New Database from the Database Template page, you will not see this screen.
  • Select the connection options desired from the Database Connection Options page. Click Next.
  • DBCA now displays the Initialization Parameters page. This page comprises a number of Tab fields. Modify the Memory settings if desired and then select the File Locations tab to update information on the Initialization Parameters filename and location. The option Create persistent initialization parameter file is selected by default. If you have a cluster file system, then enter a file system name, otherwise a raw device name for the location of the server parameter file (spfile) must be entered.  The button File Location Variables… displays variable information. The button All Initialization Parameters… displays the Initialization Parameters dialog box. This box presents values for all initialization parameters and indicates whether they are to be included in the spfile to be created through the check box, included (Y/N). Instance specific parameters have an instance value in the instance column. Complete entries in the All Initialization Parameters page and select Close. Note: There are a few exceptions to what can be altered via this screen. Ensure all entries in the Initialization Parameters page are complete and select Next.
  • DBCA now displays the Database Storage Window. This page allows you to enter file names for each tablespace in your database.
  • The Database Creation Options page is displayed. Ensure that the option Create Database is checked and click Finish.
  • The DBCA Summary window is displayed. Review this information and then click OK.  Once you click the OK button and the summary screen is closed, it may take a few moments for the DBCA progress bar to start.  DBCA then begins to create the database according to the values specified.
During the database creation process, you may see the following error: ORA-29807: specified operator does not exist
This is a known issue (bug 2925665).  You can click on the “Ignore” button to continue.   Once DBCA has completed database creation, remember to run the ‘prvtxml.plb’ script from $ORACLE_HOME/rdbms/admin independently, as the user SYS. It is also advised to run the ‘utlrp.sql’ script to ensure that there are no invalid objects in the database at this time.
A new database now exists. It can be accessed via Oracle SQL*PLUS or other applications designed to work with an Oracle RAC database.
Additional database configuration best practices can be found in the following note:
Note 240575.1 – RAC on Linux Best Practices



4.0 Administering Real Application Clusters Instances
Oracle Corporation recommends that you use SRVCTL to administer your Real Application Clusters database environment. SRVCTL manages configuration information that is used by several Oracle tools. For example, Oracle Enterprise Manager and the Intelligent Agent use the configuration information that SRVCTL generates to discover and monitor nodes in your cluster. Before using SRVCTL, ensure that your Global Services Daemon (GSD) is running after you configure your database. To use SRVCTL, you must have already created the configuration information for the database that you want to administer. You must have done this either by using the Oracle Database Configuration Assistant (DBCA), or by using the srvctl add command as described below.
To display the configuration details for, example, databases racdb1/2, on nodes racnode1/2 with instances racinst1/2 run:-
$ srvctl config
racdb1
racdb2
$ srvctl config -p racdb1 -n racnode1
racnode1 racinst1 /u01/app/oracle/product/9.2.0
$ srvctl status database -d racdb1
Instance racinst1 is running on node racnode1
Instance racinst2 is running on node racnode2
Examples of starting and stopping RAC follow:-
$ srvctl start database -d racdb2
$ srvctl stop database -d racdb2
$ srvctl stop instance -d racdb1 -i racinst2
$ srvctl start instance -d racdb1 -i racinst2
For further information on srvctl and gsdctl see the Oracle9i Real Application Clusters Administration manual
Posted by at
Labels:

No comments:

Post a Comment