Step 1: Account for Dependencies and Prerequisites
The new node should have the
- same version of the operating system as the existing nodes, including all patches required for Oracle.
- maintain the current naming convention.
- have the same required rpms installed
- have the kernel parameters set to the same values as the other nodes
- have the same oracle user and group
- have the GID of user oracle are identical to that of the other RAC nodes.
- have the same ulimits for the oracle user as the other nodes
1) Add the entries for the new host (public ip,private ip and vip) in the /etc/hosts for all the servers
2) Setup the user equivalence with SSH.
When adding nodes to the cluster, Oracle copies files from the node where the installation was originally performed to the new node in the cluster. Such a copy process is performed either by using ssh protocol where available or by using the remote copy ( rcp). In order for the copy operation to be successful, the oracle user on the RAC node must be able to login to the new RAC node without having to provide a password or passphrase.
ensure passwordless command execution
ssh rac1 hostname
ssh rac3 hostname
ssh rac1-priv hostname
ssh rac3-priv hostname
Note: When performing these tests for the first time, the operating system will display a key and request the user to accept or decline. Enter ‘Yes’ to accept and register the key. Tests should be performed on all other nodes across all interfaces in the cluster except for the VIP.
Step 2: Install Oracle Clusterware
Oracle Clusterware is already installed on the cluster; the task here is to add the new node to the clustered configuration. This task is performed by executing the Oracle provided utility called addnode located in the Clusterware’s home oui/bin directory. Oracle Clusterware has two files (Oracle cluster repository, OCR; and Oracle Cluster Synchronization service, CSS, voting disk) that contain information concerning the cluster and the applications managed by the Oracle Clusterware. These files need to be updated with the information concerning the new node. The first step in the clusterware installation process is to verify if the new node is ready for the install.
Cluster Verification. In Oracle Database 10g Release 2, Oracle introduced a new utility called Cluster Verification Utility (CVU)as part of the clusterware software. Executing the utility using the appropriate parameters determines the status of the cluster. At this stage, before beginning installation of the Oracle Clusterware, you should perform two verifications:
If the hardware and operating system configuration is complete:
a) cluvfy stage -post hwos -n rac1,rac3
b) cluvfy stage -pre crsinst -n rac1,rac3
Step 3: Configure Oracle Clusterware
Using the OUI requires that the terminal from where this installer is ran be X-windows compatible. If not, an appropriate X-windows emulator should be installed and the emulator invoked using the DISPLAY command using the following syntax.
export DISPLAY=<client IP address>:0.0
The next step is to configure the Clusterware on the new node rac3.
For this, as mentioned earlier, Oracle has provided a new executable called addNode.sh located in the <Clusterware Home>/oui/bin directory.
Execute the script <Clusterware Home>/oui/bin/addNode.sh.
Welcome - click on Next.
Specify Cluster Nodes to Add to Installation - In this screen, OUI lists existing nodes in the cluster and in the bottom half of the screen lists the new node(s) information to be added in the appropriate columns. Once the information is entered click on Next.
Public Node Name Private Node Name Virtual Host Name
rac3 rac3-priv rac3-vip
Cluster Node Addition Summary - Verify the new node is listed under the “New Nodes” drilldown and click on Install.
Once all required clusterware components are copied from rac1 to rac3, OUI prompts to execute three files:
/usr/app/oracle/oraInventory/orainstRoot.sh on node rac3
[root@rac3 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to dba.
The execution of the script is complete
Execute $ORA_CRS_HOME/install/rootaddnode.sh on node rac1.
(The addnoderoot.sh file will add the new node information to the OCR using the srvctl utility. Note the srvctl command with the nodeapps parameter at the end of the script output below.)
[root@rac1 install]# ./rootaddnode.sh
$ORA_CRS_HOME/root.sh on node rac3.
invoke VIPCA also as root. (VIPCA will also configure GSD and ONS resources on the new node.)
Welcome - click on Next.
On completion of the Oracle Clusterware installation, the following files are created in their respective directories.
[root@rac3 root]# ls -ltr /etc/init.d/init.*
The operating system provided inittab file is updated with the following entries.
[root@rac3 root]# tail -5 /etc/inittab
Click on OK after all the listed scripts have run on all nodes.
End of Installation – Click on Exit.
Verify if the Clusterware has all the nodes registered using the olsnodes command.
[oracle@rac1 oracle]$ olsnodes
Verify if the cluster services is started, using the crs_stat command.
[oracle@rac1 oracle]$ crs_stat -t
Name Type Target State Host
Verify if the VIP services are configured at the OS level. The virtual IP address is configured and added to the OS network configuration and the network services are started. The VIP configuration could be verified using the ifconfig command at the OS level.
[oracle@rac3 oracle]$ ifconfig -a
Note: eth0:1 indicates that it is a VIP address for the basic host eth0. When the node fails eth0:1 will be moved to a surviving node in the cluster. The new identifier for the VIP on the failed over server will be indicated by eth0:2 or higher, depending on what other nodes have failed in the cluster and the VIP has migrated.
Step 4: Install Oracle Software
The next step is to install the Oracle software on the new node. As mentioned earlier, Oracle has provided a new executable called addNode.sh located in the $ORACLE_HOME/oui/bin directory.
Execute the script $ORACLE_HOME/oui/in/addNode.sh.
Welcome - click on Next.
Specify Cluster Nodes to Add to Installation - In this screen OUI lists existing nodes in the cluster and in the bottom half of the screen lists the new node(s). Select the node rac3. Once the information is entered, click on Next.
Cluster Node Addition Summary - Verify if the new node is listed under the “New Nodes” drilldown and click on the Install button.
When copy of Oracle software to node rac3 is complete, OUI will prompt you to execute /u01/app/oracle/product/10.2.0/db_1/root.sh script from another window as the root user on the new node(s) in the cluster.
[root@rac3 db_1]# ./root.sh
Running Oracle10 root.sh script...
Step 5: Add New Instance(s)
DBCA has all the required options to add additional instances to the cluster.
Make a full cold backup of the database before commencing the upgrade process.
Oracle Clusterware should be running on all nodes.
Welcome screen - select Oracle Real Application Cluster database and click on Next.
Step 1 of 7: Operations - A list of all operations that can be performed using the DBCA are listed. Select Instance Management and click on Next.
Step 2 of 7: Instance Management - A list of instance management operations that can be performed are listed. Select Add an Instance and click on Next.
Step 3 of 7: List of cluster databases - A list of clustered databases running on the node are listed. In this case the database running on node rac1 is devdb; select this database. In the bottom part of the screen, DBCA requests you to “Specify a user with SYSDBA system privileges”:
Password: < > and click on Next.
Step 4 of 7: List of cluster database instances - The DBCA lists all the instances currently available on the cluster. Verify if all instances are listed and click on Next.
Step 5 of 7: Instance naming and node selection - DBCA lists the next instance name in the series and requests the node on which to add the instance. In our example the next instance name is devdb3 and the node name is rac3. Click on Next after making the appropriate selection. At this stage there is a small pause before the next screen appears as DBCA determines the current state of the new node and what services are configured on the existing nodes.
Step 6 of 7: Database Services - If the current configuration has any database services configured, this screen will appear (else is skipped). Make the appropriate selections and click on Next when ready.
Step 7 of 7: Instance Storage - In this screen, DBCA will list the instances specific files such as undo tablespaces, redo log groups, and so on. Verify if all required files are listed and click on Finish.
Database Configuration Assistant: Summary - After verifying the summary, click on OK to begin the software installation.
DBCA verifies the new node rac3, and as the database is configured to use ASM, prompts with the message “ASM is present on the cluster but needs to be extended to the following nodes: [rac3]. Do you want ASM to be extended?” Click on Yes to add ASM to the new instance.
In order to create and start the ASM instances on the new node, Oracle requires the Listener to be present and started. DBCA prompts with requesting permission to configure the listener using port 1521 and listener name LISTENER_rac3. Click on Yes if the default port is good, else click on No and manually execute NetCA on rac3 to create the listener using a different port.
Database Configuration Assistant progress screen - Once instance management is complete, the user is prompted with the message "Do you want to perform another operation?” Click on No to end.
At this stage, the following is true:
The clusterware has been installed on node rac3 and is now part of the cluster.
The Oracle software has been installed on node rac3.
The ASM5 and new Oracle instance devdb3 has been created and configured on rac3.
Verify that the upgrade is successful.
Verify that all instances in the cluster are started using the V$ACTIVE_INSTANCES view from any of the participating instances. For example:
SQL> select * from v$active_instances;
SQL> SELECT NAME,STATE,TYPE FROM V$ASM_DISKGROUP;
SQL> SELECT NAME FROM V$DATAFILE;
[oracle@rac1 oracle]$ srvctl status database -d devdb
Instance devdb3 is running on node rac3
[oracle@rac1 oracle]$ srvctl status service -d devdb
Step 6: Set the Environment and the TNS entries
For easy administration and navigation, you should define several different environment variables in the login profile.
[oracle@rac3 oracle]$ more .bash_profile
Also add the new network address to the clinet tnsnames.ora file to the appropriate connect descriptors.
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac3-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(SERVER = DEDICATED)
(SERVICE_NAME = ..)
If the servers are configured to use FAN features, add the new server address to the onsctl files on all database servers.
The ons.config file is located in
[oracle@oradb4 oracle]$ more $ORACLE_HOME/opmn/conf/ons.config