Forgot your password?    
+ Reply to Thread
Results 1 to 1 of 1

Thread: Oracle 11gR2 RAC Node Addition.

  1. #1
    Oracle Administrator
    Join Date
    Dec 2011
    Posts
    92

    Oracle 11gR2 RAC Node Addition.

    Following are the IP Addresses which i am using
    for the node.

    Public IP Address: 192.168.1.170
    Private IP Address: 192.168.2.170
    Virtual IP Address: 192.168.1.250

    OS installed
    - RHEL 5.4

    Packages to install

    Desktop Environment
    Gnome

    Applications

    Editors
    Graphical internet
    Text-based Internet

    Development
    Develoment libraries
    Development Tools
    Legacy Software development

    Servers
    Server Configuration Tools

    Base system
    administration Tools
    base
    java
    legacy software support
    system tools
    X window system



    Rpms to be installed


    binutils-2.17.50.0.6
    compat-libstdc++33*
    elfutils-libelf*
    elfutils-libelf-devel*
    elfutils-libelf-devel-static*
    gcc-4*
    gcc-c++*
    glibc-2*
    glibc-common*
    glibc-devel*
    glibc-headers*
    kernel-headers-2*
    ksh*
    libaio*
    libaio-devel*
    libgcc*
    libgomp*
    libstdc++*
    libstdc++-devel*
    make*
    pdksh*
    sysstat*
    unixODBC*
    unixODBC-devel


    Make appropiate changes in the DNS Server.

    DNS Forward lookup entries
    [root@rac1 ~]# vi /var/named/chroot/var/named/forward.zone

    Code:
    $TTL    86400
    @               IN SOA  rac1.example.com. root.example.com. (
                                            42              ; serial (d. adams)
                                            3H              ; refresh
                                            15M             ; retry
                                            1W              ; expiry
                                            1D )            ; minimum
                    IN NS           rac1.example.com. 
    rac1            IN A            192.168.1.100
    
    
    ; Oracle RAC Nodes
    rac1                    IN A        192.168.1.100
    rac2                    IN A        192.168.1.200
    rac3                    IN A        192.168.1.170
    
    rac1-priv               IN A        192.168.2.100
    rac2-priv               IN A        192.168.2.200
    rac3-priv               IN A        192.168.2.170
    
    rac1-vip                IN A        192.168.1.251
    rac2-vip                IN A        192.168.1.252
    rac3-vip                IN A        192.168.1.250
    
    ; Single Client Access Name (SCAN) virtual IP
    rac-cluster-scan    IN A        192.168.1.150
    rac-cluster-scan    IN A        192.168.1.151
    rac-cluster-scan    IN A        192.168.1.152


    DNS Reverse lookup entries

    [root@rac1 ~]# vi /var/named/chroot/var/named/reverse.zone
    Code:
    $TTL    86400
    @       IN      SOA     rac1.example.com. root.rac1.example.com.  (
                                          1997022700 ; Serial
                                          28800      ; Refresh
                                          14400      ; Retry
                                          3600000    ; Expire
                                          86400 )    ; Minimum
            IN      NS      rac1.example.com.
    100     IN      PTR     rac1.example.com.
    
    ; Oracle RAC Nodes
    100                     IN PTR      rac1.example.com.
    200                     IN PTR      rac2.example.com.
    170                     IN PTR      rac3.example.com.
    
    251                     IN PTR      rac1-vip.example.com.
    252                     IN PTR      rac2-vip.example.com.
    250                     IN PTR      rac3-vip.example.com.
    
    
    ; Single Client Access Name (SCAN) virtual IP
    150                     IN PTR      rac-cluster-scan.example.com.
    151                     IN PTR      rac-cluster-scan.example.com.
    152                     IN PTR      rac-cluster-scan.example.com.

    After making forward and reverse lookup entries
    restart the DNS server.


    [root@rac1 ~]# service named restart
    Code:
    Stopping named: .                                          [  OK  ]
    Starting named:                                            [  OK  ]

    Insert into rac3 for dns lookup.

    [root@rac3 ~]# vi /etc/resolv.conf

    search example.com
    nameserver 192.168.1.100

    New /etc/hosts file on all nodes


    Code:
    # Public Network - (eth0)
    192.168.1.100    rac1.example.com           rac1
    192.168.1.200    rac2.example.com           rac2
    192.168.1.170    rac3.example.com           rac3
    
    # Private Interconnect - (eth1)
    192.168.2.100    rac1-priv.example.com      rac1-priv
    192.168.2.200    rac2-priv.example.com      rac2-priv
    192.168.2.170    rac3-priv.example.com      rac3-priv
    
    # Public Virtual IP (VIP) addresses - (eth0:1)
    192.168.1.251    rac1-vip.example.com       rac1-vip
    192.168.1.252    rac2-vip.example.com       rac2-vip
    192.168.1.250    rac3-vip.example.com       rac3-vip

    Deconfigure NTP services on RAC3.

    Code:
    /sbin/service ntpd stop
    chkconfig ntpd off
    mv /etc/ntp.conf /etc/ntp.conf.original
    rm /var/run/ntpd.pid

    Create necessary groups and users for Grid on RAC3.



    [root@rac3 ~]#groupadd -g 1000 oinstall
    [root@rac3 ~]#groupadd -g 1200 asmadmin
    [root@rac3 ~]#groupadd -g 1201 asmdba
    [root@rac3 ~]#groupadd -g 1202 asmoper
    [root@rac3 ~]#useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,
    asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid

    [root@rac3 ~]# id grid
    uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba), 1202(asmoper)


    Assign password to grid user on RAC3.


    [root@rac3 ~]#passwd grid


    Set Bash Profile for grid user on RAC3.



    [root@rac3 ~]#su - grid

    Replace the content in .bash_profile with the following of grid user on both nodes.



    Code:
    # ---------------------------------------------------
    # .bash_profile
    # ---------------------------------------------------
    # OS User:      grid
    # Application:  Oracle Grid Infrastructure
    # Version:      Oracle 11g Release 2
    # ---------------------------------------------------
    
    # Get the aliases and functions
    if [ -f ~/.bashrc ]; then
          . ~/.bashrc
    fi
    
    alias ls="ls -FA"
    
    # ---------------------------------------------------
    # ORACLE_SID
    # ---------------------------------------------------
    # Specifies the Oracle system identifier (SID)
    # for the Automatic Storage Management (ASM)instance
    # running on this node.
    # Each RAC node must have a unique ORACLE_SID.
    # (i.e. +ASM1, +ASM2,...)
    # ---------------------------------------------------
    ORACLE_SID=+ASM3; export ORACLE_SID
    
    # ---------------------------------------------------
    # JAVA_HOME
    # ---------------------------------------------------
    # Specifies the directory of the Java SDK and Runtime
    # Environment.
    # ---------------------------------------------------
    JAVA_HOME=/usr/local/java; export JAVA_HOME
    
    # ---------------------------------------------------
    # GRID_BASE
    # ---------------------------------------------------
    # Specifies the base of the Oracle directory structure
    # for Optimal Flexible Architecture (OFA) compliant
    # installations. The Oracle base directory for the
    # grid installation owner is the location where
    # diagnostic and administrative logs, and other logs
    # associated with Oracle ASM and Oracle Clusterware
    # are stored.
    # ---------------------------------------------------
    GRID_BASE=/u01/app/grid; export GRID_BASE
    
    ORACLE_BASE=$GRID_BASE; export ORACLE_BASE
    
    # ---------------------------------------------------
    # GRID_HOME
    # ---------------------------------------------------
    # Specifies the directory containing the Oracle
    # Grid Infrastructure software. For grid
    # infrastructure for a cluster installations, the Grid
    # home must not be placed under one of the Oracle base
    # directories, or under Oracle home directories of
    # Oracle Database installation owners, or in the home
    # directory of an installation owner. During 
    # installation, ownership of the path to the Grid 
    # home is changed to root. This change causes 
    # permission errors for other installations.
    # ---------------------------------------------------
    GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
    
    ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
    
    # ---------------------------------------------------
    # ORACLE_PATH
    # ---------------------------------------------------
    # Specifies the search path for files used by Oracle
    # applications such as SQL*Plus. If the full path to
    # the file is not specified, or if the file is not
    # in the current directory, the Oracle application
    # uses ORACLE_PATH to locate the file.
    # This variable is used by SQL*Plus, Forms and Menu.
    # ---------------------------------------------------
    ORACLE_PATH=/u01/app/oracle/dba_scripts/sql; export ORACLE_PATH
    
    # ---------------------------------------------------
    # SQLPATH
    # ---------------------------------------------------
    # Specifies the directory or list of directories that
    # SQL*Plus searches for a login.sql file.
    # ---------------------------------------------------
    # SQLPATH=/u01/app/oracle/dba_scripts/sql; export SQLPATH
    
    # ---------------------------------------------------
    # ORACLE_TERM
    # ---------------------------------------------------
    # Defines a terminal definition. If not set, it
    # defaults to the value of your TERM environment
    # variable. Used by all character mode products. 
    # ---------------------------------------------------
    ORACLE_TERM=xterm; export ORACLE_TERM
    
    # ---------------------------------------------------
    # NLS_DATE_FORMAT
    # ---------------------------------------------------
    # Specifies the default date format to use with the
    # TO_CHAR and TO_DATE functions. The default value of
    # this parameter is determined by NLS_TERRITORY. The
    # value of this parameter can be any valid date
    # format mask, and the value must be surrounded by 
    # double quotation marks. For example:
    #
    #         NLS_DATE_FORMAT = "MM/DD/YYYY"
    #
    # ---------------------------------------------------
    NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
    
    # ---------------------------------------------------
    # TNS_ADMIN
    # ---------------------------------------------------
    # Specifies the directory containing the Oracle Net
    # Services configuration files like listener.ora, 
    # tnsnames.ora, and sqlnet.ora.
    # ---------------------------------------------------
    TNS_ADMIN=$GRID_HOME/network/admin; export TNS_ADMIN
    
    # ---------------------------------------------------
    # ORA_NLS11
    # ---------------------------------------------------
    # Specifies the directory where the language,
    # territory, character set, and linguistic definition
    # files are stored.
    # ---------------------------------------------------
    ORA_NLS11=$GRID_HOME/nls/data; export ORA_NLS11
    
    # ---------------------------------------------------
    # PATH
    # ---------------------------------------------------
    # Used by the shell to locate executable programs;
    # must include the $GRID_HOME/bin directory.
    # ---------------------------------------------------
    PATH=.:${JAVA_HOME}/bin:$JAVA_HOME/db/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
    PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
    PATH=${PATH}:/u01/app/oracle/dba_scripts/bin
    export PATH
    
    # ---------------------------------------------------
    # LD_LIBRARY_PATH
    # ---------------------------------------------------
    # Specifies the list of directories that the shared
    # library loader searches to locate shared object
    # libraries at runtime.
    # ---------------------------------------------------
    LD_LIBRARY_PATH=$GRID_HOME/lib
    LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$GRID_HOME/oracm/lib
    LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
    export LD_LIBRARY_PATH
    
    # ---------------------------------------------------
    # CLASSPATH
    # ---------------------------------------------------
    # Specifies the directory or list of directories that
    # contain compiled Java classes.
    # ---------------------------------------------------
    CLASSPATH=$GRID_HOME/JRE
    CLASSPATH=${CLASSPATH}:$GRID_HOME/jdbc/lib/ojdbc6.jar
    CLASSPATH=${CLASSPATH}:$GRID_HOME/jlib
    CLASSPATH=${CLASSPATH}:$GRID_HOME/rdbms/jlib
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/oc4j/ant/lib/ant.jar
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/oc4j/ant/lib/ant-launcher.jar
    CLASSPATH=${CLASSPATH}:$JAVA_HOME/db/lib/derby.jar
    CLASSPATH=${CLASSPATH}:$GRID_HOME/network/jlib
    export CLASSPATH
    
    # ---------------------------------------------------
    # THREADS_FLAG
    # ---------------------------------------------------
    # All the tools in the JDK use green threads as a
    # default. To specify that native threads should be
    # used, set the THREADS_FLAG environment variable to
    # "native". You can revert to the use of green
    # threads by setting THREADS_FLAG to the value
    # "green".
    # ---------------------------------------------------
    THREADS_FLAG=native; export THREADS_FLAG
    
    # ---------------------------------------------------
    # TEMP, TMP, and TMPDIR
    # ---------------------------------------------------
    # Specify the default directories for temporary
    # files; if set, tools that create temporary files
    # create them in one of these directories.
    # ---------------------------------------------------
    export TEMP=/tmp
    export TMPDIR=/tmp
    
    # ---------------------------------------------------
    # UMASK
    # ---------------------------------------------------
    # Set the default file mode creation mask
    # (umask) to 022 to ensure that the user performing
    # the Oracle software installation creates files
    # with 644 permissions.
    # ---------------------------------------------------
    umask 022


    Make sure of setting for RAC3 ORACLE_SID=+ASM3; export ORACLE_SID



    Activate the bash_profile on RAC3 using.


    [grid@rac3 ~]$ . ./.bash_profile


    Create necessary users and groups for Oracle software




    [root@rac3 ~]#groupadd -g 1300 dba
    [root@rac3 ~]#groupadd -g 1301 oper
    [root@rac3 ~]#useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d
    /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle

    [root@rac3 ~]# id oracle
    uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301( oper)



    Create password for oracle user


    [root@rac3 ~]#passwd oracle


    Set Bash Profile for oracle user on RAC3.


    [root@rac3 ~]#su - oracle


    Replace the content in .bash_profile with the following



    Code:
    # ---------------------------------------------------
    # .bash_profile
    # ---------------------------------------------------
    # OS User:      oracle
    # Application:  Oracle Database Software Owner
    # Version:      Oracle 11g Release 2
    # ---------------------------------------------------
    
    # Get the aliases and functions
    if [ -f ~/.bashrc ]; then
          . ~/.bashrc
    fi
    
    alias ls="ls -FA"
    
    # ---------------------------------------------------
    # ORACLE_SID
    # ---------------------------------------------------
    # Specifies the Oracle system identifier (SID) for
    # the Oracle instance running on this node.
    # Each RAC node must have a unique ORACLE_SID.
    # (i.e. orcl1, orcl2,...)
    # ---------------------------------------------------
    ORACLE_SID=orcl3; export ORACLE_SID
    
    # ---------------------------------------------------
    # ORACLE_UNQNAME
    # ---------------------------------------------------
    # In previous releases of Oracle Database, you were 
    # required to set environment variables for
    # ORACLE_HOME and ORACLE_SID to start, stop, and
    # check the status of Enterprise Manager. With
    # Oracle Database 11g Release 2 (11.2) and later, you
    # need to set the environment variables ORACLE_HOME 
    # and ORACLE_UNQNAME to use Enterprise Manager. 
    # Set ORACLE_UNQNAME equal to the database unique
    # name.
    # ---------------------------------------------------
    ORACLE_UNQNAME=orcl; export ORACLE_UNQNAME
    
    # ---------------------------------------------------
    # JAVA_HOME
    # ---------------------------------------------------
    # Specifies the directory of the Java SDK and Runtime
    # Environment.
    # ---------------------------------------------------
    JAVA_HOME=/usr/local/java; export JAVA_HOME
    
    # ---------------------------------------------------
    # ORACLE_BASE
    # ---------------------------------------------------
    # Specifies the base of the Oracle directory structure
    # for Optimal Flexible Architecture (OFA) compliant
    # database software installations.
    # ---------------------------------------------------
    ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
    
    # ---------------------------------------------------
    # ORACLE_HOME
    # ---------------------------------------------------
    # Specifies the directory containing the Oracle
    # Database software.
    # ---------------------------------------------------
    ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME
    
    # ---------------------------------------------------
    # ORACLE_PATH
    # ---------------------------------------------------
    # Specifies the search path for files used by Oracle
    # applications such as SQL*Plus. If the full path to
    # the file is not specified, or if the file is not
    # in the current directory, the Oracle application
    # uses ORACLE_PATH to locate the file.
    # This variable is used by SQL*Plus, Forms and Menu.
    # ---------------------------------------------------
    ORACLE_PATH=/u01/app/oracle/dba_scripts/sql:$ORACLE_HOME/rdbms/admin; export ORACLE_PATH
    
    # ---------------------------------------------------
    # SQLPATH
    # ---------------------------------------------------
    # Specifies the directory or list of directories that
    # SQL*Plus searches for a login.sql file.
    # ---------------------------------------------------
    # SQLPATH=/u01/app/oracle/dba_scripts/sql; export SQLPATH
    
    # ---------------------------------------------------
    # ORACLE_TERM
    # ---------------------------------------------------
    # Defines a terminal definition. If not set, it
    # defaults to the value of your TERM environment
    # variable. Used by all character mode products. 
    # ---------------------------------------------------
    ORACLE_TERM=xterm; export ORACLE_TERM
    
    # ---------------------------------------------------
    # NLS_DATE_FORMAT
    # ---------------------------------------------------
    # Specifies the default date format to use with the
    # TO_CHAR and TO_DATE functions. The default value of
    # this parameter is determined by NLS_TERRITORY. The
    # value of this parameter can be any valid date
    # format mask, and the value must be surrounded by 
    # double quotation marks. For example:
    #
    #         NLS_DATE_FORMAT = "MM/DD/YYYY"
    #
    # ---------------------------------------------------
    NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
    
    # ---------------------------------------------------
    # TNS_ADMIN
    # ---------------------------------------------------
    # Specifies the directory containing the Oracle Net
    # Services configuration files like listener.ora, 
    # tnsnames.ora, and sqlnet.ora.
    # ---------------------------------------------------
    TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
    
    # ---------------------------------------------------
    # ORA_NLS11
    # ---------------------------------------------------
    # Specifies the directory where the language,
    # territory, character set, and linguistic definition
    # files are stored.
    # ---------------------------------------------------
    ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
    
    # ---------------------------------------------------
    # PATH
    # ---------------------------------------------------
    # Used by the shell to locate executable programs;
    # must include the $ORACLE_HOME/bin directory.
    # ---------------------------------------------------
    PATH=.:${JAVA_HOME}/bin:$JAVA_HOME/db/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
    PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
    PATH=${PATH}:/u01/app/oracle/dba_scripts/bin
    export PATH
    
    # ---------------------------------------------------
    # LD_LIBRARY_PATH
    # ---------------------------------------------------
    # Specifies the list of directories that the shared
    # library loader searches to locate shared object
    # libraries at runtime.
    # ---------------------------------------------------
    LD_LIBRARY_PATH=$ORACLE_HOME/lib
    LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
    LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
    export LD_LIBRARY_PATH
    
    # ---------------------------------------------------
    # CLASSPATH
    # ---------------------------------------------------
    # Specifies the directory or list of directories that
    # contain compiled Java classes.
    # ---------------------------------------------------
    CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6.jar
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/oc4j/ant/lib/ant.jar
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/oc4j/ant/lib/ant-launcher.jar
    CLASSPATH=${CLASSPATH}:$JAVA_HOME/db/lib/derby.jar
    CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
    export CLASSPATH
    
    # ---------------------------------------------------
    # THREADS_FLAG
    # ---------------------------------------------------
    # All the tools in the JDK use green threads as a
    # default. To specify that native threads should be
    # used, set the THREADS_FLAG environment variable to
    # "native". You can revert to the use of green
    # threads by setting THREADS_FLAG to the value
    # "green".
    # ---------------------------------------------------
    THREADS_FLAG=native; export THREADS_FLAG
    
    # ---------------------------------------------------
    # TEMP, TMP, and TMPDIR
    # ---------------------------------------------------
    # Specify the default directories for temporary
    # files; if set, tools that create temporary files
    # create them in one of these directories.
    # ---------------------------------------------------
    export TEMP=/tmp
    export TMPDIR=/tmp
    
    # ---------------------------------------------------
    # UMASK
    # ---------------------------------------------------
    # Set the default file mode creation mask
    # (umask) to 022 to ensure that the user performing
    # the Oracle software installation creates files
    # with 644 permissions.
    # ---------------------------------------------------
    umask 022


    Make sure setting for rac3 should be ORACLE_SID=orcl3; export ORACLE_SID


    Activate bash profile on RAC3


    [oracle@rac3 ~]$ . ./.bash_profile


    verify that user nobody exists on RAC3.


    [root@rac3 ~]# id nobody
    uid=99(nobody) gid=99(nobody) groups=99(nobody)




    Create directory structures on RAC3.



    [root@rac3 ~]# mkdir -p /u01/app/grid
    [root@rac3 ~]# mkdir -p /u01/app/11.2.0/grid
    [root@rac3 ~]# chown -R grid:oinstall /u01
    [root@rac3 ~]# mkdir -p /u01/app/oracle
    [root@rac3 ~]# chown oracle:oinstall /u01/app/oracle
    [root@rac3 ~]# chmod -R 775 /u01


    Insert the following in /etc/security/limits.conf on RAC3.


    grid soft nproc 2047
    grid hard nproc 16384
    grid soft nofile 1024
    grid hard nofile 65536
    oracle soft nproc 2047
    oracle hard nproc 16384
    oracle soft nofile 1024
    oracle hard nofile 65536

    Insert the following in /etc/pam.d/login on RAC3.

    session required pam_limits.so


    Append the following in root user /etc/profile on RAC3.



    Code:
    if [ \$USER = "oracle" ] || [ \$USER = "grid" ]; then 
        if [ \$SHELL = "/bin/ksh" ]; then
            ulimit -p 16384
            ulimit -n 65536
        else
            ulimit -u 16384 -n 65536
        fi
        umask 022
    fi

    Edit kernel parameters on RAC3 /etc/sysctl.conf.


    leave the fields for

    kernel.shmall
    kernel.shmmax

    as it is and append the following values in /etc/sysctl.conf.

    Code:
    # Controls the maximum number of shared memory segments system wide
    kernel.shmmni = 4096
    
    # Sets the following semaphore values:
    # SEMMSL_value  SEMMNS_value  SEMOPM_value  SEMMNI_value
    kernel.sem = 250 32000 100 128
    
    # Sets the maximum number of file-handles that the Linux kernel will allocate
    fs.file-max = 6815744
    
    # Defines the local port range that is used by TCP and UDP
    # traffic to choose the local port
    net.ipv4.ip_local_port_range = 9000 65500
    
    # Default setting in bytes of the socket "receive" buffer which
    # may be set by using the SO_RCVBUF socket option
    net.core.rmem_default=262144
    
    # Maximum setting in bytes of the socket "receive" buffer which
    # may be set by using the SO_RCVBUF socket option
    net.core.rmem_max=4194304
    
    # Default setting in bytes of the socket "send" buffer which
    # may be set by using the SO_SNDBUF socket option
    net.core.wmem_default=262144
    
    # Maximum setting in bytes of the socket "send" buffer which 
    # may be set by using the SO_SNDBUF socket option
    net.core.wmem_max=1048576
    
    # Maximum number of allowable concurrent asynchronous I/O requests
    fs.aio-max-nr=1048576




    Activate kernel settings on RAC3.

    [root@rac3 ~]# sysctl -p

    Install asm on RAC3.


    [root@rac3 asm11g]# rpm -Uvh oracleasm-2.6.18-164.el5-2.0.5-1.el5.i686.rpm \
    > oracleasmlib-2.0.4-1.el5.i386.rpm \
    > oracleasm-support-2.1.7-1.el5.i386.rpm


    Output

    Code:
    warning: oracleasm-2.6.18-164.el5-2.0.5-1.el5.i686.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
    Preparing...                ########################################### [100%]
       1:oracleasm-support      ########################################### [ 33%]
       2:oracleasm-2.6.18-164.el########################################### [ 67%]
       3:oracleasmlib           ########################################### [100%]


    Check on RAC3

    [root@rac3 ~]# rpm -qa | grep asm
    Code:
    oracleasm-2.6.18-164.el5-2.0.5-1.el5
    ibmasm-3.0-9
    oracleasm-support-2.1.7-1.el5
    nasm-0.98.39-3.2.2
    ibmasm-xinput-2.1-1.el5
    oracleasmlib-2.0.4-1.el5

    Configure asm on RAC3.

    [root@rac3 ~]# /usr/sbin/oracleasm configure -i

    Code:
    Configuring the Oracle ASM library driver.
    
    This will configure the on-boot properties of the Oracle ASM library
    driver.  The following questions will determine whether the driver is
    loaded on boot and what permissions it will have.  The current values
    will be shown in brackets ('[]').  Hitting  without typing an
    answer will keep that current value.  Ctrl-C will abort.
    
    Default user to own the driver interface []: grid
    Default group to own the driver interface []: asmadmin
    Start Oracle ASM library driver on boot (y/n) [n]: y
    Scan for Oracle ASM disks on boot (y/n) [y]: y
    Writing Oracle ASM library driver configuration: done

    Execute the following on RAC3.

    [root@rac3 ~]# /usr/sbin/oracleasm init

    Code:
    Creating /dev/oracleasm mount point: /dev/oracleasm
    Loading module "oracleasm": oracleasm
    Mounting ASMlib driver filesystem: /dev/oracleasm
    Execute the following from RAC3.

    [root@rac3 ~]# /usr/sbin/oracleasm scandisks
    Code:
    Reloading disk partitions: done
    Cleaning any stale ASM disks...
    Scanning system for ASM disks...
    Instantiating disk "VOL1"
    Instantiating disk "VOL2"
    Instantiating disk "VOL3"
    Instantiating disk "VOL4"
    Instantiating disk "VOL5"
    Instantiating disk "VOL6"
    Instantiating disk "VOL7"


    Execute the following from RAC3.

    [root@rac3 ~]# /usr/sbin/oracleasm listdisks
    Code:
    VOL1
    VOL2
    VOL3
    VOL4
    VOL5
    VOL6
    VOL7


    Enable user equivalence for RAC3 as Grid user from RAC1.


    [root@rac1 ~]# su - grid
    [grid@rac1 ~]$ /u01/app/11.2.0/grid/oui/bin/runSSHSetup.sh -user grid -hosts "rac2 rac3" -advanced -exverify
    Code:
    This script will setup SSH Equivalence from the host 'rac1.example.com' to specified remote hosts. 
    
    ORACLE_HOME = /u01/app/11.2.0/grid
    JAR_LOC = /u01/app/11.2.0/grid/oui/jlib
    SSH_LOC = /u01/app/11.2.0/grid/oui/jlib
    OUI_LOC = /u01/app/11.2.0/grid/oui
    PROP_LOC = /u01/app/11.2.0/grid/sysman/prov/resources
    JAVA_HOME = /u01/app/11.2.0/grid/jdk
    
    Checking if the remote hosts are reachable.
    formulateLocalCmd rac2/bin/ping-c 1 -w 5
    formulateLocalCmd rac3/bin/ping-c 1 -w 5
    /bin/ping#rac2#-c#1#-w#5
    /bin/ping#rac3#-c#1#-w#5
    Remote host reachability check succeeded.
    All hosts are reachable. Proceeding further...
    
    
    NOTE :
    As part of the setup procedure, this script will use ssh and scp to copy
    files between the local host and the remote hosts. You may be prompted for
    the password during the execution of the script.
    AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
    AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
    directories.
    
    Do you want to continue and let the script make the above mentioned changes (yes/no)?
    yes
    
    If The files containing the client public and private keys already exist on the local host.
    The current private key may or may not have a passphrase associated with it.
    In case you remember the passphrase and do not want to re-run ssh-keygen, type 'no'. If you type 'yes',
    the script will remove the old private/public key files and, any previous SSH user setups would be reset.
    Enter 'yes', 'no'  
    no
    
    Enter the password:****     
     Logfile Location : /tmp/SSHSetup2012-12-02_02-09-40-PM
    Doing SSHSetup...
    Please be patient, this operation might take sometime...Dont press Ctrl+C...
    
    Local Platform:- Linux 
    
    ------------------------------------------------------------------------
    Verifying SSH setup
    ===================
    The script will now run the date command on the remote nodes using ssh
    to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
    THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
    PASSWORDS. If you see any output other than date or are prompted for the
    password, ssh is not setup correctly and you will need to resolve the
    issue and set up ssh again.
    The possible causes for failure could be:
    1. The server settings in /etc/ssh/sshd_config file do not allow ssh
    for user grid.
    2. The server may have disabled public key based authentication.
    3. The client public key on the server may be outdated.
    4. ~grid or ~grid/.ssh on the remote host may not be owned by grid.
    5. User may not have passed -shared option for shared remote users or
    may be passing the -shared option for non-shared remote users.
    6. If there is output in addition to the date, but no password is asked,
    it may be a security alert shown as part of company policy. Append the
    additional text to the /sysman/prov/resources/ignoreMessages.txt file.
    ------------------------------------------------------------------------
    --rac2:--
    Running /usr/bin/ssh -x -l grid rac2 date to verify SSH connectivity has been setup from local host to rac2.
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being 
    prompted for a password is ERROR.
    Sun Dec  2 14:11:05 IST 2012
    ------------------------------------------------------------------------
    --rac3:--
    Running /usr/bin/ssh -x -l grid rac3 date to verify SSH connectivity has been setup from local host to rac3.
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being 
    prompted for a password is ERROR.
    Sun Dec  2 14:11:06 IST 2012
    ------------------------------------------------------------------------
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac2 to rac2
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:11:07 IST 2012
    ------------------------------------------------------------------------
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac2 to rac3
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:11:08 IST 2012
    ------------------------------------------------------------------------
    -Verification from rac2 complete-
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac3 to rac2
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:11:08 IST 2012
    ------------------------------------------------------------------------
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac3 to rac3
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:11:10 IST 2012
    ------------------------------------------------------------------------
    -Verification from rac3 complete-
    SSH verification complete.

    Enable user equivalence for RAC3 as Oracle user from RAC1.



    [root@rac1 ~]# su - oracle


    [oracle@rac1 ~]$ /u01/app/oracle/product/11.2.0/dbhome_1/oui/bin/runSSHSetup.sh -user oracle -hosts "rac2 rac3" -advanced -exverify

    Code:
    This script will setup SSH Equivalence from the host 'rac1.example.com' to specified remote hosts. 
    
    ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1
    JAR_LOC = /u01/app/oracle/product/11.2.0/dbhome_1/oui/jlib
    SSH_LOC = /u01/app/oracle/product/11.2.0/dbhome_1/oui/jlib
    OUI_LOC = /u01/app/oracle/product/11.2.0/dbhome_1/oui
    PROP_LOC = /u01/app/oracle/product/11.2.0/dbhome_1/sysman/prov/resources
    JAVA_HOME = /u01/app/oracle/product/11.2.0/dbhome_1/jdk
    
    Checking if the remote hosts are reachable.
    formulateLocalCmd rac2/bin/ping-c 1 -w 5
    /bin/ping#rac2#-c#1#-w#5
    formulateLocalCmd rac3/bin/ping-c 1 -w 5
    /bin/ping#rac3#-c#1#-w#5
    Remote host reachability check succeeded.
    All hosts are reachable. Proceeding further...
    
    
    NOTE :
    As part of the setup procedure, this script will use ssh and scp to copy
    files between the local host and the remote hosts. You may be prompted for
    the password during the execution of the script.
    AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
    AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
    directories.
    
    Do you want to continue and let the script make the above mentioned changes (yes/no)?
    yes
    
    If The files containing the client public and private keys already exist on the local host. 
    The current private key may or may not have a passphrase associated with it. 
    In case you remember the passphrase and do not want to re-run ssh-keygen, type 'no'. If you type 'yes', 
    the script will remove the old private/public key files and, any previous SSH user setups would be reset.
    Enter 'yes', 'no'  
    no
    
    Enter the password:****      
     Logfile Location : /tmp/SSHSetup2012-12-02_02-16-53-PM
    Doing SSHSetup...
    Please be patient, this operation might take sometime...Dont press Ctrl+C...
    
    Local Platform:- Linux 
    
    ------------------------------------------------------------------------
    Verifying SSH setup
    ===================
    The script will now run the date command on the remote nodes using ssh
    to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
    THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
    PASSWORDS. If you see any output other than date or are prompted for the
    password, ssh is not setup correctly and you will need to resolve the
    issue and set up ssh again.
    The possible causes for failure could be:
    1. The server settings in /etc/ssh/sshd_config file do not allow ssh
    for user oracle.
    2. The server may have disabled public key based authentication.
    3. The client public key on the server may be outdated.
    4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.
    5. User may not have passed -shared option for shared remote users or
    may be passing the -shared option for non-shared remote users.
    6. If there is output in addition to the date, but no password is asked,
    it may be a security alert shown as part of company policy. Append the
    additional text to the /sysman/prov/resources/ignoreMessages.txt file.
    ------------------------------------------------------------------------
    --rac2:--
    Running /usr/bin/ssh -x -l oracle rac2 date to verify SSH connectivity has been setup from local host to rac2.
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but 
    being prompted for a password is ERROR.
    Sun Dec  2 14:18:29 IST 2012
    ------------------------------------------------------------------------
    --rac3:--
    Running /usr/bin/ssh -x -l oracle rac3 date to verify SSH connectivity has been setup from local host to rac3.
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being 
    prompted for a password is ERROR.
    Sun Dec  2 14:18:31 IST 2012
    ------------------------------------------------------------------------
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac2 to rac2
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:18:32 IST 2012
    ------------------------------------------------------------------------
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac2 to rac3
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:18:34 IST 2012
    ------------------------------------------------------------------------
    -Verification from rac2 complete-
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac3 to rac2
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:18:35 IST 2012
    ------------------------------------------------------------------------
    ------------------------------------------------------------------------
    Verifying SSH connectivity has been setup from rac3 to rac3
    ------------------------------------------------------------------------
    IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, 
    IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
    Sun Dec  2 14:18:37 IST 2012
    ------------------------------------------------------------------------
    -Verification from rac3 complete-
    SSH verification complete.

    Install cvuqdisk rpm on rac3.

    [root@rac1 ~]# scp /home/grid/software/oracle/grid/rpm/cvuqdisk-1.0.7-1.rpm rac3:/tmp
    root@rac3's password:
    cvuqdisk-1.0.7-1.rpm 100% 7831 7.7KB/s 00:00


    [root@rac3 tmp]# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP
    [root@rac3 tmp]# rpm -iv cvuqdisk-1.0.7-1.rpm
    Preparing packages for installation...
    cvuqdisk-1.0.7-1


    [root@rac3 tmp]# ls -l /usr/sbin/cvuqdisk
    -rwsr-xr-x 1 root oinstall 8272 May 28 2009 /usr/sbin/cvuqdisk




    Verify readiness of the new node from hardware and operating system point of view.


    [root@rac1 ~]# su - grid
    [grid@rac1 ~]$ /u01/app/11.2.0/grid/bin/cluvfy stage -post hwos -n rac3

    Code:
    Performing post-checks for hardware and operating system setup 
    
    Checking node reachability...
    Node reachability check passed from node "rac1"
    
    
    Checking user equivalence...
    User equivalence check passed for user "grid"
    
    Checking node connectivity...
    
    Checking hosts config file...
    
    Verification of the hosts config file successful
    
    Node connectivity passed for subnet "192.168.1.0" with node(s) rac3
    TCP connectivity check passed for subnet "192.168.1.0"
    
    Node connectivity passed for subnet "192.168.2.0" with node(s) rac3
    TCP connectivity check passed for subnet "192.168.2.0"
    
    
    Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
    rac3 eth0:192.168.1.170
    
    Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
    rac3 eth1:192.168.2.170
    
    Node connectivity check passed
    
    Check for multiple users with UID value 0 passed 
    
    Post-check for hardware and operating system setup was successful.



    Do prechecks as grid user for node addition.


    [root@rac1 ~]# su - grid
    [grid@rac1 ~]$ /u01/app/11.2.0/grid/bin/cluvfy stage -pre nodeadd -n rac3 -fixup -verbose

    Code:
    Performing pre-checks for node addition 
    
    Checking node reachability...
    
    Check: Node reachability from node "rac1"
      Destination Node                      Reachable?              
      ------------------------------------  ------------------------
      rac3                                  yes                     
    Result: Node reachability check passed from node "rac1"
    
    
    Checking user equivalence...
    
    Check: User equivalence for user "grid"
      Node Name                             Comment                 
      ------------------------------------  ------------------------
      rac3                                  passed                  
    Result: User equivalence check passed for user "grid"
    
    Checking node connectivity...
    
    Checking hosts config file...
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac2          passed                                            
      rac1          passed                                            
      rac3          passed                                            
    
    Verification of the hosts config file successful
    
    
    Interface information for node "rac1"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.100   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.150   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.251   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.152   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth1   192.168.2.100   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:9A 1500  
    
    
    Interface information for node "rac2"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.200   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.151   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.252   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth1   192.168.2.200   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:C5 1500  
    
    
    Interface information for node "rac3"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.170   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth1   192.168.2.170   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:66 1500  
    
    
    Check: Node connectivity for interface "eth0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
    Result: Node connectivity passed for interface "eth0"
    
    Result: Node connectivity check passed
    
    
    Checking CRS integrity...
    The Oracle clusterware is healthy on node "rac1"
    The Oracle clusterware is healthy on node "rac2"
    
    CRS integrity check passed
    
    Checking shared resources...
    
    Checking CRS home location...
    The location "/u01/app/11.2.0/grid" is not shared but is present/creatable on all nodes
    
    
    Result: Shared resources check for node addition passed
    
    
    Checking node connectivity...
    
    Checking hosts config file...
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac2          passed                                            
      rac1          passed                                            
      rac3          passed                                            
    
    Verification of the hosts config file successful
    
    
    Interface information for node "rac1"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.100   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.150   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.251   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.152   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth1   192.168.2.100   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:9A 1500  
    
    
    Interface information for node "rac2"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.200   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.151   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.252   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth1   192.168.2.200   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:C5 1500  
    
    
    Interface information for node "rac3"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.170   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth1   192.168.2.170   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:66 1500  
    
    
    Check: Node connectivity of subnet "192.168.1.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac2:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
    Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rac1,rac2,rac3
    
    
    Check: TCP connectivity of subnet "192.168.1.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:192.168.1.100              rac1:192.168.1.150              passed          
      rac1:192.168.1.100              rac1:192.168.1.251              passed          
      rac1:192.168.1.100              rac1:192.168.1.152              passed          
      rac1:192.168.1.100              rac2:192.168.1.200              passed          
      rac1:192.168.1.100              rac2:192.168.1.151              passed          
      rac1:192.168.1.100              rac2:192.168.1.252              passed          
      rac1:192.168.1.100              rac3:192.168.1.170              passed          
    Result: TCP connectivity check passed for subnet "192.168.1.0"
    
    
    Check: Node connectivity of subnet "192.168.2.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:eth1                       rac2:eth1                       yes             
      rac1:eth1                       rac3:eth1                       yes             
      rac2:eth1                       rac3:eth1                       yes             
    Result: Node connectivity passed for subnet "192.168.2.0" with node(s) rac1,rac2,rac3
    
    
    Check: TCP connectivity of subnet "192.168.2.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:192.168.2.100              rac2:192.168.2.200              passed          
      rac1:192.168.2.100              rac3:192.168.2.170              passed          
    Result: TCP connectivity check passed for subnet "192.168.2.0"
    
    
    Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
    rac1 eth0:192.168.1.100 eth0:192.168.1.150 eth0:192.168.1.251 eth0:192.168.1.152
    rac2 eth0:192.168.1.200 eth0:192.168.1.151 eth0:192.168.1.252
    rac3 eth0:192.168.1.170
    
    Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
    rac1 eth1:192.168.2.100
    rac2 eth1:192.168.2.200
    rac3 eth1:192.168.2.170
    
    Result: Node connectivity check passed
    
    
    Check: Total memory 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          1.98GB (2075500.0KB)      1.5GB (1572864.0KB)       passed    
    Result: Total memory check passed
    
    Check: Available memory 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          1.92GB (2008364.0KB)      50MB (51200.0KB)          passed    
    Result: Available memory check passed
    
    Check: Swap space 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          4GB (4192956.0KB)         2.97GB (3113250.0KB)      passed    
    Result: Swap space check passed
    
    Check: Free disk space for "rac3:/tmp" 
      Path              Node Name     Mount point   Available     Required      Comment     
      ----------------  ------------  ------------  ------------  ------------  ------------
      /tmp              rac3          /             17.45GB       1GB           passed      
    Result: Free disk space check passed for "rac3:/tmp"
    
    Check: User existence for "grid" 
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac3          exists                    passed                  
    Result: User existence check passed for "grid"
    
    Check: Run level 
      Node Name     run level                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          5                         3,5                       passed    
    Result: Run level check passed
    
    Check: Hard limits for "maximum open file descriptors" 
      Node Name         Type          Available     Required      Comment         
      ----------------  ------------  ------------  ------------  ----------------
      rac3              hard          65536         65536         passed          
    Result: Hard limits check passed for "maximum open file descriptors"
    
    Check: Soft limits for "maximum open file descriptors" 
      Node Name         Type          Available     Required      Comment         
      ----------------  ------------  ------------  ------------  ----------------
      rac3              soft          1024          1024          passed          
    Result: Soft limits check passed for "maximum open file descriptors"
    
    Check: Hard limits for "maximum user processes" 
      Node Name         Type          Available     Required      Comment         
      ----------------  ------------  ------------  ------------  ----------------
      rac3              hard          16384         16384         passed          
    Result: Hard limits check passed for "maximum user processes"
    
    Check: Soft limits for "maximum user processes" 
      Node Name         Type          Available     Required      Comment         
      ----------------  ------------  ------------  ------------  ----------------
      rac3              soft          2047          2047          passed          
    Result: Soft limits check passed for "maximum user processes"
    
    Check: System architecture 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          i686                      x86                       passed    
    Result: System architecture check passed
    
    Check: Kernel version 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          2.6.18-164.el5            2.6.18                    passed    
    Result: Kernel version check passed
    
    Check: Kernel parameter for "semmsl" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          250                       250                       passed    
    Result: Kernel parameter check passed for "semmsl"
    
    Check: Kernel parameter for "semmns" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          32000                     32000                     passed    
    Result: Kernel parameter check passed for "semmns"
    
    Check: Kernel parameter for "semopm" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          100                       100                       passed    
    Result: Kernel parameter check passed for "semopm"
    
    Check: Kernel parameter for "semmni" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          128                       128                       passed    
    Result: Kernel parameter check passed for "semmni"
    
    Check: Kernel parameter for "shmmax" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          4294967295                536870912                 passed    
    Result: Kernel parameter check passed for "shmmax"
    
    Check: Kernel parameter for "shmmni" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          4096                      4096                      passed    
    Result: Kernel parameter check passed for "shmmni"
    
    Check: Kernel parameter for "shmall" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          268435456                 2097152                   passed    
    Result: Kernel parameter check passed for "shmall"
    
    Check: Kernel parameter for "file-max" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          6815744                   6815744                   passed    
    Result: Kernel parameter check passed for "file-max"
    
    Check: Kernel parameter for "ip_local_port_range" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          between 9000 & 65500      between 9000 & 65500      passed    
    Result: Kernel parameter check passed for "ip_local_port_range"
    
    Check: Kernel parameter for "rmem_default" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          262144                    262144                    passed    
    Result: Kernel parameter check passed for "rmem_default"
    
    Check: Kernel parameter for "rmem_max" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          4194304                   4194304                   passed    
    Result: Kernel parameter check passed for "rmem_max"
    
    Check: Kernel parameter for "wmem_default" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          262144                    262144                    passed    
    Result: Kernel parameter check passed for "wmem_default"
    
    Check: Kernel parameter for "wmem_max" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          1048576                   1048576                   passed    
    Result: Kernel parameter check passed for "wmem_max"
    
    Check: Kernel parameter for "aio-max-nr" 
      Node Name     Configured                Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          1048576                   1048576                   passed    
    Result: Kernel parameter check passed for "aio-max-nr"
    
    Check: Package existence for "make-3.81" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          make-3.81-3.el5           make-3.81                 passed    
    Result: Package existence check passed for "make-3.81"
    
    Check: Package existence for "binutils-2.17.50.0.6" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          binutils-2.17.50.0.6-12.el5  binutils-2.17.50.0.6      passed    
    Result: Package existence check passed for "binutils-2.17.50.0.6"
    
    Check: Package existence for "gcc-4.1.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          gcc-4.1.2-46.el5          gcc-4.1.2                 passed    
    Result: Package existence check passed for "gcc-4.1.2"
    
    Check: Package existence for "gcc-c++-4.1.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          gcc-c++-4.1.2-46.el5      gcc-c++-4.1.2             passed    
    Result: Package existence check passed for "gcc-c++-4.1.2"
    
    Check: Package existence for "libgomp-4.1.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          libgomp-4.4.0-6.el5       libgomp-4.1.2             passed    
    Result: Package existence check passed for "libgomp-4.1.2"
    
    Check: Package existence for "libaio-0.3.106" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          libaio-0.3.106-3.2        libaio-0.3.106            passed    
    Result: Package existence check passed for "libaio-0.3.106"
    
    Check: Package existence for "glibc-2.5-24" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          glibc-2.5-42              glibc-2.5-24              passed    
    Result: Package existence check passed for "glibc-2.5-24"
    
    Check: Package existence for "compat-libstdc++-33-3.2.3" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          compat-libstdc++-33-3.2.3-61  compat-libstdc++-33-3.2.3  passed    
    Result: Package existence check passed for "compat-libstdc++-33-3.2.3"
    
    Check: Package existence for "elfutils-libelf-0.125" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          elfutils-libelf-0.137-3.el5  elfutils-libelf-0.125     passed    
    Result: Package existence check passed for "elfutils-libelf-0.125"
    
    Check: Package existence for "elfutils-libelf-devel-0.125" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125  passed    
    Result: Package existence check passed for "elfutils-libelf-devel-0.125"
    
    Check: Package existence for "glibc-common-2.5" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          glibc-common-2.5-42       glibc-common-2.5          passed    
    Result: Package existence check passed for "glibc-common-2.5"
    
    Check: Package existence for "glibc-devel-2.5" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          glibc-devel-2.5-42        glibc-devel-2.5           passed    
    Result: Package existence check passed for "glibc-devel-2.5"
    
    Check: Package existence for "glibc-headers-2.5" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          glibc-headers-2.5-42      glibc-headers-2.5         passed    
    Result: Package existence check passed for "glibc-headers-2.5"
    
    Check: Package existence for "libaio-devel-0.3.106" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          libaio-devel-0.3.106-3.2  libaio-devel-0.3.106      passed    
    Result: Package existence check passed for "libaio-devel-0.3.106"
    
    Check: Package existence for "libgcc-4.1.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          libgcc-4.1.2-46.el5       libgcc-4.1.2              passed    
    Result: Package existence check passed for "libgcc-4.1.2"
    
    Check: Package existence for "libstdc++-4.1.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          libstdc++-4.1.2-46.el5    libstdc++-4.1.2           passed    
    Result: Package existence check passed for "libstdc++-4.1.2"
    
    Check: Package existence for "libstdc++-devel-4.1.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          libstdc++-devel-4.1.2-46.el5  libstdc++-devel-4.1.2     passed    
    Result: Package existence check passed for "libstdc++-devel-4.1.2"
    
    Check: Package existence for "sysstat-7.0.2" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          sysstat-7.0.2-3.el5       sysstat-7.0.2             passed    
    Result: Package existence check passed for "sysstat-7.0.2"
    
    Check: Package existence for "unixODBC-2.2.11" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          unixODBC-2.2.11-7.1       unixODBC-2.2.11           passed    
    Result: Package existence check passed for "unixODBC-2.2.11"
    
    Check: Package existence for "unixODBC-devel-2.2.11" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          unixODBC-devel-2.2.11-7.1  unixODBC-devel-2.2.11     passed    
    Result: Package existence check passed for "unixODBC-devel-2.2.11"
    
    Check: Package existence for "ksh-20060214" 
      Node Name     Available                 Required                  Comment   
      ------------  ------------------------  ------------------------  ----------
      rac3          ksh-20080202-14.el5       ksh-20060214              passed    
    Result: Package existence check passed for "ksh-20060214"
    
    Checking for multiple users with UID value 0
    Result: Check for multiple users with UID value 0 passed 
    
    Checking to make sure user "grid" is not in "root" group
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac3          does not exist            passed                  
    Result: User "grid" is not part of "root" group. Check passed
    
    Starting Clock synchronization checks using Network Time Protocol(NTP)...
    
    NTP Configuration file check started...
    Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time 
    Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
    
    Result: Clock synchronization check using Network Time Protocol(NTP) passed
    
    
    Pre-check for node addition was successful.

    Extend grid infrastructure to the new node from RAC1.


    [root@rac1 ~]# xhost +
    access control disabled, clients can connect from any host
    [root@rac1 ~]# su - grid
    [grid@rac1 ~]$ cd /u01/app/11.2.0/grid/oui/bin
    [grid@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac3-vip}"
    Code:
    Starting Oracle Universal Installer...
    
    Checking swap space: must be greater than 500 MB.   Actual 3933 MB    Passed
    Oracle Universal Installer, Version 11.2.0.1.0 Production
    Copyright (C) 1999, 2009, Oracle. All rights reserved.
    
    
    Performing tests to see whether nodes rac2,rac3 are available
    ............................................................... 100% Done.
    
    .
    -----------------------------------------------------------------------------
    Cluster Node Addition Summary
    Global Settings
       Source: /u01/app/11.2.0/grid
       New Nodes
    Space Requirements
       New Nodes
          rac3
             /: Required 3.45GB : Available 16.25GB
    Installed Products
       Product Names
          Oracle Grid Infrastructure 11.2.0.1.0 
          Sun JDK 1.5.0.17.0 
          Installer SDK Component 11.2.0.1.0 
          Oracle One-Off Patch Installer 11.2.0.0.2 
          Oracle Universal Installer 11.2.0.1.0 
          Oracle Configuration Manager Deconfiguration 10.3.1.0.0 
          Enterprise Manager Common Core Files 10.2.0.4.2 
          Oracle DBCA Deconfiguration 11.2.0.1.0 
          Oracle RAC Deconfiguration 11.2.0.1.0 
          Oracle Quality of Service Management (Server) 11.2.0.1.0 
          Installation Plugin Files 11.2.0.1.0 
          Universal Storage Manager Files 11.2.0.1.0 
          Oracle Text Required Support Files 11.2.0.1.0 
          Automatic Storage Management Assistant 11.2.0.1.0 
          Oracle Database 11g Multimedia Files 11.2.0.1.0 
          Oracle Multimedia Java Advanced Imaging 11.2.0.1.0 
          Oracle Globalization Support 11.2.0.1.0 
          Oracle Multimedia Locator RDBMS Files 11.2.0.1.0 
          Oracle Core Required Support Files 11.2.0.1.0 
          Bali Share 1.1.18.0.0 
          Oracle Database Deconfiguration 11.2.0.1.0 
          Oracle Quality of Service Management (Client) 11.2.0.1.0 
          Expat libraries 2.0.1.0.1 
          Oracle Containers for Java 11.2.0.1.0 
          Perl Modules 5.10.0.0.1 
          Secure Socket Layer 11.2.0.1.0 
          Oracle JDBC/OCI Instant Client 11.2.0.1.0 
          Oracle Multimedia Client Option 11.2.0.1.0 
          LDAP Required Support Files 11.2.0.1.0 
          Character Set Migration Utility 11.2.0.1.0 
          Perl Interpreter 5.10.0.0.1 
          PL/SQL Embedded Gateway 11.2.0.1.0 
          OLAP SQL Scripts 11.2.0.1.0 
          Database SQL Scripts 11.2.0.1.0 
          Oracle Extended Windowing Toolkit 3.4.47.0.0 
          SSL Required Support Files for InstantClient 11.2.0.1.0 
          SQL*Plus Files for Instant Client 11.2.0.1.0 
          Oracle Net Required Support Files 11.2.0.1.0 
          Oracle Database User Interface 2.2.13.0.0 
          RDBMS Required Support Files for Instant Client 11.2.0.1.0 
          Enterprise Manager Minimal Integration 11.2.0.1.0 
          XML Parser for Java 11.2.0.1.0 
          Oracle Security Developer Tools 11.2.0.1.0 
          Oracle Wallet Manager 11.2.0.1.0 
          Enterprise Manager plugin Common Files 11.2.0.1.0 
          Platform Required Support Files 11.2.0.1.0 
          Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 
          RDBMS Required Support Files 11.2.0.1.0 
          Oracle Ice Browser 5.2.3.6.0 
          Oracle Help For Java 4.2.9.0.0 
          Enterprise Manager Common Files 10.2.0.4.2 
          Deinstallation Tool 11.2.0.1.0 
          Oracle Java Client 11.2.0.1.0 
          Cluster Verification Utility Files 11.2.0.1.0 
          Oracle Notification Service (eONS) 11.2.0.1.0 
          Oracle LDAP administration 11.2.0.1.0 
          Cluster Verification Utility Common Files 11.2.0.1.0 
          Oracle Clusterware RDBMS Files 11.2.0.1.0 
          Oracle Locale Builder 11.2.0.1.0 
          Oracle Globalization Support 11.2.0.1.0 
          Buildtools Common Files 11.2.0.1.0 
          Oracle RAC Required Support Files-HAS 11.2.0.1.0 
          SQL*Plus Required Support Files 11.2.0.1.0 
          XDK Required Support Files 11.2.0.1.0 
          Agent Required Support Files 10.2.0.4.2 
          Parser Generator Required Support Files 11.2.0.1.0 
          Precompiler Required Support Files 11.2.0.1.0 
          Installation Common Files 11.2.0.1.0 
          Required Support Files 11.2.0.1.0 
          Oracle JDBC/THIN Interfaces 11.2.0.1.0 
          Oracle Multimedia Locator 11.2.0.1.0 
          Oracle Multimedia 11.2.0.1.0 
          HAS Common Files 11.2.0.1.0 
          Assistant Common Files 11.2.0.1.0 
          PL/SQL 11.2.0.1.0 
          HAS Files for DB 11.2.0.1.0 
          Oracle Recovery Manager 11.2.0.1.0 
          Oracle Database Utilities 11.2.0.1.0 
          Oracle Notification Service 11.2.0.0.0 
          SQL*Plus 11.2.0.1.0 
          Oracle Netca Client 11.2.0.1.0 
          Oracle Net 11.2.0.1.0 
          Oracle JVM 11.2.0.1.0 
          Oracle Internet Directory Client 11.2.0.1.0 
          Oracle Net Listener 11.2.0.1.0 
          Cluster Ready Services Files 11.2.0.1.0 
          Oracle Database 11g 11.2.0.1.0 
    -----------------------------------------------------------------------------
    
    
    Instantiating scripts for add node (Sunday, December 2, 2012 2:49:04 PM IST)
    .                                                                 1% Done.
    Instantiation of add node scripts complete
    
    Copying to remote nodes (Sunday, December 2, 2012 2:49:15 PM IST)
    ...............................................................................................96% Done.
    Home copied to new nodes
    
    Saving inventory on nodes (Sunday, December 2, 2012 3:12:50 PM IST)
    .                                                               100% Done.
    Save inventory complete
    WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been 
    registered as the central inventory of this system. 
    To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root 
    privileges on nodes 'rac3'.
    If you do not register the inventory, you may not be able to update or patch the products you installed.
    The following configuration scripts need to be executed as the "root" user in each cluster node.
    /u01/app/oraInventory/orainstRoot.sh #On nodes rac3
    /u01/app/11.2.0/grid/root.sh #On nodes rac3
    To execute the configuration scripts:
        1. Open a terminal window
        2. Log in as "root"
        3. Run the scripts in each cluster node
        
    The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
    Please check '/tmp/silentInstall.log' for more details.


    Run scripts as the root user on the new node.


    [root@rac3 ~]# /u01/app/oraInventory/orainstRoot.sh
    Code:
    Creating the Oracle inventory pointer file (/etc/oraInst.loc)
    Changing permissions of /u01/app/oraInventory.
    Adding read,write permissions for group.
    Removing read,write,execute permissions for world.
    
    Changing groupname of /u01/app/oraInventory to oinstall.
    The execution of the script is complete.
    [root@rac3 ~]# /u01/app/11.2.0/grid/root.sh
    Code:
    Running Oracle 11g root.sh script...
    
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/app/11.2.0/grid
    
    Enter the full pathname of the local bin directory: [/usr/local/bin]: 
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    
    
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2012-12-02 15:17:37: Parsing the host name
    2012-12-02 15:17:37: Checking for super user privileges
    2012-12-02 15:17:37: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE 
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, 
    number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac3'
    CRS-2676: Start of 'ora.mdnsd' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac3'
    CRS-2676: Start of 'ora.gipcd' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac3'
    CRS-2676: Start of 'ora.gpnpd' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac3'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac3'
    CRS-2676: Start of 'ora.diskmon' on 'rac3' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac3'
    CRS-2676: Start of 'ora.ctssd' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac3'
    CRS-2676: Start of 'ora.drivers.acfs' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac3'
    CRS-2676: Start of 'ora.asm' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'rac3'
    CRS-2676: Start of 'ora.crsd' on 'rac3' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'rac3'
    CRS-2676: Start of 'ora.evmd' on 'rac3' succeeded
    clscfg: EXISTING configuration version 5 detected.
    clscfg: version 5 is 11g Release 2.
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    
    rac3     2012/12/02 15:25:08     /u01/app/11.2.0/grid/cdata/rac3/backup_20121202_152508.olr
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded
    Updating inventory properties for clusterware
    Starting Oracle Universal Installer...
    
    Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    'UpdateNodeList' was successful.


    Do post node addition checks from rac1.

    [root@rac1 ~]# su - grid
    [grid@rac1 ~]$ /u01/app/11.2.0/grid/bin/cluvfy stage -post nodeadd -n rac3 -verbose

    Code:
    Performing post-checks for node addition 
    
    Checking node reachability...
    
    Check: Node reachability from node "rac1"
      Destination Node                      Reachable?              
      ------------------------------------  ------------------------
      rac3                                  yes                     
    Result: Node reachability check passed from node "rac1"
    
    
    Checking user equivalence...
    
    Check: User equivalence for user "grid"
      Node Name                             Comment                 
      ------------------------------------  ------------------------
      rac3                                  passed                  
    Result: User equivalence check passed for user "grid"
    
    Checking node connectivity...
    
    Checking hosts config file...
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac2          passed                                            
      rac1          passed                                            
      rac3          passed                                            
    
    Verification of the hosts config file successful
    
    
    Interface information for node "rac2"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.200   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.151   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.252   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth1   192.168.2.200   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:C5 1500  
    
    
    Interface information for node "rac1"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.100   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.150   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.251   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth1   192.168.2.100   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:9A 1500  
    
    
    Interface information for node "rac3"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.170   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth0   192.168.1.152   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth0   192.168.1.250   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth1   192.168.2.170   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:66 1500  
    
    
    Check: Node connectivity for interface "eth0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac3:eth0                       rac3:eth0                       yes             
      rac3:eth0                       rac3:eth0                       yes             
      rac3:eth0                       rac3:eth0                       yes             
    Result: Node connectivity passed for interface "eth0"
    
    Result: Node connectivity check passed
    
    
    Checking cluster integrity...
    
      Node Name                           
      ------------------------------------
      rac1                                
      rac2                                
      rac3                                
    
    Cluster integrity check passed
    
    
    Checking CRS integrity...
    The Oracle clusterware is healthy on node "rac2"
    The Oracle clusterware is healthy on node "rac1"
    The Oracle clusterware is healthy on node "rac3"
    
    CRS integrity check passed
    
    Checking shared resources...
    
    Checking CRS home location...
    The location "/u01/app/11.2.0/grid" is not shared but is present/creatable on all nodes
    
    
    Result: Shared resources check for node addition passed
    
    
    Checking node connectivity...
    
    Checking hosts config file...
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac2          passed                                            
      rac1          passed                                            
      rac3          passed                                            
    
    Verification of the hosts config file successful
    
    
    Interface information for node "rac2"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.200   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.151   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth0   192.168.1.252   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:BB 1500  
     eth1   192.168.2.200   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:08:B8:C5 1500  
    
    
    Interface information for node "rac1"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.100   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.150   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth0   192.168.1.251   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:90 1500  
     eth1   192.168.2.100   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:C7:15:9A 1500  
    
    
    Interface information for node "rac3"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     eth0   192.168.1.170   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth0   192.168.1.152   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth0   192.168.1.250   192.168.1.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:5C 1500  
     eth1   192.168.2.170   192.168.2.0     0.0.0.0         192.168.1.1     00:0C:29:27:02:66 1500  
    
    
    Check: Node connectivity of subnet "192.168.1.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac2:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac1:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac2:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac1:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac1:eth0                       rac3:eth0                       yes             
      rac3:eth0                       rac3:eth0                       yes             
      rac3:eth0                       rac3:eth0                       yes             
      rac3:eth0                       rac3:eth0                       yes             
    Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rac2,rac1,rac3
    
    
    Check: TCP connectivity of subnet "192.168.1.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:192.168.1.100              rac2:192.168.1.200              failed          
      rac1:192.168.1.100              rac2:192.168.1.151              passed          
      rac1:192.168.1.100              rac2:192.168.1.252              passed          
      rac1:192.168.1.100              rac1:192.168.1.150              passed          
      rac1:192.168.1.100              rac1:192.168.1.251              passed          
      rac1:192.168.1.100              rac3:192.168.1.170              passed          
      rac1:192.168.1.100              rac3:192.168.1.152              passed          
      rac1:192.168.1.100              rac3:192.168.1.250              passed          
    Result: TCP connectivity check failed for subnet "192.168.1.0"
    
    
    Check: Node connectivity of subnet "192.168.2.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac2:eth1                       rac1:eth1                       yes             
      rac2:eth1                       rac3:eth1                       yes             
      rac1:eth1                       rac3:eth1                       yes             
    Result: Node connectivity passed for subnet "192.168.2.0" with node(s) rac2,rac1,rac3
    
    
    Check: TCP connectivity of subnet "192.168.2.0"
      Source                          Destination                     Connected?      
      ------------------------------  ------------------------------  ----------------
      rac1:192.168.2.100              rac2:192.168.2.200              passed          
      rac1:192.168.2.100              rac3:192.168.2.170              passed          
    Result: TCP connectivity check passed for subnet "192.168.2.0"
    
    
    Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
    rac2 eth0:192.168.1.200 eth0:192.168.1.151 eth0:192.168.1.252
    rac1 eth0:192.168.1.100 eth0:192.168.1.150 eth0:192.168.1.251
    rac3 eth0:192.168.1.170 eth0:192.168.1.152 eth0:192.168.1.250
    
    Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
    rac2 eth1:192.168.2.200
    rac1 eth1:192.168.2.100
    rac3 eth1:192.168.2.170
    
    Result: Node connectivity check passed
    
    
    Checking node application existence...
    
    Checking existence of VIP node application 
      Node Name     Required                  Status                    Comment   
      ------------  ------------------------  ------------------------  ----------
      rac2          yes                       online                    passed    
      rac1          yes                       online                    passed    
      rac3          yes                       online                    passed    
    Result: Check passed. 
    
    Checking existence of ONS node application 
      Node Name     Required                  Status                    Comment   
      ------------  ------------------------  ------------------------  ----------
      rac2          no                        online                    passed    
      rac1          no                        online                    passed    
      rac3          no                        online                    passed    
    Result: Check passed. 
    
    Checking existence of GSD node application 
      Node Name     Required                  Status                    Comment   
      ------------  ------------------------  ------------------------  ----------
      rac2          no                        does not exist            ignored   
      rac1          no                        does not exist            ignored   
      rac3          no                        does not exist            ignored   
    Result: Check ignored. 
    
    Checking existence of EONS node application 
      Node Name     Required                  Status                    Comment   
      ------------  ------------------------  ------------------------  ----------
      rac2          no                        online                    passed    
      rac1          no                        online                    passed    
      rac3          no                        online                    passed    
    Result: Check passed. 
    
    Checking existence of NETWORK node application 
      Node Name     Required                  Status                    Comment   
      ------------  ------------------------  ------------------------  ----------
      rac2          no                        online                    passed    
      rac1          no                        online                    passed    
      rac3          no                        online                    passed    
    Result: Check passed. 
    
    
    Checking Single Client Access Name (SCAN)...
      SCAN VIP name     Node          Running?      ListenerName  Port          Running?    
      ----------------  ------------  ------------  ------------  ------------  ------------
      rac-cluster-scan  rac2          true          LISTENER      1521          true        
    
    Checking name resolution setup for "rac-cluster-scan"...
      SCAN Name     IP Address                Status                    Comment   
      ------------  ------------------------  ------------------------  ----------
      rac-cluster-scan  192.168.1.151             passed                              
      rac-cluster-scan  192.168.1.152             passed                              
      rac-cluster-scan  192.168.1.150             passed                              
    
    Verification of SCAN VIP and Listener setup passed
    
    Checking to make sure user "grid" is not in "root" group
      Node Name     Status                    Comment                 
      ------------  ------------------------  ------------------------
      rac3          does not exist            passed                  
    Result: User "grid" is not part of "root" group. Check passed
    
    Checking if Clusterware is installed on all nodes...
    Check of Clusterware install passed
    
    Checking if CTSS Resource is running on all nodes...
    Check: CTSS Resource running on all nodes
      Node Name                             Status                  
      ------------------------------------  ------------------------
      rac3                                  passed                  
    Result: CTSS resource check passed
    
    
    Querying CTSS for time offset on all nodes...
    Result: Query of CTSS for time offset passed
    
    Check CTSS state started...
    Check: CTSS state
      Node Name                             State                   
      ------------------------------------  ------------------------
      rac3                                  Active                  
    CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
    Reference Time Offset Limit: 1000.0 msecs
    Check: Reference Time Offset
      Node Name     Time Offset               Status                  
      ------------  ------------------------  ------------------------
      rac3          0.0                       passed                  
    
    Time offset is within the specified limits on the following set of nodes: 
    "[rac3]" 
    Result: Check of clock time offsets passed
    
    
    Oracle Cluster Time Synchronization Services check passed
    
    Post-check for node addition was successful.


    Do following checks on the new node.


    [grid@rac3 ~]$ crsctl check crs
    Code:
    CRS-4638: Oracle High Availability Services is online
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    [grid@rac3 ~]$ crs_stat -t -v
    Code:
    Name           Type           R/RA   F/FT   Target    State     Host        
    ----------------------------------------------------------------------
    ora.CRS.dg     ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
    ora.DATA.dg    ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
    ora.FRA.dg     ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
    ora....ER.lsnr ora....er.type 0/5    0/     ONLINE    ONLINE    rac1        
    ora....N1.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac2        
    ora....N2.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac3        
    ora....N3.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac1        
    ora.asm        ora.asm.type   0/5    0/     ONLINE    ONLINE    rac1        
    ora.dell.db    ora....se.type 0/2    0/1    ONLINE    ONLINE    rac1        
    ora.eons       ora.eons.type  0/3    0/     ONLINE    ONLINE    rac1        
    ora.gsd        ora.gsd.type   0/5    0/     OFFLINE   OFFLINE               
    ora....network ora....rk.type 0/5    0/     ONLINE    ONLINE    rac1        
    ora.oc4j       ora.oc4j.type  0/5    0/0    OFFLINE   OFFLINE               
    ora.ons        ora.ons.type   0/3    0/     ONLINE    ONLINE    rac1        
    ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    rac1        
    ora....C1.lsnr application    0/5    0/0    ONLINE    ONLINE    rac1        
    ora.rac1.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
    ora.rac1.ons   application    0/3    0/0    ONLINE    ONLINE    rac1        
    ora.rac1.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac1        
    ora....SM2.asm application    0/5    0/0    ONLINE    ONLINE    rac2        
    ora....C2.lsnr application    0/5    0/0    ONLINE    ONLINE    rac2        
    ora.rac2.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
    ora.rac2.ons   application    0/3    0/0    ONLINE    ONLINE    rac2        
    ora.rac2.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac2        
    ora....SM3.asm application    0/5    0/0    ONLINE    ONLINE    rac3        
    ora....C3.lsnr application    0/5    0/0    ONLINE    ONLINE    rac3        
    ora.rac3.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
    ora.rac3.ons   application    0/3    0/0    ONLINE    ONLINE    rac3        
    ora.rac3.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac3        
    ora....ry.acfs ora....fs.type 0/5    0/     ONLINE    ONLINE    rac1        
    ora.scan1.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac2        
    ora.scan2.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac3        
    ora.scan3.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac1

    [grid@rac3 ~]$ olsnodes -n
    Code:
    rac1    1
    rac2    2
    rac3    3
    [grid@rac3 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
    Code:
    LISTENER_SCAN2
    LISTENER
    [grid@rac3 ~]$ srvctl status asm -a
    Code:
    ASM is running on rac1,rac2,rac3
    ASM is enabled.
    [grid@rac3 ~]$ ocrcheck
    Code:
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       2948
             Available space (kbytes) :     259172
             ID                       :  576957443
             Device/File Name         :       +CRS
                                        Device/File integrity check succeeded
    
                                        Device/File not configured
    
                                        Device/File not configured
    
                                        Device/File not configured
    
                                        Device/File not configured
    
             Cluster registry integrity check succeeded
    
             Logical corruption check bypassed due to non-privileged user

    [grid@rac3 ~]$ crsctl query css votedisk
    Code:
    ##  STATE    File Universal Id                File Name Disk group
    --  -----    -----------------                --------- ---------
     1. ONLINE   57e4ff96d9b64fdabfc724b7b1e7d4f9 (ORCL:VOL1) [CRS]
     2. ONLINE   7869d5fd844c4f0bbf1a022259193dfc (ORCL:VOL2) [CRS]
     3. ONLINE   73f4a7ede9bf4f4dbfb86e1b24167a3a (ORCL:VOL7) [CRS]
    Located 3 voting disk(s).
    Extend oracle software to the new node from RAC1.


    [root@rac1 ~]# su - oracle
    [oracle@rac1 ~]$ cd /u01/app/oracle/product/11.2.0/dbhome_1/oui/bin/
    [oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac3}"
    Code:
    Starting Oracle Universal Installer...
    
    Checking swap space: must be greater than 500 MB.   Actual 3903 MB    Passed
    Oracle Universal Installer, Version 11.2.0.1.0 Production
    Copyright (C) 1999, 2009, Oracle. All rights reserved.
    
    
    Performing tests to see whether nodes rac2,rac3 are available
    ............................................................... 100% Done.
    
    ..
    -----------------------------------------------------------------------------
    Cluster Node Addition Summary
    Global Settings
       Source: /u01/app/oracle/product/11.2.0/dbhome_1
       New Nodes
    Space Requirements
       New Nodes
          rac3
             /: Required 3.89GB : Available 13.09GB
    Installed Products
       Product Names
          Oracle Database 11g 11.2.0.1.0 
          Sun JDK 1.5.0.17.0 
          Installer SDK Component 11.2.0.1.0 
          Oracle One-Off Patch Installer 11.2.0.0.2 
          Oracle Universal Installer 11.2.0.1.0 
          Oracle Configuration Manager Deconfiguration 10.3.1.0.0 
          Oracle DBCA Deconfiguration 11.2.0.1.0 
          Oracle RAC Deconfiguration 11.2.0.1.0 
          Oracle Database Deconfiguration 11.2.0.1.0 
          Oracle Configuration Manager 10.3.1.1.0 
          Oracle ODBC Driverfor Instant Client 11.2.0.1.0 
          LDAP Required Support Files 11.2.0.1.0 
          SSL Required Support Files for InstantClient 11.2.0.1.0 
          Bali Share 1.1.18.0.0 
          Oracle Extended Windowing Toolkit 3.4.47.0.0 
          Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 
          Oracle Real Application Testing 11.2.0.1.0 
          Oracle Database Vault J2EE Application 11.2.0.1.0 
          Oracle Label Security 11.2.0.1.0 
          Oracle Data Mining RDBMS Files 11.2.0.1.0 
          Oracle OLAP RDBMS Files 11.2.0.1.0 
          Oracle OLAP API 11.2.0.1.0 
          Platform Required Support Files 11.2.0.1.0 
          Oracle Database Vault option 11.2.0.1.0 
          Oracle RAC Required Support Files-HAS 11.2.0.1.0 
          SQL*Plus Required Support Files 11.2.0.1.0 
          Oracle Display Fonts 9.0.2.0.0 
          Oracle Ice Browser 5.2.3.6.0 
          Oracle JDBC Server Support Package 11.2.0.1.0 
          Oracle SQL Developer 11.2.0.1.0 
          Oracle Application Express 11.2.0.1.0 
          XDK Required Support Files 11.2.0.1.0 
          RDBMS Required Support Files for Instant Client 11.2.0.1.0 
          SQLJ Runtime 11.2.0.1.0 
          Database Workspace Manager 11.2.0.1.0 
          RDBMS Required Support Files 11.2.0.1.0 
          Oracle Globalization Support 11.2.0.1.0 
          Exadata Storage Server 11.2.0.1.0 
          Provisioning Advisor Framework 10.2.0.4.2 
          Enterprise Manager Database Plugin -- Repository Support 11.2.0.1.0 
          Enterprise Manager Repository Core Files 10.2.0.4.2 
          Enterprise Manager Database Plugin -- Agent Support 11.2.0.1.0 
          Enterprise Manager Grid Control Core Files 10.2.0.4.2 
          Enterprise Manager Common Core Files 10.2.0.4.2 
          Enterprise Manager Agent Core Files 10.2.0.4.2 
          Agent Required Support Files 10.2.0.4.2 
          regexp 2.1.9.0.0 
          Parser Generator Required Support Files 11.2.0.1.0 
          Oracle 11g Warehouse Builder Required Files 11.2.0.1.0 
          Oracle Notification Service (eONS) 11.2.0.1.0 
          Oracle Text Required Support Files 11.2.0.1.0 
          Precompiler Required Support Files 11.2.0.1.0 
          Oracle Database 11g Multimedia Files 11.2.0.1.0 
          Oracle Multimedia Java Advanced Imaging 11.2.0.1.0 
          Oracle Multimedia Annotator 11.2.0.1.0 
          Oracle JDBC/OCI Instant Client 11.2.0.1.0 
          Oracle Multimedia Locator RDBMS Files 11.2.0.1.0 
          Oracle Core Required Support Files 11.2.0.1.0 
          Oracle Help For Java 4.2.9.0.0 
          Sample Schema Data 11.2.0.1.0 
          Oracle Starter Database 11.2.0.1.0 
          Oracle Message Gateway Common Files 11.2.0.1.0 
          Oracle XML Query 11.2.0.1.0 
          XML Parser for Oracle JVM 11.2.0.1.0 
          Expat libraries 2.0.1.0.1 
          Installation Plugin Files 11.2.0.1.0 
          Enterprise Manager Common Files 10.2.0.4.2 
          Perl Modules 5.10.0.0.1 
          Deinstallation Tool 11.2.0.1.0 
          Oracle Quality of Service Management (Client) 11.2.0.1.0 
          Perl Interpreter 5.10.0.0.1 
          JAccelerator (COMPANION) 11.2.0.1.0 
          Oracle Containers for Java 11.2.0.1.0 
          Oracle Code Editor 1.2.1.0.0I 
          Oracle Net Required Support Files 11.2.0.1.0 
          Secure Socket Layer 11.2.0.1.0 
          Oracle Universal Connection Pool 11.2.0.1.0 
          Oracle JDBC/THIN Interfaces 11.2.0.1.0 
          Oracle Multimedia Client Option 11.2.0.1.0 
          Oracle Java Client 11.2.0.1.0 
          Character Set Migration Utility 11.2.0.1.0 
          Oracle Locale Builder 11.2.0.1.0 
          PL/SQL Embedded Gateway 11.2.0.1.0 
          OLAP SQL Scripts 11.2.0.1.0 
          Database SQL Scripts 11.2.0.1.0 
          Oracle Globalization Support 11.2.0.1.0 
          Required Support Files 11.2.0.1.0 
          SQL*Plus Files for Instant Client 11.2.0.1.0 
          Oracle ODBC Driver 11.2.0.1.0 
          Oracle Database User Interface 2.2.13.0.0 
          Oracle Notification Service 11.2.0.0.0 
          Enterprise Manager Minimal Integration 11.2.0.1.0 
          XML Parser for Java 11.2.0.1.0 
          Oracle Security Developer Tools 11.2.0.1.0 
          Oracle Wallet Manager 11.2.0.1.0 
          Cluster Verification Utility Common Files 11.2.0.1.0 
          Oracle Clusterware RDBMS Files 11.2.0.1.0 
          Oracle UIX 2.2.24.5.0 
          Enterprise Manager plugin Common Files 11.2.0.1.0 
          HAS Common Files 11.2.0.1.0 
          Precompiler Common Files 11.2.0.1.0 
          Installation Common Files 11.2.0.1.0 
          Oracle Help for the  Web 2.0.14.0.0 
          Oracle LDAP administration 11.2.0.1.0 
          Buildtools Common Files 11.2.0.1.0 
          Assistant Common Files 11.2.0.1.0 
          Oracle Recovery Manager 11.2.0.1.0 
          PL/SQL 11.2.0.1.0 
          Generic Connectivity Common Files 11.2.0.1.0 
          Oracle Database Gateway for ODBC 11.2.0.1.0 
          Oracle Programmer 11.2.0.1.0 
          Oracle Database Utilities 11.2.0.1.0 
          Enterprise Manager Agent 10.2.0.4.2 
          Oracle Netca Client 11.2.0.1.0 
          SQL*Plus 11.2.0.1.0 
          Oracle Call Interface (OCI) 11.2.0.1.0 
          Oracle Multimedia Locator 11.2.0.1.0 
          Oracle Multimedia 11.2.0.1.0 
          Oracle Net 11.2.0.1.0 
          Database Configuration and Upgrade Assistants 11.2.0.1.0 
          Oracle XML Development Kit 11.2.0.1.0 
          Oracle JVM 11.2.0.1.0 
          Oracle Advanced Security 11.2.0.1.0 
          Oracle Internet Directory Client 11.2.0.1.0 
          HAS Files for DB 11.2.0.1.0 
          Oracle Enterprise Manager Console DB 11.2.0.1.0 
          Oracle Net Listener 11.2.0.1.0 
          Oracle Text 11.2.0.1.0 
          Oracle Net Services 11.2.0.1.0 
          Oracle Database 11g 11.2.0.1.0 
          Oracle OLAP 11.2.0.1.0 
          Oracle Spatial 11.2.0.1.0 
          Oracle Partitioning 11.2.0.1.0 
          Enterprise Edition Options 11.2.0.1.0 
    -----------------------------------------------------------------------------
    
    
    Instantiating scripts for add node (Sunday, December 2, 2012 3:41:44 PM IST)
    .                                                                 1% Done.
    Instantiation of add node scripts complete
    
    Copying to remote nodes (Sunday, December 2, 2012 3:42:00 PM IST)
    ...............................................................................................96% Done.
    Home copied to new nodes
    
    Saving inventory on nodes (Sunday, December 2, 2012 3:59:32 PM IST)
    .                                                               100% Done.
    Save inventory complete
    WARNING:
    The following configuration scripts need to be executed as the "root" user in each cluster node.
    /u01/app/oracle/product/11.2.0/dbhome_1/root.sh #On nodes rac3
    To execute the configuration scripts:
        1. Open a terminal window
        2. Log in as "root"
        3. Run the scripts in each cluster node
        
    The Cluster Node Addition of /u01/app/oracle/product/11.2.0/dbhome_1 was successful.
    Please check '/tmp/silentInstall.log' for more details.

    Run the script as root on new node.


    [root@rac3 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh
    Code:
    Running Oracle 11g root.sh script...
    
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/oracle/product/11.2.0/dbhome_1
    
    Enter the full pathname of the local bin directory: [/usr/local/bin]: 
    The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
    [n]: n
    The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
    [n]: n
    The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
    [n]: n
    
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    Finished product-specific root actions.


    Since we are using job role separation (grid,oracle), The $ORACLE_HOME/bin/oracle binary
    may not have proper group priviledges on the new node.

    This may prevent the oracle software owner from accessing the ASM disks.

    To resolve this problem do the following on the new node.

    [root@rac3 ~]# su - grid

    [grid@rac3 ~]$ cd $GRID_HOME/bin

    [grid@rac3 bin]$ ls -l /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
    -r-sr-s--x 1 oracle oinstall 173515905 Dec 2 15:55 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle*

    [grid@rac3 bin]$ ./setasmgidwrap o=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle

    [grid@rac3 bin]$ ls -l /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
    -r-sr-s--x 1 oracle asmadmin 173515905 Dec 2 15:55 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle*






    Add new instance to the cluster using DBCA from rac1.



    [root@rac1 ~]# xhost +
    access control disabled, clients can connect from any host
    [root@rac1 ~]# su - oracle
    [oracle@rac1 ~]$ dbca























    Verify


    [oracle@rac1 ~]$ srvctl status database -d dell
    Code:
    Instance dell1 is running on node rac1
    Instance dell2 is running on node rac2
    Instance dell3 is running on node rac3

    copy tnsnames.ora file from rac1 to the new node.

    [root@rac1 ~]# su - oracle
    [oracle@rac1 ~]$ cd $ORACLE_HOME/network/admin
    [oracle@rac1 admin]$ scp tnsnames.ora rac3:$ORACLE_HOME/network/admin
    tnsnames.ora 100% 347 0.3KB/s 00:00
    Last edited by Hemant; 03-10-2013 at 09:04 AM.

+ Reply to Thread

Similar Threads

  1. Oracle 11gR2 RAC Node Deletion.
    By ajaychandi in forum RAC Installation, ASM Install , ASM Administration
    Replies: 0
    Last Post: 12-11-2012, 09:14 PM
  2. Find Master Node in Oracle RAC
    By Hemant in forum RAC Concepts and Administration
    Replies: 0
    Last Post: 11-15-2012, 06:12 PM
  3. Add Node to a Cluster
    By Hemant in forum RAC Installation, ASM Install , ASM Administration
    Replies: 0
    Last Post: 12-17-2011, 10:05 PM
  4. Add a node to an Oracle 11i System with Rapid Clone
    By Amit in forum Oracle Apps Patching and Cloning, Installation , Migration,Upgrades
    Replies: 0
    Last Post: 11-03-2011, 04:49 PM
  5. Cloning Oracle Applications from Multinode to Single Node - Merge APPL_TOP
    By Hemant in forum Oracle Apps Patching and Cloning, Installation , Migration,Upgrades
    Replies: 0
    Last Post: 11-01-2011, 01:13 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

DBA Lounge (P) Ltd. deals in Oracle Technologies on Consulting, Resourcing, Corporate Training


Online and corporate training available on Oracle Database 11g, Oracle 11g Real Application Cluster (RAC), Oracle Applications 11i/R12, Oracle Fusion Middleware 11g, Oracle Identity Management-OIM, Oracle Internet Directory 11g-OID, Oracle Business Intelligence Enterprise Edition-OBIEE, Oracle Golden Gate, Oracle Access Management-OAM, Oracle Internet Directory-ODS, Oracle Identity Analytics Architecture-OIA Statistics