安装完毕后，两个虚拟机的VIP又非常的不稳定，经常自动Offline或者failover到另外的一个节点。

[    RACG][1] [716836][1][ora.oradb31.vip]: clsrcexecut: rc = 1, time = 5.305s
[    RACG][1] [716836][1][ora.oradb31.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/app/crs/oracle
[    RACG][1] [716836][1][ora.oradb31.vip]: clsrcexecut: cmd = /u01/app/crs/oracle/bin/racgeut -e _USR_ORA_DEBUG=0 54 /u01/app/crs/oracle/bin/racgvip check oradb31
[    RACG][1] [716836][1][ora.oradb31.vip]: clsrcexecut: rc = 1, time = 1.202s
[    RACG][1] [716836][1][ora.oradb31.vip]: end for resource = ora.oradb31.vip, action = start, status = 1, time = 6.598s
[    RACG][1] [786632][1][ora.oradb31.vip]: Invalid parameters, or failed to bring up VIP (host=oradb31)

搜索metalink原来是因为Default Gateway [ID 399213.1]的原因。因为是虚拟机，所以default gateway(比方说192.168.2.1)无法ping通。导致Oracle认为节点的VIP有问题，自动failover VIP。通过修改$ORA_CRS_HOME/bin/racgvip中默认的DEFAULTGW为虚拟机的public IP解决。

奇怪是同样的IP设置，安装Oracle11gR1 Clusterware的时候就没有问题，或许检查VIP状态的机制有了改进。

BTW:在server上使用screen的时候出现了如下错误：Cannot open your terminal ‘/dev/pts/0′ - please check. 万能的google告诉我: 可以使用script命令来记录这个终端会话,执行script /dev/null 然后就可以打开screen了

NOTE:298895.1 - Modifying the default gateway address used by the Oracle 10g VIP

VIP Going Offline Intermittantly - Slow Response from Default Gateway [ID 399213.1]

OCR can be added & replaced while CRS is running.

ENV: Oracle Clusterware 10.2.0.4

Detailed logfile:

While CRS is still running…

# ./crsctl add css votedisk /dev/rdsk/ c4t60060E80056F160000006F1600000669d0s1

Cluster is not in a ready state for online disk addition

# ./crsctl add css votedisk /dev/rdsk/ c4t60060E80056F160000006F1600000669d0s1  -force

Now formatting voting disk: /dev/rdsk/ c4t60060E80056F160000006F1600000669d0s1

successful addition of votedisk /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1.

# ./crsctl query css votedisk

0.     0    /dev/rdsk/c4t60060E80056F160000006F1600000469d0s1

1.     0    /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1

located 2 votedisk(s).

Looks good, continue add another one.

# ./crsctl add css votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

Cluster is not in a ready state for online disk addition

# ./crsctl add css votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1 -force

Now formatting voting disk: /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

successful addition of votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1.

It’s corrupted. Entry 1 and entry 2 both point to “*869d0s1”.

# ./crsctl query css votedisk

0.     0    /dev/rdsk/c4t60060E80056F160000006F1600000469d0s1

1.     0    /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

2.     0    /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

located 3 votedisk(s).

Has to stop CRS on all nodes.

# ./crsctl stop crs

Stopping resources. This could take several minutes.

Successfully stopped CRS resources.

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

Then I want to delete corrupted entry 1 and entry 2.

# ./crsctl delete css votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

Cluster is not in a ready state for online disk removal

# ./crsctl delete css votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1 -force

no votedisk found matching path specified /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1.

It shows “0” for entry 1 and entry 2.

# ./crsctl query css votedisk

0.     0    /dev/rdsk/c4t60060E80056F160000006F1600000469d0s1

1.     0    0

2.     0    0

Add another 2 votedisk while CRS is down.

# ./crsctl add css votedisk /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1 -force

Now formatting voting disk: /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1

successful addition of votedisk /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1.

# ./crsctl add css votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1 -force

Now formatting voting disk: /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

successful addition of votedisk /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1.

# ./crsctl query css votedisk

0.     0    /dev/rdsk/c4t60060E80056F160000006F1600000469d0s1

1.     0    0

2.     0    0

3.     0    /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1

4.     0    /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

located 5 votedisk(s).

Delete “0” votedisk. Looks good.

# ./crsctl delete css votedisk 0 -force

successful deletion of votedisk 0.

# ./crsctl query css votedisk

0.     0    /dev/rdsk/c4t60060E80056F160000006F1600000469d0s1

1.     0    0

2.     0    /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1

3.     0    /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

located 4 votedisk(s).

# ./crsctl delete css votedisk 0 -force

successful deletion of votedisk 0.

# ./crsctl query css votedisk

0.     0    /dev/rdsk/c4t60060E80056F160000006F1600000469d0s1

1.     0    /dev/rdsk/c4t60060E80056F160000006F1600000669d0s1

2.     0    /dev/rdsk/c4t60060E80056F160000006F1600000869d0s1

located 3 votedisk(s).

Want to add mirrored OCR while CRS is down. It reports error “PROT-1”.

# ./ocrconfig -replace ocrmirror /dev/rdsk/c4t60060E80056F160000006F1600000669d0s5

PROT-1: Failed to initialize ocrconfig

After start CRS on all nodes. Mirrored OCR can be added.

# ./ocrconfig -replace ocrmirror /dev/rdsk/c4t60060E80056F160000006F1600000669d0s5

# ./ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :    1151724

Used space (kbytes)      :       4608

Available space (kbytes) :    1147116

ID                       : 1986342521

Device/File Name         : /dev/rdsk/c4t60060E80056F160000006F1600000469d0s5

Device/File integrity check succeeded

Device/File Name         : /dev/rdsk/c4t60060E80056F160000006F1600000669d0s5

Device/File integrity check succeeded

Cluster registry integrity check succeeded

# cat /var/opt/oracle/ocr.loc

#Device/file  getting replaced by device /dev/rdsk/c4t60060E80056F160000006F1600000669d0s5

ocrconfig_loc=/dev/rdsk/c4t60060E80056F160000006F1600000469d0s5

ocrmirrorconfig_loc=/dev/rdsk/c4t60060E80056F160000006F1600000669d0s5

local_only=false#

[detached]

Bloom’s fuel cell works like this: Oxygen is pumped in on one side and natural gas on the other. The two combine inside the cell to create a chemical reaction that produces electricity. No burning, no combustion, no power lines from outside.

fuel cell的internal如此，不管结果怎么样，有VC投钱就有大把的工作岗位。

过了年就不去上海了。在北京做remote DBA，基本上work@home。离老家近了，再也不用发愁过年回家的问题了。

今天用expdp作用户级别的逻辑备份，该用户有2200多个表，导出文件共85G，没有使用并行大概花费了3个小时不到。

运行命令后的半个小时内，导出文件大小没有任何变化，察看active session，有个在长时间等待”wait for unread message on broadcast channel”，给人的感觉是hang住，不靠普。

后来准备开窗口使用传统的exp,继续观察expdp。”假寐”后,EXPDP先把最大的一个表给导出来了，后来又导出了几个10g的表。原来我没有碰见真死的bug。观察日志文件，expdp基本按照表从大到小进行数据导出。倒是比exp快，可惜用户体验不行。

详细命令和日志如下

yumianfeilong$> more backup_expdp.log
Export: Release 10.2.0.4.0 - 64bit Production on Wednesday, 03 February, 2010 19:54:02
Copyright (c) 2003, 2007, Oracle. All rights reserved.
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
FLASHBACK automatically enabled to preserve database integrity.
Starting “SCOTT”.”SYS_EXPORT_SCHEMA_01″: SCOTT/******** schemas=SCOTT directory=EXPDP dumpfile=backup_expdp.dmp logfile=backup_expdp.log
Estimate in progress using BLOCKS method…
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 122.4 GB
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/SEQUENCE/SEQUENCE
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/COMMENT
Processing object type SCHEMA_EXPORT/VIEW/VIEW
Processing object type SCHEMA_EXPORT/VIEW/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/TRIGGER
Processing object type SCHEMA_EXPORT/TABLE/INDEX/FUNCTIONAL_AND_BITMAP/INDEX
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
. . exported “SCOTT”.”TABLE725″ 11.39 GB 39455120 rows
. . exported “SCOTT”.”TABLE1085″ 10.87 GB 225445 rows
. . exported “SCOTT”.”adfadsfdsf” 11.07 GB 43002 rows
. . exported “SCOTT”.”adfasdf” 163.2 MB 640381 rows
. . exported “SCOTT”.”adfasdf” 6.170 GB 5758755 rows
. . exported “SCOTT”.”adfasdf” 4.362 GB 3667986 rows
. . exported “SCOTT”.”adfsdf” 1.325 GB 39455120 rows
. . exported “SCOTT”.”werwer” 1.303 GB 4759513 rows
. . exported “SCOTT”.”fadf” 102.9 MB 58882 rows
. . exported “SCOTT”.”zzzzzzzzzzzz” 21.27 MB 77610 rows
…………………………
. . exported “SCOTT”.”xxxxxxxxxxx” 381.5 MB 4443678 rows
. . exported “SCOTT”.”cccccccccc” 314.1 MB 50703 rows
. . exported “SCOTT”.”vvvvvvvv” 360.8 MB 19679 rows
. . exported “SCOTT”.”nnnnnnnnnn” 199.3 MB 79876 rows
…………………………
. . exported “SCOTT”.”mmmmmmmmmmmmm” 55.57 MB 4719 rows
. . exported “SCOTT”.”jjjjjjjjjjjjjj” 50.01 MB 27200 rows
. . exported “SCOTT”.”kkkkkkkkkkkkk” 36.18 MB 14629 rows
. . exported “SCOTT”.”llllllll” 49.89 MB 173582 rows
……………………
. . exported “SCOTT”.”ttttttttttttt” 3.667 MB 5810 rows
. . exported “SCOTT”.”tttttttttttttt” 3.843 MB 20287 rows
. . exported “SCOTT”.”ttttttttttttttt” 2.710 MB 79876 rows
. . exported “SCOTT”.”tttttttttttttttttt” 565.6 KB 16233 rows

. . exported “SCOTT”.”ahhhhhhhhhhhhhhh” 1.826 MB 38701 rows
. . exported “SCOTT”.”qqqqqqqqqqqqqqqq” 1.122 MB 65065 rows
. . exported “SCOTT”.”beeeeeeeeeee” 1.122 MB 65065 rows

. . exported “SCOTT”.”Tttttttt1119″ 2.600 MB 12544 rows
. . exported “SCOTT”.”T515″ 1.810 MB 15267 rows
. . exported “SCOTT”.”T518″ 2.571 MB 34564 rows
. . exported “SCOTT”.”Tttttttttt668″ 1.937 MB 1339 rows
. . exported “SCOTT”.”Tttttttttttttt994″ 984.5 KB 1502 rows
. . exported “SCOTT”.”Taaaaaaa541″ 1.455 MB 1153 rows
. . exported “SCOTT”.”Taaaaaa593″ 1.189 MB 768 rows
. . exported “SCOTT”.”aaaaaaaaaaaaaaa” 1.512 MB 3475 rows
…………………………………………
. . exported “SCOTT”.”ABC” 28.21 KB 177 rows
. . exported “SCOTT”.”jjjjjjjjjjjjj” 761.1 KB 43002 rows
. . exported “SCOTT”.”yyyyyyyyyy” 904.7 KB 3607 rows
. . exported “SCOTT”.”ffffffffffff” 91.74 KB 797 rows
. . exported “SCOTT”.”wwwwwwwww” 259.1 KB 1 rows
. . exported “SCOTT”.”rrrrrrrrrrrrrr” 7.687 KB 140 rows
. . exported “SCOTT”.”ddddddddddddde” 7.687 KB 140 rows
……………………………….
. . exported “SCOTT”.”T991″ 12.60 KB 3 rows
. . exported “SCOTT”.”T993″ 38.60 KB 429 rows
. . exported “SCOTT”.”TTTTTTTT” 0 KB 0 rows
. . exported “SCOTT”.”bbbbbbbbbb” 0 KB 0 rows
………………………………………
Master table “SCOTT”.”SYS_EXPORT_SCHEMA_01″ successfully loaded/unloaded
******************************************************************************
Dump file set for SCOTT.SYS_EXPORT_SCHEMA_01 is:
/oracle/TRACE/archive/EXPDP/backup_expdp.dmp
Job “SCOTT”.”SYS_EXPORT_SCHEMA_01″ completed with 0 error(s) at 22:46:43

单表expdp测试没有假死问题。

周末去朋友家做客后的感想

如果你觉的时间过得快，便是吃饱混天黑。相反觉得慢的，便是享受生活。

./crs_stat -p ora.rac1.LISTENER_RAC1.lsnr >> /tmp/ora.rac1.LISTENER_RAC1.lsnr.cap
./crs_stat -p ora.rac2.LISTENER_RAC2.lsnr >> /tmp/ora.rac2.LISTENER_RAC2.lsnr.cap
./crs_stat -p ora.rac1.ASM1.asm >> /tmp/ora.rac1.ASM1.asm.cap
./crs_stat -p ora.rac2.ASM2.asm >> /tmp/ora.rac2.ASM2.asm.cap
./crs_stat -p ora.orcl.orcl1.inst >> /tmp/ora.orcl.orcl1.inst.cap
./crs_stat -p ora.orcl.orcl2.inst >> /tmp/ora.orcl.orcl2.inst.cap

–modify REQUIRED_RESOURCES item for all tempfiles.(多个dependence 资源用空格隔开)

./crs_register -u ora.rac1.LISTENER_RAC1.lsnr -dir /tmp
./crs_register -u ora.rac2.LISTENER_RAC2.lsnr -dir /tmp
./crs_register -u ora.rac1.ASM1.asm -dir /tmp
./crs_register -u ora.rac2.ASM2.asm -dir /tmp
./crs_register -u ora.orcl.orcl1.inst -dir /tmp
./crs_register -u ora.orcl.orcl2.inst -dir /tmp

Removing a Node from a 10gR1 RAC Cluster [ID 269320.1]
Modified 27-MAY-2009     Type BULLETIN     Status PUBLISHED

Note:  This article is only relevant for 10gR1 RAC environments.

For 10gR2 RAC environments please follow the documented procedures in the manual:
Oracle? Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide
10g Release 2 (10.2)
Part Number B14197-03

PURPOSE
——-

The purpose of this note is to provide the user with a  document that
can be used as a guide to remove a cluster node from an Oracle 10g Real
Applications environment.

SCOPE & APPLICATION
——————-

This document can be used by DBAs and support analsyts who need to
either remove a cluster node or assist another in removing a cluster
node in a 10g Unix Real Applications environment.

REMOVING A NODE FROM A 10g RAC CLUSTER
————————————–

If you have to remove a node from a RAC 10g database, even if the node
will no longer be available to the environment, there is a certain
amount of cleanup that needs to be done.  The remaining nodes need to
be informed of the change of status of the departing node.  If there are
any steps that must be run on the node being removed and the node is no
longer available those commands can be skipped.

The most important 3 steps that need to be followed are;

A.    Remove the instance using DBCA.
B.    Remove the node from the cluster.
C.    Reconfigure the OS and remaining hardware.

Here is a breakdown of the above steps.

A.    Remove the instance using DBCA.
————————————–

1.      Verify that you have a good backup of the OCR (Oracle Configuration
Repository) using ocrconfig -showbackup.

2.    Run DBCA from one of the nodes you are going to keep.  Leave the
database up and also leave the departing instance up and running.

3.    Choose “Instance Management”

4.    Choose “Delete an instance”

5.      On the next screen, select the cluster database from which you
will delete an instance.  Supply the system privilege username
and password.

6.    On the next screen, a list of cluster database instances will
appear.  Highlight the instance you would like to delete then
click next.

7.    If you have services configured, reassign the services.  Modify the
services so that each service can run on one of the remaining
instances.  Set “not used” for each service regarding the instance
that is to be deleted.  Click Finish.

8.    If your database is in archive log mode you may encounter the
following errors:
ORA-350
ORA-312
This may occur because the DBCA cannot drop  current log, as
it needs archiving.  This issue is fixed in tthehe 10.1.0.3
patchset. But previous to this patchset you should click the
ignore button and when the DBCA completes, manually archive
the logs for the deleted instance and dropt the log group.

SQL>  alter system archive log all;
SQL>  alter database drop logfile group 2;

9.    Verify that the dropped instance’s redo thread has been removed by
querying v$log.  If for any reason the redo thread is not disabled
then disable the thread.

SQL> alter database disable thread 2;

10.     Verify that the instance was removed from the OCR (Oracle
Configuration Repository) with the following commands:

srvctl config database -d <db_name>
cd <CRS_HOME>/bin
./crs_stat

11.    If this node had an ASM instance and the node will no longer be a
part of the cluster you will now need to remove the ASM instance with:

srvctl stop asm -n <nodename>
srvctl remove asm -n <nodename>

Verify that asm is removed with:

srvctl config asm -n <nodename>

B.    Remove the Node from the Cluster
—————————————-

Once the instance has been deleted.  The process of removing the node
from the cluster is a manual process. This is accomplished by running
scripts on the deleted node to remove the CRS install, as well as
scripts on the remaining nodes to update the node list.  The following
steps assume that the node to be removed is still functioning.

1.    To delete node number 2 first stop and remove the nodeapps on the
node you are removing.  Assuming that you have removed the ASM
instance as the root user on a remaining node;

# srvctl stop nodeapps -n <nodename>

2.      Run netca.  Choose “Cluster Configuration”.

3.    Only select the node you are removing and click next.

4.    Choose “Listener Configuration” and click next.

5.     Choose “Delete” and delete any listeners configured on the node
you are removing.

6.      Run <CRS_HOME>/bin/crs_stat.  Make sure that all database
resources are running on nodes that are going to be kept.  For
example:

NAME=ora.<db_name>.db
TYPE=application
TARGET=ONLINE
STATE=ONLINE on <node2>

Ensure that this resource is not running on a node that will be
removed.  Use <CRS_HOME>/bin/crs_relocate to perform this.
Example:

crs_relocate ora.<db_name>.db

7.     As the root user, remove the nodeapps on the node you are removing.

# srvctl remove nodeapps -n <nodename>

8.    Next as the Oracle user run the installer with the
updateNodeList option on any remaining node in the cluster.

a.  DISPLAY=ipaddress:0.0; export DISPLAY
This should be set even though the gui does not run.

b.  $ORACLE_HOME/oui/bin/runInstaller -updateNodeList
ORACLE_HOME=<Oracle Home Location> CLUSTER_NODES=<node1>,
<node3>,<node4>

With this command we are defining the RDBMS $ORACLE_HOME’s that
now are part of the cluster in the Oracle inventory.  If there is
no $ORACLE_HOME this step can be skipped.

9.    Change to the root user to finish the removal on a node that
is being removed.  This command will stop the CRS stack
and delete the ocr.loc file on the node to be removed.  The
nosharedvar option assumes the ocr.loc file is not on a shared
file sytem.  If it does exist on a shared file system then
specify sharedvar instead.  The nosharedhome option specifies
that the CRS_HOME is on a local filesystem.  If the CRS_HOME is
on a shared file system, specify sharedhome instead.
Run the rootdelete.sh script from <CRS_HOME>/install.  Example:

# cd <CRS_HOME>/install
# ./rootdelete.sh local nosharedvar nosharedhome

10.    On a node that will be kept, the root user should run the
rootdeletenode.sh script from the <CRS_HOME>/install directory.
When running this script from the CRS home specify both the node
name and the node number.  The node name and the node number are
visiable in olsnodes -n.  Also do NOT put a space after the
comma between the two.

# olsnodes -n
<node1>       1
<node2>       2

# cd <CRS_HOME>/install
# ./rootdeletenode.sh <node2name>,2

11.    Confirm success by running OLSNODES.

<CRS_HOME>/bin>: ./olsnodes -n
<node1>          1

12.    Now switch back to the oracle user account and run the same
runInstaller command as before.  Run it this time from the
<CRS_HOME> instead of the ORACLE_HOME.  Specify all of the
remaining nodes.

a.  DISPLAY=ipaddress:0.0; export DISPLAY

b.  <CRS_HOME>/oui/bin/runInstaller -updateNodeList
ORACLE_HOME=<CRS Home> CLUSTER_NODES=<node1>,
<node3>, <node4> CRS=TRUE

With this command we are defining the CRS HOME’s that now are
part of the cluster in the Oracle inventory.

13.    Once the node updates are done you will need to manually delete
the $ORACLE_HOME and $CRS_HOME from the node to be expunged,
unless, of course, either of these is on a shared file system
that is still being used.

a.  $ORACLE_HOME>: rm -rf *
b.  $CRS_HOME> : rm -rf *   (as root)

14.    Next, as root, from the deleted node, verify that all init scripts
and soft links are removed:

Sun:

rm /etc/init.d/init.cssd
rm /etc/init.d/init.crs
rm /etc/init.d/init.crsd
rm /etc/init.d/init.evmd
rm /etc/rc3.d/K96init.crs
rm /etc/rc3.d/S96init.crs
rm -Rf /var/opt/oracle/scls_scr
rm -Rf /var/opt/oracle/oprocd

Linux:

rm -f /etc/init.d/init.cssd
rm -f /etc/init.d/init.crs
rm -f /etc/init.d/init.crsd
rm -f /etc/init.d/init.evmd
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
rm -Rf /etc/oracle/scls_scr

HP-UX:

rm /sbin/init.d/init.cssd
rm /sbin/init.d/init.crs
rm /sbin/init.d/init.crsd
rm /sbin/init.d/init.evmd
rm /sbin/rc3.d/K960init.crs
rm /sbin/rc3.d/S960init.crs
rm /sbin/rc2.d/K960init.crs
rm /sbin/rc2.d/K001init.crs
rm -Rf /var/opt/oracle/scls_scr
rm -Rf /var/opt/oracle/oprocd

HP Tru64:

rm /sbin/init.d/init.cssd
rm /sbin/init.d/init.crs
rm /sbin/init.d/init.crsd
rm /sbin/init.d/init.evmd
rm /sbin/rc3.d/K96init.crs
rm /sbin/rc3.d/S96init.crs
rm -Rf /var/opt/oracle/scls_scr
rm -Rf /var/opt/oracle/oprocd

IBM AIX:

rm /etc/init.cssd
rm /etc/init.crs
rm /etc/init.crsd
rm /etc/init.evmd
rm /etc/rc.d/rc2.d/K96init.crs
rm /etc/rc.d/rc2.d/S96init.crs
rm -Rf /etc/oracle/scls_scr
rm -Rf /etc/oracle/oprocd

16.    You can also remove the /etc/oracle directory, the
/etc/oratab file, and the Oracle inventory (if desired)

17.     To remove an ADDITIONAL ORACLE_HOME, ASM_HOME, or EM_HOME from the
inventory on all remaining nodes, run the installer to update the
node list.  Example (if removing node 2):

runInstaller -updateNodeList -local \
ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=node1,node3,node4
(If you are using private home installations, please ignore the “-local” flag.)

RELATED DOCUMENTS
—————–

Oracle? Real Application Clusters Administrator’s Guide 10g Release 1 (10.1)
Part Number B10765-02
Chapter 5
Oracle Series/Oracle Database 10g High Availabilty Chapter 5 28-34.
Note 239998.1
Oracle Clusterware and RAC Admin and Deployment Guide - Ch. 10 and 11

Show Related Information Related
Products

* Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition

Below is steps to add removed instance back. Steps also apply to 10gR2 without big modification.

http://www.oracle.com/technology/pub/articles/vallath-nodes.html

PURPOSE
——-

The purpose of this note is to provide the user with a document that
can be used as a guide to add a cluster node from an Oracle 10g Real
Applications environment.

SCOPE & APPLICATION
——————-

This document can be used by DBAs and support analsyts who need to
either add a cluster node or assist another in adding a cluster
node in a 10g Unix Real Applications environment. If you are on
10gR2 (10.2.0.2 or higher), please refer to the documentation for
more updated steps.

ADDING A NODE TO A 10g RAC CLUSTER
———————————-

The most important steps that need to be followed are;

A. Configure the OS and hardware for the new node.
B. Add the node to the cluster.
C. Add the RAC software to the new node.
D. Reconfigure listeners for new node.
E. Add instances via DBCA.

Here is a breakdown of the above steps.

A. Configure the OS and hardware for the new node.
——————————————————-

Please consult with available OS vendor documentation for this step.

See Note 264847.1 for network requirements. Also verify that the OCR and
voting files are visible from the new node with correct permissions.

B. Add the node to the cluster.
————————————

1. If the CRS Home is owned by root and you are on a version < 10.1.0.4, change
the ownership of the CRS Home directories on all nodes to the Oracle user
so that OUI can read and write to these directories.
2. Set the DISPLAY environment variable and run the addNode.sh script from
$ORA_CRS_HOME/oui/bin on one of the existing nodes as the oracle user.
Example:

DISPLAY=ipaddress:0.0; export DISPLAY
cd $ORA_CRS_HOME/oui/bin
./addNode.sh

3. The OUI Welcome screen will appear, click next.

4. On the “Specify Cluster Nodes to Add to Installation” screen, add the
public and private node names (these should exist in /etc/hosts and
should be pingable from each of the cluster nodes), click next.

5. The “Cluster Node Addition Summary” screen will appear, click next.

6. The “Cluster Node Addition Progress” screen will appear. You will
then be prompted to run rootaddnode.sh as the root user. First verify
that the CLSCFG information in the rootaddnode.sh script is correct.
It should contain the new public and private node names and node
numbers. Example:

$CLSCFG -add -nn <node2>,2 -pn <node2-private>,2 -hn <node2>,2

Then run the rootaddnode.sh script on the EXISTING node you ran the
addNode.sh from. Example:

su root
cd $ORA_CRS_HOME
sh -x rootaddnode.sh

Once this is finished, click OK in the dialog box to continue.

7. At this point another dialog box will appear, this time you are
prompted to run $ORA_CRS_HOME/root.sh on all the new nodes.
If you are on version < 10.1.0.4 then
- Locate the highest numbered NEW cluster node using “$ORA_CRS_HOME/bin/olsnodes -n”.
- Run the root.sh script on this highest numbered NEW cluster node.
- Run the root.sh script on the rest of the NEW nodes in any order.
For versions 10.1.0.4 and above the root scritps can be run on the NEW
nodes in any order.

Example:

su root
cd $ORA_CRS_HOME
sh -x root.sh

If there are any problems with this step, refer to Note 240001.1

Once this is finished, click OK in the dialog box to continue.

8. After running the CRS root.sh on all new nodes, run
$ORA_CRS_HOME/bin/racgons add_config rac2:4948
from any node.

9. Next you will see the “End of Installation” screen. At this point you
may exit the installer.

10. Change the ownership of all CRS Homes back to root.

C. Add the Oracle Database software (with RAC option) to the new node.
—————————————————————————

1. On a pre-existing node, cd to the $ORACLE_HOME/oui/bin directory and
run the addNode.sh script. Example:

DISPLAY=ipaddress:0.0; export DISPLAY
cd $ORACLE_HOME/oui/bin
./addNode.sh

2. The OUI Welcome screen will appear, click next.

3. On the “Specify Cluster Nodes to Add to Installation” screen, specify
the node you want to add, click next.

4. The “Cluster Node Addition Summary” screen will appear, click next.

5. The “Cluster Node Addition Progress” screen will appear. You will
then be prompted to run root.sh as the root user.

su root
cd $ORACLE_HOME
./root.sh

Once this is finished, click OK in the dialog box to continue.

6. Next you will see the “End of Installation” screen. At this point you
may exit the installer.

7. Cd to the $ORACLE_HOME/bin directory and run the vipca tool with the
new nodelist. Example:

su root
DISPLAY=ipaddress:0.0; export DISPLAY
cd $ORACLE_HOME/bin
./vipca -nodelist <node1>,<node2>

8. The VIPCA Welcome Screen will appear, click next.

9. Add the new node’s virtual IP information, click next.

10. You will then see the “Summary” screen, click finish.

11. You will now see a progress bar creating and starting the new CRS
resources. Once this is finished, click ok, view the configuration
results, and click on the exit button.

12. Verify that interconnect information is correct with:

oifcfg getif

If it is not correct, change it with:

oifcfg setif <interface-name>/<subnet>:<cluster_interconnect|public>

For example:

oifcfg setif -global eth1/10.10.10.0:cluster_interconnect

or

oifcfg setif -node <nodename> eth1/10.10.10.0:cluster_interconnect

D. Reconfigure listeners for new node.
——————————————-

1. Run NETCA on the NEW node to verify that the listener is configured on
the new node. Example:

DISPLAY=ipaddress:0.0; export DISPLAY
netca

2. Choose “Cluster Configuration”, click next.

3. Select all nodes, click next.

4. Choose “Listener configuration”, click next.

5. Choose “Reconfigure”, click next.

6. Choose the listener you would like to reconfigure, click next.

7. Choose the correct protocol, click next.

8. Choose the correct port, click next.

9. Choose whether or not to configure another listener, click next.

10. You may get an error message saying, “The information provided for this
listener is currently in use by another listener…”. Click yes to
continue anyway.

11. The “Listener Configuration Complete” screen will appear, click next.

12. Click “Finish” to exit NETCA.

13. Run crs_stat to verify that the listener CRS resource was created.
Example:

cd $ORA_CRSHOME/bin
./crs_stat

14. The new listener will likely be offline. Start it by starting the
nodeapps on the new node. Example:

srvctl start nodeapps -n <newnode>

15. Use crs_stat to confirm that all VIP’s, GSD’s, ONS’s, and listeners are
ONLINE.

E. Add instances via DBCA. (for standby databases see section F)
———————————————————————

1. To add new instances, launch DBCA from a pre-existing node. Example:

DISPLAY=ipaddress:0.0; export DISPLAY
dbca

2. On the welcome screen, choose “Oracle Real Application Clusters”,
click next.

3. Choose “Instance Management”, click next.

4. Choose “Add an Instance”, click next.

5. Choose the database you would like to add an instance to and specify
a user with SYSDBA privileges, click next. Click next again…

6. Choose the correct instance name and node, click next.

7. Review the storage screen, click next.

8. Review the summary screen, click OK and wait a few seconds for the
progress bar to start.

9. Allow the progress bar to finish. When asked if you want to perform
another operation, choose “No” to exit DBCA.

10. To verify success, log into one of the instances and query from
gv$instance, you should now see all nodes.

F. Adding Instances to the Standby Database
————————————————

1. If you are using a RAC primary, make sure the steps from section E have
been performed. If you are using a single instance primary, add the redo
log groups and threads to the primary database via “alter database”
commands. Example commands:

alter database add logfile thread 2
group 3  size 51200k,
group 4  size 51200k;

alter database enable public thread 2;

2. Create a new standby controlfile from the primary database and copy it
to the standby. Example commmand:

alter database create standby controlfile as “/u01/stby.ctl”;

3. Shut down the standby database, back up the existing standby
controlfile on the standby database, and copy the new standby
controlfile into place.

4. Adjust any init.ora or spfile parameters such as thread, instance_name,
instance_number, local_listener, undo_tablespace, etc… for any new
instances.

5. Recover the standby database.

SELECT concat(’+'||gname, sys_connect_by_path(aname, ‘/’)) full_alias_path FROM
(SELECT g.name gname, a.parent_index pindex, a.name aname,
a.reference_index rindex FROM v$asm_alias a, v$asm_diskgroup g
WHERE a.group_number = g.group_number)
START WITH (mod(pindex, power(2, 24))) = 0
CONNECT BY PRIOR rindex = pindex;