ASM磁盘组冗余的三种类型external、normal、high,这里恢复的是normal状态,模拟OCR磁盘或votedisk不可用时RAC会出现什么现象给出故障定位的整个过程。在11.2.0.3中表决盘是放到了ocr中所以 OCR磁盘或votedisk不可用的两个实验一起做。在11.2.0.3中可ASM磁盘组冗余的三种类型external、normal、high,这里恢复的是normal状态,模拟OCR磁盘或votedisk不可用时RAC会出现什么现象给出故障定位的整个过程。在11.2.0.3中表决盘是放到了ocr中所以OCR磁盘或votedisk不可用的两个实验一起做。在11.2.0.3中可以手动备份OCR但手动备份是无效的。ocrconfig -export /u01/ocr.exp检查OCR有哪些备份[rootrac1 ~]# ocrconfig -showbackuprac1 2013/07/22 05:39:51 /u01/grid/crs/cdata/rac/backup00.ocrrac1 2013/07/22 01:39:51 /u01/grid/crs/cdata/rac/backup01.ocrrac1 2013/07/21 21:39:50 /u01/grid/crs/cdata/rac/backup02.ocrrac2 2013/07/21 01:52:54 /u01/grid/crs/cdata/rac/day.ocrrac2 2013/07/09 01:52:25 /u01/grid/crs/cdata/rac/week.ocrPROT-25: Manual backups for the Oracle Cluster Registry are not available注意orcle明确给出了手动备份是无效的查看表决盘信息[rootrac1 ~]# crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 745716af7e5b4faebfc8d948d096aa55 (/dev/oracleasm/disks/OCR_VOT1) [OCR_VOT]2. ONLINE 7092079f66c04f9dbf65974d0dcc611a (/dev/oracleasm/disks/OCR_VOT2) [OCR_VOT]3. ONLINE 6510631353284f5fbf3d4c8839822dbd (/dev/oracleasm/disks/OCR_VOT3) [OCR_VOT]Located 3 voting disk(s).#停库[rootrac1 ~]# srvctl stop database -d orcl -o immediate#停集群[rootrac1 ~]# crsctl stop cluster -all -f#破坏OCR和VOT[rootrac1 ~]# dd if/dev/zero f/dev/mapper/mpathap1 bs1024K count1记录了10 的读入记录了10 的写出1048576字节(1.0 MB)已复制0.0160613 秒65.3 MB/秒[rootrac1 ~]# dd if/dev/zero f/dev/mapper/mpathap2 bs1024K count1记录了10 的读入记录了10 的写出1048576字节(1.0 MB)已复制0.00800275 秒131 MB/秒[rootrac1 ~]# dd if/dev/zero f/dev/mapper/mpathap3 bs1024K count1记录了10 的读入记录了10 的写出1048576字节(1.0 MB)已复制0.00927389 秒113 MB/秒注意破坏后各节点服务一切正常[rootrac1 ~]# crs_stat -tName Type Target State Host------------------------------------------------------------ora.DATA.dg ora....up.type ONLINE ONLINE rac1ora.FRA.dg ora....up.type ONLINE ONLINE rac1ora....ER.lsnr ora....er.type ONLINE ONLINE rac1ora....N1.lsnr ora....er.type ONLINE ONLINE rac1ora.OCR_VOT.dg ora....up.type ONLINE ONLINE rac1ora.asm ora.asm.type ONLINE ONLINE rac1ora.orcl.db ora....se.type ONLINE ONLINE rac1ora.cvu ora.cvu.type ONLINE ONLINE rac1ora....SM1.asm application ONLINE ONLINE rac1ora....C1.lsnr application ONLINE ONLINE rac1ora....ac1.gsd application OFFLINE OFFLINEora....ac1.ons application ONLINE ONLINE rac1ora....ac1.vip ora....t1.type ONLINE ONLINE rac1ora....SM2.asm application ONLINE ONLINE rac2ora....C2.lsnr application ONLINE ONLINE rac2ora....ac2.gsd application OFFLINE OFFLINEora....ac2.ons application ONLINE ONLINE rac2ora....ac2.vip ora....t1.type ONLINE ONLINE rac2ora.gsd ora.gsd.type OFFLINE OFFLINEora....network ora....rk.type ONLINE ONLINE rac1ora.oc4j ora.oc4j.type ONLINE ONLINE rac1ora.ons ora.ons.type ONLINE ONLINE rac1ora....ry.acfs ora....fs.type ONLINE ONLINE rac1ora.scan1.vip ora....ip.type ONLINE ONLINE rac1所有节点重启操作系统后集群服务启不来了[rootrac1 ~]# reboot如果只是停止集群服务后面的重新创建ASM磁盘组会失败但重启操作系统后就可以创建成功。检查CRS[gridrac1 ~]$ crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4530: Communications failure contacting Cluster Synchronization Services daemonCRS-4534: Cannot communicate with Event Manager启动集群服务[rootrac1 ~]# crsctl start cluster -allCRS-2672: 尝试启动 ora.cssdmonitor (在 rac1 上)CRS-2672: 尝试启动 ora.cssdmonitor (在 rac2 上)CRS-2676: 成功启动 ora.cssdmonitor (在 rac1 上)CRS-2676: 成功启动 ora.cssdmonitor (在 rac2 上)CRS-2672: 尝试启动 ora.cssd (在 rac1 上)CRS-2672: 尝试启动 ora.diskmon (在 rac1 上)CRS-2672: 尝试启动 ora.cssd (在 rac2 上)CRS-2672: 尝试启动 ora.diskmon (在 rac2 上)CRS-2676: 成功启动 ora.diskmon (在 rac1 上)CRS-2676: 成功启动 ora.diskmon (在 rac2 上)# 直停在这里其他终端使用其他命令启动集群服务[rootrac1 ~]# crsctl start crsCRS-4640: Oracle High Availability Services is already activeCRS-4000: Command Start failed, or completed with errors.操作系统及crs日志中没看到特别有用的信息[rootrac1 ~]# vi /var/log/messages[gridrac1 ~]# vi $ORACLE_HOME/log/rac1/crsd/crsd.logocss日志中提示vi $ORACLE_HOME/log/rac1/cssd/ocssd.log2013-07-21 21:15:08.550: [ CSSD][1095031104]clssnmvFindInitialConfigs: No voting files found发现部分ASM磁盘没有了[rootrac1 ~]# /etc/init.d/oracleasm scandisksScanning the system for Oracle ASMLib disks: [ OK ][rootrac1 ~]# /etc/init.d/oracleasm listdisksDATAFRA依照RAC安装文档重建ASM磁盘[rootrac1 ~]# /etc/init.d/oracleasm createdisk OCR_VOT1 /dev/mapper/mpathap1Marking disk OCR_VOT1 as an ASM disk: [ OK ][rootrac1 ~]# /etc/init.d/oracleasm createdisk OCR_VOT2 /dev/mapper/mpathap2Marking disk OCR_VOT2 as an ASM disk: [ OK ][rootrac1 ~]# /etc/init.d/oracleasm createdisk OCR_VOT3 /dev/mapper/mpathap3Marking disk OCR_VOT3 as an ASM disk: [ OK ]停掉集群服务要加-f否则可能停止非常慢[rootrac1 ~]# crsctl stop crs -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac1CRS-2673: Attempting to stop ora.mdnsd on rac1CRS-2673: Attempting to stop ora.crf on rac1CRS-2677: Stop of ora.mdnsd on rac1 succeededCRS-2677: Stop of ora.crf on rac1 succeededCRS-2673: Attempting to stop ora.gipcd on rac1CRS-2677: Stop of ora.gipcd on rac1 succeededCRS-2673: Attempting to stop ora.gpnpd on rac1CRS-2677: Stop of ora.gpnpd on rac1 succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac1 has completedCRS-4133: Oracle High Availability Services has been stopped.以-excl -nocrs 方式启动集群这将启动ASM实例 但不启动CRS[rootrac1 ~]# crsctl start crs -excl -nocrsCRS-4123: Oracle High Availability Services has been started.CRS-2672: Attempting to start ora.mdnsd on rac1CRS-2676: Start of ora.mdnsd on rac1 succeededCRS-2672: Attempting to start ora.gpnpd on rac1CRS-2676: Start of ora.gpnpd on rac1 succeededCRS-2672: Attempting to start ora.cssdmonitor on rac1CRS-2672: Attempting to start ora.gipcd on rac1CRS-2676: Start of ora.cssdmonitor on rac1 succeededCRS-2676: Start of ora.gipcd on rac1 succeededCRS-2672: Attempting to start ora.cssd on rac1CRS-2672: Attempting to start ora.diskmon on rac1CRS-2676: Start of ora.diskmon on rac1 succeededCRS-2676: Start of ora.cssd on rac1 succeededCRS-2672: Attempting to start ora.drivers.acfs on rac1CRS-2679: Attempting to clean ora.cluster_interconnect.haip on rac1CRS-2672: Attempting to start ora.ctssd on rac1CRS-2681: Clean of ora.cluster_interconnect.haip on rac1 succeededCRS-2672: Attempting to start ora.cluster_interconnect.haip on rac1CRS-2676: Start of ora.drivers.acfs on rac1 succeededCRS-2676: Start of ora.ctssd on rac1 succeededCRS-2676: Start of ora.cluster_interconnect.haip on rac1 succeededCRS-2672: Attempting to start ora.asm on rac1CRS-2676: Start of ora.asm on rac1 succeeded此时crs仍然报错[rootrac1 ~]# crs_stat -tCRS-0184: Cannot communicate with the CRS daemon.[rootrac1 ~]# crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4530: Communications failure contacting Cluster Synchronization Services daemonCRS-4534: Cannot communicate with Event Manager重建原ocr和votedisk所在磁盘组注意这里是在grid用户下SQL col path for a50SQL set lines 300SQL select path,header_status from v$asm_disk;SQL create diskgroup OCR_VOT normal redundancy disk /dev/oracleasm/disks/OCR_VOT1,/dev/oracleasm/disks/OCR_VOT2,/dev/oracleasm/disks/OCR_VOT3attribute compatible.rdbms 11.2,compatible.asm 11.2;ASM磁盘组冗余的三种类型external、normal、high我这里之前用的是normal。从ocr backup中恢复OCR在每个节点grid用户下cd $ORACLE_HOME/cdata/racocrconfig -restore /u01/grid/crs/cdata/rac/backup00.ocr恢复表决盘的准备工作show parameter asm_diskstring如果asm_diskstring没有值表示ASM磁盘用的是默认ASM磁盘搜索路径。修改成实际的ASM磁盘搜索路径alter system set asm_diskstring/dev/oracleasm/disks/*;恢复表决盘[rootrac1 ~]# crsctl replace votedisk OCR_VOTSuccessful addition of voting disk 4ad2b9cc0a754fffbf1515281199a78f.Successful addition of voting disk 9f8dc1c013df4f39bfd85c64051a0bc1.Successful addition of voting disk a4aea7a1aa434fb3bff161f6ea8ce102.Successfully replaced voting disk group with OCR_VOT.CRS-4266: Voting file(s) successfully replacedocr和vot恢复后crs等服务就会自动起来了。[rootrac1 ~]# crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4529: Cluster Synchronization Services is onlineCRS-4534: Cannot communicate with Event Manager[rootrac1 ~]# crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 4ad2b9cc0a754fffbf1515281199a78f (/dev/oracleasm/disks/OCR_VOT1) [OCR_VOT]2. ONLINE 9f8dc1c013df4f39bfd85c64051a0bc1 (/dev/oracleasm/disks/OCR_VOT2) [OCR_VOT]3. ONLINE a4aea7a1aa434fb3bff161f6ea8ce102 (/dev/oracleasm/disks/OCR_VOT3) [OCR_VOT]Located 3 voting disk(s).重启集群服务检查是否已经恢复正常[rootrac1 ~]# crsctl stop crs[rootrac1 ~]# crsctl start crsCRS-4123: Oracle High Availability Services has been started.其他节点也可以启动了[rootrac2 ~]# crsctl start crs等一会儿检查服务已经起来了[rootrac1 ~]# crs_stat -tName Type Target State Host------------------------------------------------------------ora.DATA.dg ora....up.type ONLINE ONLINE rac1ora.FRA.dg ora....up.type ONLINE ONLINE rac1ora....ER.lsnr ora....er.type ONLINE ONLINE rac1ora....N1.lsnr ora....er.type ONLINE ONLINE rac1ora.OCR_VOT.dg ora....up.type ONLINE ONLINE rac1ora.asm ora.asm.type ONLINE ONLINE rac1ora.orcl.db ora....se.type ONLINE ONLINE rac1ora.cvu ora.cvu.type ONLINE ONLINE rac1ora....SM1.asm application ONLINE ONLINE rac1ora....C1.lsnr application ONLINE ONLINE rac1ora....ac1.gsd application OFFLINE OFFLINEora....ac1.ons application ONLINE ONLINE rac1ora....ac1.vip ora....t1.type ONLINE ONLINE rac1ora....SM2.asm application ONLINE ONLINE rac2ora....C2.lsnr application ONLINE ONLINE rac2ora....ac2.gsd application OFFLINE OFFLINEora....ac2.ons application ONLINE ONLINE rac2ora....ac2.vip ora....t1.type ONLINE ONLINE rac2ora.gsd ora.gsd.type OFFLINE OFFLINEora....network ora....rk.type ONLINE ONLINE rac1ora.oc4j ora.oc4j.type ONLINE ONLINE rac1ora.ons ora.ons.type ONLINE ONLINE rac1ora....ry.acfs ora....fs.type ONLINE ONLINE rac1ora.scan1.vip ora....ip.type ONLINE ONLINE rac1到这里为止OCR和