某業(yè)務(wù)系統的監聽(tīng)每過(guò)10天左右，就異常終止一次

1 查看監聽(tīng)日志、發(fā)現監聽(tīng)終止是通過(guò)CRS發(fā)出的。

2 查看CRS的alert日志，發(fā)現CRS日志中有報錯信息

3  查看 crsd.log 的 日志、發(fā)現監聽(tīng)的狀態(tài)切換有異常。發(fā)現此問(wèn)題同《 VIP, SCAN VIP/Listener Fails Over and Listener Stops After Short Public Network Hiccup ( 文檔 ID 1333165.1) 》文檔一樣，以下4、5點(diǎn)都同此文檔一致。

4 查看 orarootagent_root.log 日志

5  查看 oraagent_<user>.log 日志： 發(fā)現監聽(tīng)強制停止，重啟失敗。

原因：

結合 1-5 點(diǎn)，此種情況和《 VIP, SCAN VIP/Listener Fails Over and Listener Stops After Short Public Network Hiccup ( 文檔 ID 1333165.1) 》文檔相符，按照此文檔進(jìn)行修改。

以下為 oracle 官方相關(guān)文檔：

VIP, SCAN VIP/Listener Fails Over and Listener Stops After Short Public Network Hiccup ( 文檔 ID 1333165.1)

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.1 [Release 11.2 to 12.1]

Information in this document applies to any platform.

SYMPTOMS

After check timed out, 11gR2 Grid Infrastructure network resource (usually ora.net1.network) goes to INTERMEDIATE state, then goes back to ONLINE very shortly. This note will not discuss cause of check time out, but most common cause is public network hiccup.

Once network resource goes into INTERMEDIATE state, it may trigger VIP, service, SCAN VIP/SCAN listener, ora.cvu and ora.ons etc to be failed over/go offline due to resource dependence, which could result in unnecessary connectivity issue for that period of time. After network resource is back online, affected resources may not come back online.

·  $GRID_HOME/log/<node>/crsd/crsd.log

  2011-06-12 07:12:31.261: [    AGFW][10796] {0:1:2881} Received state change for ora.net1.network racnode1 1 [old state = ONLINE, new state = UNKNOWN]

  2011-06-12 07:12:31.261: [    AGFW][10796] {0:1:2881} Received state LABEL change for ora.net1.network racnode1 1 [old label  = , new label = CHECK TIMED OUT]

  ..

  2011-06-12 07:12:31.297: [   CRSPE][12081] {0:1:2881} RI [ora.net1.network racnode1 1] new external state [INTERMEDIATE] old value: [ONLINE] on racnode1 label = [CHECK TIMED OUT] 

  ..

  2011-06-12 07:12:31.981: [    AGFW][10796] {0:1:2882} Received state change for ora.net1.network racnode1 1 [old state = UNKNOWN, new state = ONLINE]

  ..

  2011-06-12 07:12:32.307: [   CRSPE][12081] {0:1:2881} RI [ora.LISTENER.lsnr racnode1 1] new internal state: [STOPPING] old value: [STABLE]

  2011-06-12 07:12:32.308: [   CRSPE][12081] {0:1:2881} CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode1'

·  $GRID_HOME/log/<node>/agent/crsd/orarootagent_root/orarootagent_root.log

?  2011-06-12 07:12:08.965: [    AGFW][2070] {1:27767:2} Created alert : (:CRSAGF00113:) :  Aborting the command: check for resource: ora.net1.network racnode1 1

?  2011-06-12 07:12:08.966: [ora.net1.network][2070] {1:27767:2} [check] clsn_agent::abort {

?  ..

?  2011-06-12 07:12:31.257: [    AGFW][2070] {1:27767:2} Command: check for resource: ora.net1.network racnode1 1 completed with status: TIMEDOUT

?  2011-06-12 07:12:31.258: [    AGFW][2314] {1:27767:2} ora.net1.network racnode1 1 state changed from: ONLINE to: UNKNOWN

?  2011-06-12 07:12:31.258: [    AGFW][2314] {1:27767:2} ora.net1.network racnode1 1 would be continued to monitored!

?  2011-06-12 07:12:31.258: [    AGFW][2314] {1:27767:2} ora.net1.network racnode1 1 state details has changed from:  to: CHECK TIMED OUT

?  ..

?  2011-06-12 07:12:31.923: [ora.net1.network][2314][F-ALGO] {1:27767:2} CHECK initiated by timer for: ora.net1.network racnode1 1

?  ..

?  2011-06-12 07:12:31.973: [ora.net1.network][8502][F-ALGO] {1:27767:2} [check] Command check for resource: ora.net1.network racnode1 1 completed with status ONLINE

?  2011-06-12 07:12:31.978: [    AGFW][2314] {1:27767:2} ora.net1.network racnode1 1 state changed from: UNKNOWN to: ONLINE

·  $GRID_HOME/log/<node>/agent/crsd/oraagent_<user>/oraagent_<user>.log

?  2011-06-12 07:12:32.335: [    AGFW][2314] {0:1:2881} Agent received the message: RESOURCE_STOP[ora.LISTENER.lsnr racnode1 1] ID 4099:14792

?  2011-06-12 07:12:32.335: [    AGFW][2314] {0:1:2881} Preparing STOP command for: ora.LISTENER.lsnr racnode1 1

?  2011-06-12 07:12:32.335: [    AGFW][2314] {0:1:2881} ora.LISTENER.lsnr racnode1 1 state changed from: ONLINE to: STOPPING

·  $GRID_HOME/log/<node>/alert<node>.log

?  2012-01-10 06:48:18.474 [/ocw/grid/bin/orarootagent.bin(10485902)]CRS-5818:Aborted command 'check for resource: ora.net1.network racnode1 1' for resource 'ora.net1.network'. Details at (:CRSAGF00113:) {1:24200:2} in /ocw/grid/log/racnode1/agent/crsd/orarootagent_root/orarootagent_root.log.

?  2012-01-10 06:48:43.481 [/ocw/grid/bin/oraagent.bin(8847542)]CRS-5016:Process "/ocw/grid/bin/lsnrctl" spawned by agent "/ocw/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/ocw/grid/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log"

?  2012-01-10 06:48:43.552 [/ocw/grid/bin/oraagent.bin(8847542)]CRS-5016:Process "/ocw/grid/opmn/bin/onsctli" spawned by agent "/ocw/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/ocw/grid/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log"

CAUSE

SOLUTION

The issue is fixed in a few different bugs:

1.  bug 12680491  fixes the dependence between network and VIP

The fix of  bug 12680491  will add intermediate modifier to stop dependency between network resource and VIP to avoid unnecessary resource state change, it's included in 11.2.0.2 GI PSU4, 11.2.0.3 GI PSU3, 11.2.0.3 Windows Patch 7, 11.2.0.4 and above. This fix is recommended instead of fix for  bug 12378938  to avoid the issue in first place. 

Once patch for this bug is applied, the following needs to be executed to change the dependence for all VIPs:

# $GRID_HOME/bin/crsctl modify res ora.<racnode1>.vip -attr "STOP_DEPENDENCIES=hard(intermediate:ora.<net1>.network)"

For example:

# /ocw/grid/bin/crsctl modify res ora.racnode1.vip -attr "STOP_DEPENDENCIES=hard(intermediate:ora.net1.network)"

Once the attribute is changed, a restart of nodeapps/VIP is needed to be in effect

2.  bug 13582411  fixes the dependence between network and SCAN VIP/listener

The fix of  bug 13582411  will add intermediate modifyer to stop dependency between network resource and SCAN VIP to avoid unnecessary resource state change, it's included in 11.2.0.3 GI PSU4, 11.2.0.4 and above.  

Once patch for this bug is applied, the following needs to be executed to change the dependence for all SCAN VIPs and to restart SCAN VIPs:

# $GRID_HOME/bin/crsctl modify res ora.scan<n>.vip -attr "STOP_DEPENDENCIES=hard(intermediate:ora.net<n>.network)"

For example:

# /ocw/grid/bin/crsctl modify res ora.scan1.vip -attr "STOP_DEPENDENCIES=hard(intermediate:ora.net1.network)"

# /ocw/grid/bin/crsctl modify res ora.scan2.vip -attr "STOP_DEPENDENCIES=hard(intermediate:ora.net1.network)"

# /ocw/grid/bin/crsctl modify res ora.scan3.vip -attr "STOP_DEPENDENCIES=hard(intermediate:ora.net1.network)"

# /ocw/grid/bin/srvctl stop scan -f

$ /ocw/grid/bin/srvctl start scan_listener 

3.  bug 17435488  fixes the dependence between network and ora.cvu and ora.ons

The fix will add intermediate modifyer to stop dependency between network resource and ora.cvu and ora.ons to avoid unnecessary resource state change, it's included in 12.1.0.2