ORA-00600: internal error code, arguments: [kcblasm_1], [103], [] bug

巡检发现存在alert 日志存在ORA-600

1.0 查询alter 对应的Trace日志

/oracle/admin/fgsquery/udump/fgsquery_ora_21777.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5. - 64bit Production
Machine:    x86_64
Redo thread mounted by this instance:
Oracle process number:
Unix process pid: , image: oracle@pquerydb02
 
*** ACTION NAME:(SQL 窗口 - 新建) -- ::13.082
*** MODULE NAME:(PL/SQL Developer) -- ::13.082
*** SERVICE NAME:(shangdi) -- ::13.082
*** SESSION ID:(2125.22785) -- ::13.082
*** -- ::13.082
ksedmp: internal or fatal error
ORA-: internal error code, arguments: [kcblasm_1], [], [], [], [], [], [], []
Current SQL statement for this session:
select *
  from ucontevent
 where eventtype= 'IBQ0010'
   and eventid in
       (select eventid from ulcinsureacctrace where confdate is null/* and cvalidate>date'2018-10-01'*/)
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ssd_unwind_bp: unhandled instruction at 0x3d12d0e instr=f
ksedst()+          call     ksedst1()             ?  ?
                                                   7FFF993CB020 ? 7FFF993CB080 ?
                                                   7FFF993CAFC0 ?  ?
ksedmp()+         call     ksedst()              ?  ?
                                                   7FFF993CB020 ? 7FFF993CB080 ?
                                                   7FFF993CAFC0 ?  ?
ksfdmp()+          call     ksedmp()              ?  ?
                                                   7FFF993CB020 ? 7FFF993CB080 ?
                                                   7FFF993CAFC0 ?  ?
kgerinv()+        call     ksfdmp()             006AE9A20 ?  ?
                                                   7FFF993CB020 ? 7FFF993CB080 ?
                                                   7FFF993CAFC0 ?  ?
kgeasnmierr()+    call     kgerinv()            006AE9A20 ? 2AC380BF1168 ?
                                                   7FFF993CB080 ? 7FFF993CAFC0 ?
                                                    ?  ?
kcblasm1()+       call     kgeasnmierr()        006AE9A20 ? 2AC380BF1168 ?
                                                   7FFF993CB080 ? 7FFF993CAFC0 ?
                                                    ?  ?
kcblasm()+         call     kcblasm1()           2AC380ED9D08 ? 7FFF993CAFC0 ?
                                                   7FFF993CB080 ? 7FFF993CAFC0 ?
                                                    ?  ?
kxhfFndFreeSlot()+  call     kcblasm()            2AC380ED9D08 ?  ?
                                                 7FFF993CB080 ? 7FFF993CAFC0 ?
                                                    ?  ?
kxhfNewBuffer()+  call     kxhfFndFreeSlot()    2AC380ED7A78 ?  ?
                                                   7FFF993CB080 ? 7FFF993CAFC0 ?
                                                    ?  ?
qerhjGetNewBuffer()  call     kxhfNewBuffer()      2AC380ED7A78 ?  ?
+                                                 ? 7FFF993CC1F8 ?
                                                    ?  ?
ksxb1bqb()+       call     qerhjGetNewBuffer()  7FFF993CC1F0 ?  ?
                                                    ? 7FFF993CC1F8 ?
                                                    ?  ?
kxhrPack()+       call     ksxb1bqb()           7FFF993CC1F0 ? 143F7BA64B ?
                                                    ? 7FFF993CC1F8 ?
                                                    ?  ?
qerhjSplitBuild()+  call     kxhrPack()            ? 143F7BA64B ?
                                                  ? 7FFF993CC1F0 ?
                                                   7FFF21F8B302 ?  ?
qertbFetchByRowID()  call     qerhjSplitBuild()    7FFF993CC810 ? 143F7BA64B ?
+                                              200D263A60 ?  ?
                                                   7FFF21F8B302 ?  ?
rwsfcd()+          call     qertbFetchByRowID()  200D264800 ? 002E88560 ?
                                                   7FFF993CC810 ? 000007FFE ?
                                                   7FFF21F8B302 ?  ?
qerhjFetch()+    call     rwsfcd()             200D264800 ? 002E88560 ?
                                                   7FFF993CC810 ? 000007FFE ?
                                                   7FFF21F8B302 ?  ?
opifch2()+       call     qerhjFetch()         200D263588 ? 003230DC2 ?
                                                   7FFF993CCE80 ?  ?
                                                   7FFF21F8B302 ?  ?
opifch()+          call     opifch2()             ?  ?
                                                   7FFF993CD060 ? 2AC380EDDAE0 ?
                                                   7FFF21F8B302 ?  ?
opiodr()+        call     opifch()              ?  ?
                                                   7FFF993CD060 ?  ?
                                                    ? 0013B0002 ?
ttcpip()+        call     opiodr()              ?  ?
                                                   7FFF993CFDE8 ?  ?
                                                   005BEAF88 ? 0013B0002 ?
opitsk()+        call     ttcpip()             006AF1FB0 ? 0054A51E0 ?
                                                   7FFF993CFDE8 ?  ?
                                                   7FFF993CF8E8 ? 7FFF993CFF50 ?
opiino()+        call     opitsk()              ?  ?
                                                   7FFF993CFDE8 ?  ?
                                                    ?  ?
opiodr()+        call     opiino()             00000003C ?  ?
                                                   7FFF993D0FE8 ?  ?
                                                    ?  ?
opidrv()+         call     opiodr()             00000003C ?  ?
                                                   7FFF993D0FE8 ?  ?
                                                   005BEB820 ?  ?
sou2o()+          call     opidrv()             00000003C ?  ?
                                                   7FFF993D0FE8 ?  ?
                                                   005BEB820 ?  ?
opimai_real()+    call     sou2o()              7FFF993D0FC0 ? 00000003C ?
                                                    ? 7FFF993D0FE8 ?
                                                   005BEB820 ?  ?
main()+           call     opimai_real()         ? 7FFF993D1050 ?
                                                    ? 7FFF993D0FE8 ?
                                                   005BEB820 ?  ?
__libc_start_main()  call     main()                ? 7FFF993D1050 ?
+                                                ? 7FFF993D0FE8 ?
                                                   005BEB820 ?  ?
_start()+          call     __libc_start_main()  00072D108 ?  ?
                                                   7FFF993D11A8 ?  ?
                                                   005BEB820 ?  ?
 
--------------------- Binary Stack Dump ---------------------

2.0 匹配Mos Bug

ORA-600 [kcblasm_1] In 10.2.0.5. (文档 ID 1133845.1)  
Oracle 内部错误，bug,解决方式安装小布丁，psu，大版本升级······
 
 ORA-00600: internal error code, arguments: [kcblasm_1], [103], [], [], [], [], [], []
The call stack in the generated trace file looks similar to:
... kcblasm1 kcblasm kxhfFndFreeSlot kxhfNewBuffer qerhjGetNewBuffer ksxb1bqb kxhrPack ... 根据函数堆栈调用，发现顺序非常匹配
CAUSE
This is is a known and common problem hit in 10.2.0.5, investigated and corrected in unpublished Bug 7612454.
 
The problem was introduced in 10.2.0.5, by the fix of Bug:7523755.
This has been corrected in PSU 10.2.0.5.4 and 11.2.
 
Please refer to:
Note 7612454.8 - Bug 7612454 - More "direct path read" operations / OERI:kcblasm_1
 
CHANGES
This problem is hit only in 10.2.0.5, up to PSU 10.2.0.5.4 in which problem is fixed.
Problem is affecting 10.2.0.5 to 10.2.0.5.3.
 
SOLUTION
The quickest way to solve the problem is to apply PSU 10.2.0.5.4, Patch 12419392 or later.
All alternative solutions for this problem are listed below:
Upgrade the database to 11.2.
- OR -
Apply 10.2.0.5.4 Patch Set Update (Patch 12419392) or later PSUs where bug is fixed.
The available PSUs are mentioned in "10.2.0.5 Patch Set Updates - List of Fixes in each PSU" (Document 1337394.1)
- OR -
Apply interim Patch 7612454 on top of 10.2.0.5 (10.2.0.5.0-10.2.0.5.3):
For UNIX / Linux platforms apply Patch 7612454 available for download on MOS.
For Windows platforms apply Patch 3 or higher.
Please check Document 342443.1 for latest patches available for Windows on top of 10.2.0.5.
解决方案：1.升级数据库版本-11.2
         2.patch 12419392 apply 10.2.0.5.4
         3. patch 7612454 on top of 10.2.0.5 (10.2.0.5.0-10.2.0.5.3):

根据Mos的查询反馈结果，基本不会采纳，对问题sql进行查询

3.0 数据库查询信息

查询会话信息，ash已被刷出，使用awr视图查询
SQL>selelct sample_time,user_id,sql_id,event from v$active_session_history where sample_time between to_date('20181113 09','yyyymmdd hh24')
  and to_date('20181113 13','yyyymmdd hh24') and session_id= and session_serial#=;
 
SQL>select owner,object_type,status,object_name from dba_objects where object_name =upper('ucontevent') and object_type like '%TABLE%'
OWNER                   OBJECT_TYPE       STATUS  OBJECT_NAME
------------------------------ ------------------- ------- ------------------------------
LISPRDD                TABLE           VALID   UCONTEVENT

SQL> select sample_time,user_id,sql_id,event from dba_hist_active_sess_history where sample_time between to_date('20181113 11','yyyymmdd hh24') and to_date('20181113 12','yyyymmdd hh24') and session_id=2125 and session_serial#=22785;

SAMPLE_TIME USER_ID SQL_ID EVENT
------------------------------ ---------- ------------- ----------------------------------------------------------------
13-NOV-18 11.04.18.479 AM 249 cxp0p771cnx1f
13-NOV-18 11.02.18.293 AM 249 42wvj1k385aun
13-NOV-18 11.01.18.193 AM 249 4bbwc4gsfxu33

 SQL>  select username,sql_id,event,program,action,machine,port from dba_hist_active_sess_history a,dba_users b where a.user_id=b.user_id and sample_time between to_date('20181113 11','yyyymmdd hh24') and to_date('20181113 12','yyyymmdd hh24') and session_id=2125 and session_serial#=22785;

USERNAME   SQL_ID        EVENT      PROGRAM                        ACTION                           MACHINE                              PORT
---------- ------------- ---------- ------------------------------ -------------------------------- ------------------------------ ----------
ZKRWANGYB  4bbwc4gsfxu33            plsqldev.exe                   SQL 窗口 - 新建                  WORKGROUP\PQZBLJVAPP1               59450
ZKRWANGYB  42wvj1k385aun            plsqldev.exe                   SQL 窗口 - 新建                  WORKGROUP\PQZBLJVAPP1               59450
ZKRWANGYB  cxp0p771cnx1f            plsqldev.exe                   SQL 窗口 - 新建                  WORKGROUP\PQZBLJVAPP1               59450 

 

 

 

SQL>  select * from table(dbms_xplan.display_awr('&sql_id'))
Enter value for sql_id: 42wvj1k385aun

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------
SQL_ID 42wvj1k385aun
--------------------
select *   from ucontevent  where eventtype = 'IBQ0010'    and eventid in (select eventid
     from ulcinsureacctrace where confdate is null )
Plan hash value: 3993290469
----------------------------------------------------------------------------------------------------------
| Id  | Operation      | Name  | Rows  | Bytes |TempSpc| Cost (%CPU)| Time  |
----------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT      |  |  |  |  |  1218K(100)|   |
|   1 |  HASH JOIN SEMI       |  |  2282K|   298M|   274M|  1218K  (2)| 04:03:46 |
|   2 |   TABLE ACCESS BY INDEX ROWID| UCONTEVENT  |  2282K|   248M|  |   114K  (1)| 00:22:50 |
|   3 |    INDEX RANGE SCAN      | UCONTEVENT_INDEX1 |  2282K|  |  |  6260   (1)| 00:01:16 |
|   4 |   TABLE ACCESS FULL      | ULCINSUREACCTRACE |   104M|  2289M|  |   915K  (2)| 03:03:09 |
----------------------------------------------------------------------------------------------------------
17 rows selected. 

SQL>select owner,segment_name,sum(bytes)/1024/1024 from dba_segments where segment_name in('UCONTEVENT','ULCINSUREACCTRACE') group by segment_name,owner

OWNER SEGMENT_NAME SUM(BYTES)/1024/1024
------------------------------ ------------------------------ --------------------
LISPRDD ULCINSUREACCTRACE 33170
LISPRDD UCONTEVENT 8198

!SQL能否改写~全表扫描33G的数据量--只取部分数据

 SQL> select count(*) from (select eventid from LISPRDD.ulcinsureacctrace where confdate is null);
  COUNT(*)
----------
 106790272
SQL> select num_rows from dba_tables where owner='LISPRDD' and table_name=upper('ulcinsureacctrace');
  NUM_ROWS
----------
 105814076
SQL> select count(*) from LISPRDD.ulcinsureacctrace sample block(10);
  COUNT(*)
----------
  10946783
SQL> select last_analyzed from dba_tables where owner='LISPRDD' and table_name=upper('ulcinsureacctrace');

LAST_ANALYZED
-------------------
2018-11-16 22:10:25
SQL>  select column_name,data_type,num_distinct,num_nulls,last_analyzed from dba_tab_columns where  owner='LISPRDD' and table_name=upper('ulcinsureacctrace') and column_name in('EVENTID','CONFDATE');

COLUMN_NAME        DATA_TYPE       NUM_DISTINCT  NUM_NULLS LAST_ANALYZED
------------------------------ ------------------------------ ------------ ---------- -------------------
EVENTID         VARCHAR2    62115722     0 2018-11-16 22:10:25
CONFDATE        DATE        150  105578361 2018-11-16 22:10:25

4.0 分析结论

问题分析后，此ora-600为一个bug，遭遇此报错的是一个pl/sql 开发人员，且写的sql效率非常低，非业务sql，因此此报错可忽略。 如果是业务sql频繁报错，可能计划打补丁；
 
                    本次建议，让开发人员对此sql进行优化处理，sql的查询逻辑是否有改进的空间。