Datapatch AND What to do if the status of a datapatch action was not SUCCESS due to finding non-ignorable errors

Enterprise Manager:

Starting version 12.1 Enterprise Manager now calls datapatch to complete post patch actions upon any 12c or later database restart.
As noted above, datapatch contains the logic to identify if any post-patch SQL actions are pending.

OPatchAuto :

OPatchAuto calls datapatch to complete post patch actions upon installation of the binary patch and restart of the database.
As noted earlier, datapatch identifies the post install instructions which are necessary and automatically completes the same.
Post patch activity include both the application and the removal or rollback of SQL changes in the database.

OPatch :

Datapatch integration with OPatch is not possible as OPatch is executed when the database is down and
datapatch requires the database to be opened to complete its activity.
When
patches are installed or rollbacked using OPatch then datapatch needs
to be explicitly invoked if instructed to do so in the patch readme.
Datapatch is designed to be idempotent and can be run for all patches and
it is recommended that datapatch be run post any patch apply or rollback activity when OPatch is used.
(In
computing, an idempotent operation is one that has no additional effect
if it is called more than once with the same input parameters.)

Datapatch :

Datapatch is the patching tool to complete the post-patch SQL actions for RDBMS patches.

Windows specific : On Windows platforms the script name is datapatch.bat.

RAC specific: For a RAC environment, after the binary patch has been
applied on all nodes run Datapatch to complete the post-install SQL
deployment for the PSU only from one node. Datapatch need not be run on
all the nodes.

Datapatch determines the requisite apply/rollback actions by matching an internal repository with the patch inventory.
Datapatch should be invoked when the database is restarted after a patching session.
Enterprise Manager and OPatchAuto call datapatch automatically during every patching session.
If
OPatch is used to install RDBMS patches then datapatch has to be
explictly called to complete any patching actions after database
restart.
Datapatch will automatically determine the set of patches
that need to be installed and a set of patches that need to be rolled
back.

Datapatch ensures that a patch has been installed/rolled back in all
RAC instances before to initiate any post-patch SQL actions on the
database.
In Oracle Multitenant environment datapatch will patch the root and any pluggable databases that are opened.
In
order to patch all pluggable databases, it should be ensured that
before to invoke datapatch all pluggable databases are opened.
If an
unpatched pluggable database is opened in Oracle Multitenant then
calling datapatch will complete the pending patch actions.

Datapatch usage :

All arguments are optional, if there are no arguments then datapatch will automatically determine which SQL scripts need to
be run in order to complete the installation of any patches that contain post-patch SQL instructions.

Optional arguments:

-db <sid>
Use the specified database's SID rather than $ORACLE_SID

-apply <patch1,patch2,...,patchn>
Only consider the specified patch list for apply operations

-rollback <patch1,patch2,...,patchn>
Only consider the specified patch list for rollback operations

-force
Run the apply and/or rollback scripts even if not necessary per the SQL registry

-prereq
Run prerequisite checks only, do not actually run any scripts

-oh <oracle_home value>
Use the specified directory to determine what patches are installed

-verbose
Output additional information used for debugging

-help
Output usage information and exit

-version
Output build information and exit

Datapatch in a Multitenant Environment:

When datapatch is run to invoke any SQL actions in a Multitenant
environment, datapatch only applies the changes to the ROOT, SEED and
any open PDBs based on the pending actions. For any PDB that is not
currently open, the SQL patch will not be applied. When such a PDB is
reopened, it will be opened in restricted mode as there will be a patch
level mismatch between the PDB and the ROOT.

Call DBMS_PDB.CHECK_PLUG_COMPATIBILITY on 12.1.0.1. PDB xml file will return these two violations, which are expected.
SQL Patch ERROR

SQL patch bug # mismatch: Installed in the PDB but not in the CDB. Install the SQL patch in the PDB or the CDB

This error can be cleared by running datapatch again and closing and reopening the affected PDB(s)

1) Invoke datapatch:

% cd $ORACLE_HOME/OPatch
% datapatch

2) Close and reopen the affected PDB

Connect as SYS
alter pluggable database <PDB Name> close instances =all;
alter pluggable database <PDB Name> open read write instances =all;

3) Check again for plugin violations (by
querying pdb_plug_in_violations in cdb$root), there should be none and
the pluggable database should be opened in normal mode

Changes for DB release 12.1.0.2

Miscellaneous

Catbundle Integration: Datapatch now assumes the role of
catbundle when applying bundles/PSU. As a result of this change
catbundle.sql has been deprecated and patch registry is maintained only
in registry$sqlpatch. Application of the PSU does not update
registry$history table anymore. The status for patch application is now
entirely maintained in registry$sqlpatch
Support for patch UID: Datapatch now uses patch-ID and
patch-UID to uniquely identify a patch. With this change, datapatch now
distinguishes between different versions of a single patch. This allows
datapatch to handle the case of re-issued patches correctly.
Support for –force –rollback all: Datapatch now provides
the option to rollback all patches from a given container DB using the
–force option. This is useful for unplug/plug of container databases
Integration with upgrade/downgrade scripts
Improved error checking for PLS errors and warnings.

Prerequisite Checks

In 12.1.0.2 Datapatch now has pre-req checks for a number of
conditions. These pre-req checks prevent patching and have the
following reasons and potential issues:

Upgrade mode check:

If a patch requires upgrade mode startup,
datapatch enforces that via a pre-req check. To overcome this error,
DB/PDB needs to be re-started in Upgrade/Migrate mode to complete
patching. Please note that when there are a number of simultaneous
patch applications pending, datapatch will fail the upgrade mode pre-req
check if any of the pending patches requires upgrade mode startup. The
database being started in Upgrade mode will have no impact on other
patches being applied which do not require Upgrade mode.

Alternatively,
any patches not requiring upgrade mode can be installed by invoking
datapatch with the –apply < bug # > option
Check for existence of Apply/Rollback scripts:
Datapatch now
ensures that apply/rollback scripts are present for any pending patch
apply/rollback. If the files are not available datapatch will fail with
a pre-req error. For out of place rollback please copy any sqlpatch
files to the $OH in order to overcome this check and allow rollback to
proceed
Queryable Inventory connectivity :

Datapatch flags a
Queryable inventory error if for some reason Queryable Inventory returns
an error status. Further details can be derived by calling select
dbms_qopatch.get_pending_activity from dual and noting the error
message. Further details on resolving Queryable Inventory issues can be
found at Note 1530108.1, "Oracle Database 12.1 : FAQ on Queryable Patch Inventory"
DB connectivity: datapatch has a pre-req check to ensure DB connectivity.
Pre-req to ensure we can create directories under cfgtoollogs dir.

Symptoms

datapatch reports non-ignoreable errors were found during an apply or rollback. For example:

Errors during an apply

% datapatch -verbose

Connecting to database...OK
Determining current state...
Currently installed SQL Patches:
C patch 17027533 is a bundle patch and prereq for a later bundle
Currently installed C Patches: 17552800
Nothing to roll back
The following patches will be applied: 17552800
Adding patches to installation queue...
Installing patches...
Validating logfiles...
Patch 17552800 apply: WITH ERRORS

logfile:
/u01/app/oracle/product/12.1.0.1.0/sqlpatch/17552800/17552800_apply_BOD1211A_BOD1211A_2014Jan02_15_27_04.log
(no errors)
catbundle generate logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_GENERATE_2014Jan02_15_27_04.log
(no errors)
catbundle apply logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_APPLY_2014Jan02_15_27_05.log
(errors)
Error at line 12: SP2-0310: unable to open file "/u01/app/oracle/product/12.1.0.1.0/patch/scripts/bug16825779.sql"

See support note 1609718.1 for information on how to resolve
the installation attempts which had non ignorable errors.

SQL Patching tool complete on Thu Jan 2 15:27:09 2014

Additionally, the view dba_registry_sqlpatch will show "WITH ERRORS"

PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTION
---------- ---------- --------------- ------------------------------ --------------------
17552800 APPLY WITH ERRORS 02-JAN-14 03.27.09.020177 PM bundle:PSU

Errors during a rollback

Errors returned by datapatch

% datapatch -verbose

Connecting to database...OK
Determining current state...
Currently installed SQL Patches: 17552800
Currently installed C Patches:
The following patches will be rolled back: 17552800
Nothing to apply
Adding patches to installation queue...
Installing patches...
Validating logfiles...
Patch 17552800 rollback: WITH ERRORS

logfile:
/u01/app/oracle/product/12.1.0.1.0/sqlpatch/17552800/17552800_rollback_BOD1211A_BOD1211A_2013Dec31_19_15_54.log
(no errors)
catbundle generate logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_GENERATE_2013Dec31_19_15_54.log
(no errors)
catbundle rollback logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_ROLLBACK_2013Dec31_19_15_55.log
(errors)
Error at line 1595: ORA-02303: cannot drop or replace a type with type or table dependents

See support note 1609718.1 for information on how to resolve
the installation attempts which had non ignorable errors.

SQL Patching tool complete on Tue Dec 31 19:15:58 2013

Additionally, the view dba_registry_sqlpatch will show "WITH ERRORS"

PATCH_ID ACTION     STATUS          ACTION_TIME                    DESCRIPTION
---------- ---------- --------------- ------------------------------ --------------------
17552800 APPLY      SUCCESS         31-DEC-13 07.09.49.755424 PM   bundle:PSU
17552800 ROLLBACK   WITH ERRORS     31-DEC-13 07.15.58.746868 PM   bundle:PSU

Changes

Datapatch was executed to perform SQL patch apply or rollback

Cause

During the apply or rollback, datapatch found errors in a log file

Solution

Solution for errors during an apply

Fix the issue and rerun SQL apply

Resolve the issue causing the SQL error
Rerun datapatch

% datapatch -verbose

dba_registry_sqlpatch will now show:

PATCH_ID ACTION     STATUS          ACTION_TIME                    DESCRIPTION
---------- ---------- --------------- ------------------------------ --------------------
17552800 APPLY      WITH ERRORS     02-JAN-14 03.27.09.020177 PM   bundle:PSU
17552800 APPLY      SUCCESS         02-JAN-14 03.32.43.991067 PM   bundle:PSU

Rollback patch from ORACLE_HOME and rollback SQL from db (via force rollback)

NOTE: Should a patch be rolled back from the ORACLE_HOME having a dba_registry_sqlpatch STATUS of "WITH ERROR", datapatch will not re-execute the SQL necessary to rollback the SQL to the database unless the -force option is utilized. The datapatch output will be similar to:

Connecting to database...OK
Determining current state...
Currently installed SQL Patches:
Currently installed C Patches:
Nothing to roll back
Nothing to apply
SQL Patching tool complete on Thu Jan 2 15:52:32 2014

Rollback the patch from the ORACLE_HOME
Rerun datapatch with the force option

% datapatch -rollback < patch number > -force -verbose

The datapatch output will be similar to:

% datapatch -rollback 17552800 -force -verbose

Connecting to database...OK
Determining current state...
-force specified, skipping rollback list check...
The following patches will be rolled back: 17552800
Nothing to apply
Adding patches to installation queue...
Installing patches...
Validating logfiles...
Patch 17552800 rollback: SUCCESS

logfile:
/u01/app/oracle/product/12.1.0.1.0/sqlpatch/17552800/17552800_rollback_BOD1211A_BOD1211A_2014Jan02_15_53_18.log
(no errors)
catbundle generate logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_GENERATE_2014Jan02_15_53_18.log
(no errors)
catbundle rollback logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_ROLLBACK_2014Jan02_15_53_19.log
(no errors)
SQL Patching tool complete on Thu Jan 2 15:53:22 2014

dba_registry_sqlpatch will now show:

PATCH_ID ACTION     STATUS          ACTION_TIME                    DESCRIPTION
---------- ---------- --------------- ------------------------------ --------------------
17552800 APPLY      WITH ERRORS     02-JAN-14 03.46.47.835994 PM   bundle:PSU
17552800 ROLLBACK   SUCCESS         02-JAN-14 03.53.22.343742 PM   bundle:PSU

Solution for errors during a rollback

Fix the issue and rerun SQL rollback

Resolve the issue causing the SQL error
Rerun datapatch

% datapatch -verbose

dba_registry_sqlpatch will now show:

PATCH_ID ACTION     STATUS          ACTION_TIME                    DESCRIPTION
---------- ---------- --------------- ------------------------------ --------------------
17552800 APPLY      SUCCESS         31-DEC-13 07.09.49.755424 PM   bundle:PSU
17552800 ROLLBACK   WITH ERRORS     31-DEC-13 07.15.58.746868 PM   bundle:PSU
17552800 ROLLBACK   SUCCESS         31-DEC-13 08.58.14.104658 PM   bundle:PSU

Resolve the issue causing the SQL errors and reinstall the patch

NOTE: Should a patch be reinstalled to the
ORACLE_HOME having a dba_registry_sqlpatch STATUS of "WITH ERROR",
datapatch will not re-execute the SQL necessary to apply the SQL to the
database unless the -force option is utilized. The datapatch output
will be similar to:

Connecting to database...OK
Determining current state...
Currently installed SQL Patches: 17552800
C patch 17027533 is a bundle patch and prereq for a later bundle
Currently installed C Patches: 17552800
Nothing to roll back
Nothing to apply
SQL Patching tool complete on Tue Dec 31 20:33:49 2013

Resolve the issue causing the SQL error
Reinstall the patch to the ORACLE_HOME
Rerun datapatch with the force option

% datapatch -apply < patch number > -force -verbose

The datapatch output will be similar to:

% datapatch -apply 17552800 -force -verbose

Connecting to database...OK
Determining current state...
-force specified, skipping apply list check...
Nothing to roll back
The following patches will be applied: 17552800
Adding patches to installation queue...
Installing patches...
Validating logfiles...
Patch 17552800 apply: SUCCESS

logfile:
/u01/app/oracle/product/12.1.0.1.0/sqlpatch/17552800/17552800_apply_BOD1211A_BOD1211A_2013Dec31_20_36_28.log
(no errors)
catbundle generate logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_GENERATE_2013Dec31_20_36_28.log
(no errors)
catbundle apply logfile:
/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_BOD1211A_BOD1211A_APPLY_2013Dec31_20_36_29.log
(no errors)

dba_registry_sqlpatch will now show:

   PATCH_ID ACTION     STATUS          ACTION_TIME                    DESCRIPTION
---------- ---------- --------------- ------------------------------ --------------------
17552800 APPLY      SUCCESS         31-DEC-13 07.09.49.755424 PM   bundle:PSU
17552800 ROLLBACK   WITH ERRORS     31-DEC-13 07.15.58.746868 PM   bundle:PSU
17552800 APPLY      SUCCESS         31-DEC-13 08.36.33.355797 PM   bundle:PS