this is not a raw technical discussion of how to install oracle patch sets or how to circumvent known patch set bugs. this is a run down presentation of my most annoying experience of installing oracle software lately. an expression of anger about intransparent documentation structures and versioning, permanent manual intervention, enormous effort in time and cost and, finally, inefficiency. but let’s see what happened.
i recently installed the so called Release 10.1.0.5 Patch 28 for Microsoft Windows (32-Bit) on a lab- and a couple of production-machines. so far, everyone knows that installing oracle patches is not an easy job. you need to figure out what patch to install actually and download the patch for a dedicated database and operating system software version. mostly, the actual patch, to be applied properly, requires other patches to be applied a priori and a posteriori. in the case of the mentioned patch, which is a bundle of interim patches, even the patch installer OPatch had to be patched in advance. but that’s not the whole story. wrapped within the newer OPatch oracle automatically distributes OCM, the oracle configuration manager.
to sum it up i had to take regard of the following patches, in that order:
- p6880880_101000_WINNT – installs a newer OPatch and hooks in OCM
- p5746866_10105_GENERIC – fixes timezone files
- p7367493_10105_WINNT – my actual intend
applying this series of patches may be executed in two ways, depending on whether you already know about the hidden OCM-introduction.
the way of the unaware
for the unaware of us, OCM will, for the first time, jingle in step #2, p5746866_10105_GENERIC, just after executing opatch apply. my first thought: oh my dear, an installation dialogue? OCM, yes i know about that but i’m doing OPatch here etc. consulted the patch set readme’s again, nothing. went to metalink and studied the OPatch dedicated note 293369.1, nothing. finally explored the contents of p6880880_101000_WINNT, being installed in step #1, to find out about my astonishment (...\ocm\ccr\README.TXT) and to figure out that there is a way to silently install OCM (...\ocm\ccr\bin\emocmrsp.bat) using a response file:
This is the README for OPatch (22.214.171.124.59), the Oracle Interim Patching Tool. OPatch is Oracle’s only supported method of installing Interim patches. It updates the central and per-product inventories with the details of the patch. OPatch is used for patching Oracle software. If you have an older version of opatch it is strongly recommended to back it up.
This patch installs the “OPatch” utility and the Oracle Configuration Manager (OCM) version 10.3.0.
This latest version of OPatch also includes the opportunity to register and upload configuration details with OCM. This configuration information allows customers to leverage the personalized, proactive features offered by Oracle Software Configuration Manager (SCM) in MetaLink, and it can help Support solve customer issues more efficiently.
Note: OCM must be configured in order to be used. There are two ways to configure OCM with OPatch. The first one is through interactive mode by following the instructions during the command line invocation of OPatch. The second is by providing a response file for OCM when invoking OPatch. For more details,Refer to Oracle® Configuration Manager Installation and Administration Guide Version 10.3 available at link: http://www.oracle.com/technology/documentation/ocm.html
up to this stage i already spent more than two hours constructing the ordered list of patches above and developing a dedicated script of patch/shell commands and database bounces. it took me another good hour to get comfortable with the creation and the employment of the OCM installation response file, excavating the -ocmrf option of opatch apply, and prove testing this procedure.
nearly four hours had been passed, lunch time now and not even finished the first test case execution in the lab.
the way of the aware
ok, being illuminated about OPatch and OCM, i restarted the whole procedure like this, completing step #1, p6880880_101000_WINNT:
# have a fresh shell window and switch to the patch base dir cd /D E:\temp\10g-1.x.x-WinNT-OPatch-126.96.36.199.59\p6880880_101000_WINNT\ # set the oracle home and the opatch path set ORACLE_HOME=e:\oracle\product\10.1.0\db_1 set path=e:\oracle\product\10.1.0\db_1\OPatch;%path% # check the current opatch version (should emit somethimg like: "OPatch succeeded.") # currently we have "OPatch Version: 188.8.131.52.53" opatch version # back up the current opatch installation, make clean, install xcopy %ORACLE_HOME%\OPatch %ORACLE_HOME%\OPatch.backup /I/E/V/Y rmdir /S/Q %ORACLE_HOME%\OPatch xcopy OPatch %ORACLE_HOME%\OPatch /I/E/V/Y</pre> <pre># again check the current opatch version (should emit somethimg like: "OPatch succeeded.") # now we have "OPatch Version: 184.108.40.206.59" opatch version
then, on the updated OPatch, i created the above mentioned response file for the OCM installation upon the next opatch apply:
# create a response file for the forced ocm-installation with patch 5746866 below # the response file creation dialog goes like this: # <--- # Provide your email address to be informed of security issues, install and # initiate Oracle Configuration Manager. Easier for you if you use your MetaLink # Email address/User Name. # Visit http://www.oracle.com/support/policies.html for details. # Email address/User Name: ... # Provide your MetaLink password to receive security updates via your MetaLink account. # Password (optional): ... # ---> # iff there is no internet connection to oracle ... this follows: # <--- # Unable to establish a network connection to Oracle. If your systems require a # proxy server for outbound Internet connections, enter the proxy server details # in this format: # [ <proxy-user>@] <proxy-host>[: <proxy-port>] # If you want to remain uninformed of critical security issues in your # configuration, enter NONE # ---> %ORACLE_HOME%\OPatch\ocm\bin\emocmrsp.bat move ocm.rsp E:\temp\ocm.rsp
now, step #2, p5746866_10105_GENERIC, was due to be executed again:
# have a fresh shell window and switch to the patch base dir cd /D e:\temp\10g-1.0.5-WinNT-Patch-028\p5746866_10105_GENERIC\5746866\ # set the oracle home and the opatch path set ORACLE_HOME=e:\oracle\product\10.1.0\db_1 set path=e:\oracle\product\10.1.0\db_1\OPatch;%path% set OPATCH_PLATFORM_ID=0 # stop all oracle relevant nt services (best all instances in order) net stop "OracleServiceMYSERVICE" net stop "OracleOraDb10g_home1TNSListener" net stop "Distributed Transaction Coordinator" # test opatch (should emit somethimg like: "OPatch succeeded.") opatch lsinventory # install the interim patch (should emit somethimg like: "OPatch succeeded.") opatch apply -ocmrf E:\temp\ocm.rsp # restart all oracle relevant nt services (best all instances in reverse order) net start "Distributed Transaction Coordinator" net start "OracleOraDb10g_home1TNSListener" net start "OracleServiceMYSERVICE" # run the script installed by the patch and check for an (hopefully) empty # result set when selecting from the created temp table sqlplus "/ as sysdba" @e:\oracle\product\10.1.0\db_1\RDBMS\ADMIN\utltzuv2.sql select * from sys_tzuv2_temptab; exit
done on that. however, not really, because the (silent) OCM installation left OCM in a so called connected mode. that is, OCM phones home somehow, even initially. can’t be true? have a look at this screenshot:
puuhh, the overall OCM switch off script may look like this:
# - clear all existing diagnostic upload files # - disable automatic download and installation of ocm software updates # - stop automatic collection and upload of data to an oracle "repository" # - stop the ocm service (or whatever actually runs in the back) # - switch the ocm-installation to "disconnected mode" %ORACLE_HOME%\ccr\bin\emCCR clear -diagnostic %ORACLE_HOME%\ccr\bin\emCCR automatic_update off %ORACLE_HOME%\ccr\bin\emCCR hold %ORACLE_HOME%\ccr\bin\configCCR -d
step #3, p7367493_10105_WINNT, was in line yet, which was my actual intend (up to six hours passed, the first lab execution still on the go). do you already guess, that step #3 will also fail in some way? ok, you’re right, it was:
- pending file locks within %ORACLE_BIN% that demanded a host reboot with services being set to manual in advance,
- manual population of %ORACLE_HOME%\bundle\Patch28 because opatch apply just forgot about it …
not left to be mentioned is the bothersome view recompilation that requested another two instance bounces.
# have a fresh shell window and switch to the patch base dir cd /D e:\temp\10g-1.0.5-WinNT-Patch-028\p7367493_10105_WINNT\7367493\ # set the oracle home, the opatch and the perl path set ORACLE_HOME=e:\oracle\product\10.1.0\db_1 set path=e:\oracle\product\10.1.0\db_1\OPatch;%path% set PERL5LIB=e:\oracle\product\10.1.0\db_1\perl\5.6.1\lib;%PERL5LIB% set PATH=e:\oracle\product\10.1.0\db_1\perl\5.6.1\bin\MSWin32-x86;%PATH% # stop all oracle relevant nt services (best all instances in order) net stop "OracleServiceMYSERVICE" net stop "OracleOraDb10g_home1TNSListener" net stop "Distributed Transaction Coordinator" # test opatch (should emit somethimg like: "OPatch succeeded.") opatch lsinventory # install the interim patch (should emit somethimg like: "OPatch succeeded.") # this step probably fails with file copy errors due to persisting file-locks # set all services to manual startup and reboot the machine before continuing # say Y with this dialogue, then repeat the step # Replying 'Y' will terminate the patch installation immediately. It WILL NOT # restore any updates that have been performed to this point. It WILL NOT update # the inventory. # Replying 'N' will update the inventory showing the patch has been applied. # NOTE: After replying either 'Y' or 'N' it is critical to review: # Metalink Note 312767.1 How to rollback a failed Interim patch installation. # Do you want to STOP? # Please respond Y|N > # Y # File Patching Error! # ERROR: OPatch failed during patching, possibly due to missing files. # OPatch returns with error code = 200 opatch apply # run the remove_demo.js script to remove vulnerable Oracle HTTP Server demos cscript //nologo remove_demo.js # restart all oracle relevant nt services (best all instances in reverse order) net start "Distributed Transaction Coordinator" net start "OracleOraDb10g_home1TNSListener" net start "OracleServiceMYSERVICE" # the migrate script may not find itself where expected, so help it along the way # but, however, check first using > dir %ORACLE_HOME%\bundle xcopy .\files\bundle %ORACLE_HOME%\bundle /I/E/V/Y # run the migrate script cd /D %ORACLE_HOME%\bundle\Patch28 sqlplus "/ as sysdba" shutdown immediate; startup migrate; @catcpu.sql shutdown immediate; startup; # review d:\oracle\product\10.1.0\db_1\Bundle\Patch28\Apply_<ORACLE_SID>_<timestamp>.log # for "ora-*" errors (ignore any ORA-02303 errors) # recompile all invalid objects @e:\oracle\product\10.1.0\db_1\rdbms\admin\utlprp.sql 0 # to check for invalid objects, execute the following statement: SELECT OBJECT_NAME FROM DBA_OBJECTS WHERE STATUS = 'INVALID'; exit # check whether view recompilation (whatever this means) has already been performed # if the view recompilation has not been performed, this statement returns no rows. cd /D %ORACLE_HOME%\bundle\view_recompile sqlplus "/ as sysdba" SELECT * FROM registry$history where ID = '6452863'; exit # iff recompilation iss nessecary ... sqlplus "/ as sysdba" @recompile_precheck_jan2008cpu.sql shutdown immediate; startup upgrade; @view_recompile_jan2008cpu.sql shutdown immediate; startup; alter package XDB.DBMS_XDBZ0 compile; exit # review %ORACLE_HOME%\bundle\view_recompile\vcomp_<ORACLE_SID>_<timestamp>.log # for "ora-*" errors (ignore any ORA-02303 errors) # recompile all invalid objects sqlplus "/ as sysdba" @e:\oracle\product\10.1.0\db_1\rdbms\admin\utlprp.sql 0 # to check for invalid objects and the view recompilation, execute the following statements: SELECT OBJECT_NAME FROM DBA_OBJECTS WHERE STATUS = 'INVALID'; SELECT * FROM registry$history where ID = '6452863'; exit
… at the end of the day
this is in fact literally spoken. this patch installation course took nine instance bounces and one host cold reboot. i needed to crawl a multitude of documents and web sites (beside the oracle one’s, also). i was forced to inspect the installation source code to get an idea of what can be achieved and how.
and for the curious: the latest download of OPatch, which features version 220.127.116.11.60 for oracle 10gR1 on windows, no longer contains a trace of a hidden OCM installation within (...\ocm\ccr\README.TXT). what the heck is going on here?
frustration and anger!
- sven vetter of trivadis also was quite surprised but took it just easy
- the ioug sap sig guys were not impressed either and discouraged installing OCM at all