Behind the scenes 2. - the 12.2 online patching

Not advertised "features" of the online patching function

One of the most advertised new feature of the 12.2 system is the new online patching function. The Oracle's marketing was that this new feature will help to cut down the scheduled maintenance outages of an EBS system.

But what are the not publicated, not advertised "(dis)advantages" of this feature?

1. new patch tool
2. doubled/tripled file system
3. new "patch top" directory
4. new patching lifecycle
5. new restrictions
6. new multi node patch procedure
7. online patching db tools
8. FNDLOAD and Online Patching
9. Cloning and Online patching
10. Directory Structure and Online Patching

New patch tool

There is new utility for patching in a 12.2 system, the adop utility (AD Online Patching). The dminstrators has to use this tool not only for applying, but also for handling the whole patching lifecycle. The old adpatch tool is still available, but it is not recommend for calling directly the adpatch tool, the adop tool still use this patch during patch apply phase. (If you call the adpatch tool directly - without a special paramater - it will respond a "do not use directly this tool" message)

So what tasks handle the new adop utility? (Cite from the Maintenance Guide)

• Reads patch metadata to determine patch dependencies and requirements
• Uploads patch information from a prior patch session to the database (if applicable)
• Reads and validate the patch driver file and reads the product driver files
• Compares version numbers of object modules from the product libraries and version numbers of the existing files against the patch files
• Backs up all existing files that will be changed by the patch
• Copies files
• Archive files in libraries
• Relinks executables
• Generates forms, reports, messages, graphics, and Java archive (JAR) files
• Compiles JSP files and invalid database objects
• Updates database objects
• Runs AutoConfig to update configuration files if any template files are introduced or updated by the patch
• Saves patch information to the database

The adop utility requirements: 

• has to be called from the actual run edition system, but it will patching the patch edition system (until cutover phase).
• at least the administration server has to be in runned state at the run edition system. 
• the adop command starter user has to know the apps, the system user password and the weblogic admin user password
• the administrator has to be familiar in handling the patching phases

The adop utility "extra features"

• without explicitly parameterized the utilty searching for unzipped patch-s in the main 'patch_top' directory
• most of the patch phases takes more time then what was the "marketing" expected earlier
• will not ask for parameters, most of his parameter has to be specific at the command line (for example if you not specify the worker number, the tool automatically set  it based on the actual CPU core number)
• good feature that the administrator could use an input file for parameterizing the adop tool. (you could set patch number, custom "patch top", workers number and so on)
• could automatically merge patches if you gave him more than one patch number and set merge parameter to yes.
• NLS patches should apply with these example parameter: 'patches=222_HU :u222.drv' where HU is an example for language code.
• could restart or abandon a previously failed patch applying process.
• has got a special log directory structure under NE file system, check 2-24 page of the maitenance guide. (sadly not under $LOG_HOME)
• has got a non-interactive patching feature. Fullfill an input_file file and the adop will not ask for any parameter.
• has got a hotpatch mode. In this case the toll will apply the patch on the actual run edition system.

Doubled/tripled file system

Because of the online patching feature the whole application tier file stucture has been changed by Oracle developers. The APPL_TOP, COMMON_TOP has been doubled and a new NE file system has been created for common, or non online patchable components.

Please check my previous post "Behind the scenes 1." for some other details.

New "patch top" directory

A new "patch top" directory has been introduced in 12.2 file system. If you do not specify the patch top directory at every adop running you should unzip patches into fs_ne/EBSapps/patches ($PATCH_TOP) directory. 

New patching lifecycle

The online patching cycle consists of a number of high level phases:

1. prepare
2. apply
3. finalize
4. cutover
5. cleanup

And two special phases

1. abort
2. fs_clone

What the maintenance guide says about these phases

prepare - Prepares the environment for patching.
apply - Applies the patch the environment.
finalize - Performs the final steps in patch application. Can be run at any point after prepare and before cutover. Facilitates verification of customizations or other objects that may be affected by a patch.
cutover - Performs the actions needed to make the former patch edition the new run edition, and the former run edition the new patch edition. Cutover also prepares the environment for users, and is the only phase that involves a (brief) period of downtime.
cleanup - Drops now-redundant columns and other objects.
abort - Abandons the patch application process and restores the previous state of the environment. Cannot be specified with any other phase.
fs_clone - Used for file system cloning. This special phase must be run from the run file system, and cannot be specified with any other phase.

And a good diagram for the phases from the same document

And my comments based on some use experience of the adop utility

1. The prepare phase takes me about 2 hour, using one CPU core, memory and many IO operation during runnig the prepare adop command. During the prepration there are some strange patch synchronization steps.
2. A patch apply takes more time than earlier because of the new extra function of the adop utility
3. The cutover phase is not so fast too. For example it is doing autoconfigs, for changing the port on the patch edition into run edition port, doing some file synchronization, checking, starting the whole new run editioned application tier - it doesn't matter if the original was shutdowned - and so on, 
4. Earlier if you apply a small EBS patch you could be ready with it on live system within 15 minutes. Now you couldn't because you have to go through all patching lifecycle, through every phases. (except if you use hotpatch of course)
5. The administrators, developeres has to relearn, rethink their own patching, development procedures.
6. Is it possible to turn off this new online patching feature? :)

New restrictions during a patching cycle

Cite from the maintenance guide:

During an online patching cycle, some interesting product restrictions will apply.

Subledger Accounting:

• Users will not be able to Validate the Application Accounting definitions.

Order Management:

• Creation of a new Defaulting Condition in the Attribute Defaulting Rules form is disabled, unless the same seeded condition already exists for a given attribute.

For other product specific restrictions please check the maintenance guide.

New multi node patch procedure

Setup ssh for application tier nodes

If you are using the typical configuration of a multi-node application tier, and want to take advantage of adop remote invocation, you must set up Secure Shell (ssh)  on all your application tier nodes. Check maintenance guide for more details.

Addition steps for every phase when maintaining a multi-node system

There are extra information at every phase for a multi node enviroment. The document does not contain an independent chapter that collect all mulit node information, so you should carefully read every phase description and search for "multi-node" phrase.

Fortunately the document contain a good example scanarios for adop arguments what when has to use at what situation on a multi node system. Check the 3-28 page in the document.

Online patching db tools

There are many many new scripts and features for maintaining online patching feature in the database.

When you are installing the 12.2.2 system before applying the 12.2.2 patch you should enable the online patching feature in the database. (I don't know why is it not enabled by default in new, empty system...) During this enablement you could meet with many, many SQL script for maintaining, monitoring and reporting the DB part of the online patching feature. For more information first read the "Enabling Online Patching" chapter in "Oracle® E-Business Suite Upgrade Guide"

FNDLOAD and Online Patching

What the documentation said about this topic? You could find the above in the Setup Guide

"Each seed data table is modified to support editioned storage of seed data. A new edition name column is added so that Run Edition data and Patch Edition data can be stored in the same table. A maintenance trigger is added to populate the column, as well as a VPD filter policy to select the correct data set per edition. During online patching, seed data changes from the Run Edition will be copied to the Patch Edition using a forward cross edition trigger. After patch cutover, obsolete data editions and code will be deleted."

Cloning and Online patching

There are some good relevant information in the documentations about this connection.

"Be aware that standard cloning is different from file system cloning. Standard cloning is creating a copy of an Oracle E-Business Suite system using adclone. For example, cloning a complete Oracle E-Business Suite system to create a test copy from your production environment. In contrast, file system cloning is copying the run file system to the patch file system in Online Patching, and can only be undertaken with the adop phase=fs_clone command."

"Before cloning a system, be sure to complete any in-progress online patching cycle all the way through to cleanup. Then run fs_clone to synchronize with the other file system, so that synchronization does not have to be performed in the next patching cycle."

Directory Structure and Online Patching

Cite from the Setup Guide.

"In Release 12.2, you can set the base directory to any desired location. However, the subdirectory structure cannot be changed because of dependencies on both the WLS domain and the dual file system required for online patching. Also, the
base directory must be the same across all nodes in multi-node configurations."