Wednesday 14 August 2013

Determining the Patch Set Update Version


Patch Set Updates are referenced by their 5-place version number. The following OPatch commands are a simple way to determine the latest PSU installed in the Oracle Home.

For Database PSUs, enter the following command:

opatch lsinventory -bugs_fixed | egrep -i 'PSU|DATABASE PATCH SET UPDATE'

A list will be output, and the highest release version reported is the current PSU installed. In the following example output, 11.2.0.2.6 is the installed Database PSU:

10248523   13696224  Fri Mar 30 14:00:25 EDT 2012   DATABASE PSU 11.2.0.2.1
11724916   13696224  Fri Mar 30 14:00:25 EDT 2012   DATABASE PSU 11.2.0.2.2 (INCLUDES CPUAPR2011)
12419331   13696224  Fri Mar 30 14:00:25 EDT 2012   DATABASE PSU 11.2.0.2.3 (INCLUDES CPUJUL2011)
12827726   13696224  Fri Mar 30 14:00:25 EDT 2012   DATABASE PSU 11.2.0.2.4 (INCLUDES CPUOCT2011)
13343424   13696224  Fri Mar 30 14:00:25 EDT 2012   DATABASE PATCH SET UPDATE 11.2.0.2.5 (INCLUDES CPUJAN2012)
13696224   13696224  Fri Mar 30 14:00:25 EDT 2012   DATABASE PATCH SET UPDATE 11.2.0.2.6 (INCLUDES CPUAPR2012)


For CRS (Cluster Ready Services) PSUs, enter the following command:

opatch lsinventory -bugs_fixed | grep -i 'TRACKING BUG' | grep -i 'PSU'


For GI (Grid Infrastructure) PSUs, enter the following command:

opatch lsinventory -bugs_fixed | grep -i 'GRID INFRASTRUCTURE PATCH SET UPDATE'

For Enterprise Manager Agent PSUs, enter the following command:
opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER' | grep -i 'AGENT'

For Enterprise Manager OMS PSUs, enter the following command:
opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER' | grep -i 'OMS'

For WebLogic Server PSUs, enter the following commands, where WL_HOME is the path of the WebLogic home:
$ . $WL_HOME/server/bin/SetWLSEnv.sh 
$ java weblogic.version
In the following example output, 10.3.4.0.1 is the installed WebLogic Server PSU.
WebLogic Server 10.3.4.0.1 PSU Patch for Bug11677325 Wed Feb 23 10:52:30 IST 2011
WebLogic Server 10.3.4.0  Fri Dec 17 20:47:33 PST 2010 1384255

Tuesday 13 August 2013

ORA-15097: cannot SHUTDOWN ASM instance with connected client.

While stopping the ASM instance , on any or all of the nodes (three nodes in my current environment), I got the the below error.


ORA-15097: cannot SHUTDOWN ASM instance with connected client.

I had to do a ps-ef | grep of the grid user, and kill (kill -9 ) a process that was taking up the connection to the ASM instance. I had to do this in all the three servers, then only could execute the below commands.



  export ORACLE_SID=+ASM1
sqlplus / as sysasm
shutdown immediate

  export ORACLE_SID=+ASM2
sqlplus / as sysasm
shutdown immediate

  export ORACLE_SID=+ASM3
sqlplus / as sysasm
shutdown immediate

De-installation of Oracle Grid Software 11g R2

After the installation of Oracle Grid Infrastructure ASM, it was decided that we need to change the location of the Grid Base Location and Grid Home location (due to various issues).
So, as briefly said below, we had to UNINSTALL ORACLE GRID INFRASTRUCTURE SOFTWARE. 

Environment

This is 11.2.0.1 ASM 11gR2 in Oracle Enterprise Linux 5.8

We have 3 node RAC.

SCAN address is configured in RAC.

So, after stopping the database, and cleanly removing the database, we now need to uninstall ASM CLEANLY.

So, went to Grid software Home, 

e.g. $GRID_HOME/deinstall
then ran
./deinstall

This will take some time to run, as it will connect to all the other nodes as well. It will then come to a stage, where it will ask to run some scripts, The first script will have to be run from the third node, the second node, and then the last script will have to be run from the primary node, i.e. this node where we are executing the deinstall command Those scripts will have to be executed as root user or a user with administrative privileges.

After the script is run, there will not be any processes running from the secondary nodes.

Come back to the primary node, and complete the activity, which will also delete the Oracle Homes and do a clean deletion of the Grid Base location, the Grid Home location and the Oracle Inventory in all the three nodes.


In short, one command ./deinstall (and 3 subcommands) will deinstall everything, and remove all the binaries, and will also remove the Oracle Inventory. Clever !!

I rebooted the 3 servers as well, just for a satisfaction of my mind.

Now, off for the installation of Grid Software.

Wednesday 17 July 2013

Unable to flush restart file in aiurof()

While doing adpatch for HRMS RUP 6 patch (16000686) in R12.1.3 instance, I could see that the patch session was frozen.

I tried the usual checks on the database node to see if there was any space issue or if the archive log destination is full, but there were no problem from that side. 

I then looked at adctrl

Adctrl was bringing the below.

Enter your choice [1] :


        Control
Worker  Code      Context            Filename                    Status
------  --------  -----------------  --------------------------  --------------
     1  Run       AutoPatch R120                                 Wait
     2  Run       AutoPatch R120                                 Wait
     3  Run       AutoPatch R120                                 Wait
     4  Run       AutoPatch R120                                 Wait
     5  Run       AutoPatch R120                                 Wait
     6  Run       AutoPatch R120                                 Wait
     7  Run       AutoPatch R120                                 Wait
     8  Run       AutoPatch R120                                 Wait
     9  Run       AutoPatch R120                                 Wait
    10  Run       AutoPatch R120                                 Wait
    11  Run       AutoPatch R120                                 Wait
    12  Run       AutoPatch R120                                 Wait
    13  Run       AutoPatch R120                                 Wait
    14  Run       AutoPatch R120                                 Wait
    15  Run       AutoPatch R120                                 Wait
    16  Run       AutoPatch R120                                 Wait
    17  Run       AutoPatch R120                                 Wait
    18  Run       AutoPatch R120                                 Wait
    19  Run       AutoPatch R120                                 Wait
    20  Run       AutoPatch R120                                 Wait
    21  Run       AutoPatch R120                                 Wait
    22  Run       AutoPatch R120                                 Wait
    23  Run       AutoPatch R120                                 Wait
    24  Run       AutoPatch R120                                 Wait
    25  Run       AutoPatch R120                                 Wait
    26  Run       AutoPatch R120                                 Wait
    27  Run       AutoPatch R120                                 Wait
    28  Run       AutoPatch R120                                 Wait
    29  Run       AutoPatch R120                                 Wait
    30  Run       AutoPatch R120                                 Wait
    31  Run       AutoPatch R120                                 Wait
    32  Run       AutoPatch R120     gh52doup.pkb                Assigned



I looked at what is there in the worker 32.
cd $APPL_TOP/admin/$TWO_TASK/log

tail -100 adwork032.log | more

Below is the log.

Time when worker started job: Wed Jul 17 2013 17:16:01


Start time for file is: Wed Jul 17 2013 17:16:01

package /u01/PATC/oracle/apps/apps_st/appl/ben/12.0.0/patch/115/sql/bendsgel.pkb

Done deactivating PL/SQL compilation.

Statement executed.

Done resetting PL/SQL compilation.


Time when worker completed job: Wed Jul 17 2013 17:16:01


Time when worker started job: Wed Jul 17 2013 17:16:02

AD Worker error:
Unable to flush restart file in aiurof()

Error opening worker restart file in aijwro().

Time when worker failed: Wed Jul 17 2013 17:16:02

AD Worker is complete.

Errors and warnings are listed in the log file
/u01/PATC/oracle/apps/apps_st/appl/admin/EBSPATCH/log/adwork032.log

and in other log files in the same directory.



------------------- 

So, I went back to adctrl, and did the command to restart the worker 32. 


                    AD Controller Menu
     ---------------------------------------------------

     1.    Show worker status

     2.    Tell worker to restart a failed job

     3.    Tell worker to quit

     4.    Tell manager that a worker failed its job

     5.    Tell manager that a worker acknowledges quit

     6.    Restart a worker on the current machine

     7.    Exit







Enter your choice [1] : 6


You chose 6 - Restart a worker on the current machine.
Before you enter the worker number below, make sure that there is no
worker process already running with that worker number.

Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : 32

Control code changed to 'Startup' for worker 32.
Review the messages above, then press [Return] to continue.

                    AD Controller Menu
     ---------------------------------------------------

     1.    Show worker status

     2.    Tell worker to restart a failed job

     3.    Tell worker to quit

     4.    Tell manager that a worker failed its job

     5.    Tell manager that a worker acknowledges quit

     6.    Restart a worker on the current machine

     7.    Exit







Enter your choice [1] :


        Control
Worker  Code      Context            Filename                    Status
------  --------  -----------------  --------------------------  --------------
     1  Run       AutoPatch R120                                 Wait
     2  Run       AutoPatch R120                                 Wait
     3  Run       AutoPatch R120                                 Wait
     4  Run       AutoPatch R120                                 Wait
     5  Run       AutoPatch R120                                 Wait
     6  Run       AutoPatch R120                                 Wait
     7  Run       AutoPatch R120                                 Wait
     8  Run       AutoPatch R120                                 Wait
     9  Run       AutoPatch R120                                 Wait
    10  Run       AutoPatch R120                                 Wait
    11  Run       AutoPatch R120                                 Wait
    12  Run       AutoPatch R120                                 Wait
    13  Run       AutoPatch R120                                 Wait
    14  Run       AutoPatch R120                                 Wait
    15  Run       AutoPatch R120                                 Wait
    16  Run       AutoPatch R120                                 Wait
    17  Run       AutoPatch R120                                 Wait
    18  Run       AutoPatch R120                                 Wait
    19  Run       AutoPatch R120                                 Wait
    20  Run       AutoPatch R120                                 Wait
    21  Run       AutoPatch R120                                 Wait
    22  Run       AutoPatch R120                                 Wait
    23  Run       AutoPatch R120                                 Wait
    24  Run       AutoPatch R120                                 Wait
    25  Run       AutoPatch R120                                 Wait
    26  Run       AutoPatch R120                                 Wait
    27  Run       AutoPatch R120                                 Wait
    28  Run       AutoPatch R120                                 Wait
    29  Run       AutoPatch R120                                 Wait
    30  Run       AutoPatch R120                                 Wait
    31  Run       AutoPatch R120                                 Wait
    32  Run       AutoPatch R120     gh52doup.pkb                Running

Review the messages above, then press [Return] to continue.



After that was done, the patch session started resuming normally.