ADOP Prepare phase fails while cloning Run Context file to refresh Patch context file

Jan

ADOP Prepare phase fails while cloning Run Context file to refresh Patch context file

Author: Shakeeb Petkar | Category: EBS 12.2.x, TroubleshootingComments Off | Tags: AC-00005, apps, ebs, error, failed, fs_clone, oracle, permission, R12

Issue:

While running adop phase=prepare, it fails on one node while trying to clone Run Context file to refresh Patch context file.

Error:

It fails with the following error in the ADOP Session.
ADOP prepare phase fails with [UNEXPECTED] occurred during CONFIG_CLONE Patch File System from Run File System

The log file has the following error:

ERROR: Could not clone the Run context file due to the following exception:
AC-00005: No write permissions for creating the Context file - /tmp/temp.xml
Raised by oracle.apps.ad.context.AppsContext
ERROR: Failed to clone Run Context file to refresh Patch context file

applpreprod@ebsapp01:~$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Initializing.
Run Edition context : /u01/PREPROD/fs1/inst/apps/PREPROD_ebsapp01/appl/admin/PREPROD_ebsapp01.xml
Patch edition context: /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/appl/admin/PREPROD_ebsapp01.xml
Patch file system free space: 83.74 GB

Validating system setup.

Node registry is valid.
Log: /u01/PREPROD/fs1/inst/apps/PREPROD_ebsapp01/logs/appl/rgf/TXK/verifyssh.log
Output: /u01/PREPROD/fs1/inst/apps/PREPROD_ebsapp01/logs/appl/rgf/TXK/out.xml
Remote execution is operational.

Checking for existing adop sessions.
Continuing with existing session [Session ID: 56].
Session Id : 56
Prepare phase status : NOT COMPLETED
Apply phase status : NOT COMPLETED
Cutover phase status : NOT COMPLETED
Abort phase status : NOT COMPLETED
Session status : FAILED

===========================================================================
ADOP (C.Delta.7)
Session ID: 56
Node: ebsapp01
Phase: prepare
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/adop_20161116_111849.log
===========================================================================

Verifying existence of context files in database.

Checking if adop can continue with available nodes in the configuration.
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01
Returning SUCCESS

Validating configuration on admin node: [ebsapp01].
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01/remote_execution_result_20161116_112021.xml
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01
Returning SUCCESS

Validating configuration on node(s): [ebsapp02,ebsdmz01 and ebsdmz02].
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01/remote_execution_result_20161116_112323.xml
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01
Returning SUCCESS

Verifying data dictionary.

Creating list of nodes where prepare phase needs to be run.
The prepare phase needs to be run on node: ebsapp01
The prepare phase needs to be run on node: ebsapp02
The prepare phase needs to be run on node: ebsdmz01
The prepare phase needs to be run on node: ebsdmz02

Running prepare phase on node(s): [ebsapp01,ebsapp02,ebsdmz01 and ebsdmz02].
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01/remote_execution_result_20161116_112704.xml
[ERROR] adop phase=prepare failed on Node: "ebsdmz02"
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01
Returning SUCCESS

Stopping services on patch file system.
Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10

Stopping WLS Admin Server...
Refer /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at /u01/PREPROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PPD/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/logs/appl/admin/log/adadminsrvctl.txt for more information ...

Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

The Node Manager is already shutdown

NodeManager log is located at /u01/PREPROD/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/logs/appl/admin/log/adnodemgrctl.txt for more information ...

Generating node-specific status report.
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_111849/PREPROD_ebsapp01/adzdnodestat.out

Summary report for current adop session -
Node ebsdmz02: FAILED
- Prepare status: NOT COMPLETED
[UNEXPECTED] Error occurred running "perl /u01/PREPROD/fs1/EBSapps/appl/ad/12.0.0/patch
/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/PREPROD/fs1/i
nst/apps/PREPROD_ebsdmz02/appl/admin/PREPROD_ebsdmz02.xml -patchcontextf
ile=/u01/PREPROD/fs2/inst/apps/PREPROD_ebsdmz02/appl/admin/PREPROD_ebsdmz02.
xml -promptmsg=hide -console=off -mode=migrate -sessionid=56 -timestam
p=20161116_112707 -outdir=/u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_2
0161116_112707/PREPROD_ebsdmz02"
[UNEXPECTED] occurred during CONFIG_CLONE Patch File System from Run File System, r
unning command: "perl /u01/PREPROD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bi
n/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/PREPROD/fs1/inst/app
s/PREPROD_ebsdmz02/appl/admin/PREPROD_ebsdmz02.xml -patchcontextfile=/u0
1/PPD/fs2/inst/apps/PREPROD_ebsdmz02/appl/admin/PREPROD_ebsdmz02.xml -pr
omptmsg=hide -console=off -mode=migrate -sessionid=56 -timestamp=20161
116_112707 -outdir=/u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116
_112707/PREPROD_ebsdmz02".
[UNEXPECTED] fs_clone has failed.
[UNEXPECTED] Error calling runPendingConfigClone subroutine.
[ERROR] Prepare phase has failed.
Node ebsdmz01: COMPLETED
- Prepare status: COMPLETED
Node ebsapp02: COMPLETED
- Prepare status: COMPLETED
Node ebsapp01: COMPLETED
- Prepare status: COMPLETED
For more details, run the command: adop -status -detail

adop exiting with status = 0 (Success)

applpreprod@ebsapp01:~$ adop -status

Enter the APPS password:

==============================================================
ADOP (C.Delta.7)
Session Id: 56
Command: status
Output: /u01/PREPROD/fs_ne/EBSapps/log/status_20161116_124859/adzdshowstatus.out
===============================================================

Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
ebsapp01 master PREPARE COMPLETED 2016/11/16 11:28:06 2016/11/16 12:20:52 0:52:46
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

ebsapp02 slave PREPARE COMPLETED 2016/11/16 11:28:10 2016/11/16 12:32:25 1:04:15
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

ebsdmz01 slave PREPARE COMPLETED 2016/11/16 11:28:06 2016/11/16 12:30:05 1:01:59
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

ebsdmz02 slave PREPARE FAILED 2016/11/16 11:28:10 2016/11/16 12:17:43 0:49:33
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

File System Synchronization Type: None

adop exiting with status = 0 (Success)

The prepare phase fails on the 4th node i.e ebsdmz02

The logfile shows the following error:

/u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_112707/PREPROD_ebsdmz02/TXK_SYNC_migrate/cloneContext_apply/FSCloneApplyAppsTier_11161217.log

WARNING:
.end std out.
cp: cannot create /tmp/temp.xml: Permission denied

.end err out.

ERROR: Could not clone the Run context file due to the following exception:
AC-00005: No write permissions for creating the Context file - /tmp/temp.xml
Raised by oracle.apps.ad.context.AppsContext

ERROR: Failed to clone Run Context file to refresh Patch context file

Solution:

1) Check if the APPS user has write permissions on the /tmp directory.

Example :

As root:
# chmod 777 /tmp

# ls -al | grep tmp
2) Check if you have a temp.xml file or other RapidClone files under either of those directories owned by a different user? If so please delete them and retry adcfgclone.pl.

root@ebsdmz02:~# cd /tmp root@ebsdmz02:/tmp# ls -ltr *.xml -rw-r--r-- 1 root root 170336 Oct 25 07:20 temp.xml -rw-r--r-- 1 root root 109409 Oct 25 07:20 PREPROD_ebsdmz02.xml

We see that there are files with root user which have to deleted.

Delete these files.

root@ebsdmz02:/tmp# rm temp.xml
root@ebsdmz02:/tmp# rm PREPROD_ebsdmz02.xml

root@ebsdmz02:/tmp# ls -ltr *.xml
*.xml: No such file or directory
root@ebsdmz02:/tmp#

Run adop phase=prepare again

applpreprod@ebsapp01:~$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials.

Validating system setup.

Checking for existing adop sessions.
Continuing with existing session [Session ID: 56].

===========================================================================
ADOP (C.Delta.7)
Session ID: 56
Node: ebsapp01
Phase: prepare
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/adop_20161116_164144.log
===========================================================================

Verifying existence of context files in database.

Checking if adop can continue with available nodes in the configuration.
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_164144/PREPROD_ebsapp01
Returning SUCCESS

Skipping configuration validation on admin node: [ebsapp01]

Validating configuration on node(s): [ebsdmz02].
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_164144/PREPROD_ebsapp01/remote_execution_result_20161116_164318.xml
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_164144/PREPROD_ebsapp01
Returning SUCCESS

Verifying data dictionary.

Creating list of nodes where prepare phase needs to be run.
The prepare phase has already been run on node: ebsapp01
The prepare phase has already been run on node: ebsapp02
The prepare phase has already been run on node: ebsdmz01
The prepare phase needs to be run on node: ebsdmz02

Starting admin server on patch file system.

Running prepare phase on node(s): [ebsdmz02].
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_164144/PREPROD_ebsapp01/remote_execution_result_20161116_164837.xml
Log: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_164144/PREPROD_ebsapp01
Returning SUCCESS

Stopping services on patch file system.
Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10

Stopping WLS Admin Server...
Refer /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at /u01/PREPROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PPD/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/logs/appl/admin/log/adadminsrvctl.txt for more information ...

Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

NodeManager log is located at /u01/PREPROD/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 0

adnodemgrctl.sh: check the logfile /u01/PREPROD/fs2/inst/apps/PREPROD_ebsapp01/logs/appl/admin/log/adnodemgrctl.txt for more information ...

Generating node-specific status report.
Output: /u01/PREPROD/fs_ne/EBSapps/log/adop/56/prepare_20161116_164144/PREPROD_ebsapp01/adzdnodestat.out

Summary report for current adop session -
Node ebsdmz02: COMPLETED
- Prepare status: COMPLETED
Node ebsdmz01: COMPLETED
- Prepare status: COMPLETED
Node ebsapp02: COMPLETED
- Prepare status: COMPLETED
Node ebsapp01: COMPLETED
- Prepare status: COMPLETED
For more details, run the command: adop -status -detail

adop exiting with status = 0 (Success)

applpreprod@ebsapp01:~$ adop -status

Enter the APPS password:

==============================================================
ADOP (C.Delta.7)
Session Id: 56
Command: status
Output: /u01/PREPROD/fs_ne/EBSapps/log/status_20161116_170920/adzdshowstatus.out
===============================================================

ebsapp02 slave PREPARE COMPLETED 2016/11/16 11:28:10 2016/11/16 12:32:25 1:04:15
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

ebsdmz01 slave PREPARE COMPLETED 2016/11/16 11:28:06 2016/11/16 12:30:05 1:01:59
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

ebsdmz02 slave PREPARE COMPLETED 2016/11/16 11:28:10 2016/11/16 17:05:47 5:37:37
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED

File System Synchronization Type: None

adop exiting with status = 0 (Success)

DBA Arena

Share More | Learn More