STEPS TO troubleshoot long running concurrent request in R12.2.x

USUALLY we are getting concurrent program running longer and concurrent request not picking jobs, etc. those are very frequent issue we expected from development team.

Collect basis level of information to development team

A. Oracle seeded program/Custom program?

B. How much time it used earlier?

C. Is there any recent code change done in concurrent program?

D. Is this program fetching higher data compare to last run?

E. Does this job running any specific time/ It can be run any time?

F. Does this job fetching data using DB link?

G. Does the problem happen on both the test and production instance?

Once asked above questions to development team in meanwhile we need to start basic sanity check in our environment.

A. Verify the status of the concurrent program

select REQUEST_ID,phase_code,status_code,ORACLE_SESSION_ID from apps.fnd_concurrent_requests where request_id=’23432345’;

B. Find which concurrent manger ran a specific concurrent request

select b.USER_CONCURRENT_QUEUE_NAME from fnd_concurrent_processes a,fnd_concurrent_queues_vl b, fnd_concurrent_requests c where a.CONCURRENT_QUEUE_ID = b.CONCURRENT_QUEUE_ID and a.CONCURRENT_PROCESS_ID = c.controlling_manager and c.request_id ='65434543';

C. Verify the actual and target of the CM

D. Verify the concurrent manager status

E. Find SID for the concurrent request use below query

set lines 200

set pages 200

col USERNAME for a10

col MODULE for a25

col ACTION for a25

col PROGRAM for a18

SELECT a.request_id, d.sid, d.serial# ,d.osuser,d.process , c.SPID,d.EVENT FROM apps.fnd_concurrent_requests a, apps.fnd_concurrent_processes b, v$process c, v$session d WHERE a.controlling_manager = b.concurrent_process_id AND c.pid = b.oracle_process_id AND b.session_id=d.audsid AND a.request_id = &Request_ID AND a.phase_code = 'R';

F. Find any blocking session for concurrent request use below query and verify concurrent program SID

select (select username from v$session where sid=a.sid) blocker, a.sid, ' is blocking ' "IS BLOCKING", (select username from v$session where sid=b.sid) blockee,b.sid from v$lock a, v$lock b where a.block = 1 and b.request > 0 and a.id1 = b.id1 and a.id2 = b.id2;

If any blocking session there use below query to find the concurrent request id 

SELECT s.inst_id,a.request_id,s.sid,s.serial#,c.spid FROM apps.fnd_concurrent_requests a, gv$process c, gv$session s WHERE s.sid in ('&sid') AND s.paddr = c.addr

AND a.oracle_process_id = c.spid AND a.phase_code = UPPER ('R');

G. Find client identifier and module, action for concurrent request

H. Then check with module owner with the concurrent request

select sid,serial#,USERNAME,CLIENT_IDENTIFIER,status,LAST_CALL_ET/60,SQL_HASH_VALUE,module,action,program from v$session where sid='&n';

I. Find which sql query is running and inform to module owner

select sql_text from v$sql where hash_value='7654345333';

J. Find stale value of the tables which is used by the concurrent program

select table_name, stale_stats, last_analyzed from dba_tab_statistics where stale_stats='YES';

K. You may have to run the gather stats against those tables which are having stale value.

L. Use OEM and monitor session ID

M. If development team ask tkprof,AWR,ASH report please generate and analysis.