Location of Different Logfiles in Exadata Environment

On the cell nodes

================

1. Cell alert.log file

/opt/oracle/cell/log/diag/asm/cell/{node name}/trace/alert.log

or

if the CELLTRACE parameter is set just do cd $CELLTRACE

2. MS logfile

/opt/oracle/cell/log/diag/asm/cell/{node name}/trace/ms-odl.log.
or

if the CELLTRACE parameter is set just do cd $CELLTRACE

3. OS watcher output data

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :
cd /opt/oracle.oswatcher/osw/archive
find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} \; 

where 12- year 01- Month 13-day

4. Os message logfile

/var/log/messages

5. VM Core files

/var/log/oracle/crashfiles 

More details can be found in the following note:

Where / How to find OS crashcore file in Exadata Systems [Linux] (Doc ID 1389225.1)

6. SunDiag output files.

/tmp/sundiag_.tar.bz2

7. Cell patching issues related logfiles:   

/var/log/cellos

The major logfile of patch application output you will find in the db node from where you are patching in the location /tmp/<cell version>/patchmgr.stdout and patchmgr.err

8. Disk controller firmware logs:

/opt/MegaRAID/MegaCli/MegaCli64 -fwtermlog -dsply -a0

On the Database nodes

=====================

1. Database alert.log

$ORACLE_BASE/diag/rdbms/{DBNAME}/{sid}/trace/alert_{sid}.log
Ex: /u01/app/oracle/diag/rdbms/dbfs/DBFS2/trace/alert_DBFS2.log

2. ASM alert.log

$ORACLE_BASE/diag/asm/+asm/+ASM{instance number}/trace/ alert_+ASM {instance number}.log
Ex: /u01/app/oracle/diag/asm/+asm/+ASM2/trace/alert_+ASM2.log

3. Clusterware CRS alert.log

$GRID_HOME/log/{node name}/alert{node name}.log

Ex: /u01/app/11.2.0/grid/log/dmorldb02/alertdmorldb02.log

4. Diskmon logfiles

$GRID_HOME/log/{node name}/diskmon/diskmon.lo*
Ex: /u01/app/11.2.0/grid/log/dmorldb02/diskmon/diskmon.log

5. OS Watcher output files

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :
cd /opt/oracle.oswatcher/osw/archive
find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} \; 

where 12- year 01- Month 13-day

6. Os message logfile

/var/log/messages

7. VM Core files for Linux

/u01/crashfiles  

More details can be found in the following note:

Where / How to find OS crashcore file in Exadata Systems [Linux] (Doc ID 1389225.1)

8. Disk controller firmware logs:
    

/opt/MegaRAID/MegaCli/MegaCli64 -fwtermlog -dsply -a0

Workflow Directory Services User/Role Validation

This is often experienced in Oracle Applications 11i/R12 that user has been assigned a responsibility but not able to see it while accessing it. However , everything seems fine with responsibility ,it is not end dated and looks good. To overcome this issue , there is a concurrent request that does sync up of such users and responsibility in 11i version.

Login as System Administrator --> Submit Concurrent Program ""Workflow Directory Services User/Role Validation' with parameters 10000, yes, yes

This request would check all users and assigned responsibilities and should sync up users with attached responsibilities , users should be able to view assigned responsibility now.

p_BatchSize – 10000 (Default Value 10000)
p_Check_Dangling – Yes (Default value No)
Add missing user/role assignments – Yes (Default Value No)
Update WHO columns in WF tables – No (Default Value No)

Form process is not getting cleared and occupying /tmp space

LSOF:

lsof is a command meaning "list open files", which is used in many Unix-like systems to report a list of all open files and the processes that opened them. This open source utility was developed and supported by Victor A. Abell, the retired Associate Director of the Purdue University Computing Center. It works in and supports several Unix flavors.

Open files in the system include disk files, pipes, network sockets and devices opened by all processes. One use for this command is when a disk cannot be unmounted because (unspecified) files are in use. The listing of open files can be consulted (suitably filtered if necessary) to identify the process that is using the files.


# lsof /var
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
syslogd     350     root    5w  VREG  222,5        0 440818 /var/adm/messages
syslogd     350     root    6w  VREG  222,5   339098   6248 /var/log/syslog
cron        353     root  cwd   VDIR  222,5      512 254550 /var -- atjobs

To view the port associated with a daemon:

# lsof -i -n -P | grep sendmail
sendmail  31649    root    4u  IPv4 521738       TCP *:25 (LISTEN)

From the above one can see that "sendmail" is listening on its standard port of "25".
-i Lists IP sockets.
-n Do not resolve hostnames (no DNS).
-P Do not resolve port names (list port number instead of its name).
One can also list Unix Sockets by using lsof -U.

The /tmp directory keeps filling up but the space reported with user space tools ("du" for example) is next to nothing.

According to the standard system tools:


Digging deeper with "lsof" we see the following:


[root@mail tmp]# lsof | grep "/tmp"
bash       1970    root  cwd       DIR        9,3       4096          2 /tmp
screen    13507   jgray    3r     FIFO        9,3                 15747 /tmp/uscreens/S-jgray/13507.pts-0.mail
perl      19932  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
perl      19932  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    4w      REG        9,3 1884374605         26 /tmp/zmlogger.out (deleted)
mysqld_sa 21321  zimbra    1w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21321  zimbra    2w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
logswatch 21391  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
logswatch 21391  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
mysqld    21402  zimbra    5u      REG        9,3          0         20 /tmp/ibD4jO0l (deleted)
mysqld    21402  zimbra    6u      REG        9,3          0         21 /tmp/ibzy1fOB (deleted)
mysqld    21402  zimbra    7u      REG        9,3          0         22 /tmp/ibXukIBR (deleted)
mysqld    21402  zimbra    8u      REG        9,3          0         23 /tmp/ibJj1dq7 (deleted)
mysqld    21402  zimbra   12u      REG        9,3          0         25 /tmp/iblBE0Vn (deleted)
perl      21423  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
perl      21423  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out

Regards,
Jaagadish.