Showing posts with label OEM. Show all posts

Thursday, February 14, 2013

EM12c Startup & Shutdown scripts

Hi,

Here below are the 2 scripts for bouncing process of EM12c Server including repository services and Databases.

Slightly hardcoded for Agent/OMS Locations.

Stop12c.sh

export ORACLE_SID=em12c

. /home/oracle/em12c.env
echo ""
echo "Stopping EM Agent..."
echo ""
/e01/app/oracle/Middleware/agent/agent_inst/bin/emctl stop agent
echo ""
echo "Stopped EM Agent..."

echo ""
echo "Stopping OMS Services..."
echo ""
/e01/app/oracle/Middleware/oms/bin/emctl stop oms -all
echo ""
echo "Stopped OMS Services..."

echo ""
echo "Stopping Database Listener..."
echo ""
$ORACLE_HOME/bin/lsnrctl stop
echo ""
echo "Stopped Database Listener..."

echo ""
echo "Shuting down EM respository database..."
echo ""
sqlplus "/ as sysdba" << EOF
shutdown immediate;
exit;
echo ""
echo "EM respository database is down now..."
EOF

echo ""
echo "12c Grid Shutdown is Complete..."
echo ""

Start12c.sh

export ORACLE_SID=em12c

. /home/oracle/em12c.env

echo ""

echo "Starting EM respository database..."

echo ""

sqlplus "/ as sysdba" << EOF

startup;

exit;
echo ""

echo "EM respository database is UP..."

EOF

echo ""

echo "Starting Database Listener..."

echo ""

$ORACLE_HOME/bin/lsnrctl start

echo ""
echo "Database Listener is UP..."

echo ""

echo "Starting OMS Services..."

echo ""

/e01/app/oracle/Middleware/oms/bin/emctl start oms

echo ""
echo "OMS Services are UP..."

echo ""

echo "Starting EM Agent..."

echo ""

/e01/app/oracle/Middleware/agent/agent_inst/bin/emctl start agent

echo ""
echo "EM Agent is UP..."

echo ""

echo "Grid Startup is Complete..."

echo ""

Sunday, November 18, 2012

EM Incident: Critical:New: - Row(1): Supplied date is in the future : now =

Here the below alerts/Incidents will be coming to the EM Users once after the DST Time Change.

Ex:
Host=HOST.domain.com
Target type=Database Instance
Target name=ABCDB
Incident creation time=Nov 15, 2012 5:21:01 AM PST
Last updated time=Nov 15, 2012 7:21:05 PM PST
Message= Row(1): Supplied date is in the future : now = Thu Nov 15 19:21:01 PST 2012 supplied value = Thu Nov 15 20:20:25 PST 2012Severity=Critical
Incident ID=41705
Event count=1
Incident Status=New
Escalation level=2
Priority=None
Incident owner=
Incident Acknowledged By Owner=No
Categories=Error
Rule Name=Rule set for non production group targets,Rule to notify team about metric evaluation error
Rule Owner=SYSMAN
Update Details:
Escalation level set to 2 by rule (Name = Rule set for non production group targets,
Rule to notify team about metric evaluation error; Owner = SYSMAN).

Solution:

Oracle says this is a known behavior in instances that cant handle DST change automatically. Please do the below steps on all agents that see this problem

./emctl stop agent

./emctl clearstate agent

./emctl start agent

./emctl upload agent

This should help you to get rid of the problem

perl: warning: Falling back to the standard locale ("C") while running emctl status agent

Here Perl executable are throwing warning alerts as below

[oracle@<HOST>:/u01/app/oracle/linux/<HOST>/agent12c/agent_inst/bin]$./emctl status agentperl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "en_US.UTF-8",
        LC_ALL = "ISO-8859-1",
        LANG = "ISO-8859-1"    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version : 12.1.0.1.0
Agent Home        : /u01/app/oracle/linux/<HOST>/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/linux/<HOST>/agent12c/core/12.1.0.1.0
Agent Process ID : 5203
Parent Process ID : 5151
Agent URL         : https://<HOST>.domain.com:3872/emd/main/
Repository URL    : https://<OMS_HOST>.domain.com:4900/empbs/upload
Started at        : 2012-11-17 19:39:34
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2012-11-17 19:40:33
Last attempted upload                        : 2012-11-17 19:40:33
Total Megabytes of XML files uploaded so far : 0.05
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 85.29%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2012-11-17 19:40:06
Last successful heartbeat to OMS             : 2012-11-17 19:40:06
---------------------------------------------------------------
Agent is Running and Ready

Solution:

Now check for the bash_profile and see for the entry
vi .bash_profile
export LC_CTYPE=en_US.UTF-8
export LC_ALL=en_US.UTF-8

If the above are not set, please set it and run the bash_profile and re-check the status of the EM Agent.
It should hlep to resolve the perl issue.

[oracle@<HOST>:~]$. .bash_profile

[oracle@<HOST>:~]$ cd /u01/app/oracle/linux/<HOST>/agent12c/agent_inst/bin

[oracle@<HOST>:/u01/app/oracle/linux/<HOST>/agent12c/agent_inst/bin]$./emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version : 12.1.0.1.0
Agent Home        : /u01/app/oracle/linux/<HOST>/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/linux/<HOST>/agent12c/core/12.1.0.1.0
Agent Process ID : 5203
Parent Process ID : 5151
Agent URL         : https://<HOST>.domain.com:3872/emd/main/
Repository URL    : https://<OMS_HOST>.domain.com:4900/empbs/upload
Started at        : 2012-11-17 19:39:34
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2012-11-17 19:42:37
Last attempted upload                        : 2012-11-17 19:42:37
Total Megabytes of XML files uploaded so far : 0.05
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 85.29%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2012-11-17 19:42:06
Last successful heartbeat to OMS             : 2012-11-17 19:42:06
---------------------------------------------------------------
Agent is Running and Ready

Friday, November 16, 2012

EM Incident: Warning:New: - Agent resident memory utilization in KB is 251,848

If an EM Agent consuming more memory on the server, please do the below work around to resolve the issue.

1. ./emctl stop agent

Ensure no EMAgent process is running with the following commands:

<AGENT_HOME>/bin>./emctl status agent

$ps -ef | grep TMMain
$ps -ef | grep emagent
If you see any process, please kill them.

2. Take a Backup and edit the .../agent_inst/sysman/config/emd.properties file:

from:
agentJavaDefines=-Xmx128M -XX:MaxPermSize=96M

to:
agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M

and save the file.

3. Start the Agent and check the status:

<AGENT_HOME>/bin>./emctl start agent
<AGENT_HOME>/bin>./emctl status agent

It should Fix the issue.

If still does not resolve the issue please change the below Open Files parameter value.

-- Check the Open files value

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 774874
max locked memory       (kbytes, -l) 74136141
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 131072
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

--To set the Open files value
$ ulimit -n 65536

-- Recheck the Open files value
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 774874
max locked memory       (kbytes, -l) 74136141
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 131072
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Wednesday, October 31, 2012

'emctl status agent' Command Shows "Collection Status Disabled By Upload Manager"

./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home        : /fg01/app/linux/hostname/agent11g
Agent binaries    : /fg01/app/linux/hostname/agent11g
Agent Process ID : 27736
Parent Process ID : 27698
Agent URL         : https://domain.com:3872/emd/main/
Repository URL    : https://domain.com:4900/em/upload
Started at        : 2012-10-30 20:52:45
Started by user   : oracle
Last Reload       : 2012-10-30 21:14:01
Last successful upload                       : 2012-10-31 06:55:18
Total Megabytes of XML files uploaded so far :     8.91
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :     4.39%
Collection Status                            : Disabled by Upload Manager
Last successful heartbeat to OMS             : 2012-10-30 19:05:14
---------------------------------------------------------------
Agent is Running and Ready

./emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: Upload was successful but collections currently disabled - disk full

Solution:
-----------

<AGENT_HOME>/bin/emctl stop agent
<AGENT_HOME>/bin/emctl clearstate agent

cd <AGENT_STATE>/sysman/emd/upload

ls *.xml | wc -l

- Delete any pending upload files from the agent home

rm -r <AGENT_STATE>/sysman/emd/upload/*
rm <AGENT_STATE>/sysman/emd/lastupld.xml





After deleting the files from upload re-start the agent and verify, if still did not help, please process the below.




- If needed, the values of the following parameters can be increased in the emd.properties file:







vi $AGENT_HOME/sysman/config/emd.properites





Below are the default values




UploadFileSize=2048 

UploadMaxBytesXML=50 

UploadMaxDiskUsedPct=98



Increase the above values as




UploadMaxBytesXML=60

UploadMaxDiskUsedPct=99 (Default value is 98)




 
<AGENT_HOME>/bin/emctl start agent 

<AGENT_HOME>/bin/emctl status agent

./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /fg01/app/linux/hostname/agent11g

Agent binaries    : /fg01/app/linux/hostname/agent11g

 Agent Process ID  : 27736
Parent Process ID : 27698
Agent URL         : https://domain.com:3872/emd/main/
Repository URL    : https://domain.com:4900/em/upload
Started at        : 2012-10-30 20:52:45
Started by user   : oracle
Last Reload       : 2012-10-30 21:14:01
Last successful upload                       : 2012-10-31 06:55:18
Total Megabytes of XML files uploaded so far :     8.91
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :     4.39%
Last successful heartbeat to OMS             : 2012-10-31 07:04:08
---------------------------------------------------------------
Agent is Running and Ready

  

<AGENT_HOME>/bin/emctl upload agent 

./emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

Thursday, August 23, 2012

EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

Due to communication between OMS and agent does not work. This is in most cases because something has been changed to one of the components so the SSL certificate is not valid anymore. Therefore you have to resecure the agent. Your last try shows the error that you have used the wrong password for registration
Then go to the agent and do the following:
- emctl stop agent
- delete all files in
$AGENT_HOME/sysman/emd
rm agntstmp.txt
$AGENT_HOME/sysman/emd/upload and
rm -rf *
$AGENT_HOME/sysman/emd/state
rm -rf *
- emctl clearstate agent (this should delete all state files but sometimes it is not enough, but I experienced that it is better to use this command after deletion of the files)
- emctl secure agent (and specify the password on demand)
- emctl start agent

This should solve your problem.

Incase if you still have any issues in starting the agent or uploading files is not getting processed then look into the below:

Login to EM Console --> Setup -->Agents -->

Misconfiguration Alerts and Blocked Agents

and check whether the server is blocked or not if it is blocked unblock it and then try the below:

./emctl clearstate agent
./emctl upload agent
./emctl status agent

Saturday, April 16, 2011

Monitoring a Grid Control Installation

http://www.colestock.com/blogs/labels/Grid%20Control.html

How to: Installing 10g Grid Control

Posted at Thursday, February 08, 2007

A how-to guide I drafted on installing and configuring Oracle's new 10g Grid Control product

How to Install Enterprise Manager 10g (Grid Control) - 10.2.0.4 - on Linux

http://www.colestock.com/blogs/2008/04/how-to-install-enterprise-manager-10g.html

agentTZRegion

Question: I installed agent on one of my server but when I start it I get the following error messages:
replica6-replica:</u01/oracle/product/agent/agent10g/bin> $emctl start agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Starting agent ...... failed.
The agentTZRegion value in /u01/oracle/product/agent/agent10g/sysman/config/emd.properties is not in agreement with what agent thinks it should be.Please verify your environment to
make sure that TZ setting has not changed since the last start of the agent.
If you modified the timezone setting in the environment, please stop the agent and
exectute 'emctl resetTZ agent' and also execute the script
mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>) to get the value
propagated to repository.
Consult the log files in: /u01/oracle/product/agent/agent10g/sysman/log

And when I try to reset the time zone, I get the following error message:
replica6-replica:</u01/oracle/product/agent/agent10g/bin> $emctl resetTZ agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Updating /u01/oracle/product/agent/agent10g/sysman/config/emd.properties...
----- Tue May 8 11:33:29 2007::tzOffset for US/East-Indiana is -300(min), but agent is runnning with tzOffset -240(min)
----- Tue May 8 11:33:29 2007::trying again after waiting for 1 sec to account for daylight transition
----- Tue May 8 11:33:29 2007::tzOffset for US/East-Indiana is -300(min), but agent is runnning with tzOffset -240(min)
-----
resetTZ failed.
The agentTZRegion in:
/u01/oracle/product/agent/agent10g/sysman/config/emd.properties
is not in agreement with what the agent thinks it should be.
Fix your environment.
Pick a TZ value that corresponds to time zone settings listed
in: /u01/oracle/product/agent/agent10g/sysman/admin/supportedtzs.lst

Please help.

Answer:Your agent is not running with the correct time zone or it is not matching what you have in OMS repository.

Please do the following:
Go to the OMS repository and login as sysman and query the following SQL statement.
SQL> SELECT target_name FROM mgmt_targets;
Find exact sever name and then query the following SQL statement.
SQL> SELECT timezone_region FROM mgmt_targets
SQL> WHERE target_name like '<your server-name>';

TIMEZONE_REGION
----------------------------------------------------------------
Etc/GMT+4

No, you should manually change the time zone in the emd.properties file in the sysman/config directory. To do that following the following steps:

1. Stop the agent
# cd $AGENT_HOME
# cd bin
# emctl stop agent
2. Open the File emd.properties, put the agentTZRegion=Etc/GMT+4 and save the file.

List of Oracle Time Zone# cd $AGENT_HOME
# cd sysman/config
# vi emd.properties -- add or change the time zone that you found from the OMS repository.
All the timezone are in the following place.
# cd sysman/admin
nsupportedtzs.lst
supportedtzs.lst
tzmappings.lst
3 . Now try to start the agent.
# cd $AGENT_HOME
# cd bin
# emctl start agent
-------------- OR --------------------

1. Stop the agent
# cd $AGENT_HOME
# cd bin
# emctl stop agent
2. Open the File emd.properties, make a note of agentTZRegion.

3. The go to the OMS repository and change the timezone to what you have in the emd.properties file.
# sqlplus /nolog
# SQL> connect sysman/<your password>
# SQL> exec mgmt_target.set_agent_tzrgn
# SQL> ('machine name :port number ','timezone that you found in emd.properties');If you don’t know the port number, you should find it in the emd.properties file.

4 . Now try to start the agent.
# cd $AGENT_HOME
# cd bin
# emctl start agent

If you still getting the same error message:

Do the following:

Set the TZ to your-em-time-zone

Ex: # TZ=GMT; export TZ

Go to agent home/bin

Ex: cd $AGENT_HOME/bin

Reset time zone:

Ex: # emctl resetTZ agent

Oracle Enterprise Manager 10g Release 10.2.0.1.0.

Updating /u01/oracle/product/agent2/ agent10g/sysman/config/emd. properties...

Successfully updated /u01/oracle/product/agent2/ agent10g/sysman/config/emd. properties.

exec mgmt_target.set_agent_tzrgn(' yourservername:3872','GMT')

This can be done for example by logging into sqlplus and doing

SQL> exec mgmt_target.set_agent_tzrgn(' yourservername:3872','GMT')

oradb1t-serv62: /u01/oracle/product/agent2/ agent10g/bin $emctl start agent

Oracle Enterprise Manager 10g Release 10.2.0.1.0.

Starting agent ............. started.

Then go to emd.properties to make sure the agentTZRegion parameter was changed.

Ex: # more $AGENT_HOME/sysman/config/ emd.properties

You should see: agentTZRegion=GMT

Go to sqlplus and login as sysman in em repository and executed the above procedure.

Ex: # sqlplus /nolog

# SQL> connect sysman/password

# SQL> exec mgmt_target.set_agent_tzrgn(' yourservername:3872','GMT');

Upload agent to OEM.

Ex: # emctl clearstate agent

# emctl upload agent