|May the source be with you, but remember the KISS principle ;-)|
|Contents||Bulletin||Scripting in shell and Perl||Network troubleshooting||History||Humor|
|News||HP Operations Manager||Recommended Links||Reference||HPOM Documentation||Troubleshooting HPOM agents||Typical operations|
|Listing nodes||Adding nodes||Removing nodes||Agent patching||Starting and stopping HPOM agent||Node IP address change||opcmon|
|Installation using ssh passwordless login||Installation from local files||Local installation of the HPOM agent version 11||Deinstallation of agent||History||Humor||Support Forum|
Note: HP renamed the product called now HP operations manager way too many times. Also it is very inconsistent with using abbreviations. Here we will assume that the term "HP Operations manager" and abbreviations HPOM, OMU, and OVO mean the same thing :-)
HPOM agents are complex (multicomponent) and thus extremely unreliable with some components that tend to fail spontaneously. For each 20 servers with HPOM typically there is at least one failure in 24 hours. That means for 200 servers you have around a dozen of such problem in 24 hours (same days more, some days less). This is a major problem with HPOM. If agent health is monitored, helpdesk tickets about them became major nuisance for system administrators and source of well-deserved hate of HPOM.
|Agents are extremely unreliable with some components that tend to fail spontaneously. For each 20 servers with HPOM typically there is at least one failure in 24 hours. This is a major problem with HPOM. If agent health is monitored, helpdesk tickets about them became major nuisance for system administrators and source of well-deserved hate of HPOM.|
HPOM 9 agents use HTTPS for communication which provide secure communication between agent and the "mothership". Message format based on XML. Managed nodes can be identified by their unique OvCoreID and not necessarily by their IP addresses.
The architecture of the agent as described in HTTPS Agent Concepts and Configuration Guide Software Version: 9.01 (Figure 1-3, p. 31) looks pretty similar to classic Tivoli agent.
The first agent installation is the creation of agent on the management server. This way the server becomes the first managed node. After the server is installed and configured, agents on other servers can be installed from the management server semi-automatically (ssh connection is needed).
The main difference is that Tivoli agent is just one process and HPOM agent consists of several processes. Also some tasks that in Tivoli were performed by server are delegated to the agent level. In other works HPOM has more powerful, much more heavyweight agent then Tivoli. Due to that the agent that has potential of causing many troubles during installation (troubles with certificates are pretty common during installation), but after it is installed and configured is runs pretty reliably with almost no other problem then when one of multiple processes die (restart in this case almost always cure the problem and can be done automatically). Port 383 is used for communication of HTTP agent with the server.
Port 383 is used for communication with the server
Like in Tivoli, the HPOM agent by default runs as root on UNIX, but if necessary can be configured to run under a regular account with less capabilities.
Running of the daemons is controlled by two RC scripts. On Linux the following startup scripts are used:
HTTPS agents on Linux kernel 2.6 require the standard C++ library (libstdc++.so). There are two main versions of HTTPS agents used with HPOM 9:
Version 11.0x. Later version substantially different from 8.6. Can be used with SLES 11 SP1 and with some tricks with SLES 11 SP2 (which have linux kernel version 3 that agent does not recognize but that can be fixed by manual modification of one of the script on the installation disk (or ISO file). Assuming that agent ISO disribution is mounted to /tmp/ISO) this is the script: /tmp/ISO/packages/LIN/Linux2.6_X64/oareqcheck.cfg. See SLES 11 SP2 installation trick for details
Managed nodes must have a valid, industry standard, X509 certificate issued by the HP Certificate Server to be able to communicate with HP Operations management servers. Certificates, signed by 1024 bit keys, are required to identify managed nodes in a managed environment using the Secure Socket Layer (SSL) protocol. The “SSL handshake” between two managed nodes only succeeds if the issuing authority of the certificate presented by the incoming managed node is a trusted authority of the receiving managed node. The main communication security components responsible for creating and managing certificates are:
HP Certificate Server
HP Key Store
HP Certificate Client
The files associated with the HTTPS agent are found in the following directory structures by default:
-rwxr-xr-x 1 root bin 12966 Sep 3 13:06 cpu_mon.sh* -rwxr-xr-x 1 root bin 13216 Sep 3 13:06 disk_mon.sh* -rwxr-xr-x 1 root bin 10556 Sep 3 13:06 dist_mon.sh* -rwxr-xr-x 1 root bin 9538 Aug 5 13:57 libsis2om.pm* -rwxr-xr-x 1 root bin 12611 Sep 3 13:06 mailq_l.sh* -rwxr-xr-x 1 root bin 41912 Sep 3 13:06 mondbfile.sh* -rwxr-xr-x 1 root bin 352 Aug 5 13:57 OM-SiSAlert* -rwxr-xr-x 1 root bin 366 Aug 5 13:57 OM-SiSAlert_full* -rwxr-xr-x 1 root bin 12515 Sep 3 13:06 proc_mon.sh* -rwxr-xr-x 1 root bin 886 Aug 5 13:57 sis2om_perl.bat* -rwxr-xr-x 1 root bin 13995 Aug 5 13:57 sis2om.pl* -rwxr-xr-x 1 root bin 96162 Aug 5 13:57 sis2om_samples_95* -rwxr-xr-x 1 root bin 14487 Aug 5 13:57 sis2om-setup.pl* -rwxr-xr-x 1 root bin 937 Aug 5 13:57 sisconfigdir* -rwxr-xr-x 1 root bin 2065 Aug 5 13:57 sis_control.pl* -rwxr-xr-x 1 root bin 26062 Aug 5 13:57 sis_disc.pl* -rwxr-xr-x 1 root bin 12615 Sep 3 13:06 swap_mon.sh* -rwxr-xr-x 1 root bin 27672 Aug 5 13:57 TreePP.pm* -rwxr-xr-x 1 root bin 12330 Sep 3 13:06 vp_chk.sh*
HTTPS Communication can be controlled using the following commands.
The opcragt utility is used to control agents from the HP Operations Management server. The operations includes:
There is a wrapper called opcagt on HTTPS nodes. This utility can be used to perform remote control tasks by application launch from the operator's desktop. It allows to setup a common action definition for any kind of managed nodes. Along with starting and stopping agent they can be used to verify the status of the agent
/opt/OV/bin/OpC/opcagt -status coda OV Performance Core COREXT (1603) Running opcacta OVO Action Agent AGENT,EA (1486) Running opcmsga OVO Message Agent AGENT,EA (1487) Running opcmsgi OVO Message Interceptor AGENT,EA (1605) Running
The Document KM793358 will show you the list of all nodes in a layout group, however it is required to specify the node hierarchy as a parameter.
If node hierarchy is not specified it takes "Node Bank" as default, but those nodes which are not in the Node Bank are not listed.
Attached is an SQL Query which lists all nodes under all layout groups irrespective of the node hierarchy.
IT Resource Center forums
Aug 31, 2009
HPOM 8.33 HPUX 11.31 IA Server with latest patches (Server and Agents) effective 2009-08.
Updated Linux agent from
Operations Agent 08.11.000 to
Operations Agent 08.60.005
Just prior to update
ovbbccb -verbose -status omuserver.org
succeeds with no errors.
Post agent install (no install errors reported)
root@agent> /root #
ovbbccb -verbose -status omuserver.org
NOTE: Sending status request to: 'https://omuserver.org:383/
ERROR: (bbc-303) An exception occurred while querying the server
'pluto.dpsk12.org'. Exception message: (xpl-117) Timeout occurred
while waiting for data..
On the omu server executes with no errors:
ovbbccb -verbose -status agent
NOTE: Sending status request to: 'https://agent:383/Hewlett-Packard/OpenView/BBC/status/'.
(Namespace, Port, Bind Address, Open Sockets)
<default> 383 ANY 1
HP OpenView HTTP Communication Incoming Connections
BBC 06.10.205; ovbbccb 06.10.205
From HPOM server to agent:
ovbbccb -ping agent (from omu server):
agent: status=eServiceOK coreID=b0407b96-5198-7517-0e49-8d6c1910f2e7 bbcV=06.20.050 appN=ovbbccb appV=06.20.050
conn=1 time=73 ms
From Agent to Server:
ovbbccb -ping pluto
omuserver: status=eServiceOK coreID=401c626c-6433-7537-14e4-94396352425a
bbcV=06.10.205 appN=ovbbccb appV=06.10.205 conn=39 time=77 ms
Try the following on linux node.
1) Check whether any errors logged in System.txt file
2) try ovc -kill
3) Remove the ovbbccb.dat and queue files.
4) Restart the agents using ovc -start
5) Check ovbbccb -status
8.60 is a new & different animal. It is a huge "jump" HPOM agents wise. So, I suggest that you read the 8.6 Release notes. I've just have an (unpleasant) experience, where installation of 8.6 failed, but just going back to 8.53, successful without a single problem.
Ramkumar, thanks for the suggestions...
>>>1) Check whether any errors logged in System.txt file >>>2) try ovc -kill >>>3) Remove the ovbbccb.dat and queue files. >>>4) Restart the agents using ovc -start >>>5) Check ovbbccb -status
I have repeatedly tried ovc -kill and removed the queue files to no avail.
I tried also removing the ovbbccb.dat flie with similar unsucessfull results.
The only thing of interest in the System.txt file is a repeated instance of: "... Unknown monitor 'DBSPI-0088'... " "... Unknown monitor 'DBSPI-0086'... "
I've now determined that this not only applies to linux but HPUX agents updated (at HP Support request mind you) to the latest version 8.60.005. This behavior of breaking things that work just because Support demands you run the latest patches needs to stop. HP needs to provide support for EXISTING releases not simply the most current.
I now will probably have to remove my 8.60.005 patch on the management server and push the older agent 8.53.xx to all nodes.
What a P I T A!!!
As mentioned in my previous reply, the update to the latest patch level was requested by HP support while troubleshooting other issues.
The resulting update causing additional completely unrelated problems is something rather typical of HP. I will likely have to remove the agent patches installed on the HPOM server and the redeploy the agent software to all those ugents recently updated (LINUX and HPUX).
Its experiences like this that make me so darn frustrated at HP.
Jason: Thanks for the response however, I'm not finding the 8.60 Agent Release Notes. I have read the Patch Description and nothing seems to stand out.
Can you post the release notes or provide a pointer?
OML9.01_Linux_HTTPSAgent.pdf - KM772798
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.
ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least
Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info|
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: July 04, 2015