Operations Manager Data Warehouse High CPU Utilization

Some good steps if you are having trouble with CPU on your DW.

http://www.afinn.net/2008/10/operations-manager-data-warehouse-high-cpu-utilization/

Data Warehouse Queries for Noisy Rules

Here are a few SQL queries that you can run on the Operations Manager Data warehouse to determine what are the Noisy Event and Performance rules in your enviroment.

Noisy Event Rules

select count(*) as cnt, RuleDefaultName
from Event.vEvent as ev
Left Join Event.vEventRule as evi
On ev.eventOriginID = evi.eventOriginID
Left Join vRule as r
On r.RuleRowId = evi.RuleRowId
group by RuleDefaultName
order by cnt desc

Noisy Performance Rules

select count(*) as cnt, RuleDefaultName
from Perf.vPerfRaw as pr
Left Join vPerformanceRuleInstance as pri
On pr.PerformanceRuleInstanceRowId = pri.PerformanceRuleInstanceRowId
Left Join vRule as r
On r.RuleRowId = pri.RuleRowId
group by RuleDefaultName
order by cnt desc


As expected, in a few enviroments I tested, some of the busiest performance rules were the Processor and Memory collection rules.

However there were several unexpectedly busy Event rules that accounted for a significant portion of all the event data and some of these were subsequently disabled.

Thanks to Reut for pointing me in the right direction for the initial query.

Speed up Command Shell startup

There is a post over on the Powershell team blog about how to speed up Powershell startup times. I ran this on a few of my Operations Manager Servers and it noticeably made a difference to Powershell startup times. However the Operations Manager command shell was still quite a bit slower.

I then ran the same script from within the command shell and it found something to NGEN

NGENing : Microsoft.EnterpriseManagement.OperationsManager.ClientShell.dll

In my very scientific way of couting seconds in my head before and after running this it definitley starts faster.

Check it out - http://blogs.msdn.com/powershell/archive/2008/09/02/speeding-up-powershell-startup-updating-update-gac-ps1.aspx

Clearing Console Cache and User Settings

There are a few steps to be taken to really clean up a console if you have performed some personalization.

Open regedit and delete the key HKEY_CURRENT_USER\Software\Microsoft\Microsoft Operations Manager\3.0\Console

Start the Operations Console from the command line with the option /clearcache

Example:

C:\Program Files\System Center Operations Manager 2007\Microsoft.MOM.UI.Console.exe" /clearcache

Useful methods for searching MP's

Usually when i am looking for a way to do something in a management pack I refer to the Microsoft MP's as it has likely been done in them. Here is my normal process for searching these MP's.

In my development environment I dump all the management packs out to their XML. This is done using Command Shell.

mkdir c:\unsealed
get-managementpack | export-managementpack -path C:\unsealed

Then I open Notepad++ with the XML Tools plugin installed. A fantastic function of Notepad++ is Find in Files...

So say I am looking for the syntax to use in a recovery to start a service. I can Open up Notepad++ and do a Find in Files and look for "net start"




A few comments I read on the OpsMgr newsgroups about the Agent Maintenance Mode solution that I posted pointed out that the %MOMROOT% variable was not on the path and could not be used from the command line. If you do a search across all the MP's using the methid above it is defined in several different ones ( Sharepoint 2003, System Center Core Management Pack). Looks like this is something defined only within the context of OpsMgr.

All my failed Power supplies using Powershell

Sometimes you just want to see all the systems that have a certain problem, like failed power supplies or all systems where free disk space is low.


Here is a bit of Powershell to list any server where the Health State of the HP Proliant SNMP Management Pack Power supply Class is in an Error state.


$ClassID = "HewlettPackard.Servers.ProLiant.SNMP.HPProLiantSNMPPowerSupplies"
$Class = get-monitoringclass -name:$ClassID
$ClassInstances = get-monitoringobject -monitoringclass:$Class
$Errors = $ClassInstances | where {$_.HealthState -eq "Error"}
Foreach ($Server in $errors)
{
$server."[HewlettPackard.Servers.HPHealthCollection].ServerName"
}


Free Space /Availability of Logical Disks on Server 2003


$ClassID = "Microsoft.Windows.Server.2003.LogicalDisk"
$Class = get-monitoringclass -name:$ClassID
$ClassInstances = get-monitoringobject -monitoringclass:$Class
$Errors = $ClassInstances | where {$_.HealthState -eq "Error"}
Foreach ($Server in $errors)
{
$Server.Path + "," + $Server.name
}


A good way to catch issues where the alerts may have been missed.

Of course you can just use the Discovered Inventory View on the Operations Console and change the target type to Windows Server 2003 Logical Disk but the console has no easy way to get this data out except a one by one copy and paste.

Hp Storageworks Management Pack

I had a quick look at the Hp Storageworks MP v1.5 for the first time today and noticed quite a few glaring issues.

There are 5 separate discoveries running every 60 seconds on ALL computers. Looks like someone forgot to change the interval on these after development was finished. At least it is possible to change these intervals using overrides.

The discovery "HP StorageWorks SNMP Trap Catcher Discovery Rule" discovers every system with the SNMPTRAP service installed. Then proceeds to start monitoring all these servers. This monitoring includes an WMI query that runs every 15 seconds on all the servers that have the SNMPTRAP service and runs a script every time!!

The WMI query rule is using the backward compatibility layer so the interval cannot even be overriden.

[Off Topic] - The EOL Monster

Some stuff that has been hanging around the office for a few years.

Quick Access to OpsMgr Agent Information

Sometimes you need access to some information from Opsmgr in a hurry and you really do not want to open up the Ops Console and wait for it to load.

Here is a powershell script that extracts information from OpsMgr and drops it into several HTML files so it can be added to a website.
Download Script


The pages created are the following

AgentsCounts.html - This gives give you a count of the number of agents per management group and breaks this down also by per management server

MonitoredServers.html - This page gives a complete list of all the agents being monitored by Opsmgr.

ServersDown.html - This gives a list of servers that may be down. It is actually a list of servers where the health service watcher is in an "Error" state.

AgentsLast7days.html - This is a list of all agents installed in the last 7 days, when they were installed and includes their health state

The script does require some configuration before it can be used:

1) At the very start of the script you have to enter the name of the Root Management Server. You can uncomment the next lines lines and define multiple RMS's if you want to pull the information from more than 1 management group.

2) $outpath = "c:\" - Change this to where you want the script to write the html files.

3) If connecting to more than one management group you need to comment out line 17 and uncomment Line 19

Updated Agent Maintenance Mode

---Update---

Updated version of this MP at http://derekhar.blogspot.com/2009/11/new-agent-maintenance-mode.html

---------

There is a new version of my Maintenance Mode MP and scripts that can do maintenance mode from the Agent with no extras installed (The original post is here). The 4 files are available for download from http://cid-397bb61b75cc76c5.skydrive.live.com/browse.aspx/Public/AgentMaintModeV2?view=details

-----------

EDIT

Be aware the PowerShell execution policy might block the script from being run.

You may need to do the following:
1. Open the Command Shell, and do "set-executionpolicy remotesigned" (using "allsigned" may be considered a bit more secure.
2. Right-click on the .ps1 files, chose properties and click "Unblock" especially if you've downloaded these scripts using Internet Explorer.

Thanks to Marco for catching this. I had a few people contact with with the earlier version not working and I could never figure out why. Could be the same reason.

----------------

This version adds the ability to turn off maintenance mode from the agent as well as turn it on. The vbscript has been updated with some more options and some instructions on how to use it.

To install or update you need to import the management pack and copy the 2 ps1 files to System Center Operations Manager 2007 program folder on the RMS.

Then you need to copy the vbscript file to any agent managed server and run it to set any server in maintenance mode for the desired time period.

------------------------------

VBScript Usage:

Usage: MaintMode ON|OFF [ON Duration][D][H][M]

Examples: MaintMode ON 6H
MaintMode ON 2D

ON is a mandatory keyword to put a server in maintenance mode.

ON Duration is number of [D]ays [M]inutes or [H]ours Maintenance
Mode should remain ON.

If not specified, the default is 6 hours.
This means Maintenance mode will turn itself OFF after 6 hours.

If ON Duration is specified without [D][H][M], the number is
interpreted as hours.

------------------------------

This MP creates a new class which contains all Windows Operating Systems. There are 2 rules in this class. One listens for the Maint Mode ON event and the other listens for the Maint Mode OFF event. The script on the RMS that turns on Maintenance Mode actually turns it on for all the components of the server and then turns it off again only for the Maint Mode class. This means that even in Maint Mode it can listen for the OFF event.

NOTE: The discovery for this class runs every 12 hours so be aware that systems may take this long after install for the scripts to work

Thanks again to Boris for his examples and scripts for doing maintenance mode.

Multiple Network Name Discovery in a Cluster Resource Group

There is a known bug in OpsMgr that when you have more than one network name in a cluster resource group only the first one is discovered - http://support.microsoft.com/kb/919594. So say, for example, a cluster is not configured according to best practices and the MSDTC resources and the SQL resources are in the same group. You have a network name called "MSDTC Network Name" and one called "SQL Network Name ...". In this case the MSDTC network name will be discovered and the SQL Network name will not be discovered.

What's the problem with this exactly? Well, if the SQL Network Name is not discovered then the SQL instance in this resource group will not be monitored by the SQL MP.

 

To determine if you have clusters with this problem you can download and import the following MP - ClusterError.xml

It will create a view in your Operations Console that will list all the clustered servers that  have more than 1 network name in a resource group.

Initiate Maintenance Mode from an Agent (no extras installed)

---Update---

Updated version of this MP at http://derekhar.blogspot.com/2009/11/new-agent-maintenance-mode.html

---------

This post will detail a method to initiate Maintenance Mode from an agent managed computer using a vbscript. This method is useful in the case where the people doing patching are not the same people who use the Operations Console and when you cannot dictate what is to be installed in the server. Clive Eastwood's AgentMM for example, if run remotely requires .NET 3.0, Operations Manager specific dll's, Admin permissions on OpsMgr and the ability to connect to the SDK service on the RMS.

Using this method, the person running the command needs to have absolutely no permissions on Ops Manager. All of the components required to have this function are part of a standard OS with and agent installed. It works through Gateways so can be used in firewalled or untrusted environments if you have a gateway in place.

There are 3 parts to this solution:

1) A VBScript on each managed computer that writes an event to the local event log.

2) A Management pack containing an rule that detects this event and runs a response on the RMS.

3) The response executes a Powershell script on the RMS that puts the server in to Maintenance Mode. I have borrowed pieces of the powershell script from Boris.

VBScript

Powershell Script

Management Pack

The vbscript is pretty simple and just writes an event to the event log that indicates the length of time Maintenance Mode should remain on. This time is passed as a parameter to the script.

Usage: MaintModeOn 4

- This command turns on Maintenance Mode for 4 hours

The single rule in the management pack picks up the event from the Application Log with event ID of 0, with a source of WSH and where the description contains the words "Maintenance Mode". In response, the rule runs the Powershell script MaintModeON.ps1 and passes to it two parameters -the name of the computer where the script was run and the event description.

The Powershell script puts all three pieces of an agent in the Maintenance Mode. This Powershell script should be copied to the System Center Operations Manager 2007 program folder on the root management server. If you have clustered the RMS it should be in the folder on both nodes.

---Update---

I have updated the links since google appears to have taken down the site hosting the files.

Unsealed MP Backup with Retention Period

A small expansion of Pete's MP Export for Disaster Recovery. Here is an Management Pack that exports all the unsealed MP's once a day to a folder. The folder name is the date of the backup. Backups older than 7 days will be deleted.

Download the Management Pack

---Update---

Link to the MP has been fixed