How to upgrade or apply a hotfix to a Clustered RMS

We have had a some difficulties over the months trying to apply hotfixes to any clustered Root Management Server. The hotfix installers try and stop the 3 OpsMgr services, apply the hotfix and them restart the services. These steps result in some unexpected behavior on a Clustered RMS.

What I have seen is that after the hotfix installer stops the RMS services the Cluster Service will try and restart them again.

This results in either the services running on the node you are trying to patch or the RMS resource group failing over. If the RMS fails over then, when the hotfix tries to restart the services after it has completed, it either times out or you have the RMS services running on both Cluster nodes. It it restarts the service on the node you are trying to patch the install will fail as the files are in use.

Here are some steps that can avoid these issues:

1. In Cluster Administrator ensure the RMS resource group is on the node you are patching. Then Pause the Node you are not patching. Simply right Click on the node and select Pause. This stops the resource group failing over to the other node.

2. Set the 3 RMS services not to restart automatically. Selecting each service, right Click and select Properties. On the Advanced tab set it to "Do not restart"

3. Apply the hotfix to the current node

4. When the hotfix has completed reset the services to restart automatically and resume the other node.

Repeat this to apply the patch to the second node.

This process also worked for me in upgrading a clustered lab environment to the Release Candidate of R2

Operations Manager Data Warehouse High CPU Utilization

Some good steps if you are having trouble with CPU on your DW.

http://www.afinn.net/2008/10/operations-manager-data-warehouse-high-cpu-utilization/

Data Warehouse Queries for Noisy Rules

Here are a few SQL queries that you can run on the Operations Manager Data warehouse to determine what are the Noisy Event and Performance rules in your enviroment.

Noisy Event Rules

select count(*) as cnt, RuleDefaultName
from Event.vEvent as ev
Left Join Event.vEventRule as evi
On ev.eventOriginID = evi.eventOriginID
Left Join vRule as r
On r.RuleRowId = evi.RuleRowId
group by RuleDefaultName
order by cnt desc

Noisy Performance Rules

select count(*) as cnt, RuleDefaultName
from Perf.vPerfRaw as pr
Left Join vPerformanceRuleInstance as pri
On pr.PerformanceRuleInstanceRowId = pri.PerformanceRuleInstanceRowId
Left Join vRule as r
On r.RuleRowId = pri.RuleRowId
group by RuleDefaultName
order by cnt desc


As expected, in a few enviroments I tested, some of the busiest performance rules were the Processor and Memory collection rules.

However there were several unexpectedly busy Event rules that accounted for a significant portion of all the event data and some of these were subsequently disabled.

Thanks to Reut for pointing me in the right direction for the initial query.

Speed up Command Shell startup

There is a post over on the Powershell team blog about how to speed up Powershell startup times. I ran this on a few of my Operations Manager Servers and it noticeably made a difference to Powershell startup times. However the Operations Manager command shell was still quite a bit slower.

I then ran the same script from within the command shell and it found something to NGEN

NGENing : Microsoft.EnterpriseManagement.OperationsManager.ClientShell.dll

In my very scientific way of couting seconds in my head before and after running this it definitley starts faster.

Check it out - http://blogs.msdn.com/powershell/archive/2008/09/02/speeding-up-powershell-startup-updating-update-gac-ps1.aspx

Clearing Console Cache and User Settings

There are a few steps to be taken to really clean up a console if you have performed some personalization.

Open regedit and delete the key HKEY_CURRENT_USER\Software\Microsoft\Microsoft Operations Manager\3.0\Console

Start the Operations Console from the command line with the option /clearcache

Example:

C:\Program Files\System Center Operations Manager 2007\Microsoft.MOM.UI.Console.exe" /clearcache

Useful methods for searching MP's

Usually when i am looking for a way to do something in a management pack I refer to the Microsoft MP's as it has likely been done in them. Here is my normal process for searching these MP's.

In my development environment I dump all the management packs out to their XML. This is done using Command Shell.

mkdir c:\unsealed
get-managementpack | export-managementpack -path C:\unsealed

Then I open Notepad++ with the XML Tools plugin installed. A fantastic function of Notepad++ is Find in Files...

So say I am looking for the syntax to use in a recovery to start a service. I can Open up Notepad++ and do a Find in Files and look for "net start"




A few comments I read on the OpsMgr newsgroups about the Agent Maintenance Mode solution that I posted pointed out that the %MOMROOT% variable was not on the path and could not be used from the command line. If you do a search across all the MP's using the methid above it is defined in several different ones ( Sharepoint 2003, System Center Core Management Pack). Looks like this is something defined only within the context of OpsMgr.

All my failed Power supplies using Powershell

Sometimes you just want to see all the systems that have a certain problem, like failed power supplies or all systems where free disk space is low.


Here is a bit of Powershell to list any server where the Health State of the HP Proliant SNMP Management Pack Power supply Class is in an Error state.


$ClassID = "HewlettPackard.Servers.ProLiant.SNMP.HPProLiantSNMPPowerSupplies"
$Class = get-monitoringclass -name:$ClassID
$ClassInstances = get-monitoringobject -monitoringclass:$Class
$Errors = $ClassInstances | where {$_.HealthState -eq "Error"}
Foreach ($Server in $errors)
{
$server."[HewlettPackard.Servers.HPHealthCollection].ServerName"
}


Free Space /Availability of Logical Disks on Server 2003


$ClassID = "Microsoft.Windows.Server.2003.LogicalDisk"
$Class = get-monitoringclass -name:$ClassID
$ClassInstances = get-monitoringobject -monitoringclass:$Class
$Errors = $ClassInstances | where {$_.HealthState -eq "Error"}
Foreach ($Server in $errors)
{
$Server.Path + "," + $Server.name
}


A good way to catch issues where the alerts may have been missed.

Of course you can just use the Discovered Inventory View on the Operations Console and change the target type to Windows Server 2003 Logical Disk but the console has no easy way to get this data out except a one by one copy and paste.