System Center Orchestrator 2012 – Error Handling and Using Data in the Data Bus BETWEEN Runbooks

I will re-iterate a statement I’ve made before.  I don’t believe in re-inventing the wheel.  Before I found “Designing Orchestrator Runbooks“, I found that doing error handling using the brief bit of info I learned from the class and the other books I purchased was, frankly, a big pain.  I dropped everything I was testing with to go with the examples starting on page 36 of the book.

It takes quite a bit of thinking and experimentation with Orchestrator before you really get into how to use the tool and leverage it well.  The error handling examples are no different and they give you quite a bit of flexibility and value with your runbooks.  The examples also help clearly illustrate how to pass data between runbooks using the data bus.

I invite you to download a free copy of Designing Orchestrator Runbooks, turn to page 36 and start experimenting!

To expand on their information and also illustrate how the data bus is used, let me give you an example.  Recall that in my last blog post, from earlier this morning, I was invoking a runbook to run a report and email it.  This was done through an initiation runbook which generated a random server from a PowerShell script and used that generated server name with the invoke runbook activity.  I will show you that runbook and then an error handling runbook.

NOTE:  It is important to sit down, read through the information in Designing Orchestrator Runbooks, decide what works for you in your environment and then PLAN IT OUT and DOCUMENT it.  If you are going DNRY (Do Not Repeat Yourself) you are establishing some standards here that will be used consistently and repeatedly in all your runbook error handling.  Make sure you have everything the way you want it.  You must create your error handling runbook FIRST.  It should be present before you add an invoke runbook activity.  You have to have a road built before you can drive on it…

Runbook 1 – Error Handling

ss11

When you create your error handling runbook, you are setting the foundation for any and all data you think you would like to see in order to record errors and help indicate what problems might exist to assist in troubleshooting.

Create your variables and set their types:

ss14

Now, what do you want to do with it?  I’m emailing my data to a distribution list and writing them to the event log (event log example shown below):

ss15

That’s pretty much it for the error handling.  Now let’s get on with how to use it.

Runbook 2 – generate report and email it.  In our second runbook, we don’t care much about anything except our links and the invoke runbook activities

ss10

I have two links.  One links to an email activity, that link is utilized if my script runs ok.  If there’s an error with the script, then that link is followed.  Below is the properties of the “error” link

ss12

Notice I have two invoke runbook activites.  Both invoke runbook activities have a link with the same filters (as shown in the previous screen shot).

Both of these invoke runbook activities link to the same “Error Handling” runbook.

Now, here is where things get a bit interesting.  In your invoke runbook activity, I want to create multiple parameters.

ss16

A bit of explanation here.  I am adding parameters with names that MATCH the initialize data strings I set up in my error handling runbook.  The VALUES for these parameters are as follows:

RunbookName – subscribe to data, the runbook name from the previous activity, “Run PowerShell Script”.  What I want here is, if the script error-ed, what is the name of the runbook where this error was generated from?

ErrorMesage – subscribe to data, the error summary text from the previous activity, “Run PowerShell Script”.   What I want here is, if the script error-ed, what was the text of the error, the error message?  (if you read on in the book, they explain further how to include PowerShell error handling to the data bus as well.  So, if there’s an error WITHIN the script, you can subscribe to that too if you desire.  This only subscribes to an error message generated by the ACTIVITY, not an error in the script).

EmailAddress – subscribe to data, the global variable for my distribution list or email address of where the error should g0

Trace – I’m error handling within my script, that data is written to a temp file, so that information is being passed back via the variable “ErrorFile”.  This will become an attachment in my email or passed to the event log

Runbook – subscribe to data, the runbook name from the previous activity, “Run PowerShell Script”.  What I want here is, if the script error-ed, what server did it error out on?

ActivityName – subscribe to data, the name of the activity from the , “Run PowerShell Script”.  What I want here is, if the script error-ed, which activity did the error originate?

ActivityStatus – subscribe to data, the status of the activity from the , “Run PowerShell Script”.  What I want here is the status of the activity (failed, warning, etc).

 

Now, rinse and repeat for subsequent activities.  So, example, if my “Send Email” activity succeeds, well, that’s it.  It succeeds do nothing more.  If my “Send Email” activity fails, then invoke the “Error Handling” runbook and pass all that data to that runbook so it can use it to send an email about what my activity failed and also write to the event log on the server.

Selecting a Random Runbook Server – Part 2 – a Closer Look

It’s been quite a while since my last post.  I’ve had several people email me and ask for a closer look at selecting a random runbook server.

A bit of a review, refer to the previous post https://trwagner1.wordpress.com/2013/11/27/selecting-a-random-runbook-server/ for reference.  But, I was curious about how you could genuinely select a RANDOM server rather than rely on a “primary” and or “secondary” that were “static”.  (Please read the disclaimer at the bottom of this post)

Since this is a closer look, I’ll try to keep things simple and include screen shots.

I will show you how to subscribe to and use data within the data bus.

First, a bit of design explanation.  For simplicity, there are two runbook servers.  Their names are “SCORCH01” and “SCORCH02”.  Tying into Jon Mattivi’s post, we will create two variables.

Here’s an example of creating the first variable “RBServer1”.

ss1

You should now have two variables as in the example below:

ss2

 

I have an initiation runbook that starts another runbook.  For an explanation of what an initiation runbook folder is, download this free book “Designing System Center Runbooks“from Microsoft.

The other runbook is invoked from the initiation runbook that runs a PowerShell script, queries groups for members, creates a report and emails it to a distribution list.

For the purpose of this blog post, we don’t care about the contents of the runbook that generates a report so that won’t be discussed.

I want to point something out here.  My initiation run book is NOT using a random server!  What I want to do is edit the properties of the runbook so that I am overriding the default roles and I add any and all servers I want to be able to run this.  The top server listed is the primary.  If the primary isn’t available, then the secondary is used.

Here is a screen shot showing the properties of my initiation runbook.  click the “Runbook Servers” tab, then override and select your servers to use when this RB runs.

ss9

 

The initiation runbook looks like this:

ss5

The activity “Run .Net Script” titled “Generate Random RB Server” is our primary focus here.

Type in your PowerShell script:

<code>

$RBServer = ($Server = “”,””) | Get-Random.

</code>

Now, you could add the data to subscribe the data.  The easy way is to RIGHT click and subscribe to the data.  So, as you type in the script after the first double quotes, right click the mouse.  Then, select your first variable.  Add the closing double quotes, then the comma, then the double quotes again, right click, subscribe, then close double quotes then right parenthesis.

That will give you this:

ss3

Now, DO NOT click the finish button yet.

Click the Published Data tab, click the Add button and enter in a new variable, type string, name “RBServer”.  This is what puts the returned data into the data bus so you can use it in any subsequent activities in your runbook(s).

ss4

The last activity in my initiation runbook is “Invoke Runbook”.  I’ve linked my runbook that I want to invoke.  Right click click in the “Runbook Servers” text field, select “Subscribe > Published Data”

ss7

Since my last activity was the activity that runs the PowerShell script, I don’t need to hunt for it.  But, if you have a more complicated script that has many activities, you HAVE to pay attention to the flow of your runbook.  You can only select published data that would have been generated prior to the point at which you are subscribing to the data…. after all, you can’t use data that isn’t established.

In this example, I am looking for the published data “RBServer”.  Select it and click ok.

ss8

Now, your dialogue box for “Invoke Runbook” should look like the this:

ss6

 

Now, when this initiation runbook runs, it generates a random server name, invokes a runbook on the server name selected by the PowerShell script.

DISCLAIMER

A disclaimer?  What for?  Well, here’s why and what for.  There may be practical uses of this approach.  But, remember that I went after this with some curiosity following an Orchestrator class. But, this example is a very SIMPLE example.

So, we’ve used “HA” by adding multiple servers in the properties of the runbook.  We’ve generated a random runbook server… HOWEVER, if you pay close attention, my PowerShell script is nothing more than a simple method to select a random string from only two that I’ve provided.  That’s all it is!  I would not suggest using this example in production without a lot of improvements!

What if SCORCH01 is down?  The script doesn’t provide for any error handling or error checking either.

So, just take note.  To be much more robust, you’ll want to beef up that script quite a bit.  If your Orchestrator environment is built properly, you’ll have more than one runbook server for HA and load balancing.  To be honest, using the primary and secondary servers in the “Runbook Servers” tab on the properties of your runbook is probably sufficient.

Selecting a random runbook server

Here’s a link to a blog post that piqued my interest before taking a class on System Center Orchestrator.

http://jmattivi.blogspot.com/2012/06/high-availability-for-runbook-servers.html

I attended a remote class on orchestrator, company and instructor name withheld intentionally, shared Jon’s blog post and posed the following question, “can you randomly select a runbook server?”

The instructor said no. I thought he was a bit short-sighted.  The instructor was a proponent of saying (and implying) that you don’t need to know PowerShell.  Do you really need to know PowerShell to use System Center Orchestrator?  Technically, no.  Now, that’s “technically” a correct statement.

However, if you are going to put forth the time and effort to attend a class to learn a System Center product AND advance your skillset, AND, better leverage a product’s features, you are truly doing yourself a disservice beyond belief.  I know that won’t float well with several out there.  However, it is my opinion.  But, you have to face facts.  We are in an industry where to excel above and beyond and to be “good” you need to educate yourself and take advantage of making the fullest use of a product’s strengths as you can.  This includes spending time after hours with your own testing, learning your own, etc.  Learning PowerShell is NOT THAT HARD.  Using the command line is NOT THAT HARD.  At least familiarize yourself with PowerShell, learn the basics and see if you can use it to your advantage.  I bet you can find LOTS of ways to use PowerShell in any IT environment.  Enough said, I’ll jump off my soap box, for now.

Jon Mattivi’s blog post really made me think about this a lot. “When I design my company’s Orchestrator environment, I not only want it to be highly available, but I also want to implement the DRY principle.  I also want to extend the high availability by distributing the work load as evenly as I can.  How do I do that?”

Jon’s blog post seemed too static for my tastes.  Nothing against Jon’s post.  It’s a valid alternative.

After spending many, MANY, hours in my own home lab with Orchestrator and learning the ins and outs of building runbooks and becoming more advanced, a light clicked and I decided to try this.

I responded to Jon’s blog post with my solution.

Here’s the idea in a nutshell.  To distribute your runbook workload evenly, you can set up a global variable for each of your runbook servers.

In my example, we’ll say those two global variables are “SCORCH01” and “SCORCH02”

Next, create your initiation runbook (for more information on what an initiation runbook is, spend the best few bucks you can buy and purchase “Designing Orchestrator Runbooks” by David Ziembicki, Aaron Cushner, Andreas Rynes, and Mitch Tulloch.  It’s a very affordable book.  It is very easy to read, and in my humble opinion, the best money you can spend on a book for Orchestrator and a must have for your library).  The price is very low and frankly, it’s a must buy just because of the price!  (No, I’m not getting any commission on that, just a friendly recommendation I don’t think you should refuse.  It’s like passing up a gas station that has gasoline selling at $1.00 a gallon.  You’d be silly not to stop and top off your tank!)

In your initiation runbook, add a “Run .Net Script” activity.  Change the language to PowerShell.  Add this one liner (replace the server names in quotes and subscribe to your global variables).

<br />$RBServer = ($Server = "SCORCH01","SCORCH02") | Get-Random<br />

Set up the activity to publish the $RBServer variable to the Orchestrator bus.

Add an invoke runbook activity and link the two together. In the “Invoke Runbook” activity, in the runbook servers text box, subscribe to the published data for the data returned by the “Run .Net Script” activity.

Now, when the initiation runbook activity runs, you’ll be selecting from a random runbook server! If you have more than 2 runbook servers, this becomes a bit more useful in selecting a server at random.

So, the answer is “yes” you CAN select a random runbook server when your runbooks run.  Wasn’t that easy?

PowerShell – Refresh SCCM Updates Compliance

Here’s a quick post since I haven’t posted in a while.

Some of you SCCM users will run into circumstances where a system should be compliant with your current update deployment.  However, it doesn’t report properly in the report server within SCCM.

The “old” SMS way to do that was a common VBscript:  http://msdn.microsoft.com/en-us/library/cc146437.aspx

Ah, VBscript.  Well now, let’s do that in PowerShell!  This is so simple and only takes two lines.

You can easily do this using System Center Orchestrator as well.  Just combine the Execute PowerShell Script IP with the Configuration Manager IP for 2012 and you’re all set.  Just get collection members and then run this against each member of the collection.

Here’s the PowerShell way:

$SCCMUpdatesStore = New-Object -ComObject Microsoft.CCM.UpdatesStore
$SCCMUpdatesStore.RefreshServerComplianceState()

Voila!

(just remember to run as “Administrator”)

SCCM 2012 Client Deployment – SCEP Installation Craziness

I just finished an SCCM 2012 deployment and began upgrading all of the systems from the 2007 client to the new SCCM 2012 client.

First batch of 30 or so went fine.  Then, all of a sudden, a good sized handful (about 40 servers) failed to install SCEP.

Basically, no errors.  The only thing I saw was an entry in the \\<machinename>\admin$\ccm\logs\EndpointProtectionAgent.log was the following message:

“Unable to query registry key (SOFTWARE\Microsoft\Microsoft Security Client), return (0x80070002) means EP client is NOT installed.”

The closest I came to seeing an example of my issue was explained here:

http://social.technet.microsoft.com/Forums/en-US/configmanagersecurity/thread/872a5efc-8544-449a-8eda-777d606ac07b/

So, why not try that? It didn’t exactly fit my problem. In my case, the systems were both 2003 R2 and 2008 R2. So, I gave it a try. Won’t hurt.

Unfortunately, it didn’t work.

I’ve had stranger things happen, so I got to thinking, why not create the key and see what happens. Here’s what I did:

  • Uninstalled SCCM “ccmsetup.exe /uninstall”
  • Create the key “HKLM\Software\Microsoft\Microsoft Security Client”
  • Reinstall the SCCM client from the console

\
VOILA! The SCEP client installs just fine.

I’ve tried just creating the key and re-installing the agent, but that doesn’t seem to work.

Just as an FYI, I originally wrote this post on the 01/07/2013. The next day, I added a package, no files, with the command line:
reg add "HKLM\SOFTWARE\Microsoft\Microsoft Security Client"

I deployed that to a direct membership collection.

Within just a few minutes, the SCEP client installed on it’s own. So, I found it wasn’t necessary to uninstall the SCCM client.

Try either method. See what works. We noticed this does not always work on Windows 7 workstations.

PowerShell and Tombstoned DNS Records

I recently ran into a situation where DNS A and PTR records were mysteriously disappearing in a Windows 2008 R2 Forest.  I searched and searched for scripts to assist me in trying to “track” this problem by using event logs and even created filters with a logging product my company owns, with little luck.

The following two links are quite helpful when trying to troubleshoot, understand, setup, DNS and DHCP, etc.

http://blogs.technet.com/b/networking/archive/2011/08/17/tracking-dns-record-deletion.aspx

http://msmvps.com/blogs/acefekay/archive/2010/12/09/dns-records-disappearing-and-dns-auditing.aspx

However, a bit of a hint.  There’s some information left out!  First, you have to read Ace’s post very carefully and understand the PROCESS of how a DNS record is deleted.  In fact, don’t use the word delete.  A record isn’t deleted, it’s tombstoned, not visible and you are unable to query it through any DNS methods.  When the record goes “away”, the AD object is “cleaned/cleared” from AD.  To be simplistic, it’s a two step process.

The other information is, don’t just look for event ID 4662!  If you read Ace’s article and begin to understand the process, what’s happening is an attribute is changing on the record itself!  You also need to pay attention to event ID 5136 and 5137.

Now, you say, if the Windows security event logs don’t necessarily give you information on the actual record deletion, but only a change in the dnstombstone attribute, then is there a better way?

Certainly!  PowerShell to the rescue!  In fact, forget about the DNS console and the event logs altogether!

This short little script queries AD itself, looks at the attribute dNSTombstoned and the attribute whenChanged, creates a csv file and then emails it.  You then can see a report of any record (object DN) and when it was last changed.  You can limit the scope by altering the DN. Notice I’m using the root DN for all my zones so I’m including any reverse zones as well as additional forward zones.

Resist the urge to use the “-IncludeAllProperties” switch when using anything that queries. You can do that to dig down while developing your scripts (hopefully in a test environment), but to speed up your scripts, just include only the specific properties that you are going to use.

It’s simple enough to set up a scheduled task in windows to allow this script to run as often as you like.  Please ignore the lack of error handling here, I’m trying to keep it plain and simple.

Note the requirement of PowerShell V2 and PsSnapIn quest.activeroles.admanagement.

Also note that it’s only necessary to use the script during troubleshooting and not on a continual basis.  Probably won’t hurt anything, but if it’s not necessary to run it, why waste resources?

Also, as always, please test scripts you find on the internet in a test lab and fully understand them before bringing them into your production environments.

<#
.SYNOPSIS
   Script to generate report on DNS records marked tombstoned
.DESCRIPTION
   This script queries AD objects to find DNS records marked as tombstoned, creates a CSV file with the name of the record and when the record was changed to tombstoned.
   It then emails it to a list of individuals
.REQUIRES
	VERSION 2
	PsSnapIn quest.activeroles.admanagement 
.EXAMPLE
   PS c:\foo> .\Start-DNSTombstoneReport.ps1
#>

$TestQADSnapin = get-pssnapin | where { $_.Name -eq "Quest.ActiveRoles.ADManagement"} 
if($TestQADSnapin -eq $null){
	add-pssnapin -Name Quest.ActiveRoles.ADManagement -ErrorAction SilentlyContinue
}
$CSVFile = "C:\temp\dnsTombstoneReport.csv"

$Result = Test-Path $CSVFile
If ($Result){
	Remove-Item $CSVFile -Force
}

$TombstonedRecords = Get-QADObject -searchroot 'CN=MicrosoftDNS,DC=ForestDnsZones,DC=MyDomain,DC=Local' -IncludedProperties dNSTombstoned,name,distinguishedName,whenchanged  -SizeLimit 0 -ShowProgress:$False | Where {($_.dNSTombstoned -eq $true) -and ($_.name -NotLike "_*")} | select-object distinguishedName,@{name="DateChanged";expression={$_.whenChanged.ToShortDateString()}}
If ($TombstonedRecords){
	$TombstonedRecords | Sort-Object -Descending DateChanged | Export-Csv $CSVFile -NoTypeInformation
	Send-MailMessage -From "SomeGuy@MyDomain.Local" -To "SomeGuy@MyDomain.Local"  -subject "DNS Tombstone Records Report" -smtpServer "smtp.MyDomain.Local" -attachments $CSVFile
	Remove-Item $CSVFile
}

SCOM 2012 – Editing Product Knowledge Issues – “Could not load file or assembly ‘Microsoft.Office.Interop.Word’

Up until a few days ago, I have had nothing but constant grief trying to get the “edit” button to work with product knowledge in System Center Operations Manager 2012 since deploying the product. It has yet to be a priority.  So, spending any more than a few minutes on Google searching for this in the past hasn’t been very productive.

Here is the problem in detail:

After installing/satisfying the prerequisites, you will receive the inevitable message:

Edit Product Knowledge Error

After SCOM 2012 was deployed , I created a new custom monitor and attempted to add product knowledge and first received the error. This week, now a couple of months out from deploying SCOM 2012, I’m to the point where I do not want any further custom monitors created without having the ability to provide some knowledge information. So, off a “Google-ing” I go.

Now, let’s outline some important information. Review the prerequisites before passing go.

http://technet.microsoft.com/en-us/library/hh212900.aspx

These tidbits are critical:

  • The Operations console must be installed on a 32-bit computer. Adding and editing company knowledge must be done on this computer.
  • Microsoft Office Word 2003 with the .NET Programmability feature, or Microsoft Office Word 2007 or Office Word 2010 Professional edition (Standard edition is not supported.
  • Only the 32-bit version of Word 2010 is supported. The knowledge template will not work with the 64-bit version of Word 2010.
  • Microsoft Visual Studio 2005 Tools for Office Second Edition Runtime at http://go.microsoft.com/fwlink/?LinkId=74969. You must use this version of the Visual Studio 2005 Tools for Office.

I overlooked the first issue.

I already have all that on my management VM and management VDI. Both are Windows 7, 32-bit machines. I look at these two machines and verify the other prerequisites have been met. Indeed they were. I try to edit company knowledge, same problem.

After several days of searching, I finally found this post from June 18, 2012:  http://social.technet.microsoft.com/Forums/en-US/operationsmanagergeneral/thread/aae19511-c886-4d00-8e6f-015707b177f8

I gave that a try and BANG! Worked.

Now, those that know me know I’m as stubborn as a mule and my thick head says to try it on my laptop.  Here comes the mean old buzzer a screaming away from the game of “Operation”.

Rule #1: “The Operations console must be installed on a 32-bit computer. Adding and editing company knowledge must be done on this computer.”

So, that was wasted time.  I have to use my management VM or my VDI.

Just a word of advice:  double and triple check the following as there will be MULTIPLE lines to delete.

Microsoft.EnterpriseManagement.Monitoring.Console.exe.config

Make SURE you have a backup of the original config and make SURE you only delete the lines FOLLOWING the /dependentAssembly  tag!