System Center Orchestrator Best Practices

Updated to include System Center Orchestrator 2022

Download the guide now

Introduction

This white paper will assist anyone tasked with taking the System Center suite of solutions and using Orchestrator runbooks to integrate better, automate or orchestrate IT systems and processes to support business practices and ultimately achieve cost savings by driving greater efficiencies across the organization.

The insights in this document reflect over 13 years of experience of the collective Kelverion team in assisting customers to achieve success in the design, building and deployment of runbooks using System Center Orchestrator.

Access to Orchestrator is akin to receiving a box of LEGO® bricks with no instructions on what to build.

Therefore it is important to read and implement the recommended best practices within this guide before building your automation processes. Failing to do so can result in errors, poor service, inefficient automated processes, increased costs and contributing to a lack of IT process governance and control.

This paper will focus on Orchestrator runbook development as an integral part of IT process projects. This ensures IT professionals add real value by deploying System Center solutions and integrating System Center with other IT systems.

No two organizations are the same, regardless of why a company looks to implement IT process automation. Those in charge must first determine the success criteria of implementing IT Automation to establish clear goals to achieve and, secondly, guarantee minimal disruption to the business through the transition from inefficient to efficient processes made more effective through automation.

Some IT Professionals struggle using System Center Orchestrator as it is not a simple IT solution to be installed and learnt on the job. It is highly customizable to support the business processes you wish to automate, which can become a double-edged sword for IT staff.

For example, it is simple to install Orchestrator and create simple workflows. However, users can quickly encounter problems when designing more complex activities and representing them in Orchestrator, which involves multiple branches and loops.

From our experience, there are common mistakes to be avoided when implementing the solution. This guide will counter frustrations in your efforts by learning about our best practices and implementing them for greater success in your IT process automation projects.

What is System Center Orchestrator?

Before reading the content of this best practice guide from Kelverion, users need to understand System Center Orchestrator and runbooks clearly.

Orchestrator is one of the components of the Microsoft System Center Suite. Investing in the System Center Suite allows you to access the full capability of Orchestrator. Originally a third-party product named Opalis, which Microsoft acquired in 2009, ‘Orchestrator provides a simplified way of building complex automation’.

Rather than spending time writing lines of arcane code, users can use Orchestrator to, for example, support event monitoring, alerts and actions by using a drag-and-drop process which only takes a matter of seconds to complete.

This rapid construction and deployment of automated processes will improve IT service and remove manual, resource intensive or error-prone tasks, allowing your team to focus on resolving more strategic activities and delivering new services.

What is a Runbook?

The power of Orchestrator is driven by runbooks and the individual ‘activities’ that make up a runbook. Runbooks contain the ‘instructions’ for an automated ‘task’ or ‘process’. The individual steps throughout a runbook are called ‘activities’. Within the runbook, additional ‘controls’ provide information and instructions to govern the sequence of activities in the runbook. Runbooks, activities and each runbook control have configurable properties. These properties are modified to configure the behaviour that your runbook requires.

Fundamentally, the runbook depicts the sequence of operations or work of a person or a group. Runbooks can also be connected to form more complex solutions to the more convoluted processes in the organization.

Benefits of System Center Orchestrator

In summary, ‘Orchestrator provides a solution for IT Operations runbook Automation and provides orchestration, integration and automation of IT processes, enabling companies to define and standardize best practices and improve operational efficiency whilst reducing errors and costs.’

Integration

Orchestrator provides integration with third-party service desks and systems tools and databases, operating systems, their resources, and system functions. Numerous Integration Packs are available from Kelverion for non-Microsoft products, such as Atlassian, Amazon, BMC Helix, Cisco, ServiceNow and VMware vSphere.

Common Pitfalls to Avoid

It is important to first carefully design and plan workflows with experts on system design and those who
understand automated processes. It is unnecessary to generate reams of documentation; a visualization of the
process design via flow charts, use cases and swim lanes is sufficient.

Do NOT throw away existing paperwork documenting processes; this documentation will prove extremely  useful in analyzing the initial thought processes, future runbook and process changes, and any test issues and
personnel changes. If instructions on automated systems aren’t kept, how will what is built be remembered or how it was designed?

The Kelverion Best Practice Approach

To simplify and best structure your approach to IT process automation and the use of System Center Orchestrator, Kelverion advises our clients to follow these proven steps when building runbooks:

This is the high-level approach Kelverion always takes to gain a better understanding of our customers’ automation requirements.

Guidance for Orchestrator 2022 Migration

For organizations looking to upgrade from 2016, 2019 or earlier versions of Orchestrator to System Center Orchestrator 2022, there are several things to be aware of before considering migration, as Orchestrator 2022 is a major change to the application.

Orchestrator 2019 and all its predecessors are 32-bit applications. Orchestrator 2022 is the first version of Orchestrator, which is a native 64-bit application. It is, therefore, more compatible with the modern 64-bit versions of Windows Server.

Whilst this is great news from a supportability and product longevity viewpoint, it does cause challenges for customers looking to migrate from older versions to Orchestrator 2022, outlined as follows.

Integration Pack Compatibility

Integration Packs written for Orchestrator 2019 and earlier ARE NOT compatible with Orchestrator 2022 because they are all 32-bit applications. As such, these will not work with the 64-bit Orchestrator 2022.

Kelverion has produced 64-bit versions of our Integration Packs, which allow Orchestrator 2019 runbooks (using our Integration Packs) to be converted directly into Orchestrator 2022 runbooks.  However, many organizations have used Integration Packs from other sources, which will need to be checked for Orchestrator 2022 compatibility.

CodePlex Integration Packs

Integration Pack compatibility is particularly relevant to anyone using CodePlex Integration Packs in Orchestrator today. CodePlex was a forge website by Microsoft. While it was active, it allowed shared development of open-source software. On the initial release of Orchestrator, some individuals released free open-source Integration Packs that users downloaded and used
in their runbooks. 

CodePlex was shut down in 2017, and these Integration Packs are no longer available nor being developed or enhanced. All these Integration Packs are 32-bit and will not work in Orchestrator 2022.

Anyone using CodePlex Integration Packs will need to find alternatives to replace the activities in their runbooks. Kelverion has, over the years, developed Integration Packs, which offer similar capabilities to many of the common CodePlex Integration Packs. This is due to the demand from organizations requiring supported versions of Integration Packs to use in their
runbooks. Unfortunately, there are CodePlex Integration Packs for which no viable alternative exists, and the runbooks will need to be completely rewritten to use some other mechanism.

Microsoft Integration Packs

Table up to date February 2023

PowerShell Scripts

Historically running PowerShell scripts in Orchestrator had always been challenging because Orchestrator was 32-bit, and most PowerShell is 64-bit. It was challenging to trigger most PowerShell cmdlets from within a script running in the .Net Script Activity as these were in a different memory space to Orchestrator.

With Orchestrator 2022 this memory space challenge has been resolved but has brought a new challenge. 

Older versions of Orchestrator only supported PowerShell 1 and 2. Orchestrator 2022 supports the modern PowerShell versions supported by the Server OS, typically 4 and above. The
syntax of PowerShell subtly changed between PowerShell 2 and the more modern versions, which means that old scripts that worked in Orchestrator 2019 may not work in Orchestrator 2022. The reasons for the scripts, not running can be very obscure, and often a complete rewrite of the script is often the fastest way to fix the issue.

Top 15 Tips For Designing Runbook Solutions

Tip 1 Use a Folder & Subfolder Structure in your Runbook Layout

The best practice is ALWAYS to use a ‘Folder’ and ‘Subfolder’ structure for runbooks (similar to a windows explorer structure). For ease of manageability, only put ONE Runbook in a folder or subfolder.

WHY? This helps keep things simple. By having one folder and one runbook, the process is easier to view and understand.

Tip 2 Use Folder Numbering

To show the flow of execution and aid supportability, take time to give your folders numbers – for example, 1.0, 2.0, 3.0 etc. This additional design feature can often be overlooked, but its usefulness should not be underestimated.

Tip 3 Create Common Task Runbooks – Modularize

When building a workflow solution, an effective approach is to ALWAYS identify commonly repeated functions.

WHY? This will allow you to take advantage of the ‘build once and use many’ methodology, saving time and increasing runbook resilience.

By breaking the various steps of a process into chunks, i.e. modularising, you can reuse these chunks whenever you need to do a task many times. For example, you could modularise common tasks such as ‘send email’, ‘raise an incident ticket’ and ‘create a change request’ etc. By building this type of functionality into your runbooks, you will avoid updating runbooks in multiple locations, which may become confusing and cumbersome to maintain.

Orchestrator allows users to easily call upon any other runbooks within a runbook to complete work on behalf of the calling runbook. This means you can build runbooks which perform common functions and call on them as required.

In building common task runbooks, each task should be broken into a separate runbook. Additionally, the design of the runbook should be kept as generic as possible to enable re-usability. Orchestrator allows you to pass parameters into a runbook via the ‘Initialize Data Activity’, and then the runbook can do its operations and relay the results back to the calling runbook via the ‘Return Data Activity’.

Tip 4 Link Colors & Labels

Any Link with logic in it should ALWAYS have a label to explain its function. Any Links with decision logic should ALWAYS be color-coded, enabling users to see a decision is made.

WHY? Introducing Link colours and labels, makes it much easier to read and understand.

Here is our best practice guide for colour coding:

Green – Success Path
Red – Fail Path
Orange – Decision Paths

Tip 5 Add a Branch for ‘Nothing to Process’

If there is a runbook which looks for a state to be true and then checks for this state, ALWAYS add a branch to show the Runbook has a specific path to follow should there be nothing to process.

WHY? This makes runbooks easier to understand by staff and helps test and debug the runbook. By adhering to these guidelines, users are helping to make Runbooks foolproof.

Tip 6 Give Activities & Runbooks Meaningful Names (Avoid Defaults)

To further support the administrator managing the runbooks, ALWAYS give Runbooks and Activities meaningful names. Avoid using standard defaults provided by Orchestrator.

WHY? This will help others review the runbook to view and understand what it is trying to do immediately.

Every runbook name should start with a number which reflects where it occurs in the end-to-end process flow and end with a version number. A best practice naming convention example would look like this: 1.01 Monitor SCOM V1. This gives the administrator greater and clearer information on the process when it stops. 

Furthermore, descriptions should be included for Activities to explain processes in more detail. Often, this type of information is skipped, but it is a worthwhile inclusion, particularly when new people take over the management of runbooks, which were originally designed by others who have since moved or left the organization.

Tip 7 Include Activity Descriptions

If you put Activity Level Looping on an Activity or are using a Select Rows Activity, ALWAYS write in the description field of the Activity of the logic in use. 

WHY? By incorporating this feature, you are improving supportability. When the runbook is reviewed later, it will aid the reviewer in determining what the Activity is designed to be doing.

Tip 8 Keep Monitor Runbooks Short

ALWAYS keep the workflow very short for Runbooks monitoring an enterprise tool or similar. Grab the Event and write it to a Microsoft SQL Server Database and use a Database Activity to process it.

WHY? By keeping the workflow very short, you will maximize performance. It is better to grab the Event and write it to 

a Microsoft SQL Server Database and use a Database Activity to process it. This way, there is less chance of Orchestrator missing a new event because it is still busy processing the last set of events. There are likely many tools being monitored; therefore, this design will enable a more efficient method for monitoring, queuing, and processing.

Tip 9 Create a Robust Error Reporting Routine

At some point, an error reporting routine will be required from within Orchestrator to report runbook errors back to an Event Management system. One of the best ways to do this is to create a ‘common handler runbook’ for Error reporting, which uses an Integration Pack, such as the ServiceNow IP, to create an Incident Ticket directly in ServiceNow to inform the support team that there has been an error.

In the ServiceNow Incident Ticket, there is ALWAYS a need to include details of the error.

WHY? This will make it useful and context-sensitive to your users. This means details such as the runbook name, activity that failed, runbook server etc are needed. This information can be obtained by using the ‘Common Published Data’ available with every Activity.

Tip 10 Use Loops for Resiliency

In Orchestrator, when interacting with a System via an Integration Pack, it is a good idea to ALWAYS try a few times to complete the task before throwing an error in the runbooks.

WHY? It is common knowledge that APIs (Application Programming Interfaces) to Enterprise Management systems can be notoriously temperamental.

For example, when creating a SCOM Alert, try three times to create an individual Alert before throwing an error. The motto here is if you don’t succeed, try, try and then try again.

Right-click on an activity, select “Looping” and then set up the looping criteria.

Loop: number of attempts is an item of Common Published Data.

Tip 11 Don’t Use Counters; Use Variables Sparingly & Keep Connection
Names Generic

DO NOT use Counters in Orchestrator, as they are global values and can therefore be modified by any runbook at any time. 

WHY? You cannot trust Counters as they are not reliable. You cannot rely on their value at any one point unless you are running in a single thread. This defeats a 
fundamental benefit of Orchestrator – parallel execution and multi-threading. 

Variables in Orchestrator are also global. ALWAYS use these sparingly; they can be ideal for database server names, table names etc. which will be used in many runbooks. They can be encrypted, so are good for service account passwords. 

WHY? When a runbook is exported, all the Variables in Orchestrator are exported, not just the ones used in the runbook; therefore, it is very easy to contaminate your installation with variables gained from imported runbooks. Use with caution.

Keep Connection Names in the options tab of Orchestrator generic i.e. BEM not kelvsr13-10.10.10.1-BEM.

WHY? All Connection settings are exported when runbooks from Test / Development Orchestrator into Pre- Production / Production Orchestrator are exported. 

If all instances use the same connection name for their copy of the target tool when Orchestrator Imports the workflow, it will say, “A connection with this name already exists. Would you like to use the existing details or overwrite them with the new details?” Confirm to use the existing details, and Orchestrator will automatically remap any runbook using that connection to point to the new target system without any manual effort. 

Overall, the KISS – Keep It Short and Simple – principle applies here. Additionally, generic is better than specific.

Tip 12 Don’t Use Text Files for Data Repository

Text files can be read by many processes in parallel BUT can only be written to one process at a time. Using a Text File means your runbooks must run sequentially if they are to all write to the same text file.

Using a text file makes Orchestrator a single-threaded processor when Orchestrator is a parallel processing tool.

Tip 13 Implement Orchestrator with Another SQL Runtime Database

Kelverion implementations ALWAYS use a Runtime Database to drive a solution.

WHY?

a. Data reliability and persistency

Suppose the process is to Monitor an Event System, validate the failure and then create an Incident Ticket in a Service Desk. In that case, most users build a daisy chain of runbooks to complete the process, each passing the Event data from one runbook to the next. If any of the runbooks fails, then the whole Event’s data is lost as the information on the published database is thrown away on failure. There is no way to recover or retry that Event as the data is not persisted through the process on failure.

b. Bi-directional Event Management Interface

To implement a bi-directional interface (i.e. Event is closed in the second system, causing Event in the first system to be closed, and vice versa), a correlation
between the Event ID in the first System to the Event ID in the second system is needed. Most Event Management tools do not have a field in an Event to hold
other Event Management Tool Event IDs. There may be Incident Ticket IDs, but this is for the number of the Service Desk Incident Ticket raised for this Event and is already in use. This can be customized through Event Management systems to add an extra field, but this is often a complex and costly activity. Therefore for ease of implementation, we recommend you export this mapping table into a database table.

c. Scaleability

Event Storm – If there is a major failure and many Events are generated, there is a likelihood that the Event Management System API (used by Monitor Activity) will become overloaded. This can cause some Events to not be passed to Monitor Activity. If users are not recording which Events Orchestrator has picked up, there is no way to tell if any are missed. If the
plan is to persist Events into a database, a “sweeper” runbook can be used. 

“Sweeper” runbooks periodically go to the first system and pull back all the Events on that system that have been created since the last time the “sweeper” ran. These can then be compared to the Event IDs gathered against the persisted Events in the database. If an Event is not in the database and got missed, this new Event can be fed back into the system, as if the Monitor Activity identified it.

Therefore, under Event load conditions, all Events will arrive at the second system; possibly a little later than they were generated, ensuring they were not missed altogether.

d. Use a Database Solution for Batch Processing

If there are many Events to process continuously, then a database-driven solution provides a performance improvement as Events can be batched together. Several runbooks can each select a batch of Events and process these together in parallel, in effect multitasking. In a daisy chain process, users are limited to processing Events in a more singular method because the next runbook cannot start until the previous runbook has finished.

e. Use a Database Solution to Ensure Runbook Extensibility

Once you have an event to process ticket operating (such as SCOM to BEM), you may choose to extend or change the solution by:

If a daisy chain point-to-point process exists without using a database, users are limited on what can be amended and changed without rebuilding the entire solution from scratch.
Having a database-driven solution allows users to very easily add capabilities to a process flow, as there is a central source of data from which any runbook can feed and record results.

Why else might users want to use a Runtime Database?

• Users want a run time variable in a runbook?
• Dynamic input data?
• Look up logic?
• Audit History?

Put it in a Microsoft SQL Server Database and use a Database Activity to process it.

• Struggling to Parse Data or Complex Decision Logic?
• Got XML/CSV as an input and need to manipulate the data? 
• Need to produce a report on how often something happens or based on data from many sources?

Put it in a Microsoft SQL Server Database and use a Database Activity to process it.

Tip 14 Deploy the Kelverion Integration Pack for SQL Server

WHY? When adding a database alongside Orchestrator, there needs to be an efficient connection mechanism. The database activities provided with Orchestrator create a connection for each transaction and close it after use, even if there is more data to process to the target table. In scenarios of heavy database interaction, this puts an unsustainable load on the database
access authentication method and, for large updates, can increase the run time.

These activities also require the user to understand and write SQL code.

The Kelverion IP for SQL Server addresses these issues.

1. Maintains the connection with the SQL database to increase performance.
2. Automatically builds and executes the necessary commands without the user having to write or understand SQL, saving time, training investments and eliminating mistakes.
3. Simplifies runbook design by automatically mapping table columns to Orchestrator input properties, filters and published data items.
4. Adds the ability to monitor a SQL table for changes to execute a runbook.

This simplification of workflows is powerful for users experienced in Orchestrator workflow development and those new to Orchestrator.

Tip 15 Document Your Solution – Use Kelverion Runbook Surveyor

When a series of runbooks have been created to meet a business need, these will need to be documented. This often means taking multiple screenshots and manually recording each Activity configuration. Documentation is required to enable others to support and understand what has been built. This documentation process can take longer than the runbook design and creation phase.

What is required is an automated solution to document runbooks and their configurations, and the Kelverion Runbook Surveyor is the answer to this problem.

The Runbook Surveyor enables users to document their solutions automatically. When runbooks have been created, the Runbook Surveyor is pointed at the Orchestrator environment and automatically produces a series of Visio Diagrams and a Word Document. These outputs show how each is configured and connected to other runbooks.

Summary

IT professionals can avoid common pitfalls while implementing an Orchestrator solution by following the best practices outlined in this document. Follow our Orchestrator 2022 Migration Guidance and Top 15 Tips in designing runbooks to ensure success and efficiency in your IT Process Automation projects.

The Kelverion best practice approach to IT Process Automation and the use of System Center Orchestrator involves following the below steps when building runbooks.

Orchestrator is all the more powerful when implemented with its own runtime database. Without a runtime database, there are limits on what can be achieved. However, with a database, there are virtually no limits to Orchestrator’s potential.

Failure to follow the steps in this paper could result in errors, poor service, inefficient automated processes, increased costs and lack of IT process governance and control.

Follow Kelverion’s best practices to remove the frustration from your IT process automation projects.

Further Reading

For further information on System Center Orchestrator, read more online at Microsoft Learn

About Kelverion

Experts in Cloud, On-Premise and Hybrid automation, Kelverion provide solutions and integrations that remove the manual process tying up IT staff; transforming the productivity, efficiency, and supportability of IT service automation. Our products utilise and enhance the power of Microsoft Azure and System Center Orchestrator. 

Working closely alongside Microsoft we have developed our integrations and automation solutions to help bridge the gap between Microsoft’s automation platforms and third-party systems, in the process building key alliance partnerships with multiple vendors to ensure our products are fully certified.

Since 2010, Kelverion has expanded to become a global company, with offices now in the UK, Canada, and the US. Through this, we are able to offer and support products and professional services engagements to enterprise-level organisations no matter where they are.

There’s a smarter approach to IT Service Management
Get in touch to find out more

info@kelverion.com | US Tel: +1 289 801 0559 | UK Tel: +44 203 875 8035