System Center 2012 OrchestratorBest Practices: Lessons Learned at CargillVaughn NerdahlCargillApril 11, 2013

2

SD-B318

Who is Cargill?

3

Cargill is an international producer and marketer of food, agricultural, financial and industrial products and services. Founded in 1865, our privately held company employs 142,000 people in 65 countries.

1000+ locations globally

Orchestrator 2012 was brought into Cargill as part of our migration from Altiris to System Center.

How do we use Orchestrator?

4

• Operations Manager• Configuration Manager• Altiris• Active Directory• BMC Remedy• SQL• Oracle• SharePoint• Exchange

Organization Design

5

Central Team• COE• Best Practices• Manage Environment

Distributed development environments • Key success factors• Individuals dedicated 25%+• PowerShell

COE

Dev

DevDev

6

Runbook Promotion

Production

Q&A

Development

Promotion to Q&A

Runbook best practice review

Promotion to Production

Export / Import Runbook

Change Control

Source Control (TFS)

Export / Import Runbook

Runbook Promotion

7

• Runbook Validation• Change Control• Email Notification

8

Production Environment

Runbook Requests

Web Service

Runbook Server

Runbook Server

Management Server

Web Service

Virtual Servers8GB RAM/Quad Core

Separate physical hosts

Netw

ork

Load

B

ala

nce

r

9

Orchestrator DBDatabase where configuration information, runbooks, and logs are stored.

Shared DBDatabase for multiple purposes to store information consumed by Runbooks.

Security

10

Runbook Designer – dcomcnfg.exe • Remote Launch permission on My Computer• Launch and Access permission on omanagement

Runbooks and Global Settings folders are all security by AD group membership.

No granular security for Global Configurations and log purge settings. Results in multiple development environments.

Demo

Security

Value Capture

12

Metrics!!

Maintenance Mode

13

• Promote secondary server within Runbook Designer• Disable IIS page to trigger F5 failover• Use Orchestrator Health Checker to stop and restart monitor jobs to get

them running on the new primary server• Wait for existing running jobs to complete and/or terminate jobs as needed• Perform maintenance on server (install patches, updated IPs, etc)

Demo

Server Maintenance

Best Practices Folders

15

Folder Structure

Best Practices Runbooks

16

Parent Runbooks – 1-4 word descriptionMonitor Runbooks – Start with the word ‘Monitor’

Child Runbooks

Rename default activity names to make them descriptive

Best Practices - Links

17

Link names should only be modified from the default of “Link” if there the link is used in a logic selection or a parallel execution.

Black = Standard / Success – includes parallel processing

Red = Failure

Turquois = Selection

Orange = Expected to fire multiple times

Runbook Design

18

30 activities or lessAlways start with Initialize Data and end with Return Data

• Design runbooks so that they are generic, modular and reusable.• Consider that child runbooks may be launched by a different runbook server. • Avoid assigning runbooks to execution on specific runbook servers• Use built in actions rather than scripting if possible• Avoid long running runbooks

Error Handling

19

Best Practices

20

Counters• Fear them!!• Shared between all runbooks• Not thread safe

Variables• Use them!!• Runbook Sanitizer

Schedules• Use Global Settings• Avoid embedded schedules

Computer Groups• Rarely used

Job Concurrency

21

Max 20 per runbookWarning: Accessing text and spreadsheet files

Runbook Activities

22

Append Line – Not thread safe

Delete File/Folder – Fails if the item is read-only

Disconnect Network Path – Do not use with runbooks using job concurrency greater than 1

Invoke Runbook – Launches a new instance of policymodule.exe 3-5 second penalty. Avoid using invoke by path.

Query XML - XPath

Invoke Web Services

23

Invoke Web Service - Use username@domain.com for the username within the Security tab when cross domain authentication is needed. Disabled = anonymous authentication.

Runbook Activities

24

Junction - If the object specified within the “Return data from” is executed multiple times then the Junction action will fire multiple times.

<None> = no data from previous activities will be available on the databus.

Map Network Path - Fails if the path is already connected by another user.Connect directly to IPC$ rather than C$ to allow for multiple connections.

Query Database

25

Query Database – Protect against SQL injections by using DECLARE and SET commands in your queries.

DECLARE @ComputerName nvarchar(MAX)SET @ComputerName = ‘{Computer Name from “Intialize Data”}’Select Top 1 i.guid, i2.nameWhere i.name = @ComputerName

SELECT REPLACE([Test],';','_') FROM [OrchestratorCGLAdmin_prod].[dbo].[Test_Table]

Replace delimiter values

Run .Net Script

26

Run .Net Script • Use PowerShell• Exit = No published data• “Error initializing extension” - Field data manipulation function

with invalid variable reference.• Unstructured text

$Variable = @’ I wish this line didn’t contains any “double” quotes.‘@

• PowerShell Injection ';stop-computer -computerName "server1“’

• "Cannot invoke this function because the current host does not implement it." - Powershell Function is wanting to interact with the user. Classic example is CLS command.

• Orchestrator has its own internal Powershell 2.0 x86 engine.• PowerShell 3.0 – Use “PowerShell { <script goes here> }”

Looping

27

• Enable looping for automatic retries.

• Do Not Exit > Exit conditions = infinite loops

Front-End Web Site

28

Orchestrator needs a user friendly front-end web site!

• System Center Service Manager

• SharePoint

• 3rd party (e.g. http://eupsco.com/)

• Internally developed web site

Development Tools

29

Must have tools for runbook development

• PowerGUIUsed to create and debug PowerShell scripts

• ExpressoUsed to create and debug regular expressions to search text for patterns

• Remote Server Administration Tools (Active Directory)Used to query active directory

• SQL Management StudioUsed to query and update SQL server databases

• Oracle SQL Developer Used to query and update Oracle databases

Visio and Word Generator

30

Use the Orchestrator Visio and Word Generator utility to automatically document your runbooks.

CodePlex

31

Community IPs

SCOJobRunner.exe – command line runbook execution

Powershell examples - manage runbooks remotely.

Orchestrator Health Checker – runbook management

Additional Information

32

Orchestrator Forums

Twin Cities Orchestrator User GroupEmail: vaughn_nerdahl@cargill.com

Best Practices Document

System Center Orchestrator - Training

© Copyright Cargill, Incorporated 2012. All rights reserved.

Evaluation

Complete your session evaluations today and enter to win prizes daily. Provide your feedback at a CommNet kiosk or log on at www.2013mms.com.Upon submission you will receive instant notification if you have won a prize. Prize pickup is at the Information Desk located in Attendee Services in the Mandalay Bay Foyer. Entry details can be found on the MMS website.

We want to hear from you!

Resources

http://channel9.msdn.com/Events

Access MMS Online to view session recordings after the event.

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

System Center Orchestrator - Training

© Copyright Cargill, Incorporated 2012. All rights reserved.

Appendix

Architecture - Services

Management Server

Runbook Server

References

It is responsible for maintaining the orchestration database, communicating with the Runbook Designers, and communicating with the Deployment Manager.

It is responsible for monitoring the health of runbook servers.

It is responsible for running runbooks and for communicating with the orchestration database.

It enables the Orchestrator Deployment Manager to deploy Runbook Designer, Runbook Server, or integration packsIt is responsible for running a program or command defined in a runbook.

Run Program

38

Run Program• Leverages the Orchestrator Remoting Services service on the

Runbook Server

• Calls opexecsvc.exe = modified psexec.exe to remotely execute the command.

• Interactive mode not supported on Win 7/2008+

• The Security Credentials tab is used to define the credentials used by the opexesvc.exe to connect to the remote computers \admin$ share.

• The Run As credentials within the Advanced tab are used to configure the user account that the opexecsvc.exe should run the specified command as.

Embedded Credentials

39

1. Avoid them2. Use Encrypted Variables

2. Create an Integration Package using the Command-Line Activity Wizard

Restart System

40

Restart System• Minimum 30 second delay

• Runbook does not wait

• Message displayed to logged on users who can close the window but it does not stop the reboot.

Templates

41

Provide a place for runbook developers to share sample code, runbooks, etc.

_Templates folder

Integration Toolkit Command Line Activity Wizard

Full Integration Pack

Robocopy /MT

42

• Robocopy.exe /MT switch will allow multi-threaded file copies which greatly improved performance however each file that is copied is loaded into memory so be aware of how many instances of robocopy.exe will be allowed to run and the size of files that will be copied.

• The recommendation is to use /MT:8 by default.

• Behavior – Unable to RDP into server, new runbook executions stop, robocopy log files not updated.

Runbook Execution

43

11

Runbook ServerData Store

(Run processes)

GUI

(Store process logic) (Design, manage, report)

RunbookDesigner

Management Service

<XML><XML>

001000001010PolicyModule.exe

Operator Console001110

Troubleshooting Runbooks

44

Folder Path File Name Description\ProgramData\Microsoft System Center 2012\Orchestrator\RunbookService.exe\Logs\ RunbookService.exe.*.log Exceptions generated

by the Runbook Server service (connection errors, etc) are logged here

\ProgramData\Microsoft System Center 2012\Orchestrator\PolicyModule.exe\Logs\ PolicyModule.*.log

\ProgramData\Microsoft System Center 2012\Orchestrator\ManagementService.exe\Logs\ ManagementService.exe.*.log

Contains exceptions generated by the Runbook Server. This log is useful for finding environmental problems (e.g. database connect failures)

\ProgramData\Microsoft System Center 2012\Orchestrator\RunbookServerMonitorService.exe\Logs\

RunbookServerMonitorService.exe.*.log

Best Fit

45

• Integrate – don’t duplicate

• Avoid creating runbooks that duplicate existing tools.

• Example: System Center Operations Manager, Configuration Manager, etc.

Runbook Sanitizer

46

Use the runbook sanitizer to clean runbooks that are moved between environments.

The export file was shrunken from 676KB to 25KB so there was about 650 KB worth of useless configuration information in the file (and that was an export from a very clean environment).