lotusphere 2012: bp117 server colsolidations done right

© 2012 IBM Corporation

BP117 Shrinking Your Environment - Server Consolidations Done Right

Francie Tanner | Director | panagenda

| © 2012 IBM Corporation2

Agenda■ Introduction

■ Why Consolidate

■ Planning your Server Consolidation

■ Planning your Clients' Server Consolidation■ Consolidation Tips and Tricks

■ Summary and Q & A


Legal disclaimer© IBM Corporation 2012. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.


Introduction■ Francie Tanner, Technical Director, Americas

─ Over 14 years experience in IBM Lotus consulting

─ Managing, architecting, and supporting 10 – 100'000 user environments

─ Experienced Lotus instructor and speaker

─ Is from Switzerland, got cold, now lives in the Caribbean



■ Why Consolidate





Reasons to Consolidate■ To reduce the number of servers to support

─ Less servers = less maintenance and often less licensing costs

■ To work with reduced resources─ Less staff or budget may mean consolidating into less servers

■ To merge with a new company─ Typically there are economies of scale when acquiring another Notes company─ A consolidation and a domain merger may be in order

■ New hardware─ Purchasing more capable hardware can make it possible to accommodate twice as

many users on one server



■ Why Consolidate





Planning Your Server Consolidation■ Can you already quantify the desired end result?

─ For example: to reduce 30% of all servers─ For example: to eliminate all servers on the west coast

■ Start taking inventory and asking loads and loads of questions.....

■ What hardware and OS are all involved servers running─ “Show stat platform.net* can be helpful here

■ What IBM Lotus Domino® server version are all involved servers on─ Include hot fixes, patches etc

■ Which servers are clustered─ Map this out


Planning Your Server Consolidation■ Are the involved servers already replicating

─ If so, then what, when and how?

■ Can all involved servers currently communicate via DNS─ If not, then find out what is required to make this happen

■ Who/what is currently monitoring all involved servers─ These services as well as back-ups will need to be moved

■ Are all involved servers in the same Domino domain─ If not, then you will be consolidating AND merging domains─ If not, then you will also need to consider how to handle merging of the two Domino

Directories, Admin servers, as well as ID Vaults


Planning Your Server Consolidation

■ How are the involved servers connected from a bandwidth perspective─ Make a map of bandwidth and connection points

■ Are there mail users involved in the consolidation─ If so, what client version are they on

– Much more on this in the next section

■ Which servers have DAOS enabled─ This will need to be re-created on the destination server

■ Which servers have customized templates─ These are often forgotten about

■ ....


Deciding what remains■ After considering all of the above, deciding which servers go should

be easy─ Between hardware and bandwidth and other factors, this will be obvious

■ Now you have a list of servers who will get turned off─ Congrats!

■ Calculate the impact of decommissioning these servers─ More on this on the next slide

■ Inventory all applications and templates on servers which will get decommissioned─ Admin client, files tab, export to your favorite spreadsheet editor─ Gather template, ODS and other information


Calculating the decommissioned server impact■ How busy are the servers you are working with

─ This is huge not only before your consolidation but also during and after─ Statistics is what we're after here, start with transactions per hour, disk and memory

utilization etc

– Technotics had a brilliant session on this topic “BP106: Spreadsheets and IBM Lotus Domino Statistics in a Blender”, download those slides

■ Who exactly is accessing the servers to be decommissioned?─ Domino statistics are your best friend here!

─ Activity logging and trending helps with this

■ What mail routing functions do all involved servers have─ SMTP or NRPC routing via connection or other documents

■ What connection documents are present to/from the server to be turned off?─ Again, graphs are very helpful here


Calculating the decommissioned server impact■ Are there any hard-coded references to the soon-to-be-gone server

─ This can be in the form of agents, other code, etc– Client too but we'll discuss those in the next section– For a thorough job with this, 3rd party tools may be needed

■ Which groups are these servers in?─ And why? For ACL purposes? Mail routing?

■ Do all servers have admin rights to each other─ Without this AdminP or new replicas

won't work

■ Run the “Find Server” tool─ Domino's native way to look at connection

docs, ACL's, configuration documents etc


Calculating the decommissioned server impact■ What ports and tasks are being used by all involved servers

─ Do firewalls/ports/permissions need to be updated?─ Which tasks will need to be enabled on the remaining server?

■ Are the involved servers performing archiving or other special functions─ Don't forget to look at program documents

■ Are any of the remaining servers transaction logged?─ If not, then why not?

■ Which applications should go on which remaining server?─ Consider clustering requirements

─ Do NOT move according to db or mail file size

– You do not want to move the most used apps onto the busiest servers


Consolidation Styles: Replicate vs. OS Copy■ Consolidation =

move applications(create new replicas +delete original one) +turn off old server

■ How to get the data there: Replicate─ You can replicate single databases or entire folders

─ Pros: All databases get created by Domino in the “proper” way

─ Pros: Minor corruption often gets fixed by creating a new replica

– Requires a connection document from the original to the new server

─ Cons: Replicating an entire server may take a while (but get over it)

■ How to get the data there: OS Copy─ Pros: NONE, DON'T DO THIS!─ Cons: EVERYTHING, THIS IS NOT GOING TO WORK OUT!

■ Accelerated replication is your bestest friend here─ More on this later


Consolidation Styles: Merging vs New Server Creation■ Merging data onto an existing server

─ Create new replicas, re-direct everyone to the new server, turn old server off

■ New server consolidation─ Consolidating two (or more) servers into one new one

─ Still “just” new replicas

─ As long as all server pointers are addressed, this is no different than the above


Consolidation Styles: Gradual vs. Instant Switch-Over■ Gradual consolidation

─ The originating and destination servers are up at the same time for a while

─ Pros: This allows for a more relaxed time-line and plenty of testing

─ Pros: Consolidating this way requires no down-time

─ Cons: This type of consolidation takes longer

■ Instant Switch-Over─ Re-creating all data on the remaining server at once

─ Pros: Overall faster than the gradual approach

─ Cons: Requires down time, usually after hours

─ Cons: Leaves less time for thorough testing


AdminP Is There For You■ AdminP is the little engine that could and is in

charge of:─ Checking required access─ Creating all new replicas─ Updating some local client information

■ AdminP requires the Admin server for your names.nsf and admin4.nsf to be set─ AND to be constantly replicating - Else all this falls apart─ Also make sure you have a certlog.nsf on your Admin server

■ Check your “action required” views in Admin4.nsf frequently─ Error messages too

■ AdminP will be super busy with database movesand even more so with renames (mergers)─ MONITOR, MONITOR, MONITOR!



■ Why Consolidate





Planning your Clients' Server Consolidation■ Moving databases is the easy part, getting clients “pointers” to move

is difficult:─ Workspace icons

─ Bookmarks

─ Location documents

─ Replicator page entries and replication history

─ Connection documents

─ Doc links

– Outdated doc links are virtually impossible to “fix” but the redirector tool can help

─ Delegated mail files and calendars

■ The Redirector Tool and Policies can often help here─ NOTE: Requires the original server to be up and running in order to function

─ NOTE: This won’t help with managing replication history


Database Redirect Tool■ New to version 8, found in the Admin client, creates a .nrf file


Database Redirect Tool■ Automatically invoked when moving or deleting databases


Database Redirect Tool■ Determines which users or groups or ACL entries will be redirected across:

─ Client Bookmarks

─ User Workspace icons

─ Replicator entries

■ Database Redirector Challenges─ Requires all servers and clients and templates to be on > ND8

─ Cannot manage replication history which is used by the client to determine replication settings

• However, this is solvable via 3rd party apps

─ If you restrict a database redirect to users, this feature may remove all references to a database and act as if the database was moved or deleted

─ Requires the original server to be up and running in order to function

– For as long as you want anything redirected – this may be counter-productive

─


Policies That Help■ In addition to using the database redirect too, policies can help

─ Create new local replicas from the new servers

– But not put the new icons on top unless you are creating managed replicas

─ Add bookmarks

– Beware of the “catalog.nsf challenge” - more on this next

─ Update catalog, directory and IBM Lotus Sametime® server names

─ Adjust widget catalog servers

─ Change Passthrough servers

─ Create managed replicas

─ Upgrade your client ODS

• This is a big deal!


The Policy Challenge■ Policies depend on an already functioning/setup client

─ This is typically less than 75% of users actually receiving policies– Your $Policies view, Admin server for your mail file and other factors matter here

■ They don’t provide you with an inventory before making changed─ Client Management “in the dark”

■ They don’t adapt to your users’ unique situation─ LAN vs VPN, Citrix user, function outside the data directory

■ They aren’t entirely predictable─ Can happen after startup.... or not...

■ Most settings cannot be UNset once set─ Think about it...

■ They cannot typically repeat actions─ If the user “messes with” something it’s usually broken until they call for help


The catalog.nsf challenge■ Many Notes client features look up database information in your

home/catalog servers' catalog.nsf─ If that is outdated, you have big problems

■ Using the search bar looks up information in your servers' catalog.nsf

■ Once you turn off old servers, your client depends on the workspace and catalog.nsf on your new home/catalog server

■ The Problem: Catalog.nsf often answers Notes in alphabetical order─ AsiaMail01/panagenda may be the answer to “where is the Accounting application”

– Uh-oh!

■ Make sure you have an updated catalog on all servers─ Use selective replication or 3rd party tools to help with this issue


AdminP is there for you■ In addition to moving apps, AdminP can also change the location

document mail server fields─ But this isn't 100% reliable

─ Especially an issue if you are doing renames or a domain merger

■ AdminP also moves your roaming Files and updates your person doc

■ AdminP can't help with connection documents─ Especially problematic if hard coded with IP addresses

– DNS settings and/or custom scripts can help deal with this



■ Why Consolidate





Rights Management■ Before you move your applications, ensure all servers have appropriate rights

─ This includes to the server itself as well as ACL's

■ ACL Management “en masse”─ Add the remaining server to the appropriate ACL group

– Never hard code any name in any ACL─ Use CTRL + A, add that ACL group to ALL originating servers' applications

– WATCH OUT for error messages, no rights means neither you nor Adminp can move apps


Server Rights Management■ Consider database and template create, admin and programmability rights

─ AdminP, agents and other processes need this to move or run apps


Breaking Up and Re-Creating Clusters■ This is not difficult but can save you hours of time if done before moving apps

─ Make sure all involved servers have create/delete db rights─ Make sure the receiving server has rights to all apps you're moving

■ Put the originating and remaining server into a cluster FIRST─ And then move databases, which will use streaming replication to create the apps

– Several conditions can make this fail, however there's work-around on the next slide

■ Don't forget to exclude databases in the cluster manager that should not be clustered


Using Accelerated Replication without Clustering■ Accelerated replication will save you hours BUT:

─ Requires specific clustered version of Domino = 8

─ Doesn't work on DAOSified applications

─ Full text settings must match on the source and destination servers

─ Use the following to bypass the accelerated replica restrictions:

■ Add Adminp_Accelerated_Replica_Overrides=5 (or 1 or 4) on the source server─ Then move your apps as you wish ;)


Don't forget Unread Marks■ Enable unread marks replication BEFORE you move apps

─ Select desired apps, right click, Advanced Properties ─ AdminP is in charge of making this happen also

■ Exchange unread marks is a built in feature if you move apps


Compare and Consolidate■ Before you START/END your consolidation, compare what's different:

─ Use the Cluster Analysis tool if servers are clustered

─ Use the “Decomission Server” tool if they aren't

■ Depending on which you use, checks:─ Mismatching ACL's

─ Disabled replication on applications

─ Server configuration details

─ Missing databases

─ Connection documents

─ Selective replication formulas


Failing Users over to the new server■ If you have both the source and destination servers up, users may

still be pointing to the “old” server─ The database redirect tool can help here, but let's pretend you're in a hurry

■ You can set the source server to be restricted via:─ Server_Restricted=1 – server will be restricted to users during the server session

– A restart will set this to 0 and once again make the server available to all─ Server_Restricted=2 – keeps the server restricted to all users

– never admins though

■ You can also use this to redirect users to cluster servers if needed


Upgrading the ODS■ This is not really part of consolidations, but …

─ Your apps will run faster and require less space

─ No really!

■ Impact examples:– Startup time of a Notes 8.5.2 client with 3 ODS 20 apps in Notes data = 10 seconds

• After ODS upgrade: 2 seconds

– Reduce File I/O of your disks/SAN/NAS after ODS 41 5o 51 upgrade = 60%

– Removing 70% of all old files such as templates etc in Data directories on SAN/NAS = 45% less managed storage space to manage and backup


Making Notes run Faster: ODS Policy■ New to 8.5.2, you can force a local ODS upgrade (in most cases)

─ NOTE: If you previously deployed Create_R8_databases to the INI then this feature won't work─ NOTE: During compact databases cannot be accessed


Adjusting AdminP■ To make AdminP work more efficiently during a consolidation, try:

─ Adjusting adminp threads in your server document or via Adminp_Immediate_Thread=X or Adminp_Interval_thread=X

– CAUTION: Never adjust threads without properly monitoring performance

– There are more override parameters available, look them up in Help

─ Adjust replication frequency of admin4.nsf to be at least that of names.nsf

─ Use the “tell adminp process daily to help with unread marks or ”tell adminp process restart” as appropriate

─ Use selective replication formulas to keep admin4.nsf at a reasonable size

■ Almost all tasks involved in consolidations can be adjusted─ Router, adminp, clusters, indexers etc

─ sh st replica.cluster.workqueuedepth*

─ CLUSTER_REPLICATORS=#


Adjusting AdminP



■ Why Consolidate





Summary■ Plan, analyze, plan, question and then plan again

─ This will save you hugely on project and support time/cost AND your stress level

■ Monitor your servers, before, during and after─ You cannot add more strain to a server which is already doing badly

■ Move servers/apps/users intelligently─ Don't move by db size, for example, move by usage, bandwidth, etc

■ Use policies and the redirect tool whenever possible─ And write an inventory agent, use login script, or get a 3rd party tool if needed

■ Before you do anything, make sure AdminP is happy in your environment─ If this isn't working before you consolidate, nothing good will happen

■ Any challenge can be overcome with Knowledge and Tools


Q&A■ Got questions?

■ Feel free to contact me with further questions:

lotusphere 2012: bp117 server colsolidations done right

Technology

involved servers

ibm products

ibm logo

admin servers

number of servers

use of ibm software

standard ibm benchmarks

ibm lotus consulting