a sas r&d case study

Deploying SAS®9 on SolarisTM 10: Containing Cost, Consolidation, &

Capacity Challenges with Virtual Simplicity

A SAS R&D Case Study

Maureen ChewStaff EngineerSun MicrosystemsMarch [email protected]

2 Sun Microsystems, Inc.

Table of Contents

Many IT Challenges – Complex and Diverse.......................................................................... .........................3

Solaris 10 Containers – Addressing Cost, Consolidation and Complexity..................................................3

Lessons Learned and Best Practice Considerations.......................................... ...........................................4

● Third Party Application and Version Considerations.................................... ...........................................5

● Application and Performance Considerations.................................................... .....................................5

● Whole vs. Sparse Root Zones.......................................................................... ......................................6

● Zone Root Location................................................................................................................................. 7

● Shared Binaries................................................................................................. .....................................8

● Memory Requirements............................................................................................................................ 8

● Resource Management........................................................................................................................... 8

● The Role of ZFS.................................................................................................................................... 10

● Solaris Fingerprint Database (sfpDB)................................................... ................................................10

Summary............................................................................................................... ...........................................11

Acknowledgments................................................................................................................. ..........................11

References.......................................................................................................... .............................................11

Appendix A – Introduction to Solaris 10 Containers .............................................. .....................................12

Appendix B – Container Creation Cookbook................................................................. ...............................14

3 Many IT Challenges – Complex and Diverse

Many IT Challenges – Complex and DiverseAt SAS headquarters in Cary, NC, R&D staff responsible for both 9.1.3 and 9.2 product development, testing,

and coordination with global development teams faced major challenges. Solution development for 9.1.3 was

continuing at a rapid pace while efforts towards the next generation 9.2 platform was ramping up. Not only

were there two major development release trains, but as with typical, large software development efforts, there

were nightly and weekly builds for each development track. On the testing front, there was the obvious need

for unit and integration testing. SAS R&D realized that production enterprise deployments differed from

dev/test deployments and required modeling various production, enterprise class environments. Consequently,

the need to support enterprise class, customer centric test deployments became an added requirement.

Because of the convergence of 2 major simultaneous development efforts, a plan needed to be put into place

to address the complex and diverse needs of the various development and test communities. Faced with the

typical challenges of today's IT data centers, the cost and management complexity for multiple compute and

data servers could easily have grown out of bounds. The needs and requirement challenges could be

summarized as:

● Architect and validate realistic, enterprise ready SAS deployments

○ Discover and document system management best practices

● Create Test/Production Environments for both 9.1.3 and 9.2 releases

○ Provide ability for rapid provisioning of test environments

● Support global development teams who collectively require follow the sun support

● Foster cross product collaboration

● Create common test data repositories for testing and debugging

● Regression Testing

● Migration Path Testing

○ Inter-release, Intra-release

● Consolidation of Administrative resources

○ Reduce server and storage requirements

○ Reduce software installation duplication

Solaris 10 Containers – Addressing Consolidation, Cost, and ComplexityAfter careful evaluation of various alternatives, a project proposal was built around the use of Solaris 10

Containers to meet the urgent and critical needs of this diverse community. (If the reader is not familiar with

Solaris 10 Containers, the concepts of whole vs sparse root zones, and zone roots, see Appendix A - An

Introduction to Solaris 10 Containers). A Container is a superset of the combination of Zones and Resource

Management capabilities. In this paper, the terms Containers and Zones are

(unofficially) used interchangeably. Additionally, SAS R&D felt that Solaris provided a

rich set of tools such as dtrace(1), truss(1M), pfiles(1) that could help debug

problems should they arise.

The SAS R&D proposal was to create a Shared Production and Test Environment.

Solaris 10 Containers would be used to create virtual instances of different

deployment options. After the proposal was given the green light, the project (code

named: PRDSHARE) went forward quickly and staged on the following configuration:

● SunFire E2900

4 Solaris 10 Containers – Addressing Consolidation, Cost, and Complexity

○ 8X1.5 GHz dual core UltraSPARC IV+ processors

○ 128GB RAM

● Sun StorageTek 6540 - 6+TB

The initial deployment consisted of over 20 Containers; half for 9.2 testing and the other half dedicated to 9.1.3

testing. Figure 1 below is a logical diagram of the containers used to support 9.2 testing. A similar set for 9.1.3

was also configured.

Figure 1: PRDSHARE 9.2 Container Deployment (7 non-global containers shown)

Lessons Learned and Best Practice ConsiderationsPRDSHARE is a sophisticated environment with over 20 zones in which each zone is supporting its own

complex configuration of applications. We outline some lessons learned and best practice considerations:

● Third Party Application and Version Considerations

● Application and Performance Considerations

● Whole vs. Sparse Root Zones

● Zone Root Location

● Shared Binaries

● Memory Requirements

5 Lessons Learned and Best Practice Considerations

● Resource Management

● The Role of ZFS

● Solaris Fingerprint Database (sfpDB)

Third Party Application and Version Considerations - A sample of some of the 3rd party components

for each major SAS release is listed below. Check the SAS site below for the official current and

comprehensive support listing:

http://support.sas.com/resources/thirdpartysupport/v913sp4/index.html

Third Party Software Support 9.1.3 SP4 9.2

Java Runtime Environment (JRE) 1.4.2_09 Java 5, Update 12

BEA Weblogic 8.1 SP4, SP6 9.2

IBM Websphere 6.0.2.15, 6.0.2.19 6.1.0.2

Xythos Webdav 4.2.34.4, 4.2.35.3 4.2.34.4

At the current time, there have been 4 update releases to Solaris 10:

● Solaris 10 8/07 - Update 4




● Solaris 10 Initial General Availability (3/05)

When non-global zones are installed, the upgrade process to move from one Solaris 10 Update release to the

next requires careful planning, especially as the number of zones increases. It is highly recommended that any

new Solaris installation begin with the latest available Update release, today Solaris 10 8/07. The PRDSHARE

project launched with Solaris 10 1/06 and was subsequently upgraded to both Solaris 10 11/06 and Solaris 10

6/06 as those releases became available. The length of time to complete the upgrade increases incrementally

with each additional zone hosted on the platform. With the introduction of Solaris 10 8/07, Live Upgrade

support of systems hosting zones was added. Live Upgrade continues to be the recommended method to both

patch and upgrade Solaris systems, especially those hosting Zones. For further information, visit:

“Upgrading a Solaris 10 System That Has Installed Non-Global Zones”

http://docs.sun.com/app/docs/doc/817-1592/6mhahup0m?l=en&a=view

Application and Performance Considerations – From an application perspective, what are the

differences between running in a non-global zone and running in a global zone? Zones provide a secure and

isolated environment which cannot compromise processes in either other non-global zones or the global zone.

As such, there are a few restrictions that a user or user application may run into. First and foremost, from a

non-global zone, processes do not have access to any other processes outside of their zone. prstat(1) will only

show information about processes local to the zone running the command. Additionally, applications or utilities

that need to read kernel memory or that install kernel device drivers will not work in a non-global zone. Many

of the standard system monitoring utilities (i.e.: prstat(1), vmstat(1), zfs iostat(1), etc) can be used at the non-

global zone level but for the ones that can't, they can be run, unmodified, at the global zone level given proper

authorization.


http://docs.sun.com/app/docs/doc/817-1592/6mhahup0m?l=en&a=view


DTrace(1) has three distinct privilege sets; dtrace_kernel, dtrace_user and dtrace_proc. While dtrace_kernel

operations cannot be initiated from within a non-global zone, beginning with Solaris 10 8/07 (Update 4)

dtrace_user and dtrace_proc privileges can be assigned to a zone. This allows the opportunity for developers

and application owners within zones to leverage DTrace. Of course, in any of the Solaris 10 releases, the

ability existis for privileged administrators within the global zone to use all DTrace capabilities to

comprehensively observe all zone operations. Subsequently, DTrace can be leveraged to effectively observe

kernel, user and process interaction of all ( SAS or non SAS) applications both within and across zones.

At this time, there are no known issues with running SAS applications in zones. A few SAS binaries such as

elssrv and sasauth require setuid root privileges but that presents no issue because each zone has its own

specific set of root privileges and permissions. For further information about application heuristics in zones,

visit:

“Bringing your Application into the Zone”

http://developers.sun.com/solaris/articles/application_in_zone.html

One item to note is on the common practice of using shared home directories where a given user is presented

the same home directory regardless of the server they log into. This is not specific to running SAS in multiple

zones; its relevant if running SAS across multiple systems as well. A SAS installation requires several different

user ids. It is not good practice for these users to have a common home directory across zones or systems.

SAS installation state is stored in $HOME/vpd.properties and $HOME/.sasprefs and could become corrupted if

overwritten by an alternative installation.

Unlike other virtualization technologies which translate and/or emulate requests to I/O devices, that is not the

case with zones. I/O requests are directly handled by the global zone so there is no performance penalty for

applications that are heavily I/O dependent as is the case with many SAS applications. In general, there is little

to no performance overhead from running in a zone – typically, its less than 1%.

The benefits of containers/zones are many, with few, if any, tradeoffs and constraints. Containers serve to

maximize server utilization, create flexible, agile IT deployment platforms, provide secure, isolated

environments with little to no performance penalty. This provides a seamless foundation for virtualization

strategies.

One of the constraints of container based virtualization is that all the containers have to be, more or less,

running the same version of Solaris. One zone cannot be running Solaris 10 6/06(U2) if the global zone is

running 8/07(U4). Note: BrandZ Containers do allow different versions of the OS to run in a container (i.e.:

Solaris 8 and 9 in a Solaris 10 SPARC Container) and even different OS' (i.e.: Centos in a Solaris 10 x64

Container) but this is not, at this time, a recommended way to run SAS applications.

Early in the architecture phase, PRDSHARE was designed to be oversubscribed in terms of user population.

Similar to the way that airlines routinely overbook flights on the premise that there will be a given number of late

cancellations, no shows, missed connections, etc, PRDSHARE was configured to have many more containers

or zones than what the system could reasonably handle should everything be running concurrently. To be

more precise, Solaris could handle the load just fine, rather, the resulting service level may not be acceptable



to the user community. The current system has 16 cores and 21 zones configured. Each zone is running its

own set of SAS services or third party applications. Just considering CPU resources, if the cumulative CPU

utilization for all applications in a given zone utilizes 4 cores on average (some more, some less), the system

would need approximately 80 cores, or roughly 5 times as many as configured to service all of the applications

with minimal CPU wait time (and that isn't counting CPU resources needed for the Solaris Operating

Environment itself). While all the zones are booted with the SAS and application services running, not all of

them are in use at any given time. Suffice it to say that close monitoring is required as user communities start

and stop their testing/usage cycles since this configuration could easily be running at 100% utilization with

relatively few zones in full action.

While PRDSHARE was configured in this oversubscribed fashion, only a given subset of zones were in use at

any given time. The powerful Container resource management features provide insurance, flexibility and agility

to dynamically adjust CPU and memory allocation should the requirement for resource prioritization arisen.

Whole vs. Sparse Root Zones - When zones are configured, one of two categories of configurations exist

- Sparse and Whole Root Zone. A sparse root zone is one where one or more directories are shared from the

global zone via a read-only loopback file system (LOFS) mount. When a zone is installed, the default

configuration will be a sparse root zone where /lib, /platform, /sbin, and /usr are shared or inherited by the non-

global zone from the global zone. Obviously, the fewer directories which are shared or inherited implies that

the level of sparseness will converge towards zero and a sparse root zone will start to more closely resemble

a whole root zone.

Whole root zones are necessary if an application or its installation needs to write into directories that would

otherwise be mounted read only if it were a sparse root zone. For instance, an application might need to

create symbolic or hard links into /usr or requires installation into /usr/local. Whole root zones provide

maximum configurability options since all of the required and any selected optional Solaris packages are

installed into the private file systems of the zone. Whole root zones take on the order of 10 Gigabytes of disk

space while a sparse root zone would consume on the order of 100 Megabytes. Additionally, whole root zones

take longer to provision and upgrade.

In general, creating sparse root zones and sharing installations at a global level is recommended and more

efficient. However, there are very valid, and sometimes mandatory, reasons for choosing whole root zones

instead. Once a zone is installed, you can't switch from a whole root to sparse root or vice versa. So, in the

absence of knowing in advance whether any third party software package will need to write into /usr, the only

option would be to use a whole root zone. If it is not a problem to later re-install application software down the

road, then choose the sparse root option with the possibility that the zone might need to be re-created and

previous application installation and configuration re-done.

In the PRDSHARE environment, 3 out of the 20+ were created as whole root zones; the rest were created as

sparse root zones. Two of the whole root zones were needed because of 3rd party application requirements

(Websphere & Webseal), while the third one was to have a whole root zone available for additional

testing/comparison. Note: Per the “How to Get Started with IBM WebSphere Application Server on Solaris 10

and Zones” paper, it is not necessary to install Websphere in a whole root unless the Websphere web server

plugin needs to be installed. In this case, Websphere needs to create symbolic links in /usr and will require a


whole root zone in order for the installation to complete successfully.

Zone Root Location – The zone root directory is the directory root in which all the zone system files are

stored. For a whole root zone, it effectively contains a copy of the Solaris operating environment. PRDSHARE

was initially configured with all of the zone roots located on the primary boot disk (which was mirrored with

Solaris Volume Manager). An obvious concern was the resulting I/O workload just to service all of the zones.

Indeed, this was the case, as I/O contention to these disks surfaced early. Because the default SAS

configuration points SAS WORK to /usr/tmp (which is a symbolic link to /var/tmp), it was quickly determined

that the majority of the zones were unknowingly sending great streams of I/O to this primary boot disk which

also housed all of the Zone ROOTs and was thus creating a major I/O bottleneck. There was an easy and very

effective solution to this problem – move SAS WORK to the faster, more powerful external Sun StorageTek

6540 storage. Once that minor SAS configuration was done, I/O to the primary boot disk was no longer an

issue.

A good disk layout strategy is to install all system and application software to the boot disk and have distinct

storage pools/luns/filesystems for user home directories and other ones for data storage. Separation by some

distinct factor is generally a good idea. This strategy supported the rationale for locating all of the zone roots

on the boot disk. While different enterprise environments may warrant or require spreading out the zone root

locations, there were several over-arching reasons as to the rationale for the PRDSHARE environment:

● Simplicity – the SunFire E2900 had 2 internal disks which were configured as a mirrored pair. External

storage to the Sun StorageTek 6540 was connected via fibre channel connections. In this case, all OS

and application software resided on the internal disks while all data resided on external storage.

● Efficiency – out of the 20+ zones which were configured and installed, only 3 were configured as whole

root zones, the rest were configured as sparse root zones. Additionally, several of the SAS

installations, Java runtime environments, and database clients were installed to the global zone and

inherited by the non-global zones. Thus, application software is shared both logically and physically by

the non-global zones.

● Upgradeability / Availability- A major benefit to having the zones all co-located on the primary boot disk

was to facilitate the upgrade process. Because of the number of configured zones, the upgrade

process was fairly lengthy – for every zone, every patch must be inspected for appropriateness and

install-ability. Live Upgrade was not supported until Update 4 and downtime needed to be minimized

because of the large community of users. Since all the zones and application software were co-

located on the primary boot disk, the disk mirror was taken offline, removed, inserted into another

system and upgraded over the course of 2 days (the upgrade didn't take that long but additional time

was needed to work through peripheral complications which required manual interventions). Once the

disk was upgraded and replaced back, it was booted as an alternate boot disk. Once everything was

deemed satisfactory, a mirror sync was initiated as a background task with the newly upgraded disk as

the primary disk. The end result was that an upgrade was performed, in the absence of Update 4 Live

Upgrade support, with minimal downtime.

In this particular situation, it worked out well that all the zone roots were co-located on the primary boot disk.

Obviously, I/O contention has to be carefully monitored. For this situation, once extraneous (all non application

and system binary services) I/O was relocated to external storage, the I/O workload for application and zone

servicing was well under control.


Shared Binaries – As discussed previously in Zone Root Location Efficiencies above, sharing application

installations leads to a reduction in storage footprint and complexity of administration in having to maintain

multiple copies. Its worth noting separately that additional efficiency is gained because text code segments

can be shared at the kernel level if they are loaded from the same source resulting in a smaller kernel memory

footprint.

Memory Requirements - In additional to the cumulative memory requirements for all applications running

in a given zone, each zone will consume roughly 40-64MB of memory for zone management. There is little to

no performance impact for applications running in a zone (<=1%).

Resource Management – Solaris 10 includes a powerful hierarchy of options to provide resource

management at the CPU, memory and network level. Resources can be controlled at the higher level global

zone level as well as within a non-global zone.

From the global zone level, resources can be allocated to non-global zones through dynamic resource pools.

Solaris 10, Update 4(8/07) introduced powerful enhancements to this already rich set of dynamic resource

allocation feature set. Update 4 introduced:

● Exclusive-IP zones (default is shared-IP) which provide full IP-level functionality in which zone

assigned interfaces can be fully plumbed, modified via ifconfig and configured with IP Network

Multipathing (IPMP) and IP routing.

● Zone based memory capping – While previous releases supported memory resource allocation at a

project level (via rcapd daemon running in the non-global zones), rcapd(1M) running at the global

zone level is now supported. rcapstat(1M) is now zone aware and can report attempted and actual

mount of memory paged out. The ability to cap swap space is also new to Update 4.

● Support for locked memory – tagging memory that is not eligible for paging. This attribute affects

Intimate Shared Memory (ISM) as well as mmap(2). SAS primarily uses mmap(2) for memory

allocation.

● For more information on “What's New” in Update 4 non-global zone configuration, visit:

http://docs.sun.com/app/docs/doc/817-1592/6mhahuooj?l=en&a

In summary, at the global zone level, resources can be dynamically allocated to given containers via CPU

poolsets, CPU shares (through the Fair Share Scheduler-FSS) and via memory capping (physical, swap,

locked). Within a container, the same directives can be used as well as imposing user or application CPU time

limits, file or core size limits, or limitating the stack, heap or virtual memory of a process. Fine grained controls

within a non-global zone are supported at a project and task level. For more information, visit:

“Understanding the Basics About Solaris Containers in the Solaris 10 OS”

http://www.sun.com/bigadmin/features/articles/basics_containers.html#controlling

“Configuring Resource Controls and Attributes”

http://docs.sun.com/app/docs/doc/817-1592/6mhahuoiq?a=view

While resource allocation capabilities are very powerful, it is also very easy to configure poorly and make the

system unusable. In general, the Solaris 10 defaults, out of the box, work just fine. It's best to not change

anything unless there is a specific issue that needs to be addressed with some reasonable expectation that the


http://docs.sun.com/app/docs/doc/817-1592/6mhahuoiq?a=view










change will address the specific problem at hand. Administrators who work with other UNIX vendors are often

used to having to modify kernel parameters and feel that they are missing something if there are none to

modify. In general, with Solaris 10, it's just not necessary. If changes are made, it's recommended to start with

wide granularity and add small enhancements at a time. For the most part, changes can be made dynamically

without any system disruption, so plan and architect a change management system which allows time for true,

peak load usage to provide system feedback. Don't introduce many changes unless system performance

heuristics are well monitored and understood at the global zone level. For instance, increasing locked memory

for one zone will take away from memory generally available for the global zone and other non-global zones.

In the spirit of 'let the kernel do its job in the absence of anything specific', one example comes to mind for

discussion. CPU processor sets can be assigned to a container through the use of poolcfg(1M). The

specifications allow for primitives which define a range of CPU resources (i.e.: min 2, max 4) or, alternatively,

to dedicate a specific number of CPUs. In the example specification of min 2/max 4, 2 CPUs will be allocated

at a minimum and if utilization warrants, a maximum of could be used. If the container is idle, the 2 CPUs

which are on standby can be utilized elsewhere until called back into action to its originally assigned zone.

Sometimes administrators will want to allocate a fixed number of CPUs to a given container (i.e.: min 6, max 6

for SAS Workspace Server). Unless there are very strict service level agreements, don't restrict the kernel

from utilizing idle CPU resources.

Lastly, if there are system performance issues and a support call to Sun will be placed, if possible, try to run

without any resource controls in place to see whether that helps or hurts the performance situation. Support

personnel will likely want to know whether resource controls are helping or hindering the issue at hand.

The Role of ZFS – The ZFS file system and Zones work well together and allow for great configuration

flexibility. From the global zone, a ZFS file system can be configured for use:

● exclusively specific to a zone

● shared between non-global zones

● shared with the global zone

For more information, visit:

“Using ZFS on a Solaris System With Zones Installed”

http://docs.sun.com/app/docs/doc/819-5461/gbdaw?l=en&a=view&q=zones

ZFS has simple administration interfaces and is a high performance file system. As an anecdotal point, SAS

R&D saw an OLAP cube build time that went from 20 minutes to 3 minutes by moving to a ZFS filesystem.

While other similar performance gains have been realized, be cautious when generalizing or extrapolating the

results as the expression goes “your mileage may vary”.

ZFS provides snapshot/cloning features which can yield tremendous savings in both cost, administrative

hassle, and downtime. ZFS clones and snapshots can be a key architectural component for SAS application

backup and recovery strategies. While many storage HW and SW vendors provide hot backup solutions –

data integrity at the storage, LUN, volume or file system level, relatively few software applications are backup

aware and must be halted or quiesced in order for backups to be taken with guaranteed integrity. SAS 9.1.3

and 9.2 applications fall into this category; SAS services need to be halted in order for application data

backup. SAS R&D has used ZFS snapshots and clones to improve backup down time from 39 hours down to



15 minutes. For further information on SAS 9 Backup stratgies and using ZFS snapshots/clones in this

manner, visit:

“Backup and Disaster Recovery: When Disaster Strikes, What Will You Do?” - SAS Global Forum 2008

paper 312, Arthur Hunt, Tanya Kalich, Billy Dickerson, Chuck Antle.

While not related to the PRDSHARE project, its notable to mention how the Center for Disease Control used

ZFS clones to solve a tough IT problem; their nightly SAS SPD server update was taking too long and the data

warehouse was not available for use during this time. Since an SPD Server data store cannot be replicated on

the same system, the only alternative available was a complete server and storage replication – a costly and

very lengthy time-to-solution option. However, using ZFS clones, married to Zones, a solution was devised and

went from idea to proof of concept in a few hours and at no additional HW or SW cost. For further information,

visit:

“Zebra, Zamboni, Zen and the Art of ZFS” – SAS Global Forum 2007 Paper, Presentation

http://www.sas.com/partners/directory/sun/sas9-and-zfs.pdf

http://www.sas.com/partners/directory/sun/zebra-preso.pdf

Solaris Fingerprint Database (sfpDB) - This is not related to virtualization per se but worth a mention

under any kind of Best Practices discussion especially since its not a well known utility/service. The Solaris

Fingerprint Database is a new Sun service offering that enables you to verify the integrity of files distributed

with the Solaris Operating Environment (for example, the /bin/su executable file), Solaris patches, and

unbundled products such as the Sun Studio compilers. You can use Solaris Fingerprint Database to verify that

you are using a true file in an official binary distribution, and not an altered version that compromises system

security and causes other types of problems. SfpDB was very helpful in the PRDSHARE environment – due

to some HW failures in conjunction with many application and patch installs/un-installs, there was uncertainty

as to the integrity of some of the system files. Using sfpDB, the administrators were able to verify that all the

system files were validated to the appropriate release and not corrupted. This provided great confidence that

the current system installation integrity was as expected and that the upgrade process could proceed as

planned. For further information, visit:

“Solaris Fingerprint Database: An Identification Tool for Solaris Software and Files“

http://sunsolve.sun.com/show.do?target=content/content7

SummaryThe project requirements presented a daunting challenge - a large and diverse population of global users with

conflicting and contradictory goals: support for development, test and production environments on two

completely separate release tracks. To that end, the PRDSHARE project has met with great success; complex

appliation environments have been provisioned onto a single system maximizing server utilization. Product

teams and developers worldwide are able to focus on their tasks at hand without having to tackle infrastructure

concerns. The alternative would have required several servers and additional storage to have supported the

requirements. Deployment using Containers is administratively simple and straighforward, poses virtually no

performance overhead yet provides for great configuration flexibility via resource management, performance

monitoring, dynamic modifications, etc. Economies of scale, efficiency, increased utilization, lower costs and

agility are key benefits for a virtualization strategy for SAS deployments using Solaris 10 Containers.








12 Acknowledgments

AcknowledgmentsThanks to SAS R&D Staff responsible for the PRDSHARE architecture, deployment and support: Billy

Dickerson, Chuck Antle, Arthur Hunt, Kim Sherrill, Mark Everett, Gary Hutchison, Tim Carter – for their vision,

persistence, support, discipline and patience. Also, thanks to Phil Kinslow and the many other colleagues at

Sun who developed documentation, case studies, and papers on Solaris 10 Container technologies. Lastly,

thanks to Jason Schroeder, Greg Smith, Margaret Crevar and the many others who have answered questions

and provided feedback and review.

ReferencesUnderstanding the Basics About Solaris Containers in the Solaris 10 OS

http://www.sun.com/bigadmin/features/articles/basics_containers.html

Deploying SAS(R)9 Business Intelligence Server using SolarisTM 10 Containers, Phil Kinslow

System Administration Guide: Solaris Containers- Resource Management and Solaris Zones

http://docs.sun.com/app/docs/doc/817-1592?l=en

Using ZFS on a Solaris System With Zones Installed


Solaris 10 - System Administrator Collection -> IP Services -> IP Quality of Service (IPQoS)

http://docs.sun.com/app/docs/doc/816-4554/ipqos-config-planning-tbl-28?l=en&a=view

Solaris Operating System - Feature FAQs

http://www.sun.com/software/solaris/faq.jsp

Zones and Containers FAQ at OpenSolaris.org

http://opensolaris.org/os/community/zones/faq/

Solaris Operating System - Move a Solaris Containers How To Guide

http://www.sun.com/software/solaris/howtoguides/moving_containers.jsp

Bringing your Application into the Zone


Third Party Software Requirements for use with SAS Products


How to Get Started with IBM WebSphere Application Server on Solaris 10 and Zones

http://www.sun.com/software/whitepapers/solaris10/websphere6_sol10.pdf

WebSphere Application Server V6.0.2 detailed system requirements

http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27007256

http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27007256

http://www.sun.com/software/whitepapers/solaris10/websphere6_sol10.pdf



http://www.sun.com/software/solaris/howtoguides/moving_containers.jsp

http://opensolaris.org/os/community/zones/faq/

http://www.sun.com/software/solaris/faq.jsp

http://docs.sun.com/app/docs/doc/816-4554/ipqos-config-planning-tbl-28?l=en&a=view


http://docs.sun.com/app/docs/doc/817-1592?l=en

http://www.sun.com/bigadmin/features/articles/basics_containers.html

13 References

8.1 Supported Configurations: Sun Solaris 10 on SPARC

http://e-docs.bea.com/platform/suppconfigs/configs81/solaris10_sparc/81sp4.html

Backup and Disaster Recovery: When Disaster Strikes, What Will You Do? - SAS Global Forum 2008

Arthur Hunt, Tanya Kalich, Billy Dickerson, Chuck Antle

Zebra, Zamboni, Zen and the Art of ZFS – SAS Global Forum 2007 paper, presentation



Solaris Fingerprint Database: An Identification Tool for Solaris Software and Files


Appendix A - Introduction to Solaris 10 Containers1

In the Solaris 10 OS, Containers are an exciting, new partitioning technology allowing multiple virtualized OS

instances on a single system. Solaris Containers can be built using one or more the following technologies.

These technologies can be combined to create Containers tailored for a specific server consolidation project.

● Solaris Resource Manager, for workload resource management

● Resource Pools, for partitioning

● Zones, for isolation, security and virtualization

While Solaris Zones (“Zones”) provide a security boundary and virtualization of the underlying platform

providing a separate name space, IP addresses, file systems, unique root user, password file, and name

server, it is Solaris Resource Management that establishes the resource consumption boundary and allows the

and allows the system resources, such as processors and memory, to be allocated to each Solaris Zone.

It is important to note the terminology of Solaris Container and Zone. The terms are often used interchangeably

but that's technically incorrect. Solaris Containers are a combination of Resource Management, System

Administration (customization) and Zones. Zones is one of the technology that enables Containers. Zones

technology can be used to create a Container with certain characteristics, such as the isolation provided by the

virtual Solaris environment. But it is also possible to create another Solaris Container, without Zones, using

Resource Pools technology if the required characteristics of that Container can be met with the features

Resource Pools provide. So while a Zone is a Container, a Container is not necessarily a Zone. Note: Solaris

Resource Manager and Resource Pools technologies existed before Solaris 10. Zones are new in Solaris 10.

1 Kumar, Leigh “How to Get Started with IBM Websphere Application Server on Solaris 10 and Zones”

http://e-docs.bea.com/platform/suppconfigs/configs81/solaris10_sparc/81sp4.html




14 Appendix A - Introduction to Solaris 10 Containers1

Figure 2: Conceptual view: Single server using Solaris 10 Containers provides virtual logical environments

Zones are classified as either a global zone or non-global zone. The global zone encompasses the entire

system. Because it is equivalent to a typical Solaris OS instance, the global zone has access to the physical

hardware, and can control all processes. Non-global zones are located inside the global zone, and are isolated

from the physical characteristics of the system. For any given system, there is exactly one global zone but 0, 1

or many non-global zones.

Solaris Zones provide the following features:

● Security—Network services can be run in a zone, limiting the damage that can be done to the system

and other zones in the event of a security violation.

● Isolation—Applications requiring exclusive access to global resources, such as specific usernames or

network ports, can run on the same machine using Solaris Zones. Each zone has its own namespace,

completely separate from other zones. Users in a zone are unable to monitor other zones, such as

viewing network traffic or the activity of processes.

● virtualization—Solaris Zones present a virtualized environment to applications, removing the physical

details of the hardware from view. This eases redeployment of applications on a different physical

machine.

● Granularity—Since Solaris Zones are implemented in software, zones are not limited to the granularity

defined by hardware boundaries. Instead, zones offer sub-CPU granularity. Zones do not require

dedicated CPU resources, dedicated I/O devices such as host bus adapters and network interface

cards, or dedicated physical memory. As a result, even a system with a single processor can be used

to host several zones.

● Transparency—The environment presented to the application in a zone is nearly identical to the

standard Solaris OS environment. There are no new, zone-specific application programming interfaces

15 Appendix A - Introduction to Solaris 10 Containers1

(APIs) or application binary interfaces (ABIs) to which applications must be ported. Some restrictions

do exist due to security and isolation requirements. These restrictions mainly affect applications that

perform privileged operations or need access to physical devices

Appendix B – Container Creation Cookbook2 This appendix contains a cookbook sample to demonstrate the ease of

which Solaris Containers can be setup for a sample 3 tiered SAS

installation.

Solaris Containers and SAS EBI ServerSince the SAS EBI Server is a prototypical three tier architecture,

Solaris Containers allow customers to easily and effectively deploy SAS

BI Server. The SAS BI Server will be deployed with three functional

areas – the SAS Metadata Server, the SAS Web Application Server

(BEA Weblogic, Xythos and SAS Remote Services),

and the SAS Application Server (OLAP Server, Stored Process,

Workspace and Object Spawner). Figure 3 illustrates a sample SAS

BI Server architecture. SAS BI Server's modular architecture lends to

having three separate Solaris Containers to support this sample deployment. In this case, a single server can

be partitioned using Solaris Containers to create a development, test and production environment for SAS BI

Server. Since each Container has its own port address space, each of the multiple SAS deployments can use

the default ports. For example, the SAS Metadata Server for production, test, and development can all use

port 8561.

Creating a Solaris ContainerCreating a Solaris Container is easy. Recall that a Solaris Container is a combination of two technologies -

Resource Management and Solaris Zones, a step-by-step installation is detailed below. Notes: Solaris

Resource Management and Zones are useful independent of each other, but they are particularly powerful and

useful when used in combination. There is no required order to how Sections 1 and 2 are completed; they

could be interchanged.

Section 1: Resource management

The three BI Server components have differing resource requirements. In this section the resources, in this

case processors, will combined into a resource pool. When this resource pool is created, it will be allocated

to the zone (in section 2). The processors added to the metadata pool will only be used by the Container

that is running the SAS Metadata Server. While the Solaris kernel will limit memory allocation for all the

Containers, the initial release of Solaris 10 does not support memory limits for individual Containers. This

feature is scheduled for Solaris 10, Update 1.

Three steps are required to create a resource pool. The steps are: 1) enable, 2) configure, and 3)

instantiate. Table 1 shows these steps. In this case, two of the available twenty-four cores were allocated

for the SAS Metadata Server, four two for the SAS Web Application Server, and six for the SAS Application

Servers.

2 Phil Kinsow, 2006, “Deploying SAS®9 Business Intelligence Server using SolarisTM 10 Containers”

16 Creating a Solaris Container

Enable the resource management subsystembash# pooladm -e

Configure the resource pool for the Container using the contents of file createSASpools.bash# poolcfg -f createSASpools<<file contents of createSASpools>>create system ctclavacreate pset metadata (uint pset.max = 2)create pool metadata (string pool.scheduler="TS")associate pool metadata (pset metadata)create pset midtier (uint pset.max = 4)create pool midtier (string pool.scheduler="TS")associate pool midtier (pset midtier)create pset sasFS (uint pset.max = 6)create pool sasFS (string pool.scheduler="TS")associate pool sasFS (pset sasFS)

Instantiate the configuration and creating the resource poolbash# pooladm -c

Table 1: Limiting resource consumption of the Container by using Solaris Resource Manager

To confirm the creation and to show the configuration of the resource pool, the poolstat command is used. Note the resource pools name, the number of processors in that pool and the pool’s utilization metrics. As Table 2 shows, the system has four resource pools defined – the pool_default, metadata, midtier, and sasFS. As expected, the total of the size column is 24 processors. If the SAS workload demands more resources, more can be added to the pools at any time.

bash# poolstat pset id pool size used load 0 pool_default 12 0.00 0.17 1 metadata 2 0.00 0.00 2 midtier 4 0.00 0.00 3 sasFS 6 0.00 0.00

Table 2: Verifying the creation of the resource pools

Section 2: Solaris Zones

With the resource management complete and the resource pools created, the Zone creation requires only

four steps; they are: 1). configuring, 2). installing, 3) initializing, and 4) booting as represented in Table 3

below. The first step of creating a Zone is the most complicated and requires some planning. For this

exercise, the key Zone specifications are defined as follows:

– set zonepath=/zones/midtier: the "home" of the Container for SAS Middle Tier

– set pool=midtier: assigning the resource pool to the Zone configuration.

– set autoboot=true: the Container is automatically started a system boot time.

– add net: this section adds a network interface and IP address for the Container.

– add fs: this section adds additional file system to the Container. By default, a single a

filesystem is added to the Container. If additional are required, they are added using this

syntax.

17 Creating a Solaris Container

Create the Container with the configuration specifications listed in file midtier-create.bash# zonecfg -z midtier -f midtier-create<<file contents of midtier-create>>createset zonepath=/zones/midtierset pool=midtierset autoboot=trueadd netset physical=ce0set address=192.168.30.29endadd inherit-pkg-dirset dir=/optendadd fsset dir=/apps/sas set type=lofsadd options [rw,nodevices]set special=/san/zones/fs1 endverifycommit

Installing the Container.bash# zoneadm -z midtier installPreparing to install zone <midtier>.Creating list of files to copy from the global zone.Copying <3114> files to the zone.Initializing zone product registry.Determining zone package initialization order.Preparing to initialize <1125> packages on the zone.Initialized <1125> packages on zone.Zone <midtier> is initialized.The file </zones/midtier/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

Initializing the Container.bash# cp sysidcfg /zones/midtier/root/etc/sysidcfgbash# cat sysidcfgsystem_locale=Cterminal=xtermnetwork_interface=primary { hostname=midtier ip_address=192.168.30.29}security_policy=NONEname_service=NONEtimezone=US/Easternroot_password=dVaKf.jNJSeLE

Boot the Container.bash# zoneadm -z midtier boot

Table 3: Configuring, installing, initializing and booting the Container

Validating the ProgressBefore SAS is installed, it is worthwhile to validate the progress so far and review the environment that SAS

will be installed. Table 4 shows the output of the several commands.

List all the Containers on the system.bash# zoneadm list -cv

18 Validating the Progress

ID NAME STATUS PATH 0 global running / 1 sasFS running /zones/sasFS 2 midtier running /zones/midtier 3 metadata running /zones/metadata

Check the resources assigned to each Container (as view from the Global Zone).bash# poolstat pset id pool size used load 0 pool_default 12 0.00 0.17 1 metadata 2 0.00 0.00 2 midtier 4 0.00 0.00 3 sasFS 6 0.00 0.00

Login to a Container.bash# zlogin -C sasFSsasFS console login: rootPassword:******

Check the resources assigned to this Container (as view from inside the sasFS Container).bash# psrinfo0 on-line since 10/17/2005 10:58:261 on-line since 10/17/2005 10:58:262 on-line since 10/17/2005 10:58:263 on-line since 10/17/2005 10:58:264 on-line since 10/17/2005 10:58:265 on-line since 10/17/2005 10:58:26

bash# poolstat pset id pool size used load 3 sasFS 6 0.00 0.00

Table 4: Displaying the status of Solaris Containers

Each container has its own name space, IP addresses, file systems, unique root user, and password file. Additionally, the resource

consumption boundary is also guaranteed to ensure that each part of the SAS BI Server is guaranteed to have access to the system

resources, such as processors and memory, which were assigned to it. Since the sasFS Container is bound to the sasFS pool which

has a defined limit of six processors, the SAS Application Server can only use the six processors assigned to this Container. At this

point, the SAS installation can proceed as planned.

a sas r&d case study

Documents