campus it services tiger team...

Campus IT Services Tiger Team Report

April 30, 2009

Team Members

Chair - Marlene Shaver, VCER Deanna Richardson, CBO Karen Andrews, SVCAA Ron Spears, VCHS Jason DeFay, VCR Helen Szkorla, SVCAA Elazar Harel, ACT Jerry Wanetick, SIO David Hutches, SVCAA Raymond Young, VCER Lyle Kafeder, SDSC

Campus IT Services Tiger Team Report, April 30, 2009 p. 2

Table of Contents

Executive Summary ...................................................................................................................p. 3

IT Tiger Team Objective and Process .......................................................................................p. 5

Recommendations

Part I: Coordination and Communication..................................................................p. 7

Part II: IT Personnel....................................................................................................p. 9

Part III: Purchases and Procurement of IT Hardware and Software............................p. 13

Part IV: Server Co-Location, Space, and Energy... .....................................................p. 19

Part V: Campus Messaging/Calendaring Services .....................................................p. 23

Part VI: Implementation of Policies, Guidelines, and Alignment with

Organizational Goals ......................................................................................p 25

Appendices …........................................................................................................................p. 26

Appendix A: Chart on Information Technology Systems, excerpted from Building

Administrative Efficiency for UC

Appendix B: UCSD Health Sciences IT Centralization Review

Appendix C: Examples of Existing IT Support Service Recharges

Appendix D: Energy Efficient Data Centres in Further and Higher Education

Appendix E: Faronics Case Study – Howard Community College


Executive Summary The UC San Diego Campus Information Technology (IT) Services Tiger Team was appointed by the campus budget office. The Team was asked to explore ways of saving costs and increasing IT efficiency in our highly decentralized and ever-changing environment.

IT is a mission-critical activity that supports the University’s instruction, research, patient care, and administration. The existence of a robust IT infrastructure and high-quality IT support provides myriad benefits: 1) the ability to handle large volumes of data permits data-intensive research; 2) improved access to information results in better decision-making; and, 3) automation typically improves turnaround times, reduces error rates, and reduces the need for non-IT staff. High-quality IT services are an essential element of the campus’ success, and the quality of these services must be maintained.

Using data from a variety of sources, the Team estimates that the campus spends approximately $150 million annually from all funding sources on IT: approximately $90 million on salaries and benefits, $50 million on IT-related purchases, and $10 million on energy. Total campus IT spending therefore represents roughly 6% of the $2.4 billion in total annual UC San Diego operating expenditures.

The Team believes that significant opportunities for improvements in efficiency and potential cost savings may exist in several specific areas: personnel; hardware/software procurement; server co-location, space use, and energy costs; and messaging/calendaring services. While precise estimates are necessarily conjectural, we believe overall savings on the order of $5 to $10 million annually can be achieved through a balance of immediate actions and sustained multi-year efforts. These savings would not be realized uniformly across campus, however, as business needs and existing levels of efficiency vary.

It is important to note that ultimate success in achieving cost savings will depend on each campus unit carefully and holistically reviewing its IT operations and expenditures, and considering how it can maximize savings while continuing to effectively support the University’s mission.

A brief overview of our observations and conclusions is presented here; the body of the report contains more specific recommendations. Due to the sheer size of IT-related activity on campus, as well as Team members’ time and knowledge constraints, many of our recommendations will require further analysis by assigned experts and/or external consultants who can devote more time to detailed assessments of the issues.

1) Establish a formal campus-wide IT information-sharing network with official IT contacts in each unit. This network would provide a new locus for the ongoing collection and communication of IT-related information (e.g., costs, current best practices, and availability of recharge services) to everyone making resource use decisions. Enlightened self-interest can then be expected to result in higher-quality decisions, especially in this era of budget cuts. This network could also review reports such as Building Administrative Efficiency at UC1 and the recent UCSD Health Sciences IT Centralization Review2 to identify recommendations that should be considered for campus-wide implementation. To be successful, this network will require dedicated staff support, along with executive championship and backing.

1 See Appendix A. Also available online at http://ucadministrativeefficiencies.ucla.edu. 2 See Appendix B.


2) IT units should be properly sized. Desktop support and server administration activities are particularly amenable to consolidation of services at certain levels of the organization, either through pooling personnel or by taking advantage of existing recharge operations. Staffing ratios and job duties must be carefully assessed (in relation to their title code classification) to ensure that IT support units are correctly sized for the tasks they are assigned.

3) In the procurement arena, unit decision makers need to be able to easily access high-quality information about current best practices for IT purchases so that they can make better, more strategic, purchasing decisions. Specialized training and checklists would be helpful. Careful advance planning and pre-sales vetting of proposed hardware/software purchases by IT experts are important components of an effective procurement process. The use of open source software should be considered as an alternative to more costly proprietary offerings.

4) Servers should be consolidated into energy-efficient regional server rooms to the greatest possible extent. The energy efficiency of all computing equipment should be improved through a combination of prudent purchasing, limiting the number of devices (servers, monitors, etc.) wherever feasible, turning off non-essential equipment overnight, and implementing energy management software.

5) Administrative Computing and Telecommunications provides cost-effective centralized email/messaging services; campus units should migrate to these whenever business needs permit. Although additional resources will be required before ACT can support all campus users, that cost is lower than the cost of maintaining multiple individual email/messaging services. Provided that the legal concerns now under review by UCOP General Counsel can be resolved, it may be possible in the future to use Google email and other no-cost Google applications.

6) Establish new IT policies and guidelines, along with better cost monitoring and auditing processes, to ensure that incentives and disincentives are correctly aligned with organizational goals. When decision-makers do not directly incur the costs of inefficient resource uses, for example, there is little impetus to consider those costs during the decision-making process. One way to ensure that those costs are taken into account would be to pass along any above-normal costs to those units who choose to incur additional costs when viable alternatives exist.

In summary, a concerted, campus-wide effort is necessary to optimize IT-related costs. Executive level backing and direction will be needed. With greatly enhanced effort in the provision of information to campus units, serious reviews of staffing levels and use of personnel, more cost effective options and controls for IT-related purchases, and reduction of energy consumption in general, significant savings for individual units and for UC San Diego as a whole should be achievable.


Report IT Tiger Team Objective

The IT Tiger Team was asked to identify and pursue options for consolidating and restructuring IT services across many segments of the San Diego campus. The goal was to identify options that would save costs and increase efficiency. Over time, these options could result in:

a reduced campus workforce;

improved IT services;

reduced consumption of energy;

optimal locations of equipment and services;

potential for additional consolidation and centralization over the long term; and

reallocation of resources now being used for certain IT services to other critical campus needs.

IT Tiger Team Process

A vast array of IT equipment, software, services, and personnel have been deployed over the past few decades in response to growing instructional, research, and administrative needs for IT support in an ever-changing technological environment. Some IT activities are supported centrally, while others are supported by individual units or research grants. Resource decisions are therefore made at all levels of the organization.

Preliminary data provided to the Team made it clear that campus IT expenditures for personnel, equipment, software, and services are significant. We estimate they amount to approximately $150 million of the $2.4 billion expended annually for campus operations. During our earliest discussions, we realized that substantial benefits and cost reductions could be achieved through:

improved communication and knowledge sharing;

implementation of better overall campus standards and controls;

improved efficiencies; and

enhancements of employee skill, knowledge, and job satisfaction, all of which tend to reduce turnover.

After initial brainstorming covering the full range of campus IT services, the Team decided to focus on several primary areas where increases in service quality and substantial cost savings might be realized through sharing of best practices, cross-unit collaboration, streamlining, consolidation, and implementation and enforcement of policies. These areas are:

1. Campus-wide communication and collaboration;

2. IT personnel;

3. Purchases and procurement of hardware and software;

4. Server co-location, space, and energy;

5. Messaging/calendaring services; and

6. Policies, guidelines, and alignment with organizational goals.


The team discussed items 1 and 6 as a group. Sub teams were assigned to items 2-4 for further information gathering and reporting, while item 5 was addressed by ACT. The report is divided into parts that correspond to these areas.


Part I: Coordination and Communication Substantial improvements in coordination and communication will be necessary to achieve a more efficient and effective distribution of IT resources, while maintaining or improving the quality of IT services across campus. It would be beneficial for UC San Diego to create a defined resource where decision-makers could obtain information about the availability of IT services on campus, information about opportunities for IT-related cost savings, and advice on the provisioning of IT services. No single employee or unit is now responsible for coordinating, collecting and disseminating this type of information, and many individuals lack easy access to up-to-date information about best practices that already exist on campus, or about issues they should consider when making IT decisions. Relatively few purchasers, for example, think about issues such as warranty limitations and vendor service levels when they purchase a computer from a discount store in order to save a few dollars on the initial purchase price. Nor do they consider the added costs to the campus, overall, to process these types of purchases.

We recommend the establishment of a formal campus-wide “IT Expert Network”, consisting of official IT contacts in each significant unit. Vice Chancellors would appoint representatives to the Network as they determine appropriate. The creation of this sustained, high-level group with appropriate executive level backing would improve IT coordination and communication across the “knowledge silos” inherent in the decentralized nature of the University community. It would provide clearer channels for bidirectional discussion and dissemination of current best practices in IT across the entire campus. These official contacts would be responsible for ensuring other IT staff in their respective areas are included in communications and knowledge sharing, and held accountable for carrying out recommended actions. The Network could be a “quick win” to the campus to enable the discovery and implementation of more specific cost saving outcomes.

The Network would be tasked with capturing, updating, and regularly communicating IT-related information (costs, current best practices, availability of recharge services, etc.) among all interested parties (themselves, managers, Principal Investigators, etc.). Members of the Network would be key players in the implementation of best practices and new policies or procedures related to IT.

Dedicated staff support to assist with project management and with tasks such as conducting surveys, scheduling meetings, and posting information on Blink will be necessary for success, and will require the allocation of some campus resources. Executive-level backing and direction will also be required.

One of the first projects this group might undertake would be an inventory of IT resources at UC San Diego (people, equipment, and technical expertise). The Team quickly discovered that these data do not exist in consolidated form at present, but would be essential to decision-makers to improve the efficiency of IT services. Initially, Blink may be a logical place to publish and maintain the relevant data, but “push” strategies such as email to relevant constituencies may also be required. Creation of a more specialized web-based resource for IT-related personnel that would deliver policy, procedure, and information, along with required training and other know-how, might be a longer term outcome.

Among other things, this kind of inventory would serve to increase the visibility of centrally-provided IT services. WebCT (for course websites) is marketed to new faculty, for example, but few staff know much about it, even if they are aware of its existence. Many employees are


equally unaware of other IT resources such as the range of services supported by NGN (Next Generation Network) fees, the existence of campus-wide software licensing agreements, and the extent of IT services available through various campus recharge operations. Even individuals who know something about these services often lack reliable data about the relative costs and efficiencies of providing those services at various levels of the organization.

Another valuable project would be collecting and publishing solid data about “invisible” costs – those that may not be obvious to decision-makers at the time of the decision. The cost to the campus of running a desktop computer 24 hours per day, for example, has no direct impact on the end-user or their unit. Similarly, employees who purchase software on their personal credit cards and subsequently seek reimbursement from the University are usually completely unaware of the higher-than-usual processing costs incurred for that transaction.3 At present, little information is available about those costs, and there is little connection between the decisions people make about IT and the consequences of those decisions. Merely educating people about these “invisible” costs can change behavior in many instances.

With proper support, the IT Expert Network could also coordinate other IT-related activities. One “quick win” would be to make power management software freely available throughout the campus, along with a suggested implementation process. Longer term projects could identify new campus-wide software licensing opportunities, synchronize a process to plan for coordinated campus-wide hardware purchases, facilitate inter-unit equipment transfers, and review reports such as Building Administrative Efficiency at UC4 and the recent UCSD Health Sciences IT Centralization Review5 to identify recommendations that should be considered for campus-wide implementation.

3 Research conducted by Huron Consulting for UC SAN DIEGO indicates that the additional cost of processing a purchasing transaction outside of Marketplace is between $5 and $115 per transaction, depending on the mechanism used. 4 See Appendix A. Also available online at http://ucadministrativeefficiencies.ucla.edu. 5 See Appendix B.


Part II: IT Personnel

Background

UC San Diego currently employs over 1,000 people in information technology at a total annual payroll cost of roughly $90 million. As shown in the table below, approximately 25% of this activity ($24 million and over 250 FTE) is direct-charged to extramural funding; the remainder comes from other sources. Increasing IT personnel efficiencies within and across units could therefore yield significant benefits. Reliable estimates of potential savings in any particular area cannot be determined from the limited data6 we were able to obtain within the short timeframe allotted to this study. That said, the estimate is likely to be reasonable, and achieving only a 5% reduction in the total amount spent on IT-related payroll costs over time would translate into $4.5 million saved annually.

Technical Salary & Benefit Costs & FTE for Fiscal Year 2008

Total Salary & Benefits by Funding Source Type and Title Series

Funding Source Prog/Analyst Resource Spec Resource Mgr Other Seriesa) TOTALb) General Funds $20,007,350 $1,387666 $1,507,129 $424,007 $23,326,153 Other Core 7,245,331 677,410 462,091 0 8,384,833 Contracts & Grants 23,024,313 766,768 193,800 0 23,984,881 Auxiliary & Self-Supporting 13,992,390 1,538,999 638,639 8,159 16,178,188 Medical Center 7,882,757 242,090 377,768 559,768 9,062,383 Other 7,942,520 243,867 251,672 14,142 8,452,201 TOTALb) $80,094,662 $4,856,801 $3,431,100 $1,006,075 $89,388,638

Total FTE by Funding Source Type and Title Series

Funding Source Prog/Analyst Resource Spec Resource Mgr Other Seriesa) TOTALb)

General Funds 218 25 12 7 261 Other Core 88 12 3 0 103 Contracts & Grants 255 14 1 0 270 Auxiliary & Self-Supporting 160 27 6 0 193 Medical Center 83 4 3 10 100 Other 89 4 2 0 95 TOTALb) 892 86 28 17 1,023 a) Computer Operators and Data Processing/Production Coordinators. b) Totals may not appear to add up correctly due to rounding error.

Recommendations

1. Size IT Units Appropriately

There is considerable potential for realizing efficiencies through properly sizing IT service units. Organizational units with very small numbers of IT staff (3 or fewer) are likely to suffer from some inherent inefficiencies. A typical unit requires IT support in several different areas: desktop support; web sites; databases; application programming; system administration; local network management; access management (e.g., DSA activities); and, in some cases, server and

6 Title code and funding source data are relatively easy to obtain, but do not identify the types of IT support provided by a given individual.


database administration. When a handful of individuals must cover all of these IT functions, they may not be able to develop the skill sets and technical sophistication that can increase their effectiveness. This approach is economically inefficient in many instances, since an organization must typically pay for an individual’s highest skill set even if they spend considerable time performing lower level tasks.

The chart below illustrates the possible magnitude of the issue. Over 70 units at UC San Diego have three or fewer IT support staff, and more than 100 units appear to have no technical staff. Although some of these units are presumably already taking advantage of some sort of resource pooling or IT recharge service, there may still be substantial opportunity for improvement.

There are two areas where significant consolidation could occur: desktop support and server administration. Desktop support typically encompasses activities in standard operating systems, application support, email support, network connectivity support, and shared printing. Server administration activities usually include administering shared file servers, web servers, email servers, and application servers. Because the skills and knowledge of personnel in these jobs would tend to be universally applicable across the campus, careful consideration should be given to pooling and sharing them in a more centralized manner. Consolidation of these services at the right level of the organization could result in both cost savings and increases in service quality.

A review of the literature shows that a single desktop support staff person can support between 50 and 100 customers.7 Units with fewer than 100 employees requiring IT support could therefore benefit considerably from consolidating the desktop support function.

Additional economies of scale can be realized by adopting standardized platforms and software, since an area with a homogeneous IT environment will require fewer desktop support staff than one with a highly diverse IT environment.

It is important to recognize that some areas may prove to be understaffed with regard to IT support. Again, more detailed data will be required to correctly assess these situations. 7 http://www.coastaltech.com/hd-staff.htm, http://rms.net/lc_faq_supp.htm, and http://www.afcom.com/The_Association/ResourceCenter/Data_Center_Management/Help_Desk_Staffing.asp discuss standard desktop support staffing ratios.


2. Consolidate IT Personnel and Share Costs

One successful model for consolidation is the simple pooling of resources within a building and/or a geographic area, or among related organizational units. The Chancellor’s complex desktop support group is an example of this. The units housed in the Chancellor’s complex negotiated an agreement amongst themselves to pool their FTE resources into a single unit (overseen by the VCBA) that provides desktop support for all the units. Contributions ranged from 1 – 3 FTE.

Similarly, UCLA has used a model where they consolidate much of their IT support at the Divisional level. This approach works well as long as the service is of high quality and is monitored to ensure continued success. If service levels erode, however, or if service providers appear to lack sufficient technical sophistication, customer departments typically begin to hire more of their own IT staff.

While it is important to move toward a prudent balance of centralized and decentralized services, the Team believes that any significant consolidation of IT personnel is likely to be a longer-term issue, requiring a great deal more in-depth analysis. The review, which could be undertaken by the “IT Expert Network” noted above, would include obtaining an understanding of the business functions performed in IT units across the campus, and inventorying which of those are general needs or services that all units require, versus specialized needs or services. One possible model would be to redeploy existing personnel who perform a common set of tasks and functions into service “clusters” that would serve multiple units; specialized personnel would remain in their individual units to provide specialized support.

3. Use Recharge Units

Use of existing recharge units – or creating new ones – may be another way to consolidate personnel and apportion costs to achieve staffing efficiencies. At present, the IOD (Integrative Oceanographic Division), SDSC (San Diego Supercomputer Center), and JSOE (Jacobs School of Engineering) all have successful desktop support recharge models. Their coverage varies slightly8, but the basic concept is the same: users agree to a monthly charge for a given package of services. The charges are assessed as a fixed “head tax” or levied based on FTE, depending upon how the units are structured and what seems equitable. The campus may be able to deploy this type of model across very large divisions, or even across entire Vice Chancellor areas.

In addition to the monetary benefits that can accrue from consolidating IT operations in this fashion, personnel in larger units are likely to be able to acquire greater skill and knowledge in various areas of IT, thus improving their value to the University as well as their personal job satisfaction. This tends to lead to lower turnover rates, thus saving the campus money in the form of less recruitment and training.

4. Consolidate Server System Administration

Another area that offers some promise for personnel consolidation is server maintenance. Ratios vary significantly in the literature with regards to ideal staffing ratios. This is because there are a very large number of factors involved, including the level of automation, whether or not reference systems are employed, and whether or not the physical number of servers is limited by the use of virtual servers. In organizations that make poor use of tools and virtual servers, ratios can be as low as one system administrator to 10 or fewer servers. In organizations that are

8 See Appendix C for details.


highly automated and make good use of tools, ratios can be as high as one system administrator to 50 servers.9

The Team believes substantial personnel savings could be achieved by identifying smaller inefficient IT units, determining their relative strengths and weaknesses, and either merging those units into larger IT service units or contracting out to recharge units for those services. To be successful, this process would need to be carefully managed, probably with leadership from the divisional or Vice Chancellorial level. These offices are presumably at a level within the organizational hierarchy where they have both a reasonably good view of what resources are available campus-wide and a good understanding of the needs of the units reporting to them.

Consolidating server system administration could also reduce the number of servers, with attendant reductions in energy, space, and equipment acquisition costs.

9 Information about determining appropriate server support staffing ratios in various environments is available at:

http://www.pti-consulting.com/Samples/Snohomish%20County%20-%20Technology%20Benchmarking%20Memo.pdf and http://techupdate.zdnet.com/techupdate/stories/main/0,14179,2846915-2,00.html


Part III: Purchases and Procurement of Hardware and Software

Background

Given available IFIS data, the Team estimates that UC San Diego spends approximately $45 to $50 million annually on IT hardware and software.10 Direct charges to contract and grant funds represent about 25% of the total amount. The campus already has a wealth of IT purchasing-related resources, and there is little need for the invention of new processes or opportunities. What is primarily required is the development of a robust infrastructure, plus control and compliance processes, to effectively communicate and utilize the existing resources. The overall savings that could be achieved with better controls, oversight and information are not reliably known, but even a reduction of 5% per annum would generate more than $2 million of savings.

Recommendations

1. Appoint an Official to the IT Expert Network in Each Unit

In the context of the new Roles system, each significant unit should be required to designate an official that would be appointed to the “IT Expert Network” outlined above. These individuals would also serve as the unit’s primary source of expert advice on IT issues. Ideally, persons in the units with authority to purchase goods and services would be required to consult with these individuals before making IT-related purchases, particularly in those cases where an existing configuration and its appropriate use are not already documented. To ensure consistency, some manner of common training may be required of officially-designated IT Network appointees.

2. Provide IT Purchasing Training and Checklists

Personnel at all levels would benefit from improved access to the latest policies, procedures, and best practices associated with IT procurement.

An “IT Purchasing Checklist” (similar in concept to the existing “New Employee Checklist”) should be created and made available via:

the “IT Expert Network” described above; a broader Blink portal with an inventory of all information, resources, topics, and expert

advice on IT-related issues; and a link embedded in campus purchasing processes at the appropriate point prior to

purchase completion.

In addition, regular, specialized training should be required for designated departmental IT buyers. This would provide uniform education for those individuals and make them aware of the information resources and expert advice available on an ongoing basis from the campus-wide “IT Expert Network”. Buyers should then be held accountable for adherence to UC San Diego policies, procedures, or best practices related to IT procurement.

3. Improve IT Purchasing and Procurement Decisions

There are many opportunities to create both short- and long-term cost efficiencies by improving the quality of the decisions that are made about IT purchases. In a highly decentralized environment, long-term planning for procurement sometimes gets short shrift, and individual IT sourcing decisions can easily lead to ineffective choices about sources (vendors), products (type, 10 This figure excludes High Performance Computing (HPC) purchases at SDSC.


brand, configuration, compatibility), and total cost of ownership (shipping, warranty, return policy, workmanship, personal travel time, sustainability, energy efficiency, etc.). Although the actual impact of poor decision-making is difficult to determine precisely, the result may be ineffective capital expenditures and long-term operational costs that could have been avoided. Individuals and units making IT-related purchases must have timely and appropriate information at their disposal in order to make cost effective decisions.

At present, the campus provides minimal resources, or oversight, for planning and executing IT purchases. Information about what to buy, how to buy it, how to configure it, and whether or not it will effectively meet a purchaser's needs has historically been ad hoc, or not readily or uniformly available. Even somewhat "official" information is not always current. Although purchasing tools and business processes are already available to campus consumers, some are primarily designed for "experts" rather than "average" users. Mechanisms can and do exist to enable significant savings, but these will not be used routinely unless they are easily and broadly accessible.

There is no oversight authority, review, or audit process for IT procurement, and the current decentralized procurement decision-making process incurs avoidable costs that, while difficult to quantify without further investigation, are almost certainly significant. The procurement process must incorporate essential requirements, policies, limits, and – in some instances – prohibitions. The Procurement and Contracts Division of Business & Financial Services (BFS) can provide excellent insight about where better regulation could provide cost savings to the campus overall.

It is important to note that the specific examples cited below represent current best practices to achieve cost savings, and that the rapidly changing nature of IT means that some of these recommendations will age better than others. Because of this dynamism, we reiterate that the creation of a robust coordination and communications infrastructure is essential to ensure the timely dissemination of new information as it becomes available, and the expiration of stale information when it is no longer relevant. It is also important to note that high-quality decisions cannot be made in a vacuum; the caution statements that appear below call out some of the critical considerations.

Clients (desktops, laptops, etc.)

Units must think strategically about purchases and hardware replacement cycles. Modern computers can operate well for 4-6 years (or even longer) before becoming obsolete as a result of other technological trends. Existing systems can retain their usefulness via inexpensive, incremental component upgrades (memory, disk, etc.) versus wholesale replacement. LCD monitors, (which have an average life of about 7 years) should not automatically be replaced when new computers are purchased.

Purchases should always be made with long-term use in mind. Will a hardware platform still be viable 4-6 years from now? Is there enough cost effective expansion capacity to meet future needs? Will spending slightly more now result in significant cost savings over the long term?

Equipment should be reused whenever possible, since systems no longer appropriate for one type of activity may be fully adequate for another. For example: systems that no longer meet the needs of "power users" can be migrated to users with more modest requirements; older desktop systems inadequate to run Windows Vista or Windows 7 may be repurposed as Linux workstations in a research lab, or as thin client hosts. New mechanisms to facilitate the transfer of still-usable equipment from one unit to another should be developed to allow equipment that cannot be reused within the original unit to be redeployed elsewhere on campus. Integrate into


the purchasing process (e.g., Marketplace) information about pools of "pre-owned" systems that may now be going to Surplus Sales or simply languishing somewhere, either unused or under-used.11

UC San Diego should consider implementing a "one person, one device" model in which each individual has only one type of computing system,12 and the provision of specialized equipment (oversize or additional monitors, personal printers, etc.) is governed by demonstrated business needs. To avoid “over-buying” and to minimize energy consumption, computing power should be carefully considered and matched to technically accurate estimates of current and future computing requirements. Caution: Decisions in this area must be made holistically; it is important to balance energy efficiency, initial purchase costs, long-term reliability, ergonomic implications, systems management requirements, security concerns, etc. Research requirements must also be taken into account where appropriate.

Servers

Wherever feasible, units should use campus-wide, NGN-funded resources such as centralized email, calendaring, and web content management systems to reduce the need for departmental hardware, software, and FTE resources. Caution: Local IT resources will still be required to support the client-side component of centrally-provided services; units must plan accordingly.

Energy efficient servers using low power memory and processor technologies should be purchased whenever possible. Consolidating multiple services on a small number of servers is more efficient than a one service per server model. Similarly, virtualization technologies, in which multiple, independent operating systems all run on the same physical hardware, can result in improved server utilization. Virtual hosting environments already exist on campus, and units can minimize their equipment and management costs by taking advantage these when appropriate. Caution: Determining when virtualization and service consolidations are appropriate requires expert advice and a full understanding of the environment in which they will be deployed. An effective campus-wide IT information sharing infrastructure could both communicate the availability of these kinds of resources and provide expert advice.

Software Purchases and Licenses

Units should obtain software via negotiated licensing agreements whenever suitable agreements exist. Units that make significant use of Microsoft and/or Adobe products, for example, would acquire that software through the FTE-based Microsoft Campus Agreement and/or Adobe Acrobat Term Site License.

The first step in any software purchasing decision should be to check with campus Software Distribution13 to see if a product is covered under an existing agreement. Incorporating this step will reduce costs for already-licensed software, while effectively communicating user community interest in software that might be suitable for future site licensing or volume discounts.

11 In addition to wasting energy, underutilized and ill-maintained systems that are connected to the campus network pose a significant security risk for the campus: unpatched machines are easy prey for hackers. 12 Personnel whose duties primarily involve desk work would use desktop systems (small form factor, ultra small form factor, all-in-one). Personnel whose duties require substantial mobility would use laptop systems, plus docking stations and external keyboards and monitors if required by ergonomic and business process factors. 13 http://software.ucsd.edu or [email protected]


Concurrent licensing models, which allow units to buy only the number of licenses likely to be used simultaneously instead of buying one license for each user, should be used whenever they are cost-effective. Academic Computing Services (ACS) Software Distribution already operates concurrent license servers for most common software products. Caution: While concurrent licensing usually results in lower per seat costs, technical expertise is often required on both the server and client side.

The campus should aggressively pursue a project to standardize on one software product among a family of similar products. For example, choose one PC anti-virus product for use across campus, and block the use of licensing agreements for all others. Such standardization results in lower administrative overhead and systems support cost, and by consolidating the user community into a larger volume purchasing block, improves the University’s leverage in the negotiation of software licensing fees.

Recommendations for appropriate and available Open Source Software (OSS) alternatives to commercial software should be broadly disseminated. Ideally, these recommendations, along with pointers to additional information (web sites, discussion forums, campus experts, etc.), would be integrated into the purchasing process.14

Software Development

Units should be discouraged from local software development, unless existing tools demonstrably fail to meet basic business process requirements. Using existing tools may require some changes to a unit’s business processes, but making those changes will almost always be significantly less expensive than developing and maintaining in-house tools. All departments should be encouraged to use the free and low cost business support services available on campus. As an example, a goal should be to minimize or eliminate "shadow systems" that largely duplicate the functionality of systems in the Link family of applications.15 For instructional users, Academic Computing Services16 provides free web hosting and access to a Learning Management System. Faculty and academic staff should become familiar with services described in the Faculty Guide to Academic Information Technology.17 Caution: The motivation to build in-house systems is correlated with a perception on the part of IT professionals and/or a user community that centralized services providers are not sufficiently responsive to their needs. In a University environment, the desire for uniformity and consistency common in a business computing environment must be balanced with the innovative nature of the academic enterprise.

Free or low cost “Software as a Service” (SAAS) and/or OSS (Open Source Software) alternatives should be used in lieu of purchasing or developing local software systems. For example, SurveyMonkey18 is a free survey tool that provides excellent functionality and avoids the costs of custom development. Doodle19 is a free tool for scheduling an event when

14 UCLA has been conducting a comprehensive and thorough study of the efficacy of moving toward large scale deployment of OSS to replace proprietary software for functions such as office suites, collaboration tools, etc. A final report should be available in the near future, and their recommendations should be carefully considered. 15 http://blink.ucsd.edu/go/linkfamily 16 http://www-acs.ucsd.edu 17 http://www.libraries.ucsd.edu/facguideait 18 http://www.surveymonkey.com 19 http://www.doodle.com


participants do not have access to a common calendaring system; it is ideal for the distributed collaborations common in academia.

Pre-Purchase Vetting of IT Configurations

IT-related procurements should be vetted before purchase by assigned IT experts to eliminate flawed orders, minimize customization, and reduce costs. Potential configurations should be evaluated to ensure that they meet the requirements of the environment in which they will be deployed. Easily-accessible resources for "pre-sales/pre-sourcing" consultation will help ensure that configurations to be purchased will ultimately meet the technical needs of the users and the cost containment needs of the University. Leadership at the VC level is required in order to ensure that this consultation occurs, as existing habits are difficult to change, and without clear and consistently enforced directives, inertia and political considerations will prevail. Marketplace

Marketplace20 is conceptually the right tool for the campus to implement as a central purchasing resource. It continues to be developed, and should be provided the resources to achieve that development more rapidly. Once fully developed, it must become THE required tool to be used as the first and primary choice for all IT purchases. Marketplace can intuitively "guide" shoppers to the best source of products to meet their needs. Its cost-effective processing incorporates an approval process with automatic routing, tracking, full control, and accountability, an ordering process that can automatically transmit purchase orders to suppliers, and a payment process that allows index assignment, split-funding, and automatic invoice settlement.

For Marketplace to be effective, however, it must be responsive to customer feedback regarding items and configurations not currently available through Marketplace. It must also address the issue of customer-identified “deals” that seem to be better than those available in Marketplace, perhaps by sending appropriate explanatory or corrective information back to the customer. Annual Planning for IT-Related Purchases

The campus should develop and phase in an annual IT equipment acquisition planning process that would involve all campus units, including those in Health Sciences and Marine Sciences. Encourage bulk purchases of standard IT equipment configurations at the campus level (or possibly the system-wide level); this could significantly reduce per unit costs by leveraging UC’s purchasing power. Units would obviously need advance notice of the process, which could be coordinated through the IT Network described above.

A similar annual campus-wide planning process for software needs should also be put in place.

IT-Related Purchasing Controls and Limits UC San Diego needs to create and implement more stringent campus-wide policies pertaining to IT purchases. Consideration should be given to the possibility of requiring pre-authorization of certain IT-related purchases. These controls are important for successful implementation of not

20 Throughout this report, we use “Marketplace” to refer to the full-service version, which provides a conduit for purchasing from multiple vendors, rather than to “Marketplace Preview”, which was limited in scope.


only the recommendations laid out in this report, but also the sustainability policies established by the UC Office of the President.21

The Express Card can be an effective tool for procuring IT-related items when those items are not available through Marketplace, but Express Card holders should be specifically trained on best practices for IT procurement. Transactional reporting tools should be developed to allow both individual units and central offices to more effectively monitor IT-related purchases on Express Cards.

The ability to obtain reimbursement for IT purchases made on personal funds should be strictly limited. Allowing personal reimbursements provides a means of circumventing the above recommendations; it also costs the campus approximately $75 for each reimbursement that must be separately processed. Serious consideration should be given to assessing a “processing surcharge” when the reimbursement mechanism is used, as a disincentive to making purchases outside the most cost effective means available to the campus.

21 http://www.ucop.edu/purchserv/rfp/empuniform/exhibitd.pdf


Part IV: Server Co-Location, Space, and Energy

Background The need for facilities to support computing and data intensive research in higher education is steadily increasing. In particular, demand from faculty in disciplines that have not traditionally been heavy users of computational resources – life sciences, social sciences, arts, and humanities – has increased dramatically, with ever smaller organizational units running their own server facilities. Furthermore, the costs related to powering and cooling IT equipment over a 3-year period is typically 2 to 5 times more than the initial purchase cost, and equipment used for intensive research computing has particularly high power and cooling needs. Institutions will need to look at strategic approaches across their entire organizations in order to achieve savings and become both cost effective and sustainable.22

Server-based computational, messaging, storage, and security systems comprise a significant and increasing component of UC San Diego’s energy costs. However, in the absence of detailed metering data, it is not possible to obtain exact figures on energy expenditures related to campus IT operations. Based on the rule of thumb cited above and consultation with campus experts, it is reasonable to suppose that all of the campus computing-related power and cooling costs could represent at least 15% of the overall campus costs for energy use, or $10 million of the approximately $70 million expended annually on campus utilities.

Over the years, server room provisioning has been handled on an ad hoc basis, and certainly without any overarching campus review or plan. The result has been the creation of many smaller, less efficient rooms in both energy and space. A more coherent plan to accommodate large computing systems should entail the use of current server room space, centralized collocation facilities such as SDSC, or regional server rooms; retrofitting existing space; constructing new facilities; or procuring modular server facilities (e.g., Sun Modular Datacenters). The design of server room space has a significant impact on overall energy efficiency: two facilities housing essentially the same set of equipment can have very different energy bills based on how they are configured.

If we assume a campus average of 1.5 devices per employee, the number of computing devices in use would exceed 22,000, and saving even a small dollar amount per piece of equipment could generate significant savings for the campus. Some analyses suggest that instituting better energy management practices across all units can cut the cost of IT-related energy use by 25% or more;23 this could save approximately $2.5 million annually, using the estimated IT-related energy costs noted above.

Best practices relating to energy consumption of technology equipment and energy saving opportunities should be – but are not - widely promulgated to the campus community. The publication and maintenance of this type of information could be assigned to the “IT Expert

22 A UC San Diego working group has produced a draft report on the implications and implementation of a campus research cyberinfrastructure that includes information about how economies of scale and centrally managed services can be leveraged to provide services to the entire campus. While the implementation details are specific to the needs of UC San Diego’s research community, the relevant reasoning is also likely to apply to many production IT activities. 23 P. James & L. Hopkinson, Energy Efficient Data Centres in Further and Higher Education (draft), SustainIT, UK Centre for Economic and Environmental Development, January 2009; http://www.susteit.org.uk/uploads/DOCS/39-data_centres_v4_7_1_09.doc. Also provided as Appendix D.


Network” described in Part I of this report. Recommendations

1. Consolidate Server Locations

The campus should relocate as many servers as possible into centralized, regional server rooms on campus and at the Medical Centers within the next 1-2 years. In order to accomplish this task, the following will need to occur:

Perform a complete inventory of server locations across campus. Make use of any existing, efficient server room facilities with spare capacity (e.g.,

SDSC).

Develop plans for shared, regional server rooms to accommodate growth. Develop funding models that make housing equipment in regional server facilities –

instead of in existing local alternatives – cost neutral to the server owner.

As a general rule, prohibit the construction of small, inefficient satellite server rooms. Modify UC San Diego’s current Server Room Standards to specify minimum room

size and energy efficiency compliance limits.

Require campus-level approval and review of design specifications for all server room renovation projects to ensure that they meet appropriate standards for rack systems, electrical, cooling, networking systems, and physical security.

When units choose to operate local server rooms that do not meet campus energy standards in lieu of using available space in server rooms that do meet those standards, require those units to pay for utilities and maintenance costs that exceed the per unit value typically incurred by facilities that adhere to the server room standard.

Potential Savings and Opportunities

There are an estimated 10,000 servers, primarily in cluster configurations, in use for research computing on campus, many of which are housed in small and/or inefficient server rooms. Based on EPA energy reduction guidelines, relocating 75% of these machines into more energy efficient server rooms could reduce cooling costs enough to save up to $1 million annually.

Well-managed server rooms can generate additional energy savings through operational policies such as consolidating servers, enforcing equipment replacement policies for old, inefficient or underused equipment, and identifying which new technologies and energy-efficient systems should be used as replacements.

Taking advantage of the economies of scale available in large, regional facilities will ultimately result in savings that can be used to cover a significant component of the capital and operating expenditures of such facilities.

Geographically diverse, regional server rooms could also provide opportunities for redundant services and data storage, potentially reducing expenditures to outside storage vendors.


Reconfiguration of existing space into server rooms is costly, and results in the loss of space that would otherwise be available for core instruction and research functions. At an average cost of $845/asf, renovation of space to accommodate 100 racks costs nearly $3.5 million and requires approximately 4,000 ASF with a value of nearly $2.5 million. 24

Actual Server Room Renovation Costs

Location ASF Total Cost Cost/ASF AP&M 5301 521 $620,000 $1,190 Urey Hall 6124 1,000 $510,000 $510 BSB 1000B 1,000 $1,000,000 $1,000 Average Renovation Cost per ASF: $845

2. Improve Energy Efficiency

Improvements in energy efficiency – by using equipment that consumes less power, by sharing equipment, and by using equipment to its full capacity, for example – yields cost savings. Ways to improve energy efficiency include:

Strongly encourage the adoption of voluntary energy saving habits such as those recently circulated by UC San Diego’s Facilities Management group.

Provide no cost workstation energy management software, such as Faronics 25to all campus units and strongly encourage its deployment, reducing computer energy consumption based on user activity level and decreasing utility costs without disrupting either end-users or IT personnel.

Work with system administrators to ensure and certify that their desktop computer management practices are in line with energy savings measures.

Reduce the inventory of computing equipment by performing careful assessments of the business needs for “duplicate” equipment such as multiple monitors or stand-alone printers for individual staff.

Help units and end-users select equipment that is appropriately sized for actual business process demands, and to consider options that consume less energy, such as laptops or thin client technologies.

Encourage the use of server virtualization where appropriate. Locating servers in shared regional centers will facilitate this practice. Develop a hosted virtual server commodity market.

Potential Savings and Opportunities The campus’ Energy/Utilities Manager estimates that turning off a typical Windows

desktop system overnight yields annual savings of 520.78 lbs of CO2 and $69.17.

The use of PC energy management software is estimated to save between $16 and $24 annually per workstation. Installing that software on 10,000 workstations across

24 The average cost of construction of two lab buildings in 08/09 was $621/asf. 4,000 asf x $621/asf = $2,404,000. 25 See Appendix E for a case study of implementation of this product at Howard Community College.


campus would therefore generate between $160,000 and $240,000 in annual savings.26 The Team is not recommending any particular software, as that decision should be investigated and determined by a campus procurement process. By way of example, however, information on Faronics Power Save software and a case study of its implementation are included as Appendix D of this report.

In a hypothetical unit of 100 faculty and staff with one server room of 140 square feet, implementing significant energy savings measures relating to IT technologies, principally through changing the behavior of personnel, reducing equipment inventory and using equipment with lower power demands as shown in the table below, could save as much as $13,000 annually over all fund sources. Extrapolating these potential savings to a workforce of 15,000 employees across campus shows that up to $2 million annually could be saved on energy costs alone. (This assumes that the existing level of energy consumption is high and compares it with a best-case scenario of realizable savings.)

Potential Energy Savings in a Hypothetical 100-Person Unit

Current “Less Efficient” Unit Potential “More Efficient” Unit

How Electricity Energy How Electricity Energy Many (KwH) Cost Many (KwH) Cost Servers 10 66,680 $5,201 5 18,068 $1,409 Desktops/laptops 150 77,963 6,081 100 4,600 359 Printers 20 4,270 333 10 546 43 Monitors 150 44,280 3,454 100 3,600 281 Copiers 4 3,809 297 2 1,905 149 TOTAL 334 197,002 $15,366 217 28,719 $2,241

Concurrent savings from ultimately purchasing fewer devices could be as much as

$20,000 for the hypothetical unit, or as much as $3 million annually for the entire campus.

Taking advantage of virtualization techniques to reduce the number of servers on campus can reduce acquisition costs for users while also saving the campus money through reduced energy consumption.

26 It is assumed that at least 10,000 of the workstations on campus do not yet have such software installed and/or are not being regularly turned off when not in use.


Part V: Campus Messaging/Calendaring Services

Background

Many units individually manage their own messaging/calendaring services at present. A variety of applications are in use across the campus and there is no stated campus standard. It is estimated that there are approximately 450 servers deployed on the campus for email/calendaring needs, and 45 FTE involved as the system administrators for such needs. Maintaining these decentralized services and servers costs UC San Diego as much as $4 million annually. (Most of these savings have probably been included in one of the estimates of potential savings that could be achieved in the other areas of this report, including personnel, procurement and energy).

In the mid-1990s, ACS/Network Operations offered a centralized service called Popmail (popmail.ucsd.edu). Popmail was so under-funded and under-staffed that it couldn’t provide the reliability needed by most campus users. As a result, many units set up their own email servers.

There are now many robust (and well-managed) messaging environments in individual units that dedicate significant staffing and capital to them. The cost of a robust unit-level messaging environment (including hardware, software, service contracts and personnel) is estimated to range between $25-$200/customer/year, versus approximately $17/customer/year for the service centrally maintained by ACT. NGN charges include the ACT-managed messaging services, so there is no additional cost to units that use them.

Additionally, there are many small unmanaged email servers that exist throughout the University. These servers are inexpensive for the unit and the end-user (because they are unmanaged), but incur significant costs for the University when ACT must troubleshoot service problems or respond to security events.

Current Availability of Campus-Wide Services

In late 2007, ACT provided NGN funding and support to build a robust Enterprise email system for the campus. ACT also formed a focus group of twenty IT representatives from all VC areas to define the basic requirements of the system. The new system, mail.ucsd.edu, was put into production in July 2008, and over 12,000 staff and faculty use it at present. The number of users is projected to grow to over 20,000 by July 2009.

This ACT central messaging service is a fault-tolerant Microsoft Exchange implementation that provides email, contacts, and calendaring through a Microsoft client or a Web portal. Email can be accessed using a wide range of clients (Thunderbird, Outlook, Mac Mail, etc.) and operating systems. The mailboxes are administratively controlled by the unit, but physically managed by ACT. The messaging infrastructure can withstand multiple hardware and software failures without any outage, mail delay, or loss.

ACT provides full support to system administrators, who in turn support the customers in their individual units. ACT’s support includes setting up unit-controllable resources within the environment and troubleshooting complex problems. No desktop or end-user support is included in the central messaging service; end-user support continues to be provided at the unit level or through other campus recharge services such as ACS/Desktop Support (http://desktop.ucsd.edu). Limited telephone support is provided under NGN funding and is also available through the ACT Help Desk.

http://desktop.ucsd.edu/�


Recommendation

All units should migrate to ACT services for their centralized email/messaging environments unless they have business needs that cannot be met by the technology in use (e.g., a need for more space for email than Microsoft Exchange can handle well). This would permit the elimination or consolidation of many of the servers that operate for these purposes, along with the elimination or consolidation of system administration personnel, and result in associated cost reductions in individual units. This type of migration would take some months to complete.

At present staffing levels, ACT will soon reach the limit in its ability to handle this service for the entire campus. ACT would therefore require an infusion of some budgetary resources or an increase in the NGN fee. Once fully implemented, estimated overall annual cost savings to the campus in terms of hardware, software, service contracts and personnel, net of the increase needed to ACT, could be as much as $3 million. Again, most of these savings would overlap with savings described in other sections of this report.

Other Considerations

ACT is actively investigating a no-cost, UC-wide contract with Google for the outsourcing of email and use of other Google Apps services. Although the cost and quality of these services are attractive, there currently are some legal concerns with how and where University data would be stored and accessed. UCOP General Counsel is reviewing these concerns. It is likely that the University will be able to use Gmail in some capacity in the future.

When robust and cost-effective external tools exist that meet our business needs, UC San Diego should consider using these in lieu of developing in-house solutions or paying large sums for proprietary software.


Part VI: Implementation of Policies, Guidelines, and Alignment with Organizational Goals

Background

IT expenditures are funded from a variety of sources, and allow us to perform a very broad spectrum of activities in support of the University’s mission of research, instruction, public service, and patient care. The default approach should be to adopt and enforce a set of standard policies and practices related to IT procurements, construction, and personnel/hiring practices. During implementation of any of the Team’s recommendations, it will be vital to acknowledge and allow for some differences in business needs among campus units, as well as to avoid unintended consequences that undermine the original goals of cost savings and efficiencies. Thus, there may be business reasons for granting exceptions to standard polices and practices in some instances.

In order to make optimum decisions, units must carefully balance issues such as energy efficiency, sustainability, ergonomics, security, compatibility, acquisition costs, and ongoing maintenance/repair costs. The availability of the “IT Expert Network” noted above would provide unit decision-makers with a resource to review critical decisions on IT-related matters.

Even with additional information being provided to IT decision makers and a holistic review at each unit level, the campus may be too decentralized and the IT options too numerous to effectively self-regulate. Furthermore, existing incentives – and disincentives – sometimes encourage behavior that may not be in the best long-term interest of the institution.

Recommendation

UC San Diego should consider establishing new IT policies and guidelines, along with better expenditure monitoring and auditing processes, to ensure that incentives and disincentives are correctly aligned with organizational goals for savings and efficiencies. When decision-makers do not directly incur the expense of inefficient use of resources, for example, there is little impetus to consider the impact to the campus overall when making decisions. One way to ensure that the campus-wide expense is taken into account would be to charge any above-normal costs to units that continue to incur them if other viable, less expensive alternatives exist. Determination and deployment of new IT policies, guidelines, and monitoring requirements, as well as the costs versus benefits of such, could be overseen by the previously described IT Expert Network. The Network would require the expertise of the responsible process owners and service providers, such as campus procurement and disbursements.


APPENDICES

APPENDIX A: Chart on Information Technology Systems, excerpted from Building Administrative Efficiency for UC

APPENDIX B: UCSD Health Sciences IT Centralization Review

APPENDIX C: Examples of Existing IT Support Service Recharges at UC San Diego

Integrative Ocean Division (IOD)

IOD’s Computational Infrastructure Recharge provides specified services and software to members of IOD and any ancillary operations for a monthly per person cost based on employee effort. Primary services include:

Desktop computer support, including security management. Access to shared printers. Data storage and backup. Email account support and administration. Individual and research group website hosting. Individual and research group calendar & collaboration hosting. Access to reference operating systems and application software.

San Diego Supercomputer Center (SDSC)

Core desktop services include: Email account support and administration, including spam and anti-virus configuration

for managed systems. SharePoint. Includes maintenance of existing sharepoint service, but not the creation of

additional or specialized sharepoint sites. Home directories, including storage up to 2GB with back ups. Shared calendars. Printing, including printing consumables (paper, toner etc.) on shared printers. Network/connectivity. Machine management and support for one reference desktop/laptop up to 3 years old. Support for all installed ‘managed’ system applications (e.g., Adobe, Office 2007,

antivirus, etc.) for support and maintenance. Jacobs School of Engineering (JSOE)

Administrative and other users may elect to use OEC’s managed desktop services. A desktop “lease” system consists of:

A currently available commodity desktop computer (e.g., Dell Optiplex class), configured to run Windows XP.

A number of typical productivity applications, including the Microsoft Office suite (Access, Excel, OneNote, Outlook, PowerPoint, Project, Visio, Word).

Annual campus site licensing costs for all installed operating system and application software.

Hardware and software maintenance services, including nightly security and configuration scanning, operating system and application software upgrades, and backups of local roaming profiles (desktop and other user-specific settings).

When a desktop lease system is fully depreciated (typically 3-4 years, depending upon the system configuration) it will be replaced with current generation hardware.

Systems which fail prior to full depreciation will be repaired or replaced immediately.

The JSOE recharge unit also provides the following additional services: Consolidated email and calendaring. Windows terminal services access. Unix/Linux interactive login accounts. Database hosting (Sybase and MySQL). Tiered storage services (high availability, standard, archival; CIFS, NFS, AFP). Video streaming services. Managed and unmanaged virtual machine hosting. Web hosting (LAMP; Cold Fusion) Contracted system administration support for administrative and research computing

systems (Unix/Linux, Windows, Mac OS). Contracted web and database programming. Networking support.

APPENDIX D: Energy Efficient Data Centres in Further and Higher Education

APPENDIX E: Faronics Case Study – Howard Community College

campus it services tiger team...

Documents