college of engineering & computer science computing infrastructure upgrade & replacement

8/3/2019 College of Engineering & Computer Science Computing Infrastructure Upgrade & Replacement

http://slidepdf.com/reader/full/college-of-engineering-computer-science-computing-infrastructure-upgrade 1/28

College of Engineering & Computer ScienceComputing Infrastructure Upgrade & Replacement

Mark StanislavEngineering Computer Services (ECS)

University of Michigan - Dearborn [email protected]

August 2007 - January 2008

Abstract

This document will cover the computing infrastructure upgrades and re-placements done for the the College of Engineering and Computer Science atthe University of Michigan, Dearborn campus. The contents of this documentare limited to the information allowed for general consumption and aspects of certain server configurations or otherwise may have been withheld.

The scope of this project was vastly encompassing with an almost complete

replacement of all major infrastructure components for the department and itsresources. The College of Engineering and Computer Science (CECS) containsfive major departments and at the time of implementation, about 3,300 stu-dents, staff, and faculty within it. The plan of this project was to upgradeall existing integral components of the computing infrastructure with minimaldowntime and end-user inconvenience.

Outlined in this document will be not only a point of view into the deci-sions made for the upgrades and replacements, but also technical informationrelating to how technologies were implemented and notable aspects to eachconfiguration. The project timeline noted above is for the initial project plan-

ning, all testing, and commencing in January 2008, the launch of the newcomputing resources.

1



Contents

1 Existing Infrastructure 5

1.1 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2 Network File Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2.1 File System Quotas . . . . . . . . . . . . . . . . . . . . . . . . 61.3 E-Mail Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3.1 SMTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.3.2 IMAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.3.3 Web Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.4 Network & Service Monitoring . . . . . . . . . . . . . . . . . . . . . . 71.5 Remote Login Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . 71.6 User Account Management . . . . . . . . . . . . . . . . . . . . . . . . 81.7 PC Client Authentication . . . . . . . . . . . . . . . . . . . . . . . . 8

1.8 UNIX Lab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Existing Infrastructure - Issue Summary 9

2.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 General Security & Management . . . . . . . . . . . . . . . . . . . . 9

3 Infrastructure Replacement - Overview 10

3.1 Scope Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2 Project Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 Authentication 114.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.3 LDAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.4 Kerberos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.5 Major Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5 Network File Storage 13

5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.3 OpenAFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.4 Major Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 E-Mail Services 14

2



6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146.3 SMTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.4 IMAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.5 Webmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.6 Mailing Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.7 Additional Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.8 Major Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

7 Network & Service Monitoring 16

7.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167.2 Nagios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167.3 Cacti . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177.4 Snort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

8 Remote Login Cluster 17

9 User Account Management 18

9.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189.2 User Account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189.3 Administrator Account . . . . . . . . . . . . . . . . . . . . . . . . . . 18

10 PC Client Authentication 19

11 UNIX Lab 19

12 Data Backups 20

13 Infrastructure Details 20

13.1 Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2013.2 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2113.3 Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

14 Implementation 21

14.1 User Credentials Migration . . . . . . . . . . . . . . . . . . . . . . . . 2114.2 U ser Data Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . 2214.3 Service Cut-Over . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2214.4 Goals Reached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2314.5 Needed Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . 2314.6 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3



15 Appendix 24

15.1 New Infrastructure Servers . . . . . . . . . . . . . . . . . . . . . . . . 2415.2 Screenshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

15.2.1 my.engin - User Main . . . . . . . . . . . . . . . . . . . . . . . 2515.2.2 my.engin - Administrator User Diagnostic . . . . . . . . . . . 2615.2.3 Cacti . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2715.2.4 Nagios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

16 References 28

16.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2816.2 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2816.3 Network File Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 2816.4 E-Mail Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2816.5 Network & Service Monitoring . . . . . . . . . . . . . . . . . . . . . . 28

4



1 Existing Infrastructure

The majority of existing infrastructure was running from very outdated and some-

what failing Sun Microsystems hardware (generally Sun Ultra or Enterprise hardwarefrom approximately 8 years prior). Most of the servers were installed with Solaris8 which was originally released in February 2000. Hardware was rarely configuredwith redundancy by way of disks, backups, or fail-over. Most existing infrastructurewas dependent on the single machine which operated as both data storage and ap-plication server.

The following is an overview of the previous infrastructure that was in place beforeupgrades and replacements were made. Its inclusion is useful to see the overwhelmingscope of the project as well as noting the major improvements from the deprecated

software and hardware in place previously.

1.1 Authentication

The original authentication system implemented by ECS was a standard NetworkInformation Service (NIS) configuration, supported by a single master server whichpassed information to subnetwork specific slave machines. NIS (and not even themore recent, NIS+) had been implemented within the department nearly a decadeago. NIS offers little security (DES passwords) and has no real purpose in a mod-ern computing infrastructure. NIS is long outdated but has a particularly common

inclusion when used along side older version of NFSv3.

1.2 Network File Storage

NFSv3 (RFC from 1995) was implemented as the network file sharing mechanism forall data that required a network share. Included in this data were user’s personal filesand e-mail, departmental web sites, lab software, and faculty and staff information.In modern network storage and sharing, the abilities of NFS are far surpassed by cur-rent technologies such as NFSv4 and OpenAFS. As previously mentioned, NFSv3 iscommonly used with NIS and was a standard configuration from nearly a decade ago.

5



Actual file storage for the network was accomplished with a mash-together of a fewfiber enabled storage arrays that were drastically misused for their adequate space, aswell as some local disk space. The allocation of disk space hindered the ability for easy

expansion and wasn’t configured for the usage of hot-spares in case of disk failure.Generally speaking, a failing RAID would leave users without resources.

1.2.1 File System Quotas

Users generally were given a very small (20MB) file quota, which was often notenforced by the system, allowing for abuses of disk space with the potential for aDenial of Service (DoS) on computing resources if an account was compromised.

1.3 E-Mail Services

Of note, e-mail quotas were also improperly managed much like the general file sys-tem. E-mail data was stored within accessible reach of users, allowing for potentialsecurity risks and the potential for privacy violations. In addition, there was pre-viously no simple way to request e-mail aliases for an account, check to see quotainformation, or even set vacation messages.

1.3.1 SMTP

The incoming mail server was based on Sendmail and resided on a single serveracting the mail gateway for both incoming and outgoing e-mail connections. Theexisting build of Sendmail was approximately three years old and severely under-maintained. No ability for SMTP connections existed outside of the local networkand no SMTP-SSL/TLS connection was available to ensure e-mail was sent securelyfrom the originating sender to the server. No redundancy was available if the SMTPserver went down so users were unable to send and receive e-mail if a failure occurredwith the machine.

1.3.2 IMAP

Again, a single server was available to serve IMAP requests from users. All connec-tions to check a user’s e-mail had to process through the same server which would

6



often lock-up at seemingly random times, most likely due to a hardware failure insystem cooling or RAM. Within the project time-span, the IMAP server machinefailed nearly 20 different times. The IMAP server was running a severely outdated

version of UW-IMAP with improperly configured IMAP-SSL/TLS.

1.3.3 Web Mail

ECS had not previously supported their own webmail system as it was providedthrough the university’s technology department, ITS. This webmail interface, whilefunctional, was not maintained by any staff member of ECS and its availabilitywas outside the control of the department. Any webmail issues were unable to befixed by ECS staff, providing confusion as to whom to contact when an error wouldoccur.

1.4 Network & Service Monitoring

At the time of project start, ECS had no previously established network or servicemonitoring software in place. Server or application downtime was generally discov-ered by end-users and ECS would be alerted through a help desk ticket. Metricsof computing resources (network bandwidth, disk storage, service uptime) were un-known to the staff and their reliability and patterns of usage were only able to beguessed by staff as to how machines and services were preforming.

1.5 Remote Login Cluster

To provide members of CECS with access to remote computing resources in theform of a UNIX-like OS to perform homework on or remotely check their e-mail viathe command line, ECS had implemented a three-server login cluster which wouldprovide this access. At any given time, at least one of the three servers would bedown due to their antiquated hardware failing. The inability to keep all three serversonline caused issues for people trying to connect to a server they usually did andhaving it be down and inaccessible for extended durations.

7



1.6 User Account Management

A largely unorganized PERL script was responsible for command-line administration

of user accounts. Tasks such as adding and deleting users was done through a lackingswitch-based script that offered little information to the administrator and mademanaging of information tedious and confusing. As a result, hundreds of accountsthat were largely unused sat dormant providing a security risk as well as wastedstorage.

1.7 PC Client Authentication

The usage of a pGINA provided a way for department Windows machines to au-thenticate against the existing UNIX-based infrastructure. While this worked for

simple authentication, the need to patch a major component of Windows and usageof unreliably maintained plugins to function makes it a last resort. No network filestorage was available, as Windows network shares weren’t mounted for users uponlogging in. This inconvenience largely made network storage usage not attractive toend-users, providing a higher usage of local machine storage. While the usage of adesktop client to store files is certainly reasonable, it also provides a single point of failure for that data and is unavailable at any other machine.

1.8 UNIX Lab

The ECS UNIX computing lab was running a Solaris 8 image from approximately 5years ago, providing no actual friendly GUI for users or upgrades in software versionsthat had been available for years. Also, machines were rarely patched for securityupdates, making machines an easy target for a malicious user to take over 30 clientsat one time if a vulnerability was found. Reliability of authentication was sporadicas only one server could authenticate users, providing a single point of failure forlogging in.

8



2 Existing Infrastructure - Issue Summary

2.1 Hardware

• Primarily based around hardware that was generally 5-8 years old

• Little usage of RAID for basic data redundancy

• Missing hot-spares on RAID arrays

• Mismanagement of available disk storage resources

• Network interfaces all 10/100mbit on a 1Gbps network

• Some hardware only supporting serial connections for administration

•

No redundancy of critical hardware (e-mail, authentication, network storage)

2.2 Services

• Insecure or feature-outdated versions of all critical services

• Lack of ECS administrated web mail

• No proper Windows domain with network file access

• Network printing services not adequately setup

• Mail, authentication, e-mail services not redundantly configured

2.3 General Security & Management

• No services/hardware monitoring

• Lack of usable OS software package management

• Deprecated interface for user management

• No implemented reliable data backup procedure

• No firewalls configured

• No backup procedure for critical system configurations

9



3 Infrastructure Replacement - Overview

3.1 Scope Limitations

The project as defined was to update the core components of the infrastructure andany related entities that made sense to deal with at the same time. For instance,SQL database servers were not upgraded at this time as they are autonomous. How-ever, network printing (CUPS) was replaced as it was related to the configurationand deployment of a proper Samba-based Windows domain.

This project did not comprehensively replace the whole of ECS resources but thevital aspects to which other services could work around and with. By providing acurrent and functional infrastructure, the ability to add new servers becomes much

more trivial and easily managed.

3.2 Project Goals

• Replace old or failing hardware with new machines

• Switch from Solaris to Linux (Debian GNU/Linux, Gentoo Linux) and FreeBSD

• Replace deprecated service with more current and advanced ones

• Tightly integrate Windows PCs and the UNIX lab for maximum usability

• Provided usable interfaces for administrators and end-users to manage accounts

• Monitor systems and provided a reliable and easy back-up procedure

• Keep security at the forefront of configurations and policies

• Thoroughly focus on making e-mail as stable and featured as possible

• Reallocate disk usage to maximize disk space and logically organize it

• Prevent the upgrades from harming any remaining ’old’ infrastructure

• Transition users easily and make them want to use the new infrastructure!

10



4 Authentication

4.1 Overview

Authentication has transitioned from one authentication mechanism into two. NISwas replaced with a combination of LDAP and MIT Kerberos V. The usage of twoauthentication mechanisms actually makes things less complex by providing twomechanisms that can be leveraged by services that don’t support the other. Forinstance, the usage of Kerberos with the OpenAFS cell provided ease of integration.On the other side, LDAP provides us with a way to not only authenticate users, butmanage them.

4.2 Architecture

The previous usage of a single primary NIS server that would distribute to subnetwork-specific slaves has been replaced with two servers which provide full network coverage,sharing the load of queries using round-robin DNS. While there is still a single pri-mary server for both LDAP and Kerberos, each server can be quickly changed tobecome the master server in case the original master is down for an extended periodof time. By utilizing DNS-based load-balancing, we can help ensure that each serverbears some of the load from authentication, address book look-ups, and any otherrelated queries that involve LDAP or Kerberos.

4.3 LDAP

By utilizing LDAP we can store not only a username and password, but also in-formation such as what department they work for (or are a student in), any e-mailaliases they have, paths to their network file share, path to their Samba share, whatgroups they belong to (for file access or organization), and even their e-mail vacationreply settings.

LDAP provides network access to a myriad of needed information for user accountsin an easily managed database that is a generally widely supported authentication

11



mechanism. NIS had no such extended functionality as LDAP, but also, LDAP of-fers secure password storage in the database with ACLs restricting information fromthose who shouldn’t see it.

In addition to the uses mentioned above, LDAP is the authentication mechanism forall UNIX/Linux clients as well as the authentication backend for Samba (Windowsdomain). Also, it provides the address book functionality to all e-mail clients so thatusers can look-up other people within the scope of CECS to e-mail. By having acentral place for a user’s information we eliminate excessive and scattered storageacross multiple servers.

4.4 Kerberos

Kerberos primarily allows us to integrate easily with the OpenAFS cell for networkfile server access but also helps us integrate with different networks. For example,the usage the Kerberos configuration provides easy access to available University of Michigan, Ann Arbor network file resources in addition to the ones that ECS pro-vides. Kerberos can also provide an authentication interface for many services andhelps ensure easy future expansion of networking resources.

By utilizing Keberos, Windows PC, UNIX lab, and login cluster users are all able toaccess network files merely by logging in. The token generated by Kerberos providesthe required access to those files, as well as the ability to log into other ECS serverswithout the need to re-authenticate.

4.5 Major Improvements

• Securely replicated data between redundant servers

• Two authentication mechanisms providing vast compatibility with services

• Current and reliable technologies that will be around for 5+ years

• Organized user account information and settings storage to single database

• Provide cross-network integration for remote-file access (Kerberos + OpenAFS)

• Storage of passwords no longer in deprecated and easily crackable format

12



5 Network File Storage

5.1 Overview

OpenAFS has replaced NFSv3, providing a huge upgrade in not only technologybut also reliability. OpenAFS is widely used in academia and elsewhere, providingsmooth integration with Keberos based networks. OpenAFS has both database andfileservers. Database servers provide information as to what files are stored where.Fileservers contain the actual files.

5.2 Architecture

The OpenAFS cell (the collection of all of the OpenAFS resources) involves fourdatabase servers and two file servers. There are four database servers to providesupreme redundancy in the case of failure. Two OpenAFS file servers provided accessto over 4TB of RAID-5 data storage, spanning across two RAID storage arrays andlocal disk space allocated on an internal RAID array.

5.3 OpenAFS

OpenAFS provides a coherent namespace for all file storage and allows you to ma-nipulate file storage resources as needed. By using OpenAFS, data stores are able to

be logically named (such as user.mail or user.home) and altered as needed for sizingor naming requirements. OpenAFS gives us certain other advantages such as: tightintegration with Kerberos; hard filesystem quotas; ease-of-scalability; redundancy;manageability.


• Redundant file storage that easily scales and allows reorganization

• Simple integration with Solaris, Linux, Mac OS, and Windows clients

• Reliable and well evaluated technology being widely utilized currently

• Provides added security through highly customizable ACLs of volumes

13



6 E-Mail Services

6.1 Overview

E-mail services were one of the most heavily affected aspects of this infrastructureupgrade as they generally lacked a lot of simple functionality that a modern e-mail system should possess. While e-mail isn’t as directly critical as something likeauthentication or file storage, it is probably the most important end-user servicethat ECS provides. Great effort was taken to make sure that the e-mail system wasrobust and brought users a proper e-mail system for 2008 and beyond.

6.2 Architecture

Two servers were utilized in the configuration of e-mail services. There is essentiallyfull parity between the two machines, meaning that the services and configurationson both servers are the same. The mail servers utilized OpenAFS file space for mailstorage, providing a central location for e-mail to be read/written to, allowing formany servers to access that data without a separate multi-interface RAID storagedevice. Round-robin DNS load balancing is done for incoming IMAP connectionsand MX record based access is done for incoming SMTP connections.

6.3 SMTP

The replacement for Sendmail was Postfix. It provides for more-easily customiz-able configurations and integrations. LDAP provides the authentication backendfor SMTP, now allowing users to authenticate to the e-mail server to ensure that itdoesn’t allow just anyone to send mail through it. Also added was an SSL certificateproviding secure communication between the end-user and e-mail server. More thanever, users can now feel secure when sending e-mail through ECS mail servers.

Virus and spam filtering was also added so that users can deal with less securitythreats and annoyances in their daily usage. Incoming and outgoing e-mail is scanned

to help prevent users from being attacked and also from the ECS network from beingblacklisted for sending dangerous e-mails.

14



6.4 IMAP

The existing IMAP service was replaced with Courier-IMAP which has a very strong

security history and is widely used. E-Mail quotas were properly implemented withCourier, helping ensure that e-mail usage remained under control unlike the previoussystem. IMAP-SSL connections are also correctly configured now so that users canagain feel safe that the data transmitting across a public network is secure. LDAPis also used to authenticate IMAP users, as it is with SMTP.

6.5 Webmail

Previously missing from the scope of the ECS infrastructure was a webmail system.To help ensure the usage of webmail by CECS users, two webmail systems featur-

ing different strengths were implemented. RoundCube provides a very interactive,almost desktop-application feel to webmail. SquirrelMail on the other hand is a verysimplistic webmail client with no visual features, but has been proven reliable foralmost a decade. Each e-mail system features an address book (pulled from LDAP),information about e-mail quota usage, and ease of integration with the e-mail sys-tem. Both webmail systems authenticate against the IMAP servers that run locallyon the servers.

6.6 Mailing Lists

Upgraded from the previous infrastructure was a new version of Mailman. Mailman isthe foundation of all CECS mailing lists and the newest version provides security andusability enhancements over the previous. While not a major change, mailing listswere re-created and a method to populate mailing lists with reliable user databaseswas made through the usage of LDAP records.

6.7 Additional Features

Vacation reply messaging, e-mail alias requests, and quota information was not pre-

viously readily available to users or administrators. A web interface has been givento not only set vacation message information, but request quota increases & e-mailaliases simply. These will be mentioned later on regarding ‘my.engin’.

15




• Working e-mail quotas that can be requested to be increased and viewed easily

• Functional virus quarantining and spam filtering

• Two webmail interfaces to help appease different user requirements

• Redundant e-mail servers utilizing DNS round-robin/MX record balancing

• Securely authenticated SMTP/IMAP transactions w/ off-campus SMTP access

• Updated mailing list software and realiable list management

• Centrally-stored mail protected by ACLs within OpenAFS

7 Network & Service Monitoring

7.1 Overview

As previously mentioned, the prior ECS infrastructure had no service/server mon-itoring in place. To fulfill that need, a combination of Nagios and Cacti were usedto help not only monitor that a server or server is online, but also monitor howthe resources of that server is being used, and when. SNMP is used for the Cactimonitoring and has been installed on all servers.

7.2 Nagios

Nagios provides per-server system monitoring of critical system services as well aswatching for system metric thresholds to be broken. For instance, if a disk reachesa certain point (such as 80% full) Nagios will send an e-mail, page, and AIM instantmessage alerting the system administrators that a problem has occurred. This kindof monitoring helps to ensure that the technical staff is the first one to know of aproblem, not the last.

Each server has an individual set of monitored services and will even alert admin-istrators when package upgrades are needed. Full customization of who, when, andwhy allow Nagios to provide administrators only the alerts that are critical.

16



7.3 Cacti

By utilizing SNMP, Cacti can monitor trends in server metric usage, such as: band-

width; memory usage; disk usage; and load averages. Cacti is a great resource tomonitor server trends to find out when you have spikes in load averages or memoryusage to help find problems with scripts or services you may be running. With Cacti,you can not only find out if a server is being overtaxed but also help catch prob-lems that may arise in the near future before Nagios needs to alert you at a critical

juncture.

7.4 Snort

As an added protection, Snort helps to monitor the network for attacks. By using

Snort we can help detect if a user on an ECS subnet is port scanning other comput-ers or trying to perform a potential attack from ECS computing resources. UsingSnort helps ensure that we know that ECS network clients and servers aren’t beingused to attack any other campus machines, or worse, machines outside of the ECSnetwork.

8 Remote Login Cluster

While not a major component of the critical network infrastructure, the login cluster

is utilized by students, staff, and faculty for tasks such as code compilation, editingtheir files, and checking their mail. Previously, old hardware and Solaris 8 provedto cause many problems for these simplistic tasks. The new login cluster providesPC machines as servers and run Debian GNU/Linux rather than Solaris. They aremore reliable and easier to maintain than the previous Sun Ultra 1 machines runningSolaris.

These new machines also feature more software than previously available to end-users. Customization to system-wide configuration files has helped to make taskssuch as checking e-mail even easier. Also, because resources are more organized with

the OpenAFS and how authentication works, replacing a login cluster machine ismuch easier than before.

17



9 User Account Management

9.1 Overview

As previously mentioned, the existing method for managing user accounts was donethrough a command-line PERL script. The replacement for this is a completely newPHP-based web application that gives greater control over the user. Not only canadministrators add, edit, and delete users, but users can also login and see theirdetails and set options related to their account.

9.2 User Account

Any user that has an account with ECS is able to log into the ‘my.engin’ websiteand manage their account details to a certain extent. Features of the ‘my.engin’ website for a user include:

• Configuring vacation auto-reply settings

• Set-up e-mail forwarding

• Request e-mail aliases & e-mail quota increases

• Change the account password (updates LDAP, Kerberos, Samba all at once)

• View account information (quota usage; e-mail aliases & forwards; web address)

9.3 Administrator Account

The administrative users for ‘my.engin’ are able to perform many user manage-ment tasks, including those not previously available to simply do through the oldcommand-line PERL script. Available to administrators are the abilities to:

• Add/delete users (verified against the student Oracle database)

• View diagnostic information users to verify their configuration and settings

• Check for user quotas that are approaching a high threshold

• Reset user passwords

• Edit user quotas, e-mail aliases, vacation status, forwarding, and more

18



10 PC Client Authentication

Prior to infrastructure upgrades, PC Windows authentication was done through a

pGINA plugin that would contact NIS. Through the usage of Samba, a valid Win-dows domain is created that authenticates against LDAP. By integrating Sambawith OpenAFS and Kerberos, per-user Windows home directory mounting is accom-plished. This could also be handled by utilizing the Kerberos and OpenAFS clientsfor Windows.

CUPS is one of the services updated on the periphery as it was directly related tocreating a Windows domain. CUPS integrates well into the domain, providing load-balanced printing in labs where appropriate. With the usage of network printingand network file storage, the new infrastructure gives any generic Windows clientthe ability to be fully integrated into the new infrastructure.

Windows domain administrators are organized through LDAP for easy configuration.All computers that authenticate against Samba are also added to LDAP automati-cally.

11 UNIX Lab

The Solaris UNIX lab received a major software upgrade even though the actualhardware remained the same. Not only did the actual UNIX client image upgradefrom Solaris 8 to 10, but it also was fully integrated with Kerberos & LDAP. When

a user logs into the UNIX lab, it authenticates against LDAP and then retrieves aKerberos token so that OpenAFS is mounted as their home directory.

The UNIX lab provides the last component of a fully integrated environment, allow-ing a user to seamlessly access their same files from a Solaris lab machine, WindowsPC client, Linux remote login cluster machine, or even a Mac OS machine theybrought from home (using appropriate software).

The machine that images machines was also upgraded and the UNIX lab is now afully Solaris 10 configuration. The upgrades with the image outside of integrationinto the new infrastructure included basic package management as well as new GUI

interfaces for users to select from to have a better experience.

19



12 Data Backups

A briefly mentioned goal was to implement data backups within the new infrastruc-

ture as the existing one had none. An unused tape-changer became the cornerstoneof the new backup system. By using the built-in functionality of OpenAFS to dofull and incremental backups, tapes are utilized to do weekly full backups and thendaily incrementals until the next full. Configuration and system critical files for eachserver are also committed daily to a subversion repository on the network monitoringserver. The backups of these files help to ensure that if a server crashes, a new servercan be built and configured in a minimal amount of time.

Tape backups with the added redundancy of all RAID-5 configurations for majordata storage points help to ensure that data is rarely ‘lost’. OpenAFS easily restores

volume backups into place, allowing for an accidental mail deletion from the weekbefore to not affect a user dramatically.

Data tape backups are not kept in the same building as the servers for obvioussecurity and threat reasons.

13 Infrastructure Details

13.1 Operating Systems

The majority of new machines are all running Debian GNU/Linux. Apt providessimple binary package management and a minimalist installation, providing an easychoice for servers that have very specific service needs. The primary OpenAFS file-server runs Gentoo Linux because of initial problems getting a working OpenAFSfileserver running on Debian GNU/Linux. FreeBSD was used on the network moni-toring server as it provides a reliable and secure configuration that is trusted to bea hub of information about all of the servers. The selection of FreeBSD was oneof familiarity and not due to overwhelming technical reasons other than securityenhancements found through TrustedBSD extensions.

20



13.2 Hardware

The majority of replacement servers are HP Proliant servers. Their configurations

vary depending on the requirements of the server, but each server generally has80GB of disk space, 1-2GB of RAM, and gigabit ethernet. The OpenAFS fileserversare dramatically more high-end machines, featuring fiber channel cards, multiplegigabit-ethernet cards, hundreds of gigabytes of internal storage, etc. All machinesrun Pentium 4 or Xeon processors.

13.3 Connectivity

As mentioned, gigabit ethernet was one major upgrade to each new machine from theprevious 10/100mbit configuration. The main OpenAFS file server also has multiple

SCSI cards for connectivity to the tape changer, as well as expandability for morestorage. It also possess a fiber channel card for connectivity to a RAID-5 Sun storagearray.

Currently, only one ethernet interface is used, but any additional ones could supportdirect server-to-server file server connectivity. Of note, the network monitoring serveralso has an analog modem in the machine so that the added ability of SMS could beimplemented to alert administrators of problems.

14 Implementation

14.1 User Credentials Migration

One of the largest parts of this infrastructure upgrade was migrating users from anolder authentication system to the new ones (Kerberos/LDAP/Samba). Part of thechallenge of this procedure was to make sure that the passwords they previously hadin the lax restrictions of DES and NIS would be usable in the new system. As a wayto help publicize the transition as well as have users create new passwords if needed,a web site called ‘migrate.engin’ was created. This web software essentially verified

whether or not a user existed, and if they did, forced them to create a new passwordif their current one wouldn’t meet a realistic set of requirements.

21



By forcing users to migrate it helped to figure out who had active accounts and whodid not. It also made the idea of obtaining current passwords a lot simpler thanwaiting for users to arbitrarily login to a service and eavesdrop to essentially ‘steal’

the password. The migration effectively had a vast majority of active users hold theirspot (so to speak) for the new infrastructure before the dead-line that was imposedupon them.

14.2 User Data Migration

The old and new mail systems were incompatible so a script had to be written totake a list of users, archive their existing mail, convert it into a usable format forthe new mail server, and transfer that data onto the new server. Also, all user home

directory data had to also be transferred so that data wouldn’t be missing that auser expected after the transition.

One aspect of home directory files intentionally skipped where ‘dot’ files (files be-ginning with a period, usually related to configuration files). By not archiving ’dot’files, we helped to ensure their old settings wouldn’t mangle their settings with thenew infrastructure. A script to convert e-mail data and move both the convertede-mail and home directory data was deployed and successfully executed during thetransition downtime.

14.3 Service Cut-Over

By launching the new infrastructure at approximately 5am on a Sunday morning,there was an extreme mitigation of user interruption. Any interruptions were likelydue to auto-connecting IMAP clients left open. As new infrastructure was broughtonline and tested, data was transferred where needed. Once all hardware and con-figurations were in place, DNS was transferred to the new servers and the migrationfrom the old infrastructure components to the new ones was completed.

22



14.4 Goals Reached

All initial goals were accomplished through the implementation of the new infras-

tructure. Users had a very positive view of the new services and upgrades offered byECS and felt very little discontent with any procedures they had to endure. Greatlengths were taken to help ensure simple aspects like making sure e-mail forwardsexisted in the new infrastructure as they did in the old one helped provide a smoothtransition. While there were tweaks to be made, the overall scope of the project wentflawlessly when implemented, even exceeding expectations first hoped for.

14.5 Needed Improvements

As mentioned before, some services were not upgraded as they weren’t required to

be related to this infrastructure upgrade. Web servers, SQL database servers, andother stand-alone appliances still need to be integrated formally and upgraded forcompliance in the new infrastructure. The needed improvements are generally relatedto usability, reliability, and security.

14.6 Miscellaneous

Documentation related to back-up procedures, risk assessment, policies & proce-dures, and administrative notes were also created in the same time-frame as theseinfrastructure updates. The usefulness of these documents are related to how well

they are maintained in the future. General server installation documents were cre-ated for each server for emergency recovery situations.

23



15 Appendix

15.1 New Infrastructure Servers

joust.engin.umd.umich.edu - Debian GNU/LinuxOpenAFS DB Server, LDAP Primary Server, Kerberos Primary Server

breakout.engin.umd.umich.edu - Debian GNU/LinuxOpenAFS DB Server, LDAP Secondary Server, Kerberos Secondary Server

pitfall.engin.umd.umich.edu - FreeBSDCacti, Nagios, Snort, Subversion

gravitar.engin.umd.umich.edu - Gentoo LinuxOpenAFS DB Server, OpenAFS Fileserver, Personal Web Files

adventure.engin.umd.umich.edu - Debian GNU/LinuxPostfix, Courier-IMAP, Apache (webmail)

klax.engin.umd.umich.edu - Debian GNU/LinuxPostfix, Courier-IMAP, Apache (webmail)

threshold.engin.umd.umich.edu - Debian GNU/LinuxOpenAFS DB Server, OpenAFS Fileserver, Samba, CUPS

my.engin.umd.umich.edu - Debian GNU/LinuxApache (‘my.engin’ web site)

cluster2.engin.umd.umich.edu - Debian GNU/LinuxNo services - File/shell access

cluster3.engin.umd.umich.edu - Debian GNU/LinuxNo services - File/shell access

24



15.2 Screenshots

15.2.1 my.engin - User Main

25



15.2.2 my.engin - Administrator User Diagnostic

26



15.2.3 Cacti

15.2.4 Nagios

27



16 References

16.1 General

Gentoo Linux http://www.gentoo.org/Debian GNU/Linux, http://www.debian.org/Solaris, http://www.sun.com/software/solaris/

16.2 Authentication

OpenLDAP, http://www.openldap.org/MIT Kerberos, http://web.mit.edu/Kerberos/Samba, http://www.samba.org/

16.3 Network File Storage

OpenAFS, http://www.openafs.org/

16.4 E-Mail Services

Postfix, http://www.postfix.org/

Courier-IMAP, http://www.courier-mta.org/imap/SquirrelMail, http://www.squirrelmail.org/RoundCube, http://www.roundcube.net/Mailman, http://www.gnu.org/software/mailman/ClamAV, http://www.clamav.net/DCC, http://www.rhyolite.com/dcc/Spam Assassin, http://spamassassin.apache.org/

16.5 Network & Service Monitoring

Nagios, http://www.nagios.org/Cacti, http://www.cacti.net/Snort, http://www.snort.org/

28

college of engineering & computer science computing infrastructure upgrade & replacement

Documents