1

Mississippi State University Request for Proposals (RFP) 21-65

High Performance Computing System

ISSUE DATE: July 12, 2021

ISSUING AGENCY: Office of Procurement and Contracts

Mississippi State University 610 McArthur Hall 245 Barr Avenue Mississippi State, MS 39762

Sealed Proposals, subject to the conditions made a part hereof, will be received August 10, 2021 at 2:00 PM in the MSU Office of Procurement and Contracts, same address above, for furnishing services and potentially, optional services as described herein.

IMPORTANT NOTE: Indicate firm name, and RFP number on the front of each sealed proposal envelope or package.

All inquiries concerning this RFP should be directed to:

Jennifer Mayfield Office of Procurement and Contracts, (Same address above) jmayfield@procurement.msstate.edu 662-325-2550

Any addendum associated with this RFP will be posted at http://www.procurement.msstate.edu/procurement/bids/index.php located under RFP 21-65. It is the respondent’s responsibility to assure that all addenda have been reviewed and if applicable, signed and returned.

2

1. UNIVERSITY OVERVIEW Mississippi State University (MSU) is a comprehensive land grant university of nearly 23,000 students and approximately 5,000 faculty and staff. The main campus is located adjacent to the community of Starkville in northeast Mississippi, with a remote campus located in Meridian with an enrollment of around 700 students. Additionally, the university operates several remote agricultural experiment stations and has an Extension office located in each of the eighty-two Mississippi counties. The MSU High Performance Computing Collaboratory (HPC2), located in the Thad Cochran Research, Technology and Economic Development Park adjacent to the university campus, is a coalition of seven affiliated research centers and institutes sharing the common characteristic a of multi-disciplinary team-oriented effort that is strategically involved in the application and advancement of computation science and engineering through the utilization of high- performance computing. Additionally, the HPC2 manages the university’s Computational Engineering graduate program, offering both Master of Science and Doctor of Philosophy degrees.

Additional information about MSU can be found at our website www.msstate.edu .

2. INVITATION TO SUBMIT PROPOSAL ON RFP MSU seeks a high-performance computing cluster based on x86-64 processors, interconnected with a high-bandwidth low-latency network, maximizing performance and minimizing power consumption. MSU desires to utilize the proposed solution in support of research, educational, and service activities at the HPC2. The proposed system should be a balanced and fully integrated system including all compute nodes, support nodes (management, login, data transfer, development), networking systems, storage systems, cooling distribution units, and power distribution infrastructure. The proposed system design should be one that maximizes computing capacity and is scalable for the support of a planned expansion. The proposal should provide for a fully integrated system with a total cost not to exceed $5,500,000, heretofore referred to as the base system. In addition to the base system, the proposal should include configuration details and the incremental costs for optional expansion “units” that could be purchased within 120 days of the proposal due date. The number of expansion units that may be purchased is dependent upon funding availability. The maximum total procurement price for the base system plus any expansion units will not exceed a total cost of $15,500,000. An expansion unit must utilize the same components as proposed in the base system. The scope of an expansion unit, in terms of the quantity of components or architectural design, is to be defined by the vendor. For instance, an expansion unit could be defined as an additional rack or sets of additional racks of equipment in a configuration consistent with what is specified in the base system. The expansion unit must include all computing, networking, cooling components, power components and cabling, to allow for fully integrating with the base system. The

3

expansion unit should not include any login, development, system management, data transfer nodes, or storage systems. Vendors must describe in detail the system architecture of their proposed solutions (vendors may submit up to 2 responses to this solicitation).

3. SCOPE OF SERVICES & WORK REQUIRED a) General Requirements

• All proposed hardware, software (released commercially or as open source), and infrastructure must be supported by the Supplier at the time of system acceptance and throughout the subsequent 48 months of production system use.

• The proposed system must be a balanced, commercially-available, production-grade HPC system that contains an appropriate combination of processor, memory, interconnect, disk input/output (I/O), and operating system (OS) capabilities in order to execute complex, tightly-coupled, large-scale, scientific calculations; more specifically, the system must be able to successfully execute a variety of workloads, including jobs which stress all subsystems and which require the simultaneous, tightly-coupled use of the full number of compute nodes within the system.

• All equipment must be new; no refurbished or used equipment is allowed. • A thermal overload protection capability must be included to prevent overheating of the

nodes. A hardware-based solution is preferred. • All nodes and operating systems that are provided must include any required fixes for

Spectre/Meltdown vulnerabilities and its variants. • The system compute racks, support racks, cooling distribution units (CDU) and network

racks will be installed in the space depicted in the attached floor plan. Equipment layout must provide adequate service area / isle space in relation to existing infrastructure.

• Once award is complete, the Supplier must provide a project manager for the duration of the project to include:

o Final configuration o Site review o System build and off-site integration o Factory test o Installation at MSU facility o Acceptance testing

b) Technical Requirements

i. Compute Nodes

o Standard Compute Node. Must contain two processors. The processor must be an Intel Platinum 8358 or AMD EPYC 7543, or a higher performing processor in the same family/series.

4

o Must contain a minimum of 6 GB of memory per core, utilizing the highest available memory speed supported by the proposed processor, must be error-correcting code (ECC) memory, all memory channels populated and must be a balanced memory configuration.

o Must contain a mixed-use internal SSD with a minimum size of 240 GB. o Must support the latest stable release of RedHat Enterprise Linux (RHEL) and

CentOS (licenses for RHEL are not required) o Must include connectivity to the High-Speed Network (HSN) o Must include connectivity to the Management Network (MN) at a minimum of 1

GigE o Must support IPMI2.0 and/or Redfish

ii. Login Nodes

The proposal must include a minimum of four (4) login nodes, each of which:

o Must contain two processors, using the same processor as the Standard Compute Node o Must contain a minimum of 16 GB of memory per channel for all available memory

channels, utilizing the highest available memory speed supported by the proposed processor, must be error-correcting code (ECC) memory, and must be a balanced memory configuration

o Must contain two (2) internal SSDs for mirrored OS, each of at least 400 GB. o Must contain redundant power supplies o Same OS as the Standard Compute Node o Must include connectivity to the HSN o Must include connectivity to the MN at a minimum of 10 GigE o Must support IPMI2.0 and/or Redfish

iii. System Management Nodes

The proposal must include a minimum of two (2) system management nodes, each of which:

o Must contain two processors, using the same processor as the Standard Compute Node o Must contain a minimum of 16 GB of memory per channel for all available memory

channels, utilizing the highest available memory speed supported by the proposed processor, must be error-correcting code (ECC) memory, and must be a balanced memory configuration

o Must contain two (2) internal SSDs for mirrored OS, each of at least 1 TB o Must contain redundant power supplies o Same OS as the Standard Compute Node o Must include connectivity to the HSN

5

o Must include connectivity to the MN at a minimum of 10 GigE o Must support IPMI2.0 and/or Redfish

iv. Development Nodes

The proposal must include a minimum of four (4) development nodes, each of which:

o Must contain two processors, using the same processor as the Standard Compute Node o Must contain a minimum of 16 GB of memory per channel for all available memory

channels, utilizing the highest available memory speed supported by the proposed processor, must be error-correcting code (ECC) memory, and must be a balanced memory configuration

o Must contain two (2) internal SSDs for mirrored OS, each of at least 400 GB. o Must contain redundant power supplies o Same OS as the Standard Compute Node o Must include connectivity to the HSN o Must include connectivity to the MN at a minimum of 10 GigE o Must support IPMI2.0 and/or Redfish

v. Data Transfer Nodes

The proposal must include a minimum of four (4) data transfer nodes, each of which:

o Must contain two processors, using the same processor as the Standard Compute Node o Must contain a minimum of 16 GB of memory per channel for all available memory

channels, utilizing the highest available memory speed supported by the proposed processor, must be ECC memory, and must be a balanced memory configuration

o Must contain two (2) internal SSDs for mirrored OS, each of at least 400 GB. o Must contain redundant power supplies o Same OS as the Standard Compute Node o Must include connectivity to the HSN o Must include connectivity to the MN o Must include connectivity to the facility LAN at 40 GigE o Must support IPMI2.0 and/or Redfish

vi. High Speed Network (HSN)

o A high-throughput low-latency interconnect network, with a minimum of 200 Gb/s per node injection rate

o Fat-tree topology with a maximum of 2:1 bandwidth over-subscription using no more than two-tiers

o Redundant power supplies for any switch that would impact more than 24 nodes

6

o If a director class switch is proposed, this must come with a redundant management system

o If a fabric manager is required, it must run on the System Management Nodes

vii. Management Network (MN)

o At least one GigE connection to all equipment, including but not limited to all nodes, storage system, switches, smart/switched PDU’s, RDHx’s, etc.

o A core switch must be provided with a minimum of 10 GigE connections to all edge switches and system management nodes

o Minimum of 100 GigE connectivity to the facility LAN o 100G LR (duplex single mode) optics should be provided by the vendor; facility will

provide cables for connectivity to the facility LAN

viii. Connectivity to Existing Geographically Separated Storage (/work)

o The proposed system must be integrated with an existing InfiniBand-connected DDN ExaScalar resource. The proposed system and the DDN resource will be housed in data centers in geographically separate, but adjacent buildings.

o 24 strands of single-mode dark fiber is available between the two data centers for connectivity to the storage resource. The end-to-end distance for this fiber is less than 1000 meters.

o The proposed solution must connect to the IB fabric of the existing storage resource. This connection may be via either a) a direct fabric-to-fabric integration or b) through the utilization of LNet Routers. Fabric-to-fabric solutions: connectivity to the storage resource must be via a

Mellanox HDR InfiniBand spine network consisting of 14 HDR switches. The integration must utilize one (1) HDR connection to each of the 14 spine switches in the existing IB fabric.

LNet Router solutions: must connect to the existing fabric via no more than 10 InfiniBand ports in the existing Mellanox HDR switch within the storage resource. An air-cooled rack for all LNet Routers and associated components must be provided. The rack must include two in-rack PDU’s (each to be fed via a facility-provided 30-amp 208-volt L630 receptacle) for redundant connectivity to the LN Routers and other equipment.

o The proposed solution must provide a minimum throughput bandwidth of 600 Gigabit/second to the existing DDN storage resource.

7

ix. Home Directory and Application Storage System (/home & /apps)

o Must be simultaneously mounted on all compute nodes, login nodes, data transfer nodes, and system management nodes

o Must be NFS-based with a total usable capacity of at least 70 TB, to be sub-divided into 2 distinct file systems (/home & /apps), in a hardware-based RAID6 or equivalent drive failure protection configuration

o Must support NFSv4 o All HDDs shall be self-encrypting drives o Must contain redundant power supplies o Must include connectivity to the HSN o Must include connectivity to the MN

x. The proposal must include one networked KVM with monitor, 24 port minimum.

c) Software and Hardware Management

o Cluster deployment, configuration, and software management Not required as a deliverable. Vendor is responsible for any software and

configuration tools utilized during acceptance testing. o Hardware Management Environment

Software for the management of compute, login, DTN, development, and management nodes. This software shall manage firmware, and other updates.

A software package must be included for the management of firmware, bios, etc. on all compute, login, DTN, development, and management nodes. This software must be able to provide an inventory the major components of these nodes and firmware/bios revisions, and be able to update firmware/bios. The software must be able to provide the health status of all nodes.

d) Physical Infrastructure Requirements

i. Racks

o Each rack should include a sufficient number of internal PDUs to feed all equipment mounted within the rack. No racks shall have more than a maximum of four (4) whips per rack.

o All power connections shall be bottom fed. Network interconnect cables for adjacent compute racks may be top or bottom fed.

ii. Cooling

o MSU will provide a dedicated chilled water capability for this system via a Daikin chiller, model number AWV014B. Chiller specifications are provided upon request. Allowable chilled water setpoint range is 45-65F.

8

o The facility cooling loop includes up to three chilled water supply connections. Each of these connections utilize a 4” Victaulic flange.

o Rack Cooling Cooling will be accomplished through liquid-cooled direct thermal contact, Rear

Door Heat Exchanger (RDHx), or a combination of the two. The cooling must neutralize 100% of the heat generated by the rack (room

neutral) All cooling and power connections shall be bottom fed The cooling system must support network remote monitoring and control The cooling system must provide leak prevention and detection capabilities

o Cooling Distribution Units (CDU) The proposal must provide a CDU to be connected to the facility chilled water

loop, and in turn supply chilled water for each rack. The datacenter room temperature is approximately 70 F with a 40% relative

humidity. The CDU must be able to automatically adjust to operation using cool water

above the ambient dew point in the data center. The proposal must provide a distribution manifold and all necessary

piping/hoses to connect the CDU to rack cooling systems.

iii. Power Requirements and Availability

o 1200 kW of power is available via a 480-volt switchboard and distribution panel containing 18 Eaton 3-phase Frame Size 2 branch breakers (225A Frame) with PXR10 electronic trip units, adjustable from 30 to 225 amps. The distribution panel has available space to support 4 additional Frame Size 2 breakers. Proposals should minimize the number of breakers used in order to support

system expansion. Non-transformer Remote Power Panels (RPP) may be utilized for additional

power distribution. Vendor must provide all power cabling and make final connections to facility

breakers and RPP specified. o An additional 50 kW of UPS power provided via twenty-four 30 amp, 208-volt 1-phase

power whips terminated with L6-30 receptacles is available for infrastructure items such as network switches, system management nodes, login nodes and development nodes. The utilization of this power source to support these infrastructure systems is preferred.

o Any RPP or in-rack PDU should include networked remote monitoring with capabilities, including but not limited to, input/output voltage and input/output current for each circuit.

o All rack-level input and output connections shall be bottom fed.

9

e) R&D, Operational, and Outreach Partnerships

o Long term, sustainable partnerships have always been an important part of MSU’s strategy for advanced information technology and cyberinfrastructure. The outcome of this RFP will play a strong role in defining MSU’s partnerships in cyberinfrastructure in the years to come.

o The awardee must help MSU personnel with the tuning and benchmarking efforts of the proposed system

o The awardee is encouraged to promote MSU interests associated with the proposed system.

o The awardee is encouraged to identify any ongoing partnership opportunities available in the form of collaborative research and development programs, internship programs, hardware grants relevant to R&D, education and outreach, or economic development.

f) Scope of Work

i. General

The awardee is responsible for installation and deployment of the fully integrated system at MSU, with appropriate help from MSU personnel.

ii. Maintenance and Support

The purchase price of the proposed system and support/infrastructure components is to include 48 months of 8x5 hardware and software support with next business day parts and a comprehensive spare kit (e.g., parts locker) on site. The maintenance period will begin upon meeting all acceptance criteria specified in the following section.

iii. Acceptance Testing and Benchmarking

o Acceptance testing and final benchmarking must be performed after system installation onsite

o MSU personnel must have access to the system, both physical and remote, during the benchmark and acceptance phases of the installation.

o Fabric validation must be conducted o The Supplier must specify the Rmax of the proposed system (compute nodes only). The

proposed system must meet or exceed the specified Rmax utilizing the latest version of the HPL benchmark.

o Upon award, additional acceptance criteria may be negotiated with the Supplier to confirm that the proposed system meets the specified performance and system operability objectives.

10

4. INQUIRIES ABOUT RFP Prospective respondents may make written inquiries concerning this request for proposal to obtain clarification of requirements. Questions should be submitted either via a Word document or plainly typed in the email itself. Responses to these inquiries may be made by addendum to the Request for Proposal (RFP). Please send your inquiries to Jennifer Mayfield via electronic mail at jmayfield@procurement.msstate.edu.

All inquiries should be marked “URGENT INQUIRY. MSU RFP 21-65”

5. ADMINISTRATIVE INFORMATION a) Issuing Office

This RFP is issued by the following office:

Office of Procurement and Contracts Mississippi State University 245 Barr Avenue, 610 McArthur Hall Mississippi State, MS 39762

b) Schedule of Critical Dates

The following dates are for planning purposes only unless otherwise stated in this RFP progress towards their completion is at the sole discretion of the university.

RFP Posted July 12, 2021

Questions from Vendors Due July 23, 2021

MSU Q&A Response Due July 30, 2021

Proposal Submission Deadline – 2:00 p.m. August 10, 2021

Award Date (Estimated) September 1, 2021

Contract Effective Date (Estimated) September 1, 2021

11

6. PROPOSAL CONTENTS Indicate firm name and RFP# on the front of the outside sealed proposal envelope or package.

If you are submitting multiple responses, both can be sent in the same external package, but they need to be in individual envelopes inside that package and follow the instructions below:

At a minimum, the following items should be included in the contents of the Technical Proposal Section:

• Cover letter, indicating the scope of the proposal. The letter should include an overview of the services being offered. The letter should include a statement of exceptions to any of the terms and conditions outlined in this RFP. (Cover letter should be no more than 3 pages in length.)

• Corporate Structure and Credentials o Number of years of experience in HPC o Staffing levels and support proposed o Examples of similar previous work.

• Operations and Ability To Perform

o Provide operation plan. This should include, but not be limited to, acknowledgement and agreement with all requirements as well as explanations, where applicable, of the intended plan to achieve the requirements.

o Timeline of planned events, to include delivery, installation, and the beginning acceptance testing.

At a minimum, the following items should be included in the contents of the Cost Proposal Section:

Costs breakdowns for the major components of the proposal (e.g., computing equipment, network systems, storage systems, physical infrastructure, maintenance/support, installation, software, etc.)

7. DISCUSSIONS/EVALUATION CRITERIA/AWARD PROCESS MSU reserves the right to conduct discussions with any or all respondents, or to make an award of a contract without such discussions based only on evaluation of the written proposals. MSU reserves the right to contact and interview anyone connected with any past or present projects with which the respondent has been associated. MSU likewise reserves the right to designate a review committee to evaluate the proposals according to the criteria set forth under this section.

12

MSU may make a written determination showing the basis upon which the award was made and such determination shall be included in the procurement file.

MSU reserves the right to award this contract in whole or in part depending on what is in the best interest of MSU with MSU being the sole judge thereof.

The evaluation factors set forth in this section are described as follows:

• The Vendor’s ability to deliver an application meeting the overall objective and functions described in the RFP

• Competitive cost • Availability and access technical support • Delivery schedule • Vendor’s HPC experience in higher education institutions and federal government • Compliance with applicable State and Federal laws and regulations • The committee may invite finalists for interviews and/or presentations

Failure to attend a requested interview presentation before the committee may result in a proposal not being considered.

Upon award of contract(s), successful respondent(s) will be asked to provide a transition plan and timeline and obtain MSU’s input and concurrence before moving forward.

Proposals will be scored based on the following weights (100 points total):

• Corporate Structure/Years of Experience/References – 20 pts • Overall System Design/Technical Approach – 20 pts • System Performance (compute/networking/storage) – 30 pts • Expandability and/or Scalability of Overall System – 10 pts • Delivery Time/Time to Production – 10 pts • Added Value Provided to MSU/Partnership Value – 10 pts

8. PROPOSAL SUBMISSION Proposals shall be submitted in one package (envelopes or boxes) as set forth in Section 7. Please make sure that the RFP number is clearly visible on the outside of the package.

One (1) original and one (1) electronic copy (of the complete technical proposal in one pdf file on a flash drive) of parts 7(b)(i) (Cover Letter), 7(b)(ii) (Corporate Structure and Credentials), and 7(b)(iii) (Operations and Ability to Perform) should be sealed in a package with “Technical Proposal” in the lower left hand corner. Each submitted package should be a complete copy. The original shall be marked on the first page “Original”. If you are submitting multiple

13

responses, package each separately as stated above, but both can be sent in the same external package.

The proposal package must be received on or before 2:00 p.m. on August 10, 2021. It is the responsibility of the respondent to ensure that the proposal package arrives in the Procurement and Contracts office on-time. The proposal package should be delivered or sent by mail to:

Office of Procurement and Contracts Mississippi State University 610 McArthur Hall 245 Barr Avenue Mississippi State, MS 39762

Your response must include the signature page included in this RFP (See Appendix A) and contain the signature of an authorized representative of the respondent’s organization. The signature on the “Original” signature page should be in blue ink

MSU reserves the right to reject any and all proposals and to waive informalities and minor irregularities in proposals received and to accept any portion of a proposal or all items bid if deemed in the best interest of the University to do so.

Proposals received after the stated due date and time will be returned unopened. Submission via facsimile or other electronic means will not be accepted.

9. TWO-PHASE, BEST AND FINAL OFFER If the initial proposals do not provide MSU with a clear and convincing solution, or if MSU feels it is appropriate to offer the potential providers an opportunity to submit revised proposals, MSU reserves the right to use a two-phase approach and/or invite Best and Final Offers (BAFO). Based on the information obtained through the proposal submittals (Phase-One), MSU may choose a specific business model, and potential providers may be asked to submit revised proposals based upon that specific model.

The evaluation committee may develop, for distribution to the top-ranked firms, refined written terms with specific information on what is being requested as a result of information obtained through initial RFP submittal process. Proposers may be asked to reduce cost or provide additional clarification to specific sections of the RFP. Selected proposers are not required to submit a BAFO and may submit a written response notifying the solicitation evaluation committee that their response remains as originally submitted.

14

10. ACCEPTANCE TIME Proposal shall be valid for one-hundred and twenty (120) days following the proposal due date.

11. RFP CANCELLATION This RFP in no manner obligates MSU to the eventual purchase of any services described, implied or which may be proposed until confirmed by a written contract. Progress towards this end is solely at the discretion of MSU and may be terminated without penalty or obligations at any time prior to the signing of a contract. MSU reserves the right to cancel this RFP at any time, for any reason, and to reject any or all proposals or any parts thereof.

12. INDEPENDENT CONTRACTOR CLAUSE The contractor shall acknowledge that an independent contractor relationship is established and that the employees of the contractor are not, nor shall they be deemed employees of MSU and that employees of MSU are not, nor shall they be deemed employees of the contractor.

13. OTHER CONTRACT REQUIREMENTS Award Terms: This award shall be awarded at the discretion of the University based on the capabilities and overall reputation of the Supplier, as well as the cost. Acceptance shall be confirmed by the issuance of a contract from the University.

The Procurement Process: The following is a general description of the process by which a firm will be selected to fulfill this Request for Proposal.

• Request for Proposals (RFP) is issued to prospective suppliers. • A deadline for written questions is set. • Proposals will be received as set forth in Section 8. • Unsigned proposals will not be considered. • All proposals must be received by MSU no later than the date and time specified on the

cover sheet of this RFP. • At that date and time, the package containing the proposals from each responding firm

will be opened publicly and the name of each respondent will be announced. • Proposal evaluation: The University will review each proposal. • At their option, the evaluators may request oral presentations or discussions for the

purpose of clarification or to amplify the materials presented in the proposal • Respondents are cautioned that this is a request for proposals, not a request to contract,

and the MSU reserves the unqualified right to reject any and all proposals when such rejection is deemed to be in the best interest of the University.

• The proposals will be evaluated according to the criteria set forth in Section 7.

15

APPENDIX A: SIGNATURE PAGE

Provide information requested, affix signature and return this page with your proposal:

NAME OF FIRM: ________________________________________________

COMPLETE ADDRESS: ________________________________________________

________________________________________________

TELEPHONE NUMBER: ________________________________________________

AREA CODE/NUMBER

E-MAIL ADDRESS: ________________________________________________ AUTHORIZED SIGNATURE: ________________________________________________

PRINTED NAME: ________________________________________________

TITLE: ________________________________________________

NEW PDU

GLASS WALLHPC2

REVISION: 2021.06.25

Scout-1

Scout-head

PDU AIR HANDLER

Scout-2

Scout-3

Scout-4

Scout-5

DOOR

DOOR

GLASS WALL

EXTERIOR WALL

Scale: 24 inch tilesMechanical Facilities

installations are approximate