Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 1

MIT Sloan School of Management Final Report for S-Lab Project with Facebook, Inc.

Introduction & Objective

Facebook, Inc. (“Facebook” or “the company”) owns and operates data centers in Oregon, North Carolina, and Sweden with a new site currently under construction in Iowa. As the company scales, it is critically important for the company to grow its server infrastructure as well. Energy efficiency and sustainable design are at the heart of Facebook’s design philosophy. Its data centers are designed with the perspective that the servers, the building, and the operational systems are part of one, interconnected system. More information on Facebook’s sustainability efforts can be found at www.facebook.com/green. Facebook engaged students from the MIT Sloan School of Management (“MIT Sloan team”) to provide recommendations on how to improve the design and sustainability of its current and future portfolio of data centers. The MIT Sloan team evaluated existing and emerging initiatives for Facebook’s consideration. This includes (but is not limited to): site location, employee engagement, water and material usage, holistic design thinking, and industrial partnerships. The company expressed particularly interested in non-energy related initiatives to complement its existing efforts. The information contained within this report highlights additional background on Facebook and the industry. It then provides a brief overview of the project approach followed by data center compilation of projects and initiatives for Facebook’s consideration. These recommendations include the development of a site selection tool, the exploration of co-location opportunities, and an evaluation of cooling and ancillary systems. Details are provided on feasibility, financial metrics (where available), energy implications (if applicable), and other qualitative sustainability measurements.

Background

Company. Facebook has strong buy-in to continue increasing the efficiency and sustainability of its data centers over time. Facebook’s internal team of architects, designers, and electrical engineers is continuously working to improve its data centers’ designs and construction methods. Facebook also launched the Open Compute Project Foundation in 2011. Open Compute is an online community and knowledge share platform to improve the overall efficiency of the data center industry. It offers suggestions on the following categories designed to address serve and hardware development and construction: server, storage, data center design, networking, hardware management, certification, open rack, and solution providers. The reported result is “a data center full of vanity free servers which is 38% more efficient and 24% less expensive to build and run than other state-of-the-art data centers.”1

1 http://www.opencompute.org/

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 2

Industry. Despite a commonality in functions, organizations utilize different sustainability strategies for designing and operating their data centers. Google touts one of the most energy-efficient portfolios of data centers and servers in the world - averaging about 50 percent less energy use than comparable operations. There are two primary sets of initiatives that have allowed Google to achieve this. First, Google reduced non-server energy consumption to 12 percent of server usage, a Power Use Effectiveness (PUE) of 1.12.2 PUE is an industry metric that compares total data center energy use to energy used by the server/IT equipment. A data center’s ability to reduce energy consumption from heating, cooling and other functions thus improves the efficiency of the whole structure. Google has also implemented a “reuse and recycle” policy for its older servers - first attempting to reuse outdated equipment (over 300,000 successfully) and then recycling unusable components. In addition to Google’s work to improve its data centers efficiency, the company has also funded government research targeting the impacts of cloud computing. Google has been a strong supporter of the Cloud Energy and Emissions Research (CLEER) Model developed by the Lawrence Berkeley National Laboratory. This open-access model assesses the energy implications of varying market adoption rates for cloud-based services.3

Amazon, on the other hand, is much less transparent with its effort, and lack of transparency suggests that efficiency may not be a high priority at this time. Nevertheless, Amazon argues that by providing service to hundreds of clients in every data center, it is effectively eliminating hundreds of individual data centers by combining them into a single location. Such consolidation can reduce real estate, labor, and backup generator requirements. Parallel to Google’s fund efforts, Amazon has contributed to the Xen Project. The Xen Project is a Linux Foundation Group project dedicated to software designed for lower operating costs. Xen hypervisor is a software capable of running on low-power server chips that can run many applications in a single server, resulting in servers that consume less power, thus reducing the power needed by data centers.4

Challenges. As the world’s population grows - coupled with more people emerging from poverty and having more devices connected to the internet, data usage will continue to grow and computing requirements will become more important. Data centers are thus a growing source of energy usage and environmental impact, as well as an area for continuous improvement. A 2006 EPA report estimated that data centers and servers consumed 1.5 percent of U.S. electricity (61 billion kWh, costing $4.5 billion).5 A more recent report from Stanford University found that global and U.S. data center energy usage increased by 56 percent and 36 percent between 2005 and 2010, respectively.6

2 https://www.google.com/about/datacenters/efficiency/internal/ 3 http://cleermodel.lbl.gov/ 4 http://www.xenproject.org/ 5 U.S. EPA. “Report to Congress on Server and Data Center Energy Efficiency.” 6 Analytics Press. “Growth in data center electricity use 2005 to 2010.”

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 3

Due to advancements in processing speed, heat densities are also getting higher and causing temperatures to rise, which requires more energy to cool. Cooling typically involves evaporative cooling, which requires extensive amounts of water. Many areas of the world face drought conditions and water scarcity, making data center water use an area of growing concern. Data center expansion has other environmental and social implications as well. The need for back-up generators (typically supplied by diesel fuel and tested regularly) leads to additional energy consumption via fuel burn, as well as noise pollution. Furthermore, data centers present zoning and siting complexities for local and regional governments - such as brownfield vs. greenfield site selection and weighing land prices with security considerations.

Project Overview

Approach. In April 2014, the MIT Sloan team evaluated and analyzed potential technologies and initiatives for data center improvements. The MIT Sloan team will evaluate ideas against current Facebook and industry practices, their impact on resource consumption and regeneration, their ease of implementation based on technical and organizational complexity, and their financial impact. Potential ideas were prioritized if the fell into one of Facebook’s priority areas: site location, employee engagement, water and material usage, holistic design thinking, and industrial partnerships. Selected Initiatives. From the ideas generated, the MIT Sloan team selected the initiatives captured in Table 1 below. These initiatives represent high-level categorizations of specific technologies or actions that can be implement at new and existing data centers.

Table 1. Recommended data center sustainability initiatives

Initiative Description Sustainability Benefit

Site selection Evaluation of sites (regionally, then locally) based on financial, environmental and operational criteria using multi-decision criteria analysis

Clean energy, water usage

Co-Location Evaluation of locating with or in close proximity to organizations, input providers and governments based on feasibility, financial benefit, payback and employee engagement.

water usage, waste heat usage, greening of work spaces, employee engagement

Cooling & ancillary systems

Evaluation of liquid cooling systems based on heat transfer properties and impact on the building design. Research into developing technologies with potential energy and environmental benefits in the near future.

waste heat usage, water usage, pollution reduction, social benefit

Assumptions. Facility location and site selection is a complex decision involving specific local and regional information (e.g. potential tax incentives offered by governments) as well as proprietary,

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 4

internal/operational considerations (e.g. cost of capital, technical issues, etc). Site selection recommendations are therefore provided irrespective of geographic location. Additionally, the MIT Sloan team prioritized cutting-edge technologies and initiatives over low-hanging fruit based on preliminary conversations with Facebook employees. Consequently, detailed financial assessment was limited and difficult to perform. Where available, analysis is listed. Resources. This report was prepared by Ricky Ashenfelter (MBA 2015), Kari Hodges (MBA 2014), and Alberto “Beto” Luna (LGO 2015) under the mentorship of Professor John Sterman. Two additional MIT Sloan students supported with third-party research and knowledge sharing, including Al Rabassa (Data Center Manager for the U.S. Government) and Bruno Pimentel (MIT Sloan Visiting Faculty Fellow from Vale Institute of Technology). Bill Weihl (Manager of Energy Efficiency & Infrastructure Sustainability) served as Facebook’s main point of contact to the MIT Sloan team. He was supported by Karen Cooper (Sustainability Program Manager) and Lyrica McTiernan (Infrastructure Manager).

Site Selection

Site selection is an important consideration for data center development. While Facebook has sales offices around the world, data centers are often located in remote areas; consequently, choosing a location where Facebook already has a presence is not a priority. The location in which a data center is built carries with it many environmental implications, including water availability, temperature, cleanliness of energy generation, and susceptibility to natural disasters. Without greater visibility into Facebook’s future expansion plans, however, specific locations were not selected or recommended. Before any decisions are made, it is important that Facebook continue considering carbon accounting and additionality standards when selecting site locations. If construction of a data center in an area with low-carbon energy (e.g. hydro) leads to the displacement of someone else’s clean energy, then the cumulative environmental benefits are more difficult to discern. Location decisions that cause new, incremental no/low carbon generating capacity that would not previously have been constructed (or used, in the event of under-utilization of existing resources) are ideal. That said, data center expansion to regions at capacity can also offer financial and societal benefits if it directly or indirectly attracts future companies, governments, and/or states to “raise the bar” in these regions - such as through new renewables capacity generation, investment in more sustainable infrastructure, or broader economic improvement. Recommendation. We recommend Facebook build a multi-criteria decision analysis as a tool for future evaluation of site selection. One example is the Analytic Hierarchy Process (AHP), which was developed by Thomas Saaty in 1977 as a process for Multi-Criteria Decision Analysis.7

7

While AHP is used here as an example, other frameworks and methodologies exist and should be

T. Saaty. "Decision making with the analytic hierarchy process."

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 5

considered as well. These include Evidential Reasoning (ER) and Intelligent Design System (IDS) modeling.8

The first step is to determine a set of quantitative or qualitative criteria to be used in the decision problem and establish objective comparisons between them. In Facebook’s case, the following “environmental” criteria should be incorporated into site selection:

1. Electricity cost: price to purchase each unit of energy (quantitative, measured in USD per MWh)

2. Cleanliness of energy: takes into consideration energy generation fuel mix; more favorable for cleaner sources of energy such as hydro, wind, solar, geothermal, and other renewable energy sources; displacement vs. additionality considerations (quantitative, measured in tons CO2 per MWh)

3. Natural disaster vulnerability: an area’s susceptibility to flooding, seismic activity, tornados, hurricanes, snow, and wildfires (qualitative or quantitative, using categorized risk maps)9

4. Water availability: short term susceptibility to drought; longer term vulnerability to scarcity, water rights and policy (qualitative or quantitative, using risk maps); see Appendix 1 for examples from the USDA’s Drought Monitor and WRI’s Aqueduct tool

5. Climate: finding the right balance between temperature of ambient air and moisture content; (quantitative, measured in degrees and air moisture content); ideal conditions is approximately 70 degrees and 40 percent humidity; see Appendix 2 for examples of free cooling mapping in Europe and the United States.

Facebook should also consider fiber network connectivity; security; the availability of tax, land and economic incentives; labor and construction costs; and education levels. Facebook will need to weigh environmental, financial, and operational considerations in choosing how to perform these pairwise comparisons and minimize total cost of ownership (TCO). Proper due diligence, location selection, design, construction, and operations/maintenance can ultimately both improve sustainability and lower TCO. Another opportunity for site selection is at brownfield sites for industrial parks or with canal systems. Canals provide cheap real estate, areas of environmental reclamation, and can leverage flowing water for both cooling and turbine-electricity generation purposes. A comparable system has been implemented at the Massachusetts Green High Performance Computing Center (MGHPCC) located in Holyoke, MA. MGHPCC is a data center supporting five Boston Area research universities (including MIT) and where hydropower and solar installations make up more than 70 percent of the facilities energy use.10

8 https://php.portals.mbs.ac.uk/Portals/49/docs/jyang/XuYang_MSM_WorkingPaperFinal.pdf

Additional locations around the country and world may also yield ideal conditions.

9 FORTRUST. “Geographic and risk mitigation factors for data center site selection.” 10 http://www.mghpcc.org/about/what-are-the-green-design-aspects-of-the-mghpcc/

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 6

Implementation. Facebook can craft multi-criteria decision analysis around those factors that are most relevant. See Attachment 1 for an example buildout of a decision model - capturing eight decision criteria and evaluated based on both quantitative and qualitative criteria. The resulting analysis shows that fiber connectivity and energy fuel mix are the most important factors for deciding site selection, though this model is meant to be customized based on Facebook’s requirements, strategy, and future plans. For example, it excludes other factors of production and a community’s willingness to tolerate construction or backup (diesel) generators. It also dependent on Facebook’s ability to extract favorable tax breaks or electricity price discounts. As built, the existing equations suggests Denmark is the most optimal choice, followed by Brazil. Screenshots from the model are illustrated in Appendix 3. Of course, this analysis omits many of the factors listed above; it will largely be up to the Facebook team to decide which factors are most important and aligned to broader business strategy. Given this model is meant for a handful of locations only, it is recommended that Facebook first use a tool like this after it has first identified a region that aligns with its core business growth or strategy (e.g. continent such as Asia, Africa, Europe). The tool could then be used to identify the country (e.g. South Korea, China, or Singapore within Asia) or the region (e.g. Guangzhou, Shanghai, Beijing, or Shantou within China) that meets these expansion criteria. The larger the number of choices, the larger the number of pairwise comparisons with the qualitative criteria are needed. If the number of choices becomes too large, other approaches, such as mathematical programming may be used; this approach carries with it added complexity.

Co-Location

After business and site location considerations, co-location can greatly impact data center sustainability. By sharing location, infrastructure, and waste products, co-location maximizes the use of inputs and byproducts. For example, co-locators utilize data center waste heat as a source of hot water for local communities. Additionally, co-location creates opportunities for community and employee engagement and positive brand impact, such as named or sponsored community gardens. Decision Making Framework. During weekly phone calls, Facebook set co-location priorities. Specifically, Facebook indicated the following: co-location should be cost efficient with clear payback. Co-location should align with Facebook’s culture and uniqueness. Facebook also indicated that security restrictions might limit co-locator access to the data center campus and buildings. The MIT team worked within these parameters and with the assumption that Facebook can resolve its security needs. Because future data center locations are confidential, the MIT team did not recommend specific co-locators or countries/states/cities. Instead, we recommended a framework for assessing co-location opportunities and provided successful examples that met with the included criteria. While CHP was initially considered a co-location priority, the low temperature waste heat Facebook data

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 7

centers produce is more aligned with yet to be commercialized CHP technologies. Appendix five provides EPA resources on data center CHP as well as information on the Department of Energy’s CHP Technical Assistance Partnership, which works with private industry on CHP technology innovation.11

In Facebook’s case, the following criteria should be incorporated into co-location decision-making:

1. Water treatment plant proximity: municipal water treatment plants often provide greywater at no cost. Evaporation is a desirable alternative to releasing greywater into local rivers and stream. While many of these systems are gravity dependent, a pump can also be used to transport treated water. With Facebook’s current energy efficiency, we believe water efficiency is a reasonable next step with evident financial benefits (see appendix 4).

2. Non-potable water source access: provides a desirable source of water without taxing the local drinking supply. This minimizes water costs, increases efficiency, and provides future potential for energy/ cooling water technologies.

3. Proximity to co-locators with waste heat as an input resource: residential neighborhoods, offices and eco-industrial parks consume wasted heat, as do gardens. Close proximity helps maximize heat recovery in low temperature waste heat transfers and minimize waste heat transfer losses.

Co-locating with Facebook created entities. Given the complexity of co-locating with external organizations, we recommend Facebook’s co-location strategy start with internally derived initiatives that meet the following criteria:

4. Alignment with Facebook values: assess the Impact, Fastness, Boldness, Openness and Social Value of each co-location opportunity.

5. Community and employee engagement: for initiatives without measurable financial payback, employee and community engagement levels provide another means of assessment through surveys, social media mentions, and participation rates.

6. Potential for positive brand/ PR impacts: initiatives should have positive impact on the Facebook brand and the public’s understanding of Facebook’s commitment to sustainability.

The following sustainability recommendations are sourced from working prototypes at data centers worldwide and meet both the co-location and engagement model criteria outlined above. Onsite Co-location Incubators. Incubators provide a unique opportunity to utilize data center space and further Facebook’s community engagement priorities. For example, IBM’s innovation center in Cambridge, MA provides coworking space as well as software and systems support for

11 Charlie Goff, Consultant, and Stephanie DeGabriele. Eastern Research Group, Inc. EPA CHP Partnership. EPA CHP Partnership Helpline: 703-373-8108 or chp@epa.gov, Charlie.Goff@erg.com.

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 8

start-ups12 while Google’s backed incubator supports a social mission of creating more female run tech startups.13

With an incubator, Facebook can maximize data center use while maintaining maximum control over the physical site, the type of companies to support, and the financial investment. An onsite incubator also allows Facebook to vary its investment depending on access to funding and long-term interest or facility needs. This model has high potential for any future urban data center locations.

Greenhouses. In Iowa, recovered ethanol plant waste heat is pumped to a co-located greenhouse to support local vegetation production. Given the low temperature of the ethanol plant’s waste heat, a heat pump is used to increase the temperature from 90°F to the 120 to 140°F range. The acre greenhouse’s waste heat needs vary by month. The majority of the heat is used during Iowa winters to ensure growing temperatures are kept consistent and that food can be produced year round. A one-acre greenhouse uses approximately 7210.35 (MBtu/h) of heat annually.14

The cost of building a greenhouse varies largely by size, material, and growing method (soil vs. water). For example, the Hometown Harvest Greenhouse http://www.hometownharvestseiowa.org/ in Iowa took seven months to build and cost approximately $70,000, with much of the resources and labor donated.15

The greenhouse does not track its waste heat usage. Instead, its curator Jan Swinton attempts to maintain a 70-degree differential between the greenhouse and the surrounding environment for Iowa’s six months of cold weather while incorporating 244 Btu/ hour of lost heat into her configurations (see Appendix 7).

On weekends without waste heat production, the greenhouse relies on a propane heater to warm the facility. During the coldest season, propane costs reach about $100/ week. Cultivating the produce requires labor (an additional cost).16

Ms. Swinton, uses ten volunteers working three-hour/week shifts as well as three master growers, to harvest the 90 lbs. of produce weekly. This produce is sold to local schools for approximately two dollars per pound. Additional greenhouse costs include water and drainage infrastructure as well as 20 amps of electricity used for water pumps.

Node Greenhouse. Another example of greenhouse co-location is Notre Dame’s node data center. The Notre Dame Center for Research Computing placed “high-performance computing nodes” at the South Bend Greenhouse and Botanical Garden, in order to heat the facility. A grid scheduling software controls the temperature range of the greenhouse. “Compute jobs” are sent to

12http://www.thewire.com/technology/2014/03/google-backed-incubator-looks-increase-women-tech-25-percent/359061/ 13]https://www304.ibm.com/partnerworld/wps/servlet/ContentHandler/isv_inv_tsp_iic_cambridge_overview 14 Leopold Center. Final Report: Potential to Operate Greenhouses and Aquaculture in Conjunction with Iowa’s Ethanol Plants.

February 29, 2008. http://www.agmrc.org/media/cms/GreenhouseAquaculture_2008_11806A2892322.pdf 15 http://www.hometownharvestseiowa.org/the-greenhouse 15Jan Swinton has agreed to be a Greenhouse resource. She can be reached at: jan@pathfindersrcd.org 16 Leopold Center. Final Report: Potential to Operate Greenhouses and Aquaculture in Conjunction with Iowa’s Ethanol Plants. February 29, 2008.

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 9

the greenhouse node to increase the temperature. Early results indicate that the less than pristine environment, including higher moisture levels, do not negatively affect the node’s functionality.17

This model is also attractive because not only does it decrease the gas heating costs for the greenhouse but it may also provide an opportunity for the greenhouse to pay, in part, for the node’s electricity usage. Specific information on waste heat usage is currently unavailable from Notre Dame’s greenhouse.

While the payback for a greenhouse is hard to assess, strategically co-locating with a greenhouse has other benefits. For example, greenhouse produce could offset Facebook’s food expenditures. Facebook’s cafeterias provide an extensive selection of food for employees. Co-locating with a greenhouse could help Facebook decrease food and transportation costs and ensure year-round access to high quality low-cost organic food at its offices while decreasing costs and transport emissions. This co-location also provides positive PR, and community engagement opportunities. In the case of the Southeast Iowa Greenhouse project, the resulting press for Schaus Vorhies, which supplies the greenhouse with waste heat, has been extremely positive related to fostering healthy eating in children, supporting local communities and helping to increase organic food production. Arboretum. The Condoret data center in Telecity, France built arboretum space into its data center design to create an in-house use for waste heat. This on-site garden enhances the work environment and provides a makeshift research center for local university climate scientists. The Société Forestière and the French National Institute for Agricultural Research use the arboretum to determine plant adaptability to changing climate conditions. As universities and laboratories often have security and disclosure needs of their own, this partnership may be extremely attractive if security remains a concern. Although pricing information is not available for this option, as the arboretum is built into the data center structure, this cost is absorbed into the building process. 18

Swimming Pool. In Switzerland, waste heat will heat a community swimming pool. In this process, waste heat flows through heat exchangers to warm the water that is pumped into the pool located on the data center’s campus.19 For Facebook to maximize the use of its low temperature waste heat, the community pool and the data center can share a wall. Although exact waste heat usage information is not available for the swimming pool model, it takes 8.34 Btu to increase one gallon of water one degree Fahrenheit.20

17

The pool also works for Facebook employees. An employee pool encourages employee health and fitness, which has been linked to fewer employee sick days and better stress management. If incorporated into a larger gym or fitness center, a pool heated by waste heat can help decrease Facebook’s overall healthcare costs and expenditure on gym

http://www.datacenterknowledge.com/archives/2008/05/16/data-center-heats-a-greenhouse/ 18http://www.telecitygroup.com/our-company/news/2010/telecitygroup-opens-new-state-of-the-art-data-centre-in-paris.htm 19 http://www.datacenterknowledge.com/archives/2008/04/02/data-center-used-to-heat-swimming-pool/ 20 http://smartenergy.illinois.edu/pdf/newsletter6_6.pdf

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 10

subsidies at future sites. Additionally a pool has high community and employee engagement potential, which speak to two of Facebook’s core values: building social value and focusing on impact.21

Integration into Urban Settings. Several data centers have integrated into urban settings and used waste heat to provide heat and hot water to local communities. Since many of these initiatives are still in the experimental phase, hard data on financial benefits and waste heat usage for this type of co-location is not available. Additionally data centers we’ve contacted are hesitant to disclose their waste heat distribution infrastructure. In general, 0.24 Btu heats one cubic foot of air one degree Fahrenheit. 22

There are several successful examples of data center waste heat heating urban areas in close proximity. For example, waste heat from Telehouse West data center will produce nine megawatts of power and heat the local community immediately surrounding the urban data center. In Finland, waste heat from one underground data center warms pipes that heat 500 nearby homes. In Canada, pneumatic baffles regulate waste heat supplied to an office directly above the data center via a connected duct system. In all these instances, waste heat was used successfully in the district heating of offices, homes, and water supplies in urban areas.23

These initiatives have high impact on lowering community heating expenditures and energy use, and have resulted in positive media coverage and community engagement.

Cooling and Ancillary System

After site selection, the design of the data center is the next important consideration for its function and performance. The building design must integrate all systems needed for the operation of the data center, such as power and cooling, plus it needs to allow incorporation of co-location possibilities like the ones presented in this report. Currently, Facebook’s data centers are stand-alone structures which primarily house the servers and the employees who work in the building. The cooling systems have been designed into the building to provide temperature and humidity controlled air to the servers and extract hot air via a network of fans, ducts, and aisles. Although some of the waste heat is used to control the climate for the building, the design can be taken

21 http://www.datacenterknowledge.com/archives/2008/04/02/data-center-used-to-heat-swimming-pool/

22 Samuel C. Sugarman, Sam Monger.2000. Testing and Balancing HVAC Air and Water Systems. Fairmont Press. 23http://www.datacenterknowledge.com/archives/2013/02/14/telehouse-to-expand-in-london/ http://searchdatacenter.techtarget.com/news/1314324/Companies-reuse-data-center-waste-heat-to-improve-energy-

efficiency

http://www.treehugger.com/clean-technology/data-center-housed-under-cathedral-heats-homes-in-finland.html

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 11

further, getting cooling systems closer to the heat-generating components and integrating waste heat use beyond just climate control. Liquid-based cooling. Air is the preferred method of removing heat away from servers. However, liquid-based cooling has a higher capacity to remove and transfer heat than air-based cooling. Some benefits from liquid-based cooling include the ability to run servers at more efficient operating temperatures due to the increased heat removal, as well as the reduction in infrastructure needed for air cooling. Removing heat-using liquids also facilitates alternative uses for waste heat. Liquids store heat better than air, allowing for heat transport over longer distances and more efficient heat exchange. The ability to transfer heat over longer distances increases the possibilities for the design of co-located facilities. There are two methods of liquid-based cooling that could benefit Facebook: cooling of heat-generating components with liquid-filled heat sinks, and submerging the servers in liquid. The former brings the fluid directly to the CPU, GPU, and other components on the server, extracting heat through a network of tubes and heat exchangers at the rack24

. The latter submerges the entire rack in a non conductive, dielectric liquid which is circulated to remove heat from the servers. Both methods utilize liquid in a closed loop system to control the quality of the liquid, and then use heat exchangers to remove the heat away from the racks using facilities’ water (or possibly wastewater).

Both of these technologies are developed and available. Component liquid cooling systems may be retrofitted into existing server and rack designs, while submerged cooling would require a higher capital investment as well as a significant amount of effort to switch from standard to submerged server racks. Liquid cooling will also require additional considerations. Personnel must be trained on the new systems and on the new safety procedures that must be developed to avoid injuries, spills, and damage to the electronics. The benefits from liquid cooling systems could impact Facebook in multiple ways. For a data center site like Prineville, Oregon, where large amounts of air are circulated per minute, these systems reduce the amount of circulated air, the level of humidity and cooling control, and the need for cold and hot aisles. These reductions translate into smaller facilities with smaller air ducts, lower ceilings, and fewer ancillary equipment like fans and humidifying water lines. Liquid cooling systems also have the potential of improving the Power Usage Effectiveness (PUE) closer to 1.0525

. In addition, waste heat removal can be coupled with the site’s fresh water system to provide year-round hot water. As an alternative, the use of wastewater for heat removal could reduce the use of freshwater, freeing up potable water for the surrounding community. Uses for the hot water can range from climate control to floor and sidewalk temperature control, as well as transferred to nearby facilities, like pools and greenhouses.

24 http://asetek.com/data-center/technology/how-liquid-cooling-works.aspx 25 http://www.grcooling.com/liquid-cooling-retrofit-savings/

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 12

Ancillary Systems. There are ancillary technologies in different phases of development which could prove useful for Facebook at reducing emissions and pollution. The three explored technologies were energy storage, wastewater treatment, and air pollution reduction. Although these systems may not be considered primary systems, they can reduce dependence on backup generators, reduce freshwater usage, and reduce the carbon footprint for the company. Energy Storage. Existing energy storage systems are expensive, and their use may have environmental implications. Two developing technologies stand out due to their small environmental impact: liquid metal batteries and compressed air energy storage (CAES). Liquid metal batteries made from abundant metals and a molten salt electrolyte26 will provide a low cost, modular, silent, and emissions-free alternative for energy storage. Compressed air energy storage uses mechanical energy to store compressed air for use at peak hours. One technology in particular uses the heat generated from compression to improve energy extraction during peak hours27

. Although unconfirmed, removed heat from the servers could potentially contribute to the performance of the CAES system. These two systems are in different stages of development. Liquid metal batteries have developed through to prototype manufacturing facilities, while the CAES system has gone through several successful funding cycles. The environmental implications for Facebook would arise from a reduced dependency on diesel backup generators, thus lowering GHG emissions and noise pollution. These two technologies could benefit Facebook in the near future.

Wastewater Treatment. Facebook’s selection of evaporative cooling for its data centers has minimized its energy needs for the cooling system but requires the use of freshwater. The company’s data center design allows it to recirculate the water and maintain an annualized Water Usage Effectiveness (WUE) of 0.42 liters per kWh of energy. Facebook could reduce its freshwater use by connecting to wastewater supplies and incorporating water treatment equipment. New commercially available purification systems have become small enough to supply 60,000 liters of potable water per day without the use of chemicals28. Developing purification technology uses bacteria to purify wastewater while producing methane29

, and, subsequently, electricity. Used in combination with liquid-based cooling, these technologies would allow Facebook to replace freshwater consumption with wastewater usage, reducing costs and benefiting the surrounding communities.

Air Pollution Reduction. Although Facebook data centers only generate on-site GHG emissions when the backup generators are in operation, energy consumption and other activities contribute to the company’s carbon footprint. Facebook could compensate for its carbon emissions by using pollution reduction technology. The use of a titanium dioxide coating on architectural structures has been used in places like Mexico City to reduce nitrous oxides (NOx) to water, carbon dioxide, and

26 http://www.ambri.com/technology/ 27 http://www.lightsail.com/ 28 http://www.swiss-cleanwater-group.com/en/our-water-cleaning-products.html 29 http://cambrianinnovation.com/solutions/ecovolt/

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 13

calcium nitrate30

(a fertilizer). Titanium dioxide is a photocatalytic activated by sunlight. This characteristic has made it ideal for direct application to building structures. The modules currently available are priced based on size, but a preliminary estimate is approximately $42 per square foot. The coating has an expected life of 10 to 15 years, at which time recoating would cost around $5 per square foot. Its use at Facebook facilities in pollution intensive locations would provide the company the opportunity to compensate for its carbon impact and improve the air quality of the surrounding communities.

Conclusion and Next Steps

Energy efficiency and sustainable design are at the heart of Facebook’s design philosophy. With this in mind our MIT Sloan team evaluated existing and emerging initiatives with respect to site location, employee engagement, water and material usage, holistic design thinking, and industrial partnerships. These priority areas were distilled into three initiatives: site selection, co-location, and cooling and ancillary systems. These initiatives provide Facebook with sustainable benefits such as clean energy, water usage, and social impact. The initiatives allow Facebook to explore new opportunities for improvement. As the company scales up operations, site selection will be an increasingly important consideration. The proposed multi-criteria decision analysis tool may help identify critical elements for data center location, such as fiber connectivity and energy fuel mix. Furthermore, co-location considerations may influence site selection. Co-location has significant implications for resource consumption. Opportunities like greenhouses and arboretums can boost both employee and community involvement. In addition, proximity to wastewater sources can facilitate improvements in other aspects of the data center, such as liquid based cooling. Incorporating technologies like liquid based cooling, energy storage systems, and pollution reducing coatings may further enhance Facebook’s power and water usage effectiveness and reduce the company’s environmental impact. This report was prepared with generalized recommendations for the different initiatives. The next step for Facebook is to consider evaluating each initiative further for potential benefit and applicability. The multi-criteria decision analysis may prove to be a useful tool for site location once the relevant criteria for selection are identified. The co-location options and criteria may be incorporated into the site selection tool and the design of new data centers to improve employee health, employee engagement, and positive community impact. The liquid cooling technologies can be implemented as commercially available or integrated into the open compute design. We believe that these recommendations will benefit Facebook in multiple areas, and we look forward to seeing them implemented in the next generation of their data centers.

30 https://www.youtube.com/watch?v=g2tCnub5Zlk#t=183

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 14

Appendix 1. Water scarcity mapping31

31 USDA Drought Monitor. World Resources Institute Aqueduct.

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 15

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 16

Appendix 2. Sample free cooling maps32

32 The Green Grid

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 17

Appendix 3. Sample multi-criteria decision analysis

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 18

Appendix 4. U.S. wastewater treatment plants33

33 EPA. “U.S. wastewater treatment plants”.

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 19

Appendix 5. U.S. EPA and DOE CHP Resources.

ICF and Department of Energy Report. “Opportunities for Combined Heat and Power in Data Centers.” March 2009: https://www1.eere.energy.gov/manufacturing/datacenters/pdfs/chp_data_centers.pdf U. S. Environmental Protection Agency Combined Heat and Power Partnership Report. “The Role of Distributed Generation and Combined Heat and Power (CHP) Systems in Data Centers.” August 2007: http://www.epa.gov/chp/documents/datactr_whitepaper.pdf Data Center Energy Resources: http://energy.gov/eere/femp/data-center-energy-efficiency Waste Heat Recovery: Technology and Opportunities in U.S. Industry Report: http://www1.eere.energy.gov/manufacturing/intensiveprocesses/pdfs/waste_heat_recovery.pdf Center of Expertise for Energy Efficiency in Data Centers: (http://datacenters.lbl.gov/) DOE CHP Technical Assistance Partnerships (http://www1.eere.energy.gov/manufacturing/distributedenergy/chptaps.html#).

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 20

Appendix 6. CHP Incentives and Grants

www.epa.gov/chp/policies/database.html : Provides an interactive database of CHP related incentives and grants. The database can be narrowed to include states with high data center build potential.

Facebook, Inc. May 2014

MIT Sloan School of Management | 15.915 S-Lab | Draft Final Report | 21

Appendix 7. Iowa Hometown Harvest Greenhouse

http://www.hometownharvestseiowa.org/the-greenhouse via Jan Swinton.