1、 High Density Data CentersCase Studies and Best PracticesThis publication was prepared in cooperation with TC 9.9, Mission Critical Facilities,Technology Spaces, and Electronic Equipment.Any updates/errata to this publication will be posted on the ASHRAE Web site at www.ashrae.org/publicationupdates
2、.American Society of Heating, Refrigeratingand Air-Conditioning Engineers, Inc.High Density Data CentersCase Studies and Best PracticesISBN 978-1-933742-32-82008 American Society of Heating, Refrigeratingand Air-Conditioning Engineers, Inc.1791 Tullie Circle, NEAtlanta, GA 30329www.ashrae.orgAll rig
3、hts reserved.Printed in the United States of AmericaPrinted on 30% post-consumer waste using soy-based inks.Cover courtesy of Joe Lombardo of DLB Associates.ASHRAE has compiled this publication with care, but ASHRAE has not investigated, and ASHRAE expresslydisclaims any duty to investigate, any pro
4、duct, service, process, procedure, design, or the like that may bedescribed herein. The appearance of any technical data or editorial material in this publication does not constituteendorsement, warranty, or guaranty by ASHRAE of any product, service, process, procedure, design, or the like.ASHRAE d
5、oes not warrant that the information in the publication is free of errors, and ASHRAE does not neces-sarily agree with any statement or opinion in this publication. The entire risk of the use of any information inthis publication is assumed by the user.No part of this book may be reproduced without
6、permission in writing from ASHRAE, except by a reviewer whomay quote brief passages or reproduce illustrations in a review with appropriate credit; nor may any part of thisbook be reproduced, stored in a retrieval system, or transmitted in any way or by any meanselectronic, photo-copying, recording,
7、 or otherwithout permission in writing from ASHRAE._Library of Congress Cataloging-in-Publication DataHigh density data centers : case studies and best practices.p. cm. (ASHRAE Datacom Series)Summary: “Provides the reader a series of data center case studies and best practices that demonstrate how h
8、igh density loads can be cooled using a number of different approaches and includes a breadth of data center ventilation schemes and shows how they are deployed to cool high density IT equipment“Provided by publisher.Includes bibliographical references and index.ISBN 978-1-933742-32-8 (softcover)1.
9、Office buildings-Design and construction-Case studies. 2. Office buildings-Air conditioning-Case studies. 3. Electronic data processing departments-Equipment and supplies-Protection-Case studies. 4. Data processing service centers-Equipment and supplies-Protection-Case studies. 5. Electronic digital
10、 computers-Cooling-Case studies. 6. Data libraries-Protection-Case studies. I. American Society of Heating, Refrigerating and Air-Conditioning Engineers. TH4311.H54 2008725.23-dc222008006301ASHRAE STAFFSPECIAL PUBLICATIONSChristina HelmsEditorCindy Sheffield MichaelsAssociate EditorJames Madison Wal
11、kerAssistant EditorMichshell PhillipsAdministrative AssistantPUBLISHING SERVICESDavid SoltisGroup ManagerTracy BeckerGraphic Applications SpecialistJayne JacksonPublication Traffic AdministratorPUBLISHERW. Stephen ComstockContentsAcknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . .
12、 . . . . . . . . . . . . . .viiChapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Chapter 2 Raised-Access Floor Case Studies . . . . . . . . . . . . . . . . . 152.1 Raised-Access Floor with Perimeter Modular CRACs . . . . . . . . . 15 2.1.1 Case Study 1
13、National Center forEnvironmental Prediction (NCEP) . . . . . . . . . . . . . . . . . . 152.1.2 Case Study 2IBM Test Facilityin Poughkeepsie (2004) . . . . . . . . . . . . . . . . . . . . . . . . . . 302.1.3 Case Study 3San DiegoSupercomputer Center . . . . . . . . . . . . . . . . . . . . . . . . . .
14、 . 472.1.4 Case Study 4IBM Test Facilityin Poughkeepsie (2005) . . . . . . . . . . . . . . . . . . . . . . . . . . 632.2 Raised-Access Floor with AHUs on Subfloor. . . . . . . . . . . . . . . . 722.2.1 Case Study 5Lawrence LivermoreNational Lab Data Center . . . . . . . . . . . . . . . . . . . . . .
15、 . . . 722.3 Raised-Access Floor Supply/Ceiling Return . . . . . . . . . . . . . . . . 802.3.1 Case Study 6NYC Financial ServicesData Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 802.4 Raised-Access Floor with Heat ExchangersAdjacent to Server Racks . . . . . . . . .
16、. . . . . . . . . . . . . . . . . . . . . 922.4.1 Case Study 7Georgia Institute of TechnologyData Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 922.4.2 Case Study 8Cedars-Sinai Medical CenterData Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17、. 100vi Contents2.4.3 Case Study 9Hewlett-Packard Richardson DataCoolData Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1092.5 Raised-Access Floor withUnderfloor Supply/Ducted Ceiling Return . . . . . . . . . . . . . . . . . .1262.5.1 Case Study 10Oracle Data Center . .
18、 . . . . . . . . . . . . . .126Chapter 3 Non-Raised-Access Floor Case Studies . . . . . . . . . . . . .1413.1 Non-Raised-Access Floor with Ceiling Supply . . . . . . . . . . . . . .1413.1.1 Case Study 11Lawrence Berkeley National Lab. . . . . .141Chapter 4 Best Practices . . . . . . . . . . . . . .
19、. . . . . . . . . . . . . . . . . . . .1614.1 Data CenterNew Builds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1614.1.1 Ventilation Designs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1624.1.2 Raised-Access Floor Plenum Height . . . . . . . . . . . . . . . .1644.1.3 Roo
20、m Ceiling Height . . . . . . . . . . . . . . . . . . . . . . . . . . . .1664.1.4 Underfloor Blockages . . . . . . . . . . . . . . . . . . . . . . . . . . . .1674.1.5 CRAC Placement and Configuration . . . . . . . . . . . . . . . .1684.2 Accommodating Future Data Center Growth . . . . . . . . . . . .
21、 . . .1694.3 Raised-Access Floor Data Center. . . . . . . . . . . . . . . . . . . . . . . .1704.3.1 Perforated Tile Layout and Configuration . . . . . . . . . . . . .1704.3.2 Rack and Rack Layout-Related Effects . . . . . . . . . . . . . .1724.4 Localized Cooling . . . . . . . . . . . . . . . . . .
22、. . . . . . . . . . . . . . . . . .1744.5 Non-Raised-Access Floor Data Center. . . . . . . . . . . . . . . . . . . .1754.6 Data Center Energy Management and Efficiency. . . . . . . . . . . .175References and Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179Abbreviations a
23、nd Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189viiAcknowledgmentsThe information in this book was produced with the help and support of thecorporations, academic
24、 institutions, and organizations listed below:American Power ConversionBellsouthCedar Sinai Medical CenterCitigroupCushman and WakefieldDLBassociates Consulting EngineersEmersonGeorgia Institute of TechnologyHewlett PackardIBMJDA Consulting EngineersLawrence Berkeley National LabMicrosoftMinick Engi
25、neeringOpengate Data SystemsOracle PanduitRumsey EngineersSan Diego Supercomputer CenterTed Jacob Engineering GroupASHRAE TC9.9 wants to particularly thank the following people:John Bean, Christian Belady, Jack Glass, Jason Kutticherry, OlegLevchook, Rhonda Johnson, Bret Lehman, Mukesh Khattar, Joe
26、Prisco,Madhusudan Iyengar, and Roger Schmidt for their participation as chapterleads and for writing and performing final edits of their chapters. Dr. Roger Schmidt of IBM, Chair of TC9.9, for his vision and leadership inthe creation of this book.Joe Lombardo for developing common graphics for the e
27、leven case studies.In addition TC9.9 would like to thank Will Dahlmeier, Mike Mangan, and DonBeaty of DLB Associates, Inc., and the following people for substantial contribu-tions to the individual case studies in the book:Case 1: Thanks to Bob Wasilewski and Tom Juliano of DLB Associates, Inc.for a
28、iding in the measurements, and thanks to Donna Upright and Duane Oetjenviii Acknowledgmentsof IBM for their complete support in performing these measurements while thedata center was in full operation.Case 2: Thanks to Bob Wasilewski and Tom Juliano of DLB Associates, Inc.,for their aid in the measu
29、rements, and thanks to Donna Upright and Duane Oetjenfor their complete support in performing these measurements while the data centerwas in full operation.Case 3: Thanks to Dr. Roger Schmidt, Dr. Hendrik Hamann, Dane Miller, andHarald Zettl for their help with collection and interpretation of the d
30、ata. The char-acterization and paper would not have been possible without their contribution. Theauthor also thanks the staff of SDSC, especially Mike Datte and Jeff Filliez, for theirfull cooperation in allowing IBM to study the data center and publish the results.Case 4: Thanks to Donna Upright an
31、d Duane Oetjen for their complete supportin performing these measurements in Poughkeepsie while the data center was in fulloperation. Case 5: Thanks to Steve Holt at Livermore for helping with the data collectionat the Livermore site.Case 6: Thanks to Gerhard Haub and Patrick Calcagno of Cushman and
32、 Wake-field and Ryan Meadows and Ed Koplin of JDA Consulting Engineers for their assis-tance with field measurements and analysis.Case 7: Thanks to Dr. Bartosz Ilkowski at the Georgia Institute of Technology,Bret Lehman of IBM, Stephen Peet of BellSouth, and Steve Battenfeld of MinickEngineering for
33、 their contributions to both the design and documentation of this highdensity case study. Thanks also to Sam Toas and Rhonda Johnson of Panduit for theircontributions in the areas of temperature measurement and results documentation.Case 8: Thanks to Jonathan Lomas for field data collection and Scot
34、t Buell forCFD modeling and graphics.Case 9: Thanks to Lennart Stahl of Emerson, a great collaborator on the project,and to the supporting executives, Paul Perez of HP and Thomas Bjarnemark ofEmerson. Thanks also to Chandrakant Patel, Cullen Bash, and Roy Zeighami for alltheir technical support and
35、contributions.Case 10: Thanks to Dr. Mukesh Khattar, Mitch Martin, Stephen Metcalf, andKeith Ward of Oracle for conceptual design and implementation of the hot-aircontainment at the rack level, which permitted use of variable-speed drives on theCRACs while preventing mixing of hot and cold air in th
36、e data floor; Mark Redmondof Ted Jacob Engineering Group for system engineering and specifications; andMark Germagian, formerly of Wright Line and now with Opengate Data Systems,for building server racks with hot-air containment.Case 11: Thanks to Bill Tschudi of Lawrence Berkeley National Laborator
37、y andPeter Rumsey of Rumsey Engineers for contributing this case study, which wasperformed as part of a broader project for the California Energy Commission.1IntroductionData centers and telecommunications rooms that house datacom equipment arebecoming increasingly more difficult to adequately cool.
38、 This is a result of IT manu-facturers increasing datacom performance year after year at the cost of increasedheat dissipation. Even though performance has, in general, increased at a more rapidrate than power, the power required and the resulting heat dissipated by the datacomequipment has increase
39、d to a level that is putting a strain on data centers. However,in the struggle to improve the thermal management characteristics of data centers itis sometimes important to assess todays data center designs. The objective of thisbook is to provide a series of case studies of high density data center
40、s and a rangeof ventilation schemes that demonstrate how loads can be cooled using a number ofdifferent approaches. This introductory chapter describes the various ventilation designs most oftenemployed within data centers. This book does not present an exhaustive resource forexisting ventilation sc
41、hemes but, rather, a wide variety of schemes commonly usedin the industry. Seven primary ventilation schemes are outlined here. In the casestudies that follow, each of these will be shown with detailed measurements ofairflow, power, and temperature. The most common ventilation design for data center
42、s is the raised-access floorsupply, with racks arranged in a cold-aisle/hot-aisle layout (see Figure 1.1). Thechilled-air supply enters the room through perforated tiles in the raised floor, wash-ing the fronts of the racks facing the cold aisle. The hot exhaust air from the racksthen migrates back
43、to the inlet of the computer room air-conditioning units (CRACs)typically located on the perimeter of the data center.Another version of the raised-access floor supply is shown in Figure 1.2, wherethe air-handling units (AHUs) are located beneath the floor containing the IT equip-ment. One of the ke
44、y advantages of this arrangement is that all the mechanical equip-ment is located in a room separate from the IT equipment, which allows for ease ofmaintenance. 10 IntroductionFigure 1.1 Raised-access floor supply.Figure 1.2 Raised-access floor with air-handling unit on floor below.High Density Data
45、 CentersCase Studies and Best Practices11A slightly different version of the raised-access floor supply is the raised-accessfloor supply and ceiling return, as shown in Figure 1.3. This is an advantageousdesign for high-powered racks with hot exhaust, since the hot air is pulled from thehot aisle be
46、fore it is allowed to mix with the cold air. The higher return air temper-ature allows the CRACs to operate much more efficiently given the higher return airtemperature.A unique layout for raised-access floors is shown in Figure 1.4. In this case, themodular CRACs are laid out in the data center in
47、the hot aisle. The advantage of thisventilation scheme is that the hot-air exhaust from the racks has a short path to theinlet of the CRACs, and the chilled air exhausting from the CRACs has a short pathto the cold-aisle perforated tiles. Both paths are short, thereby minimizing theimpedance to airf
48、low. The heat load capability of the CRACs needs to be somewhatbalanced with the heat load of the racks in the immediate vicinity.Since the heat load of the racks has become quite high, there are severaloptions now offered in the industry that provide localized air-to-liquid heatexchangers. In these
49、 cases, the localized air-to-liquid heat exchanger, as shown inFigure 1.5, removes mostif not allof the heat load from the rack before itexhausts into the larger data center room. This removes any hot-spot potential inthe room. Figure 1.3 Raised-access floor supply/ceiling return.12 IntroductionFigure 1.4 Raised floor with modular CRACs in hot aisle.Figure 1.5 Raised-access floor with air-to-liquid heat exchangersadjacent to IT
