maintaining cache coherency with amd opteron...
TRANSCRIPT
![Page 1: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/1.jpg)
Parag Beeraka | February 11, 2009
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s
![Page 2: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/2.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
2
Outline
Introduction
FPGA Internals
Platforms
Results
Future Enhancements
Conclusion
![Page 3: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/3.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
3
Introduction
![Page 4: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/4.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
4
Two ways to maintain Cache Coherency
Memory is coherent with AMD Opteron™
Ability to Cache AMD Opteron™ ’s Memory
Caching Engine
Coherent Memory
![Page 5: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/5.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
5
Our Approach
Caching Engine
Coherent MemoryCoherent
HyperTransport™
FPGA
![Page 6: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/6.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
6
FPGA Internals
![Page 7: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/7.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
7
FPGA Internals
HT
![Page 8: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/8.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
8
FPGA Internals
Coherent HyperTransport™ core from University of Heidelberg
– Follows HyperTransport™ Protocol
X-Bar and Configuration Space
– Configuration Space Register Strapping's
FPGA Caching Engine (CPU) Core
– Ability to request Cache Lines from AMD Opteron™
FPGA Coherent Memory (MCT) Core
– Memory is coherent with AMD Opteron™ CPU’s
Combination of all blocks Cache Coherent Node
![Page 9: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/9.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
9
Coherent HyperTransport™ Core
![Page 10: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/10.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
10
Coherent HyperTransport™ Core
Follows full HyperTransport™ 1.0 Protocol
– Takes care of HT Init, CRC’s, NOP’s, VC’s etc
Developed by University of Heidelberg
Supports 400 / 800 Mbps Link Speed per CAD/ CTL Line
Supports 8bit / 16bit Link Width
Ability to decode and send all commands in coherent HyperTransport™ specification
![Page 11: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/11.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
11
Application Interface
![Page 12: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/12.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
12
Caching Engine (CPU) Core
Core capable of accessing Opteron™ Cache Lines maintaining Cache Coherency
Direct Mapped Cache (1K)
– 16 Cache Lines (Each Cache Line - 64 bytes)
Implements “MOESI” State Machine for Cache Coherency
Support Write Back, Write Thru and Non Cacheable Memory Type
Handles incoming System Interrupts, Snoop Requests and Lock Transactions
Handles Probe Filter related Requests and Responses (New AMD Opteron™ feature)
![Page 13: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/13.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
13
Caching Engine (CPU) Core
![Page 14: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/14.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
14
Caching Engine (CPU) Core
Automatic SrcTag, Node Information Insertion into Requests
Handles 32 Requests in-flight (Max for HyperTransport™)
Handles Out – of – Order Responses for Requests
Optional 1K Internal Buffers on Tx and Rx Interfaces
Caching Engine
![Page 15: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/15.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
15
Coherent Memory (MCT) Core
Core capable of having Cache Coherent Memory
Developed by AMD Dresden Design Center and University of Heidelberg
8K Memory - Supports 128 Cache Lines
All incoming Cache Block Requests are Routed to Memory Controller
FPGA Coherent Memory (MCT) can also be cached by Caching Engine (CPU) Core
Coherent Memory
![Page 16: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/16.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
16
X-BAR and Configuration Space
![Page 17: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/17.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
17
Configuration Space and X-BAR
X-BAR
Developed by University Of Heidelberg
Routes packets between Units (Caching Engine (CPU), Coherent Memory (MCT) Core) and Coherent HT Core
Uses Routing Table Registers from Configuration to Route Packets to Units
Configuration Space
Similar to AMD Family 10 Architecture Configuration Space
Supports Fn0, Fn1, Fn3 and Fn4 Registers
Added support for HyperTransport™ 3.0
![Page 18: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/18.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
18
Platforms
![Page 19: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/19.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
19
FPGA Cache Coherent Node Platform
Two Xilinx Virtex 5 FPGA’s
Supports 4 16-bit HyperTransport™Links with HMZD Connectors
USB Interface for Back End Control
![Page 20: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/20.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
20
FPGA Cache Coherent Node Test Platform
![Page 21: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/21.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
21
Platform BIOS
Made changes to AGESA (AMD) and IBV (Phoenix, AMI) Bios Code for Coherent Node Configuration
– Used DeviceID for configuring HT Fabric Topology
– Bypass APIC Initialization
– Bypass SMM Processing Routines
– Bypass Power Management Routines
Gained access to FPGA Coherent Memory Core (MCT) by defining a Memory Hole and changing DRAM Routing Tables
![Page 22: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/22.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
22
Platform Experiments
Verified on Barcelona and Shanghai range of Processors on AMD Internal Server Platforms
BIOS support for various 4P Configuration’s
– 1 AMD Opteron™, 1 FPGA Coherent Node
– 2 AMD Opteron™ CPU’s , 2 FPGA Coherent Nodes in a Star Topology
Developed Linux Device Driver to read and write to FPGA’s coherent memory (Verify FPGA Coherent Memory Core)
Developed scripts to access Opteron™ Cache Lines using USB Back End Interface (Verify FPGA Caching Engine)
![Page 23: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/23.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
23
Results
![Page 24: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/24.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
24
Performance Metrics – Caching Engine (CPU)
HT 8 bit 400Mbps Max Bandwidth
- 800MB/s
Bytes
![Page 25: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/25.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’s February 11, 2008
25
Future Enhancements & Conclusion
![Page 26: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/26.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
26
Future Enhancements
Try Coherent HyperTransport™ Core with 16 bit Link Width and HT400 on same platform
Perform Experiments in full 4P system with 2 FPGA Cache Coherent Nodes
Verify on real system with next generation HT Core from University Of Heidelberg
Plan to deliver whole package with next generation HT Core for Coherent Licensee’s
Port to other FPGA Vendor Devices
Run Benchmarks with real applications
![Page 27: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/27.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
27
Conclusion
Developed a Cache Coherent Node that can be used over Coherent HyperTransport™ with AMD Opteron™ Processors using FPGA’s in various environments
– Silicon Validation
– Simulation BFM’s Validation
– Accelerated Computing (Co – Processing using FPGA’s)
![Page 28: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/28.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
28
Danke!!
![Page 29: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/29.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
29
Questions ?
![Page 30: Maintaining Cache Coherency with AMD Opteron ...ra.ziti.uni-heidelberg.de/coeht/pages/events/20090211/...2009/02/11 · Caching Engine Coherent Memory Maintaining Cache Coherency](https://reader034.vdocuments.us/reader034/viewer/2022042917/5f5b32b70b0b7552676e9f6b/html5/thumbnails/30.jpg)
Maintaining Cache Coherency with AMD Opteron™ Processors using FPGA’sFebruary 11, 2008
30
Trademark Attribution
AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks of their respective owners.
©2009 Advanced Micro Devices, Inc. All rights reserved.