Download - Fast Readout Links - indico.cern.ch
23 June 201123 June 2011 [email protected]@cern.ch 00
Fast Readout Links Fast Readout Links
Optical interconnects by network applicationOptical interconnects by network application
Comparison with developments for LHC and HLComparison with developments for LHC and HL-- LHCLHC
What to expect for future High Energy Physics What to expect for future High Energy Physics experimentsexperiments
F. Vasey, CERN, PH-ESE
23 June 201123 June 2011 [email protected]@cern.ch 11
Optical Link MetricsOptical Link Metrics
BandwidthBandwidth
Energy (Energy (pJpJ/bit, /bit, mW/GbpsmW/Gbps))
1mW/Gbps = 1pJ/bit1mW/Gbps = 1pJ/bit
1 1 googlegoogle search = 1kJ search = 1kJ
1 1 PbitsPbits (@1pJ/bit)(@1pJ/bit) !!!!!!
Density (mmDensity (mm22/Gbps)/Gbps)
Cost ($/Cost ($/GbpsGbps))
Environmental ResistanceEnvironmental Resistance
23 June 201123 June 2011 [email protected]@cern.ch 22
Use and Misuse of Scaled MetricsUse and Misuse of Scaled Metrics
mW/GbpsmW/Gbps, mm, mm22/Gbps, $//Gbps, $/GbpsGbps
Systematically favour very high bandwidth Systematically favour very high bandwidth systemssystems
Cannot be used to extrapolate likely figures for Cannot be used to extrapolate likely figures for bandwidthbandwidth--distributed systemsdistributed systems
Per device (Per device (TRxTRx) scaling more appropriate) scaling more appropriate
mW/TRxmW/TRx, mm, mm22/TRx, $//TRx, $/TRxTRx
Nevertheless very useful to compare systemsNevertheless very useful to compare systems
23 June 201123 June 2011 [email protected]@cern.ch 33
Existing Optical Links in CMS (1)Existing Optical Links in CMS (1)
CMS Tracker AOHCMS Tracker AOH
39k links39k links
40MS/s x 8 bits = 320 Mbps40MS/s x 8 bits = 320 Mbps
100 100 mWmW, 3 channels , 3 channels (dependent on laser bias)(dependent on laser bias)
30mm x 23mm, 3 channels30mm x 23mm, 3 channels
300 CHF / channel300 CHF / channel
CMS ECAL GOHCMS ECAL GOH
10k links10k links
800 Mbps800 Mbps
GOL operates up to 1600 GOL operates up to 1600 MbpsMbps
Aggregate bandwidth ~ Tk: 12.5Tbps
ECAL: 8Tbps
23 June 201123 June 2011 [email protected]@cern.ch 44
Existing Optical Links in CMS (2)Existing Optical Links in CMS (2)
AOHAOH19971997
Bandwidth Bandwidth GbpsGbps 0.320.32
Reach mReach m 100100
Energy Energy pJpJ/bit, /bit, mW/GbpsmW/Gbps 150 150 * extrapolating to * extrapolating to bidibidi linklink
Density mmDensity mm22/Gbps/Gbps 10001000* extrapolating to * extrapolating to bidibidi linklink
Cost $/Cost $/GbpsGbps 900900
23 June 201123 June 2011 [email protected]@cern.ch 55
Interconnects by Network ApplicationInterconnects by Network Application
Chip to ChipChip to Chip 1 1 –– 50 cm50 cm
Data CenterData Center(Rack to Rack)(Rack to Rack) 11--100 Meters100 Meters
EthernetEthernetInfiniBandInfiniBand
FibreFibre Channel / SASChannel / SASProprietaryProprietary
Board to BoardBoard to Board ~0.5 ~0.5 --1.5M1.5M
MemoryMemoryMemoryMemory MemoryMemoryMemoryMemory
MemoryMemory
TeraTera--scale CPUscale CPU
MemoryMemory
CPU 2CPU 2
Metro &Metro &Long HaulLong Haul(0.1(0.1--80km)80km)
SONETSONETWDMWDM
EthernetEthernet
IntraIntra--RackRackEthernetEthernetInfiniBandInfiniBandFibreFibre Channel / SASChannel / SASPCIePCIeProprietaryProprietary
BackplaneBackplane
100s of100s ofThousandsThousands
10s of10s ofMillionsMillions
10s of10s ofBillionsBillions
Volume100s of100s ofMillionsMillions
23 June 201123 June 2011 [email protected]@cern.ch 66
Wide Area Network (WAN)Wide Area Network (WAN)
Chip to ChipChip to Chip 1 1 –– 50 cm50 cm
Data CenterData Center(Rack to Rack)(Rack to Rack) 11--100 Meters100 Meters
EthernetEthernetInfiniBandInfiniBand
FibreFibre Channel / SASChannel / SASProprietaryProprietary
Board to BoardBoard to Board ~0.5 ~0.5 --1.5M1.5M
MemoryMemoryMemoryMemory MemoryMemoryMemoryMemory
MemoryMemory
TeraTera--scale CPUscale CPU
MemoryMemory
CPU 2CPU 2
Metro &Metro &Long HaulLong Haul(0.1(0.1--80km)80km)
SONETSONETWDMWDM
EthernetEthernet
IntraIntra--RackRackEthernetEthernetInfiniBandInfiniBandFibreFibre Channel / SASChannel / SASPCIePCIeProprietaryProprietary
BackplaneBackplane
100s of100s ofThousandsThousands
10s of10s ofMillionsMillions
10s of10s ofBillionsBillions
Volume100s of100s ofMillionsMillions
23 June 201123 June 2011 [email protected]@cern.ch 77
Bandwidth and CapacityBandwidth and Capacity
TDM
WDM
Bits/Symbol
Bandwidth Bandwidth
#Symbols/s x bits/symbol x ##Symbols/s x bits/symbol x #
Only one fibre considered hereOnly one fibre considered here
Capacity = Reach x BandwidthCapacity = Reach x Bandwidth
Reach not an issue in typical Reach not an issue in typical detector applications (~100m)detector applications (~100m)
Only bandwidth optimization Only bandwidth optimization considered hereconsidered here
QPSK 8-PSK
16-QAM
c
23 June 201123 June 2011 [email protected]@cern.ch 88
WAN Network featuresWAN Network features
Ultimate CapacityUltimate Capacity
Design for minimum Design for minimum received energyreceived energy
Exploit TDM, WDM, Exploit TDM, WDM, Advanced modulation Advanced modulation formatsformats
Single ModeSingle Mode
Electronic Mitigation Electronic Mitigation Techniques Techniques
Ultra fast electronics Ultra fast electronics operating at line rate (operating at line rate (f.if.i. . 50GS/s ADC)50GS/s ADC)
FEC, equalizers, FEC, equalizers, ……
Metrics (very coarse guess)Metrics (very coarse guess)
Bandwidth: 10TbpsBandwidth: 10Tbps
Reach: 100kmReach: 100km
Energy: 10Energy: 10’’000 000 pJpJ/bit/bit
Density: 10Density: 10’’000 mm000 mm22/Gbps/Gbps
Cost: 100$/GbpsCost: 100$/Gbps
Not in our ballpark,
but watch out for emerging technologies
23 June 201123 June 2011 [email protected]@cern.ch 99
Access NetworkAccess Network
Chip to ChipChip to Chip 1 1 –– 50 cm50 cm
Data CenterData Center(Rack to Rack)(Rack to Rack) 11--100 Meters100 Meters
EthernetEthernetInfiniBandInfiniBand
FibreFibre Channel / SASChannel / SASProprietaryProprietary
Board to BoardBoard to Board ~0.5 ~0.5 --1.5M1.5M
MemoryMemoryMemoryMemory MemoryMemoryMemoryMemory
MemoryMemory
TeraTera--scale CPUscale CPU
MemoryMemory
CPU 2CPU 2
Metro &Metro &Long HaulLong Haul(0.1(0.1--80km)80km)
SONETSONETWDMWDM
EthernetEthernet
IntraIntra--RackRackEthernetEthernetInfiniBandInfiniBandFibreFibre Channel / SASChannel / SASPCIePCIeProprietaryProprietary
BackplaneBackplane
100s of100s ofThousandsThousands
10s of10s ofMillionsMillions
10s of10s ofBillionsBillions
Volume100s of100s ofMillionsMillions
23 June 201123 June 2011 [email protected]@cern.ch 1010
Access over Passive Optical NetworkAccess over Passive Optical NetworkFTTH
FTTC
FTTB
GPON, EPON 1.25Gbps GPON, EPON 1.25Gbps -- 2Gbps2Gbps
Consumes 18x Consumes 18x lessless energyenergy per user per user thanthan VDSL2VDSL2
Sharing of Sharing of thisthis capacitycapacity amongamong multiple multiple usersusers (time(time--division division multiplexingmultiplexing))
Distance to centralDistance to central--office office cancan go up to 60km go up to 60km whenwhen amplifieramplifier--assistedassisted, , not a few 100s not a few 100s metersmeters fromfrom setset--top box to DSLAM.top box to DSLAM.
10GPON solutions 10GPON solutions recentlyrecently proposedproposed by system by system vendorsvendors
WDM dimension WDM dimension cancan alsoalso bebe exploitedexploited to to increaseincrease capacitycapacity,,
And And provideprovide Point to Point connections, Point to Point connections, capacitycapacity on on demanddemand……whilewhile preservingpreserving scalabilityscalability
23 June 201123 June 2011 [email protected]@cern.ch 1111
PON Access Network featuresPON Access Network features
Moderate speed, reachModerate speed, reach
ScalabilityScalability
Shared resourceShared resource
High dynamic rangeHigh dynamic range
High split ratioHigh split ratio
High Volume, Low CostHigh Volume, Low Cost
Single Mode, bidirectionalSingle Mode, bidirectional
10G PON coming up, WDM PON 10G PON coming up, WDM PON nextnext
Metrics (coarse Metrics (coarse guestimatesguestimates))
Bandwidth: 2.5GbpsBandwidth: 2.5Gbps
Reach: 10kmReach: 10km
Energy: 400pJ/bitEnergy: 400pJ/bit
Density: 200mmDensity: 200mm22/Gbps/Gbps
Cost: 10$/GbpsCost: 10$/Gbps
Not in our ballpark,
Traffic flows in wrong direction
But high volume and low cost model in right bandwidth range
23 June 201123 June 2011 [email protected]@cern.ch 1212
Local Area Network (LAN)Local Area Network (LAN)
Chip to ChipChip to Chip 1 1 –– 50 cm50 cm
Data CenterData Center(Rack to Rack)(Rack to Rack) 11--100 Meters100 Meters
EthernetEthernetInfiniBandInfiniBand
FibreFibre Channel / SASChannel / SASProprietaryProprietary
Board to BoardBoard to Board ~0.5 ~0.5 --1.5M1.5M
MemoryMemoryMemoryMemory MemoryMemoryMemoryMemory
MemoryMemory
TeraTera--scale CPUscale CPU
MemoryMemory
CPU 2CPU 2
Metro &Metro &Long HaulLong Haul(0.1(0.1--80km)80km)
SONETSONETWDMWDM
EthernetEthernet
IntraIntra--RackRackEthernetEthernetInfiniBandInfiniBandFibreFibre Channel / SASChannel / SASPCIePCIeProprietaryProprietary
BackplaneBackplane
100s of100s ofThousandsThousands
10s of10s ofMillionsMillions
10s of10s ofBillionsBillions
Volume100s of100s ofMillionsMillions
23 June 201123 June 2011 [email protected]@cern.ch 1313
40/100GbE based on 10G 40/100GbE based on 10G PHYsPHYs
10G: directly modulated EE lasers or 10G: directly modulated EE lasers or VCSELsVCSELs
25G: externally modulated lasers25G: externally modulated lasers
40G40G--100G: aggregation100G: aggregation
23 June 201123 June 2011 [email protected]@cern.ch 1414
LAN Network FeaturesLAN Network Features
converging with MANconverging with MAN
TDM limits reached TDM limits reached
Aggregation via parallelismAggregation via parallelism
PhysicalPhysical
Wavelength Wavelength
Inheriting WAN featuresInheriting WAN features
High DensityHigh Density
Low PowerLow Power
High Volume Low CostHigh Volume Low Cost
Multimode OM3 & OM4 fibreMultimode OM3 & OM4 fibre
10G over 100m & 300m10G over 100m & 300m
Single Mode nextSingle Mode next
Metrics (Metrics (guestimatesguestimates))
Bandwidth: 100GbpsBandwidth: 100Gbps
Reach: 100mReach: 100m
Energy: 10pJ/bitEnergy: 10pJ/bit
Density: 2mmDensity: 2mm22/Gbps/Gbps
Cost: 1$/GbpsCost: 1$/Gbps
23 June 201123 June 2011 [email protected]@cern.ch 1515
From modules to chips: packaging issuesFrom modules to chips: packaging issues
23 June 201123 June 2011 [email protected]@cern.ch 1616
Link Type Power Efficiency (pJ/bit)
Distance
Electrical 3 < 1 mEOE 7 < 100 m
Yearm
W/G
b/s
Electrical Link Power Efficiency Progress
20002002
20042006
20082010
05
1015202530354045
Electrical Interconnects are still alive Electrical Interconnects are still alive
For short distances, EOE For short distances, EOE cannot beat EEcannot beat EE
Short range low power Short range low power electrical links are the electrical links are the subject of intense subject of intense development effortsdevelopment efforts
Moving targetMoving target
23 June 201123 June 2011 [email protected]@cern.ch 1717
Optical network metricsOptical network metrics A risky comparison* (1)A risky comparison* (1)
AOHAOH19971997
WANWAN19901990
AccessAccess20002000
LANLAN20102010
Bandwidth Bandwidth GbpsGbps
0.320.32 1010’’000000 2.52.5 100100
Reach Reach mm
100100 100100’’000000 1010’’000000 100100
Energy Energy pJpJ/bit/bit
150 150 1010’’000000 400400 1010
Density Density mmmm22/Gbps/Gbps
10001000 1010’’000000 200200 22
Cost Cost $/$/GbpsGbps
900900 100100’’ss 1010 11
*This comparison expresses the views of the author at presentation time and do not necessarily represent an accurate vision of the current and future situation Arrays
23 June 201123 June 2011 [email protected]@cern.ch 1818
Next for HEP: Front End VersatileNext for HEP: Front End Versatile--TRxTRx for HLfor HL--LHCLHC
Low Mass & VolumeLow Mass & Volume
Minimize material, avoid metalsMinimize material, avoid metals
Capable of operating in a magnetic Capable of operating in a magnetic fieldfield
Requires replacement of ferrite bead used in Requires replacement of ferrite bead used in laser bias networklaser bias network
Radiation TolerantRadiation Tolerant
BitrateBitrate determined by determined by ASICsASICs: 5 : 5 –– 10 10 GbpsGbps
850nm and 1310nm 850nm and 1310nm flavoursflavours
TOSA
ROSA
LDD
GBTIA
VTRx
23 June 201123 June 2011 [email protected]@cern.ch 1919
Front End VersatileFront End Versatile--TRxTRx Results at 5GbpsResults at 5Gbps
TxTxSM@1310nmSM@1310nm MM@850nmMM@850nm
RxRx
23 June 201123 June 2011 [email protected]@cern.ch 2020
Optical network metricsOptical network metrics A risky comparison* (2)A risky comparison* (2)
AOHAOH19971997
WANWAN19901990
AccessAccess20002000
LAN R2RLAN R2R20102010
VTRxVTRx20082008
Bandwidth Bandwidth GbpsGbps
0.320.32 1010’’000000 2.52.5 100100 55
Reach Reach mm
100100 100100’’000000 1010’’000000 100100 100100
Energy Energy pJpJ/bit/bit
150 150 1010’’000000 400400 1010 100100
Density Density mmmm22/Gbps/Gbps
10001000 1010’’000000 200200 22 100100
Cost Cost $/$/GbpsGbps
900900 100100’’ss 1010 11 60 ?60 ?
*This comparison expresses the views of the author at presentation time and do not necessarily represent an accurate vision of the current and future situation
Arrays
23 June 201123 June 2011 [email protected]@cern.ch 2121
A A TRxTRx for an allfor an all--inin--one moduleone module
50% scaling required50% scaling required
SizeSize
powerpower
23 June 201123 June 2011 [email protected]@cern.ch 2222
Optical network metricsOptical network metrics A risky comparison* (3)A risky comparison* (3)
AOHAOH19971997
WANWAN19901990
AccessAccess20002000
LAN R2RLAN R2R20102010
VTRxVTRx2008 2008 1212
Bandwidth Bandwidth GbpsGbps
0.320.32 1010’’000000 2.52.5 100100 55
Reach Reach mm
100100 100100’’000000 1010’’000000 100100 100100
Energy Energy pJpJ/bit/bit
150 150 1010’’000000 400400 1010 100100
5050
Density Density mmmm22/Gbps/Gbps
10001000 1010’’000000 200200 22 100100
5050
Cost Cost $/$/GbpsGbps
900900 100100’’ss 1010 11
*This comparison expresses the views of the author at presentation time and do not necessarily represent an accurate vision of the current and future situation
Arrays
23 June 201123 June 2011 [email protected]@cern.ch 2323
LetLet’’s not forget radiation tolerances not forget radiation tolerance
Our daily breadOur daily bread
23 June 201123 June 2011 [email protected]@cern.ch 2424
Radiation Tolerance: Is there a limit?Radiation Tolerance: Is there a limit?
Active Active optoopto devices tested up devices tested up to now do not to now do not resist resist fluencesfluences beyond 10beyond 1016 16 cmcm--22
Still OK for SLHC Still OK for SLHC TkTk
Tighter marginsTighter margins
Are there Are there alternatives for alternatives for fluencesfluences beyond beyond 101016 16 cmcm--2 2 ? ?
(Re)(Re)--consider consider Passives ?Passives ?
modulatorsmodulators
23 June 201123 June 2011 [email protected]@cern.ch 2525
Extrapolating I/O Bandwidth RequirementsExtrapolating I/O Bandwidth Requirements
Need to go optical, on chip, with WDM · Si – CMOS Photonics
23 June 201123 June 2011 [email protected]@cern.ch 2626
Evolution of high speed interconnects Evolution of high speed interconnects 20102010--20152015
Base circuit board withoptical waveguides & turning mirrors
Lens array
CPU CMOS TRX and SLC
OE50mm Organic Module
CPU
Base circuit boardPower and slow signals
HSI
Electrical or Optical engines + Opto PCB MMPIC TRx
20152015--20202020CPU
Base circuit boardPower and slow signals
Monolithic integration of TRx and Processing chip SM
OR
Will this paradigm shift occur ?
Will it bring us benefits
(density, power, radition tolerance) ?
23 June 201123 June 2011 [email protected]@cern.ch 2727
ConclusionsConclusions
Broadband optical communication is a commodity and is ubiquitousBroadband optical communication is a commodity and is ubiquitous
Also in HEPAlso in HEP
Scaling bandwidth and connectivity remains a major issue at all Scaling bandwidth and connectivity remains a major issue at all levels of the levels of the network hierarchynetwork hierarchy
HEP has a distributed bandwidth need. It cannot simply aggregatHEP has a distributed bandwidth need. It cannot simply aggregate.e.
Current optical communication technologies will not allow to meeCurrent optical communication technologies will not allow to meet the challenges t the challenges ahead without major breakthroughs:ahead without major breakthroughs:
Advanced modulation, coding and electronic mitigation schemesAdvanced modulation, coding and electronic mitigation schemes
High density parallel and wavelength multiplexed optics (opticalHigh density parallel and wavelength multiplexed optics (optical engines)engines)
Photonic integration with CMOSPhotonic integration with CMOS
Pragmatically speaking, all incremental electrical chip to chip Pragmatically speaking, all incremental electrical chip to chip interconnection interconnection approaches will be exhausted before fast interconnects go opticaapproaches will be exhausted before fast interconnects go opticall
Will this kill photonics integration?Will this kill photonics integration?
In the short/medium term: laserIn the short/medium term: laser--based EOE conversion such as proposed by the based EOE conversion such as proposed by the Versatile Link common project is the way forwardVersatile Link common project is the way forward
10Gbps, 1010Gbps, 10’’s s pJpJ/bit, 10/bit, 10’’s mms mm22/Gbps/Gbps
In the long term:In the long term:
Worth watching photonics integrationWorth watching photonics integration
Embracing a new technology is resource intensiveEmbracing a new technology is resource intensive