a look at memory issues data-transfers must occur between system memory and the network interface...
Post on 19-Dec-2015
220 views
TRANSCRIPT
![Page 1: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/1.jpg)
A look at memory issues
Data-transfers must occur between system memory and the
network interface controller
![Page 2: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/2.jpg)
Typical Chipset Layout
MCHMemory Controller Hub
(Northbridge)
ICHI/O Controller Hub
(Southbridge)
CPUCentral Processing Unit
DRAMDynamic RandomAccess Memory
NICNetwork Interface
Controller
HDCHard Disk Controller
ACAudio Controller
GraphicsController
Timer Keyboard Mouse Clock
MultimediaController
FirmwareHub
USBcontroller
![Page 3: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/3.jpg)
Typical Chipset Layout
MCHMemory Controller Hub
(Northbridge)
ICHI/O Controller Hub
(Southbridge)
CPUCentral Processing Unit
DRAMDynamic RandomAccess Memory
NICNetwork Interface
Controller
HDCHard Disk Controller
ACAudio Controller
GraphicsController
Timer Keyboard Mouse Clock
MultimediaController
FirmwareHub
USBcontroller
DMA
![Page 4: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/4.jpg)
PCI Bus Master DMA82573L i/o-memory
RX and TX FIFOs(32-KB total)
Host’s Dynamic Random Access Memory
Descriptor Queue
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
DMA
on-chip RX descriptors
on-chip TX descriptors
![Page 5: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/5.jpg)
Memory-mapped I/O
• We mentioned that Intel’s x86 architecture originally was designed with two separate address-spaces, one for memory and the other for I/O ports, unlike the designs for CPUs by many of Intel’s competitors in which I/O access was “memory-mapped”
• But now the newer Intel processors also can support memory-mapped I/O as well
![Page 6: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/6.jpg)
Address-bus widths
• The Intel Core-2 Quad processors in our classroom and Lab machines potentially could address 236 physical memory cells (i.e., 64GB), although only 4GB of RAM actually are installed at the present time
• Some PCI-compliant hardware devices were designed for a 32-bit address-bus, thus they must be “mapped” below 4G
![Page 7: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/7.jpg)
Physical-address assignments
The CPU’s physical address-space
Devices’ registers must be mapped to addresses in the bottom 4G
Dynamic Random Access Memory
![Page 8: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/8.jpg)
Virtual addresses
• Software running on the x86 processor is unable to use actual memory addresses, but instead uses ‘virtual’ addresses that map to physical addresses by means of mapping-tables which Linux dynamically defines for each different process it runs
• This complicates the steps software must take to arrange for the DMA to take place
![Page 9: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/9.jpg)
Our ‘dram.c’ module
• To help us confirm that our hardware-level network software is working as we intend, or to diagnose our ‘bugs’ if it isn’t, we can use an LKM we’ve written that implements a character-mode device-driver for system memory, allowing us to view the contents of physical memory as if it were a file; for example, by using our ‘fileview.cpp’ tool
![Page 10: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/10.jpg)
How to view system memory
• Download ‘dram.c’ from course website
• Compile it using our ‘mmake’ utility
• Install ‘dram.ko’ by using ‘/sbin/insmod
• Insure the ‘/dev/dram’ device-node exists
• Download ‘fileview.cpp’ from website and compile it with ‘g++’ (or with ‘make’)
• Execute ‘fileview /dev/dram’ and use the arrow-keys to navigate (or hit <ENTER>)
![Page 11: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/11.jpg)
“canonical” addresses0000000001000100001100100001010011000111
01000010010101001011011000110101110011111000010001100101001110100101011011010111
1100011001110101101111100111011111011111
“non-canonical”(invalid)
virtual addresses
“canonical” addresses
“canonical” addresses
0xFFFFFFFFFFFFFFFF … 0xFFFF800000000000
0x00007FFFFFFFFFFF … 0x0000000000000000
Analogy using 5-bit values
64-bit “vrtual”address space
![Page 12: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/12.jpg)
4-Levels of mapping
PageMap
Level-4Table
CR3
PageDirectoryPointerTable
PageDirectory
PageTable
PageFrame(4KB)
offset
64-bit ‘canonical’ virtual address
sign-extension PML4 PDPT PDIR PTBL 63 48 47 39 38 30 29 21 20 12 11 0
Each mapping-table contains up to 512 quadword-size entries
![Page 13: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/13.jpg)
4-level address-translation
• The CPU examines any virtual address it encounters, subdividing it into five fields
offset into page-frame
index into page-table
63 48 47 39 38 30 29 21 20 12 11 0
16-bits 9-bits 12-bits
index into page-
directory
index into page-
directorypointer table
index intolevel 4
page-maptable
9-bits9-bits9-bits
sign-extension
Any 48-bit virtual-address is sign-extended to a 64-bit “canonical” address
Only “canonical” 64-bit virtual-addresses are legal in 64-bit mode
![Page 14: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/14.jpg)
Reserved(must be 0)
Format of 64-bit table-entries
63 62 52 51 40 39 32
31 12 11 9 8 7 6 5 4 3 2 1 0
Page-frame physical base-address[31..12]
Page-framephysical
base-address[39..32]
EXB
PWUPWT
PCD
A
avl
avl
Meaning of these bits varies with the table
Legend: P = Present (1=yes, 0=no) PWT = Page Cache Disable (1=yes, 0=no) W = Writable (1=yes, 0=no) PWT = Page Write-Through (1=yes, 0=no) U = User-page (1=yes, 0=no) avl = available for user-defined purposes A = Accessed (1=yes, 0=no) EXB = Execution-disabled Bit (if EFER.NXE=1)
![Page 15: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/15.jpg)
Our ‘mem64.c’ module
• We wrote an LKM to create a pseudo-file that will let us see how the virtual memory is being utilized by an application program
• Download this file, compile it with ‘mmake’ and install ‘mem64.ko’ in the Linux kernel
• Then view the virtual-memory mapping that is being used by the ‘cat’ program:
$ cat /proc/mem64
![Page 16: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/16.jpg)
The NIC’s PCI ‘resources’
StatusRegister
CommandRegister
DeviceID0x109A
VendorID0x8086
BISTCacheLineSize
Class CodeClass/SubClass/ProgIF
RevisionID
Base Address 0
SubsystemDevice ID
SubsystemVendor ID
CardBus CIS Pointer
reservedcapabilities
pointer Expansion ROM Base Address
MinimumGrant
InterruptPin
reserved
LatencyTimer
HeaderType
Base Address 1
Base Address 2Base Address 3
Base Address 4Base Address 5
InterruptLine
MaximumLatency
31 0 31 0
16 doublewords
Dwords
1 - 0
3 - 2
5 - 4
7 - 6
9 - 8
11 - 10
13 - 12
15 - 14
![Page 17: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/17.jpg)
Mechanisms compared
kernel memory-space
NIC i/o-memory
CPU’s ‘virtual’ address-space
io
user memory-space
Each NIC register has its own address in memory (allows one-step access)
addr data
Access to all of the NIC’s registers is muliplexed through a pair of I/O-ports(requires multiple instructions)
CPU’s ‘I/O’ address-space
![Page 18: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/18.jpg)
‘nicstatus.c’
• Here’s an LKM that creates a pseudo-file (called ‘/proc/nicstatus’) which will allow a user to view the current value in our Intel 82573L Network Interface Controller’s ‘DEVICE_STATUS’ register
• It uses the I/O-port interface to the NIC’s registers, rather than a ‘memory-mapped’ interface to those device-registers
![Page 19: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/19.jpg)
0
Device Status (0x0008)
? 0 0 0 0 0 0 0 0 0 0 0GIO
MasterEN
0 0 0
0 0 0 0 PHYRA ASDV
ILOS
SLU
0 TXOFF 0 0
FD
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
FunctionID
LU
31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16
SPEED
FD = Full-DuplexLU = Link UpTXOFF = Transmission PausedSPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved)ASDV = Auto-negotiation Speed Detection ValuePHYRA = PHY Reset Asserted
82573L
some undocumented functionality?
![Page 20: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/20.jpg)
‘82573.c’
• This is a more elaborate example of an LKM which not only creates a pseudo-file (i.e., ‘/proc/82573’) that we can view using the Linux ‘cat’ command and that lets us see our NIC’s PCI Configuration Space, but also implements some device-driver functions that let us view the NIC’s device registers by using our ‘fileview.cpp’ tool
![Page 21: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/21.jpg)
Linux PCI helper-functions
#include <linux/pci.h>
struct pci_dev *devp; unsigned int mmio_base; unsigned int mmio_size; void *io;
devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL );if ( devp == NULL ) return –ENODEV;
mmio_base = pci_resource_start( devp, 0 );mmio_size = pci_resource_len( devp, 0 );io = ioremap_nocache( mmio_base, iomm_size );if ( io == NULL ) return –ENOSPC;
![Page 22: A look at memory issues Data-transfers must occur between system memory and the network interface controller](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d3e5503460f94a17517/html5/thumbnails/22.jpg)
In-class exercise
• Two of the NIC’s 32-bit device registers are used to hold its 48-bit Ethernet MAC address – which will be a different value for each of the hosts in our classroom
• These two registers are located at offsets 0x5400 and 0x5404 in device-memory
• The six bytes occur in network byte-order
• Write code to show the MAC address!