labeled risc-v: a new perspective on software-defined ... · tradeoff in data center: util. vs. qos...

Labeled RISC-V: A New Perspective on

Software-Defined Architecture

Zihao Yu, Jiuyue Ma, Bowen Huang, Xin Jin,

Huizhe Wang, Yaoyang Zhou, Zihao Chang,

Yan Cao, Sa Wang, Yungang Bao

May 9th , 2017 @ ShangHai

Institute of Computing Technology (ICT),

Chinese Academy of Sciences (CAS) 1

• Why?

• How?

• What Effect?


Agenda

Tradeoff in Data Center: Util. vs. QoS

3

• LOW utilization[1,2]: 6%~12%

• Sharing causes interference

• Sacrifice utilization to guarantee QoS

[1] http://www.gartner.com/newsroom/id/1472714.

[2] J. M. Kaplan, W. Forrest, and N. Kindler. Revolutionizing data center energy efficiency. McKinsey & Company, 2008.

Lo et al. Heracles: Improving

Resource Efficiency at Scale,

ISCA, 2015.

http://www.gartner.com/newsroom/id/1472714

More Hardware Support Needed

4

QoS-unawared Hardware

Ambulance?

• Unmanaged Sharing

• No architectural support for QoS

Unable to distinguish Shared resources

5

• Why?

• How?

• What Effect?


Agenda

Labeled Networking

• Fine-grain: every packet has a label

• Semantic association: correlate labels with users’ demand

• Propagation: propagate labels in a whole network

• DiffServ: process packets differentiately based on labels

MPLS is widely used for VPN and QoS

Yes!

The Computer as a Network • Hardware components communicate via internal

packets, e.g., PCIe packets, NoC packets, QPI packets

Core P3

Core P1

Processor

P0

Labeled von Neumann Architecture (LvNA)

9

4. Software-defined

control logic

1. Fine-grained

object

2. Sematic association

3. Propagation

Bao and Wang, Labeled von Neumann Architecture for Software-Defined Cloud,

Journal of Computer Science and Technology, 2017 Vol. 32 (2): 219-223.

Input Output

Memory

Add label registers to req. sources

10

Core

Shared Last Level Cache

I/O

Chipset Memory

Controller

Core Core

Disk NIC Disk Disk

…

VM0 VM1 VMn

DS-id DS-id DS-id

DS-id DS-id

DS-id

DS-id

DS-id DS-id

Add label

registers

Sematic association

11

Core


I/O

Chipset Memory

Controller

Core Core

Disk NIC Disk Disk

…

VM0 VM1 VMn

DS-id DS-id DS-id

DS-id DS-id

DS-id

DS-id

DS-id DS-id

Propagation

12

Core


I/O

Chipset Memory

Controller

Core Core

Disk NIC Disk Disk

…

VM0 VM1 VMn

DS-id DS-id DS-id

DS-id DS-id

DS-id

DS-id

DS-id DS-id

DMA CPU

request

Software-defined label-based control logic

13

Core


I/O

Chipset Memory

Controller

Core Core

Disk NIC Disk Disk

…

VM0 VM1 VMn

DS-id DS-id DS-id

DS-id DS-id

DS-id

DS-id

DS-id DS-id

CL

CL CL

Programmable Architecture for Resourcing-on-Demand

PARD

Leverage LvNA to perform fine-grained control

14 Ma et. al, Supporting Differentiated Services in Computers via Programmable Architecture for Resourcing-on-Demand (PARD), ASPLOS, 2015

Tables as CL

15

Cache Controller Memory Controller

Closed-loop Control

16

Trigger => Action

Trigger Table

DS-id1 Cond-1 Action-1



…

Statistics Table

DS-id1 Stat1 Stat2 …

DS-id2 Stat1 Stat2 …

DS-id3 …

…

Parameter Table

DS-id1 Param1 Param2 …

DS-id2 Param1 Param2 …

DS-id3 …

…

e.g. miss_rate > 30%

e.g. adjust way mask

action

signal firmware

action script

+

stat

op

>

<

=

threshold

Platform Resource Manager (PRM)

• Augmented IPMI

• Connect all control logics

• Run linux-based firmware

• Abstract CLs as files

17

parameter

ident

type

ldoms

cpa0

/sys/cpa

statistics

trigger

ldom0

ldom1

cpa1

cpa2

ldom2

param1

param2

Core DS-id


I/O

Chipset

Memory

Controller

Core

VM1

DS-id

Core

VMn

DS-id

Disk NIC DS-id

DS-id

DS-id Disk

DS-id

Disk DS-id

…

VM0

Centralized

PRM

Programming

Monitoring & Interrupts

CL

CL CL CL CL

CL

CL

parameter

Access Control Logics

18

ident

type

ldoms

cpa0

/sys/cpa

statistics trigger

ldom0

ldom1

cpa1

cpa2

ldom2

param1

param2

Query Parameters cat /sys/cpa/cpa0/…/parameter/param1

Setting Parameters echo 10 > /sys/cpa/cpa0/…/parameter/param2

Query Control Logic Info cat /sys/cpa/cpa0/ident cat /sys/cpa/cpa0/type

• Why?

• How?

• What Effect?


Agenda

Implementation

20

* http://github.com/fsg-ict/PARD-gem5

+ http://github.com/fsg-ict/labeled-RISC-V

• Full-system cycle-accurate simulator

• FPGA prototype on Xilinx VC709 evaluation board • MicroBlaze version

• RISC-V version

Open Sourced *

Deprecated

Open Sourced +

Xilin

x VC

70

9 Evalu

ation

Bo

ard

Core Control Logic

Core0 Core1 Core2 Core3

Cache / XBar

Cache Control Logic

Memory Controller (MIG7)

Memory Control Logic

PRM

I/O CL

Eth0

Eth1

Eth2

UART*4

SFP+

CN Switch

UART

PC IP: 192.168.1.1

(dhcp, tftp, httpd)

AXI4 Memory Bus AXI4 I/O Bus Labeled Intr. CN Bus (I2C-based)

LDom #1

LDom #2

LDom #3

LDom #4

21

Evaluation - Performance Isolation

• 4 Ldoms: 1 X 429.mcf + 3 X Attacker

• Allocate different LLC capacities

• Perf. degradation: 7%(w/ PARD) vs. 48%(w/o

PARD)

solo attacker Attacker + “T->A” 22

LvNA + RISC-V = Labeled RISC-V

• Switching to RISC-V

• Add labels in the core easier

• riscv-go to run container

• Goal - establish Labeled

RISC-V branch

23 https://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-17.pdf






RISC-V branch


DS-id DS-id






RISC-V branch


DS-id DS-id

CL






RISC-V branch


DS-id DS-id

TL2.dsid <-> axi.user

CL






RISC-V branch


DS-id DS-id


DS-id Base Len

1 0x0000 0x4000

2 0x8000 0x8000

CL

CL

Address mapping

Labeled token bucket






RISC-V branch


DS-id DS-id


DS-id Base Len

1 0x0000 0x4000

2 0x8000 0x8000

CL

CL

Address mapping

Labeled token bucket

PRM

Overheads

• 16 lines of chisel code to add labels into RocketChip – Add dsid member in the

Bundle of TileLink2

– Attach labels at the TileLink2 masters of core tiles

• < 5% resource overheads for CLs – Much more less with complex

cores, e.g. BOOM

29

Control Logics

Demo 1 - NoHype

• Push the software hypervisor down to LvNA

• Isolate the resources (address space, device) by CLs

• Partitioned into several sub-machines

30

Label-based CL

Label-based CL

Label-based CL

Demo 1 - NoHype

31

LDom0 LDom1

Demo 2 – Memory Bandwidth Control

• Use labeled token buckets to isolate the bandwidth

attacker

32

Solo

interfered

isolated

Hardware - LvNA

Hardware

Hypervisor

Operating System

Runtime Library

Compiler

Schedule Framework

Application

boom + SMT

boom + SMT

boom + SMT

boom + SMT

L2

MEM PCI-E

CL CL CL CL

CL

CL CL

PRM (rocket)

...

ROB

33

• Labels everywhere in the hardware

• Plan to tape out with 40nm TSMC!

Hypervisor - NoHype

Hardware

Hypervisor

Operating System

Runtime Library

Compiler

Schedule Framework

Application

34

• Push the software hypervisor down to LvNA

• Remove run-time overhead

• VMentry, VMexit…

LvNA

VM1 VM2 VM3 software VMM

VM4 VMn …

Isolated by NoHype

Finished

Operating System - Fine-grained labeling

Hardware

Hypervisor

Operating System

Runtime Library

Compiler

Schedule Framework

Application

35

• Add fine-grained label as context resource

• Process (integrated into Cgroup)

• Process/container-level

• Thread-level

• Address space

• Function-level

• Object-level

• Provide libraries

• pthread_create_with_dsid()

• malloc_with_dsid()

Process

relative

labeling Core

VM0

P0 P1

dsid start end

1 0x8000 0xffff

3 0x2000 0x27ff

Address space labeling

Finished

Compiler - collect QoS info. from prog

Hardware

Hypervisor

Operating System

Runtime Library

Compiler

Schedule Framework

Application

36

• Express QoS info. from source files

• Additional compilation results

• Address space labeling info

• Extra ELF sections for loader

• Resource requirement

• QoS desc. file for schedule framework

#progma qos(10s)

sort();

SLA = 10s

working set

= 64KB

…

QoS Desc.

dsid start end

1 0x8000 0xffff

3 0x2000 0x27ff

…

call sort

…

Binary

Sche. Framework - QoS resource schedule

Hardware

Hypervisor

Operating System

Runtime Library

Compiler

Schedule Framework

Application

37

• Expose QoS resources to schedule frameworks

• Integrate QoS resources into OpenStack Finished

A lot to explore!

• Theory: How does LvNA impact on RAM, PRAM, LogP models?

• Hardware/Arch: How to implement LvNA at CPU pipeline/SMT, memory, storage, networking? How to correlate LvNA and SDN by labels?

• OS/Hypervisor: How to correlate labels with VMs, containers, processes, threads? How to abstract programming interfaces for labels?

• Programing Model and Compilers: How to express users’ requirements and propagate to the hardware via labels? How to make compilers support labels?

• Distributed systems: How to correlate labels with distributed resources? How to manage distributed systems with label mechanisms?

• Measurement/Audit: How to leverage labels to gauge and audit resource usages?

38

• Finished • On-going • Have ideas • Feature work

Summary

• LvNA: a model of software-defined architecture

• PARD: a proof of concept of LvNA

• Labeled RISC-V: an implementation of LvNA

39

Labeled RISC-V: A New Perspective on Software-Defined Architecture

40

Thanks

Scan the QR code to join the discussion group with WeChat

labeled risc-v: a new perspective on software-defined ... · tradeoff in data center: util. vs. qos...

Documents