managing spp resources: system resource manager (srm) fred kuhns [email protected] applied...

Managing SPP Resources:System Resource Manager (SRM)

Fred [email protected]

Applied Research Laboratory

Washington University in St. Louis

2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

Fred Kuhns - 04/19/23

Common Definitions and Types

• For all commands the message context ID is used to identify the context within which a command is to be executed.– The special value of 0 (cid = 0) indicates a privileged

operation performed by the substrate.

– Otherwise the context ID is an identifier indicating a user specific context in which the command is to be interpreted and executing. In most cases this is the Global Fastpath ID.

• See presentation titled Types.ppt



NATD Interface• [egress_map, ingress_map]

get_sched_map(ipAddr_t link, hwaddr_t board)– Message context ID = 0.

sched_map_t {

qmID_t qmID; // 2-bit QM identifier

schID_t schID; // 3-bit qm relative scheduler identifier

qid_t qid; // 16-bit queue identifier

statID_t sindx; // Stats Index

vlanID_t vid; // VLAN identifier

}

Return:

egress_map: sched_map_t

ingress_map: sched_map_t

Arguments:

link: Interface IP address (aka external IP)

board: Board/Device MAC Address



Reserving Fastpath ResourcesSee Types.ppt for type definition and mapping to substrate API, message context ID = 0.

Reserve resources for 0..N fastpaths (on NPEs) and 0..1 GPEs• retCode_t make_reservation(plabID, rsvRecord_t)

Update an existing reservation, note because all operations need to be idempotent the update argument includes all required resources.

• retCode_t update_reservation(plabID_t, rsvRecord_t)

Cancel any reservation overlapping date_t• retCode_t cancel_reservation(plabID_t, date_t)

Return any reservation (belonging to slice) that overlaps with date_t• rsvRecord_t? get_reservation(plabID_t, date_t)

Return a list of all reservations associated with calling slice• rsvRecord_t* get_reservations(plabID_t)



Allocating Fast-path Instance• fastpath_t alloc_fastpath(plabID_t, copt, bwspec, rcnts, mem)

– Message context ID = 0 plabID_t : GPE/PlanetLab slice identifieruint16_t copt : code option type. {IPv4=1,I3=2}struct bwspec {bw_t bw-firm, bw-soft;} struct rcnts {cnt_t fltrs, queues, buffers, stats;}struct mem {cnt_t sram, dram;} // Requested size of SRAM and DRAM in Bytesstruct fastpath_t {

fpid_t fpid; // fastpath id assigned by srm, used as context ID in future messagesvlan_t vlan; // VLAN id to associate with fp’s logical interfaceipAddr_t fpIP; // IP address of logical fp’s interface on GPEipAddr_t npeIP; // IP address for fp’s data interface on NPEipAddr_t fpNM; // Netmask of fp’s logical interfaceipAddr_t scdIP; // IP address for fp’s assigned SCDipPort_t scdPort; // UDP Port number for fp’s assigned SCDhwaddr_t npeMAC;} // NPE MAC address

– bw-firm: Expresses the required aggregate processing resources in terms of BW, units are Kbps. – bw-soft: This is just a hint as to the expected maximum processing resources required. Resources are not actually allocated to the

fastpath, rather the system uses this number for load balancing best-effort fastpaths across the available NPEs..• Actions required to allocate code option instance and resources:

– Select NPE• Load balance across available NPEs• Must have sufficient resources (SRAM, TCAM space, queues, etc)

– Either allocates requested resources or returns error• Keeps memory map of SRAM so can perform allocation, though the absolute starting address is not required.• If compaction is necessary then must communicate with SCD directly.

– Allocate VLAN and configure switch.– Send commands to SCD on NPE

• set_fastpath(fpid, copt, VLAN, TParams, Mem[])• set_sched_params(sid(gpe) GPE_MIID, GPE_BW_MAX*, GPE_BW_MIN)• set_fpmi_bw(fpid, sid(gpe), GPE_MIID, GPE_BW_MIN)• set_encap_gpe(fpid, gpeIP(fpid), npeIP(fpid))

Deprecated



Releasing a fast-path instance• retCode_t free_fastpath()

– Message context ID = Global fastpath ID (fpid)

– Send command to SCD on NPE• set_sched_params(sid(gpe), GPE_MIID, GPE_BW_MAX*, GPE_BW_MIN)

• retCode_t set_fpmi_bw(fpid, sid(gpe), GPE_MIID, 0)• set_encap_gpe(fpid, 0, 0)• rem_fastpath(fpid)

Deprecated



Instantiate Fastpath and/or Slowpath reservations

• All resource allocation methods operate on the currently active reservation (i.e. must be used during the interval of time when a reservation is valid).

– If free is not called then all resources are automatically freed when reservation expires.

allocate any GPE/Slice resources reserved in the current interval• retCode_t alloc_plspec(plabID_t)

instantiate the “current” fastpath reservation with name fpName• fpInfo_t alloc_fpspec(plabID_t, name_t fpName=“”)

– if only one fastpath then the fpName parameter can be omitted.

free any GPE specific resources currently allocated to slice.• retCode_t free_plspec(plabID_t)

Destroys fastpath instance, releasing meta-interfaces and NPE resources.• retCode_t free_fpspec()

– Message context ID set to the Global fastpath ID



Managing InterfacesReserve Aggregate Interface BW for slice (traffic to/from GPE).• retcode resrv_slice_ifbw(plabID, bw, ifn)

Release slice interface BW reservation for GPE• retcode reles_slice_ifbw(plabID, bw, ifn)

Reserve aggregate Interface BW for indicated fastpath• retcode resrv_fpath_ifbw(bw, ifn)

– message context ID = fpid

Release Aggregate Interface BW allocation for indicated fastpath• retcode reles_fpath_ifbw(bw, ifn)

– message context ID = fpid

Deprecated



System Interfaces, Addresses and PortsGet list of available interfaces• if_list get_interfaces(plabID)

– Message context ID = 0. – if_list = {ifAttr_t, ...}

ifAttr_t { ifn_t ifn; // Interface number ifType_t type; // {Multi-Access = 0, Point-to-Point = 1} ipAddr_t ipaddr; // Interface/Link IP address ipAddr_t linkBW; // Raw (Gross) interface (link) BW bw_t availBW; // Available (Net) interface (link) BW}

– if type = point-to-point then may call get_ifpeer()

Map interface IP address to interface number.• ifn_t get_ifn(plabID_t, ipAddr_t)

Get updated interface attributes• ifAttr_t get_ifattrs(plabID_t, ifn_t)

Get the IP address of point-to-point link’s peer node (SPP peer).• ipAddr_t get_ifpeer(plabID_t, ifn_t)



Managing Externally Visible Address for GPE Traffic• Message context ID set to 0, plabID used to identify Slice making request• Managing bandwidth:

– SRM: verifies slice meta-interfaces and queues do not exceed interface allocations– SRM: Notifies SCD of per Scheduler slice limits

Allocate a local endpoint for slice (linecard directs matching traffic to the corresponding GPE)

• epInfo_t alloc_endpoint(plabID_t, epInfo_t)– Message context ID = 0.

Release allocated endpoint (linecard filters removed and address added bac kto available pool of addresses tuples).

• retCode_t free_endpoint(plabID_t, epoint_t)– Message context ID = 0



Internal Interfaces for NATmap{bw_tbw; # Kbps

ipAddr_t ipaddr; # local interface IP address

ipPort_t port; # if TCP/UDP and 0 then allocate

ipProto_t proto; # TCP/UDP,ICMP

dw1_t smac}; # low order 8 bits of source mac

• {fid, port} alloc_epmap(map)– Message context ID = 0.

– Allocate endpoint and add LC TCAM filter to send pkts to GPE.

• retCode_t free_epmap(fid)– Message context ID = 0.

– Allocate endpoint and add LC TCAM filter to send pkts to GPE.

Deprecated



Allocating Meta-Interfaces (tunnel endpoint)• Common Types

ep {uint32_t bw, uint32_t ipaddr; uint16_t port; uint8_t proto}dw2_t mi: Meta-interface number for an endpoint. Interface numbers start at 0.

• [mi, ep] alloc_tunnel(bw, ipaddr, tunnel_type, params) Not implemented• [mi, ep] alloc_udp_tunnel(bw, ipaddr, port)

Message context Id = fpiduint8_t tunnel_type : must be one of {UDP = 0}uint32_t bw : (Minimum guaranteed) BW assigned to endpoint, Kbpsuint32_t ipaddr: External IP address of endpoint/tunnel. There is a 1-to-1 relationship between links and ip addresses.uint16_t port: External UDP port number– Actions required to allocate code option instance and resources:

• verify requested address is available, assign port if necessary (port == 0)• Allocate new meta-interface ID• allocate and assign filter id for ingress and egress TCAM• Lookup existing or allocate qid for ingress and/or egress

– Send command to SCD on Line Card Ingress• set_sched_params(sid(fabric,NPE), ifn(fabric), BW*, BWmin)• set_queue_sched(qid, sid(fabric,NPE))• set_queue_params(qid, threshold, bw)• write_fltr(dbid_ingress, fid, key_ingress, mask, result_ingress)• write_fltr(dbid_egress, fid, key_egress, mask, result_egress)

– Send commands to Line Card Egress• set_sched_params(sid(link), ifn(link), BW*, BWmin)• set_queue_sched(qid, sid(link))• set_queue_params(qid, threshold, bw)

– Send command to SCD on NPE• set_sched_params(sid(link, miid), ifn(fabric), BW*, BWmin)• set_fpmi_bw(fpid, sid(link, miid), miid, bw*)

• ep get_endpoint(mi)Message context Id = fpid

Deprecated



Releasing Meta-Interfaces• Common Types

ep {uint32_t bw, uint32_t ipaddr; uint16_t port; uint8_t proto}uint_t mi: Meta-interface number for an endpoint. Interface numbers start at 0.

• retCode_t free_tunnel(ipaddr, tunnel_type, params)retCode_t free_udp_tunnel(ipaddr, port)

– Message context Id = fpid. uint32_t ipaddr: External IP address of endpoint/tunneluint16_t port: External UDP port number– Actions required to allocate code option instance and resources:

– Send command to SCD on Line Card Ingress• set_sched_params(sid(fabric,NPE), ifn(fabric), BW*, BWmin)• set_queue_params(qid, threshold, bw)• rem_fltr_byfid(dbid_ingress, fid)• rem_fltr.fid(dbid_egress)

– Send commands to Line Card Egress• set_sched_params(sid(link), ifn(link), BW*, BWmin)• set_queue_params(qid, threshold, bw)

– Send command to SCD on NPE• set_sched_params(sid(link, miid), ifn(fabric), BW*, BWmin)• set_fpmi_bw(fpid, sid(link, miid), miid, bw*)

Deprecated



Allocating Meta-Interfaces (tunnel endpoint)--- Simplified interface, see steps from deprecated slides ---

Allocate local Address for Slice’s fastpath (MI) and forward to NPE, will also reserve sufficient aggregate BW on corresponding internal and external interfaces.

• [miID_t, epInfo_t] alloc_udp_tunnel(bw_t, ipAddr_t, ipPort_t)

– Message context ID set to the global fastpath ID

– It is an error if the sum of requested BW exceeds the reserved fastpath BW.

– Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance.

– if port = 0 then one is allocated from the free pool

– Returns the meta-interface number representing the new endpoint

Release BW and Address associated with corresponding meta-interface• retCode_t free_udp_tunnel(ipAddr_t, ipPort_t)

Map meta-interface ID to corresponding externally visible address tuple• epInfo_t get_endpoint(miID_t)

– mi must be a valid meta-interface ID returned from a call to alloc_udp_tunnel



Classes and Tables

xsid

Entry: {plabID, xsid, gpe, npe, MI}

Mappings:plabID to Entryxsid to Enry

GPE: {GPE object reference, BW, EPs, ??}

NPE: {NPE object reference, vlan bw, sram, fltrs, qs, stats}

MI: {EP object reference, ID}

Slice TableplabID

endpoint (port) mapsresvMap availMap usedMaps xsidMap

Systemtables

Interfacesifn:{type,ipaddr,linkBW,availBW}...

NPE Tableid:{addr,BW/Port,copts,fltrs,sram,Qs}...

VLAN mapsrange:{start,end}

free {...}

GPE

servMap resvMap

endpoint references

controlIPBWmaps

Slices

Per SliceTables vlan

meta-ifacesmi:endpoint...

gpe

board id BW

plab sliceID

NPE (allocated)sram {start,size} #flts

#Qsboard ID BW #Stats

xsid plabID

managing spp resources: system resource manager (srm) fred kuhns [email protected] applied...

Documents