NetQuery: A General-Purpose Channel for Reasoning about NetworkProperties

Alan Shieh Oliver Kennedy Emin Gun Sirer Fred B. Schneider

Dept. of Computer Science, Cornell University, Ithaca, NY 14853{ashieh, okennedy, egs, fbs}@cs.cornell.edu

ABSTRACTAlthough the configuration of modern networks has a signif-icant impact on the performance, robustness, and security ofapplications, networks lack support for reporting these dif-ferences. This paper presents the design and implementationof NetQuery, a novel, general-purpose channel for dissemi-nating the properties of networks and their participants. Net-Query implements a distributed, decentralized, tuple-basedattribute store that records information about network enti-ties. Operators can add new tuples into this store and canalso annotate existing tuples with new, custom attributes,thus allowing the system to support network entities andproperties not anticipated at the time of deployment. Net-Query clients can query this attribute store for the currentnetwork state and install event triggers to detect future statetransitions, thus establishing long-running guarantees overthe behavior of the network. We have implemented Net-Query and deployed networks with NetQuery-enabled de-vices that leverage commodity trusted hardware to providestrong assurance over the accuracy of reported properties.We describe the NetQuery system, outline the types of newapplications enabled by NetQuery, and report on the perfor-mance of the system from deployments of real devices andfrom simulations of ISP networks.

1. INTRODUCTIONDepending on their configuration, administration and pro-

visioning, networks can provide drastically different features.For instance, some networks provide little failure resiliencewhile others provision failover capacity and deploy middle-boxes to protect against denial of service attacks [4]; somenetworks provide network neutral access while others selec-tively throttle applications [12]; and some networks provideconfidentiality guarantees while others monitor a user’s re-quest stream to build an advertising profile [17]. Yet thestandard IP interface masks these differences, as every net-work appears to provide the same basic “dial-tone” service.Consequently, clients resort to ad hoc techniques to detectthese differences or target the lowest common denominatorservice.

Similarly, configuration differences between end hosts areof interest to network operators, yet network operators lack a

channel for securely querying the properties of their clients.For instance, hosts configured without firewalls and viruscheckers can launch internal attacks on a protected network,while hosts with untrusted network stacks can monitor net-work traffic or consume excess network capacity. Networkadministrators often resort to expensive manual scans to checktheir clients’ configurations [15]. Similarly, lack of a stan-dard channel to the user leads ISPs to try to communicatewith the user through disruptive mechanisms such as captiveportals and interstitial web pages. These mechanisms arebrittle, complex and can break client applications.

This paper describes NetQuery, a general-purpose chan-nel for distributing and reasoning about the properties of net-work entities. By network entities, we mean both the phys-ical devices, such as routers, switches, and end hosts, andthe virtual entities, such as flows and applications, that com-prise a network. Each network entity is associated with aset of properties, represented in NetQuery as unconstrainedattribute/value pairs. These properties may be intrinsic toa device, such as a router’s routing tables, or they may bearbitrary labels assigned by third parties, such as a certifi-cate from an audit service that a router is properly config-ured. NetQuery implements a distributed, decentralized tu-plestore that stores the attribute/value pairs of network enti-ties. The tuplestore provides a query interface through whichapplications can retrieve the properties of network entitiesand a trigger interface through which applications can de-tect changes to these properties. These interfaces supportthe establishment of long-running guarantees about the net-work: the query interface enables applications to deducethat a guarantee holds for an initial network state, while thetrigger interface enables applications to detect and modify,where applicable, network changes that can invalidate theguarantee.

While NetQuery provides a globally unified interface, theimplementation and deployment of the tuplestore is decen-tralized. Every network operator deploys a NetQuery serverdedicated to storing the attribute/value pairs for its portionof the network and specifies an access policy to these at-tribute/value pairs. NetQuery enables anyone to tag any en-tity with an arbitrary property without the need for a centraladministrator to mediate conflicts.

1

Although the tuplespace may contain conflicting informa-tion from different sources, NetQuery enables applicationsto identify information from sources that they trust. The tu-plespace records for each attribute the principal, such as adevice, system administrator, or audit service, that generatedit. A policy language enables applications to establish trustin the attribute/value pairs stored in NetQuery, and to controlaccess to proprietary information and sensitive operations.

Reasoning about properties of remote network entities onlymakes sense when there is an established basis to trust theirclaims. NetQuery is based on a “clean-slate” design whereall networked components are assumed to be equipped withsecure hardware coprocessors; this trusted hardware servesas a root of trust for claims about the network. Coprocessorssuch as the Trusted Platform Module (TPM) are cheap andbecoming ubiquitous [6]. Such secure hardware makes itpossible to issue unforgeable certificates describing the soft-ware environment and dynamic state of a device through amechanism known as attestation [8]. While our work fo-cuses on translating such local guarantees into network-levelassurance, we also discuss how current systems equippedwith management interfaces such as SNMP but no securecoprocessor might evolve towards supporting NetQuery ser-vices. The base system design supports incremental deploy-ment, though such a deployment necessitates a larger trustedcomputing base (TCB).

Three sample scenarios illustrate the types of applicationsthat NetQuery enables:

• Checking end hosts. Misconfigured end hosts cancompromise the integrity of a network. A NetQuery-enabled network can restrict access based on the con-figuration of end hosts. For instance, before allowinga newly connected end host to send packets, a switchcan verify that the end host is running the latest soft-ware versions and a virus checker.

• Checking paths. Middleboxes that can perform so-phisticated deep packet inspection for large volumesof network traffic enable ISPs to monitor and modifystreams that traverse their network. A privacy con-scious user who wants to know how data from her websessions will be monitored can use NetQuery to dis-cover whether there are entities that can potentially ac-cess and record her sessions and, if applicable, to ob-tain guarantees from the network on how packets arehandled.

• Differentiating providers. Customers currently dif-ferentiate between ISPs based solely on reputation. Net-Query enables ISPs to advertise the performance androbustness features that they provide. For instance, awireless service provider can use NetQuery to adver-tise its backhaul capacity and traffic management tech-niques; clients can use this to select the best availablenetwork.

Figure 1: A simple network and its tuplespacerepresentation. Every network entity has a corre-sponding set of tuples that describes its current config-uration and state. Attribute/value pairs are prependedwith the speaker name.

This paper outlines the case for NetQuery and makes threecontributions. First, it proposes a set of abstractions fordisseminating network properties and outlines a realizabledistributed architecture for storing meta-information aboutnetwork entities. Second, it proposes a representation fornetwork entities and their properties and shows, by way ofexample applications, how analysis over this representationenables novel applications. Finally, it shows that NetQueryscales to the query load from millions of clients and to thevolume of properties needed to represent the devices of alarge network.

This paper is structured as follows. Section 2 presents theNetQuery abstractions and interfaces. Section 3 describeshow applications establish the verity of properties, and howinformation providers control access to properties. Section4 provides example applications. Section 5 describes ourimplementation, and Section 6 reports the performance ofthe system. Section 7 presents related work, and Section 8concludes.

2. THE NETQUERY TUPLESPACEThe central abstraction for disseminating network infor-

mation in NetQuery is that of a tuplespace (Figure 1). Thetuplespace presents to applications a tuple-based representa-tion of the network that reflects its topology and configura-tion and describes every network entity with a correspondingset of tuples that describes its external connections, internalconfiguration, and runtime state. In addition to the basic ac-cess interfaces for storing and retrieving data, the tuplespaceprovides attribution and extensibility for tuples and can del-egate its operation to different administrative domains, thattogether make it well-suited for representing heterogeneousnetworks such as the Internet.

Every NetQuery entity accesses the tuplespace as a prin-cipal. A principal is a user that is granted access privileges tothe tuplespace. Every principal is associated with a uniquepublic/private key pair. This key acts as an identifier for the

2

Figure 2: NetQuery tuplespace delegation. Eventhough NetQuery presents a unified tuplespace, the im-plementation is decentralized across multiple adminis-trative domains and distributed across multiple servers,including those embedded in network components.

principal, without the need to resort to PKI certificates toprovide identities and is used for authentication.

NetQuery provides a single global tuplespace for the In-ternet that is distributed across multiple tuplespace servers.The responsibility for managing subspaces, disjoint subsetsof the tuplespace, is delegated to multiple administrative do-mains (Figure 2). Every AS or third party information ser-vice manages a subspace covering its own network and an-notations. An AS or information service is responsible fordeploying tuplespace servers for storing their local tuplesand is empowered to define an access policy for the tupleswithin their subspace.

2.1 Data model and representationEvery tuple is named by an opaque, globally unique tuple

ID (TID) and stores data as typed attribute/value pairs.NetQuery provides scalar primitives, references, and aggre-gate attribute types. Integers and strings comprise the scalarprimitives. References contain pointers to NetQuery tuples,encoded as tuple IDs, and are used to chain NetQuery tuplesinto complex data structures such as trees or graphs. Aggre-gate types, such as vectors, sets, and tries, are used to repre-sent information of varying cardinality, such as the forward-ing entries used during routing. Applications and devicescreate, read, modify, and delete tuples and attribute/valuepairs using primitive operations exported by the tuplespace.

Tuples conform to schemas, which define the set of at-tribute/value pairs that all tuples for a given kind of networkentity must provide. The schema thus establishes a commonbaseline representation that enables programmers to writeanalysis code over the various network entities. A schemadiffers from a class in object systems in that it describes justthis common set of attribute/value pairs and not the code oroperations. The schema name of an tuple is stored within

Figure 3: NetQuery extensibility and annota-tions. A BGP NetQuery router has extended the baserouter representation with information about its BGPconfiguration. A third-party audit service has anno-tated router X after checking its configuration.

the tuple type attribute of the tuple. NetQuery clients use thisfield to determine how to interpret the attribute/value pairscontained in a tuple.

NetQuery provides built-in schemas for representing pro-cessing entities, which describe the devices or software end-points that generate, receive, or forward packets; and for rep-resenting protocol entities, which describe the higher-levelconversations between processing entities. Every packet isassociated with one or more protocol entities. Processing en-tities include hosts, routers, switches, network cards, and ap-plications. Protocol entities include TCP connections, TCP/UDPendpoints, and IPsec security associations.

These schemas together enable a comprehensive descrip-tion of all network entities that covers their hardware andsoftware configuration. The tuples for processing entitiesdescribe their operation in terms of their functional blocks,such as switching fabrics, network interfaces, network links,and packet filters. Tuples with this generic representationare not restricted to describing only existing network enti-ties and can be used to extend NetQuery with support fornew kinds of network components. The schemas for proto-col constructs describe the protocol state and end points.

Any NetQuery speaker, whether the creator of a tuple or athird party, can extend a tuple with additional attribute/valuepairs, called annotations (Figure 3). For instance, a routerthat implements BGP can annotate the base router tuple withBGP configuration information. A third-party audit servicecan run a configuration analysis tool such rcc [7] to annotatea router with the property

consistent config = true

asserting that it has been configured to propagate consistentroutes.

2.2 Initialization and state maintenanceEvery network device present in the network corresponds

to a tuple. The tuples and their attribute/value pairs can beinitialized and maintained by different parties, including thedevice itself, its administrator, or a third party.

In clean-slate deployments, every NetQuery device ini-tializes its own tuplespace representation and is configuredwith a local NetQuery server. On start up, a device issues

3

create() operations to this server to instantiate the tuplesthat represent it, and issues update() operations to set theattribute/value pairs based on its local configuration and ini-tial state. For instance, a freshly activated router creates tu-ples for all of its interfaces and exports its initial forward-ing table state. A newly connected endpoint creates tuplesfor each of its interfaces and defines attribute/value pairs de-scribing its system statistics such as OS version, network ser-vices provided, or any security guarantees obtainable fromtrusted hardware. Incremental deployments, wherein not alldevices embed NetQuery speakers, use management services,such as SNMP-to-NetQuery proxies, to export the same per-device information.

2.3 QueriesNetQuery applications use the contents of the tuplespace

to derive conclusions about network devices, processes, andthe network itself. In principle, NetQuery applications usethe entire tuplespace state as the basis for these conclusions.In practice, applications perform tuplespace queries to findand retrieve the relevant subset of this state and install trig-gers to detect any network changes that might impact theconclusion. The process of inferring some high-level char-acteristics about the network, such as the route between twohosts, from low-level properties, such as routing tables, iscalled analysis. Together, the combination of queries andtriggers can check a range of properties that varies in breadthof network coverage and duration of validity.

Clients discover tuples of interest by issuing query() tothe tuplespace. query() accepts a tuple pattern as a param-eter and returns the TIDs of tuples that match this pattern.Tuple patterns consist of boolean combinations of predicatesover the contents of a tuple. These predicates specify iden-tity and wildcard matches for attribute values, tuple IDs, andprincipal names. For instance, the query

tuple type = Router AND zip code = 10001 AND tid

= AT&T :: ∗returns the tuple IDs of all routers at a particular geographicregion, restricted to those tuples describing AT&T’s network.NetQuery executes queries on tuplespace servers, enablingclients to efficiently search for tuples without retrieving largeportions of the tuplespace. The pattern language specificallyexcludes operations such as allocation, loop constructs, andrecursion, that can consume unbounded amounts of mem-ory and computation. Hence, queries are safe to execute ontuplespace servers.

Clients issue retrieve() to access tuples whose tuple IDsthey already know. This operation takes a tuple ID and aset of attribute names, specified as an attribute name pattern,and returns the corresponding attribute values from the spec-ified tuple if they exist. The operation returns an error if theattribute/value pair does not exist. A client might acquiretuple IDs for use in retrieve() from the reference-typed at-tribute/value pairs of another tuple, or receive them out ofband.

Applications vary in how they derive properties and in theconsistency guarantees that they need. Some properties de-rive from the value of a single attribute. For instance, a phys-ical link is either wired or wireless and a single query suf-fices to return the relevant attribute. Other properties derivefrom multiple attribute/value pairs. For instance, NetQuerycan determine the route to a destination by iterating throughrouting tables: each iteration retrieves the next hop router forthe next iteration.

When multiple attribute values are requested by an ap-plication, NetQuery returns a recent value for each. Thisconsistency between multiple attribute values is analogousto that provided by sampling the network state with probes,such as by sending packets to perform traceroute or band-width measurement. Applications typically send multiplepackets for a characteristic: for instance, in traceroute, apacket is sent for every hop on a path. Similarly, NetQueryanalyses based on multiple attribute/value pairs generally re-trieve these with a sequence of queries. Probes reveal infor-mation about the network as packets traverses it and thusonly measure a particular slice of the network at a particularpoint of time. Should the network change during probing orattribute retrieval, both the probe- and attribute-based analy-ses will potentially use inputs based on inconsistent views ofthe network, resulting in an analysis result corresponding toa state the network never actually entered. NetQuery’s con-sistency is appropriate for existing applications for which thelevel of consistency provided by sending probe packets suf-fices. NetQuery applications benefit from a programmingmodel that enables them to directly extract, rather than infer,low-level network properties through attribute/value pairs,reserving programmer effort for deriving higher-level char-acteristics.

NetQuery also enables new applications that require strongerconsistency guarantees. NetQuery applications that dependon long-running characteristics can use triggers to react tonetwork changes that might invalidate a characteristic. Forinstance, suppose an access control agent allows LAN accessonly to end hosts that run approved software. To discoverthese hosts, the access control agent issues queries that tra-verse the LAN’s topology graph. If the graph changes dur-ing this traversal, the access control agent can miss new endhosts. These missed hosts cannot connect to the network,since the access control agent will never examine them to au-thorize access. To discover all hosts, the access control agentcan install triggers to detect topology changes and restart theanalysis as needed.

The remainder of this section describes the tuplespace ab-stractions that enable applications to detect, react to, andcontrol network changes; and to perform analyses that spanmultiple attribute/value pairs.

2.3.1 Detecting and controlling network changesChanges to the network state are propagated to the tu-

plespace through corresponding events. These changes can

4

invalidate a property that was previously detected by an ap-plication. NetQuery applications use triggers to monitor,or to maintain as valid, a network characteristic over time.A trigger registers an application’s interest in detecting andsometimes modifying events. In some cases, triggers canwarn applications when a property on which they relied isabout to change. In others, triggers can maintain a propertyon behalf of an application by responding to network eventswith remedial actions. Thus, triggers enables guarantees toclients of the form, “Either characteristic X holds, else no-tify client C.”

A trigger consists of a predicate, a trigger notification type,and the IP and port number of a recipient. The predicateis written in the same pattern language as query() invo-cations and specifies a set of attribute/value pairs that aremonitored for intended updates. Changes that match thepredicate cause the trigger to fire: when a trigger fires, Net-Query sends a notification packet describing the change tothe indicated IP/port; this packet describes which tuple andattribute/value pairs have changed, and has the semantics ofa remote upcall to the trigger recipient. Since time is definedrelative to tuplespace changes, NetQuery servers do not re-quire tightly synchronized clocks.

NetQuery does not run any application-specific code dur-ing trigger processing: it pushes the bulk of custom updateprocessing to clients, leaving the tuplespace responsible onlyfor update detection. This reduces the processing overheadand complexity of NetQuery servers. NetQuery clients canuse triggers to change how NetQuery updates the tuplespacein response to a modification request; thus, NetQuery inter-prets these requests as an intent to update.

The controllability of a change determines what actionscan be taken when a trigger fires. Some changes cannot becontrolled by software or hardware. For example, if a userphysically unplugs a network cable, the resultant changes tothe tuplespace cannot be averted. These changes give riseto advisory events. Often, these real-world changes resultin collateral events that can be controlled. With NetQuery,these can be deferred, or aborted before they take effect.NetQuery exposes to applications this control over eventsand network devices. These are delayable or vetoable. Forexample, while NetQuery has no control over when a userplugs a computer into a switch, the switch does not need toimmediately provide full network connectivity to the com-puter. Enabling full network connectivity is both delayableand vetoable, provided the switch can be instructed to re-strict connectivity. Networks can enforce policy by defer-ring such events so that they can take action before the eventoccurs. A topology management application can delay anend host’s request to attach to a switch so that it can firstreconfigure the switch to place the new end host within thecorrect VLAN. Similarly, a firewall can expose insertion ofnew TCP firewall connection table entries for new flows asa vetoable event. An access control agent can intercept suchupdates to verify that a new flow is allowed by local policy.

Applications can detect events to react to network changes.Suppose an application uses a particular route because thatroute exhibits a desired characteristic. A routing table changemight alter the route, changing this characteristic. For in-stance, applications that use NetQuery to discover high-capacityroutes for improving throughput, can use advisory eventsto adjust its optimization strategy, while applications thatuse NetQuery to discover routes with stronger confidential-ity guarantees, such as those that lack unencrypted wireless,can use delayable events give the application the opportunityto check that the new route is acceptable before switching toit.

Some events are potentially controllable but difficult tomake so under current device architectures. For example,forwarding table updates are fully under software or hard-ware control, and it is straightforward to make them de-layable, because delay can be injected with largely localcode changes. However, making such events vetoable re-quires error handling code to react to vetoed updates.

The notification type determines what a recipient can doin response to the intended update. There are three types,ask-veto, ask-delay, and FYI ; ask-veto allows the recipientto cancel the update, ask-delay allows the recipient to de-fer the update from taking effect, up to a maximum delay,while it takes corrective action, and FYI (for your informa-tion) serves as a one-way notification for a change in the net-work. In FYI notifications, information is only pushed to therecipient, while for ask-veto and ask-delay, the server waitsfor the recipient to respond with whether to veto or delay theupdate.

NetQuery servers mediate between tuplespace triggers andthe network by forwarding a veto or delay to the affectednetwork devices. Thus, vetoes or delays result in canceledor delayed changes to device state, and hence behavior. Theallowable notification types vary, depending on the type ofevent that generated the intended tuplespace update and onthe policy of the affected network entity. Vetoable eventssupport all trigger types, delayable events support ask-delayand FYI, and advisory events support only FYI.

2.4 Tuplespace access controlExporting information about network entities can cause

privacy violations, while allowing triggers to exert controlover events can allow malicious users to cause a network tomalfunction. For instance, network operators might want toprotect proprietary information by restricting who can ac-cess information about their internal networks, and preventuntrusted users from disrupting network connectivity by re-stricting ask-veto or ask-delay triggers on routing updates.Similarly, describing an end host’s software configurationcan reveal private activities about a user. To address theseprivacy and safety concerns, NetQuery associates every at-tribute/value pair with a policy provided by its creator thatspecifies which NetQuery clients are allowed to read, query,and install triggers on that attribute/ value pair (Section 3).

5

2.5 NetQuery servers and access protocolNetQuery servers export a tuplespace interface to Net-

Query clients and speakers. Every NetQuery principal isassociated with an administrative domain, whose tuplespaceservers process queries and store tuples and attribute/value-pairs on behalf of its local principals. Requests that referencea tuple or attribute/value-pair are transparently routed to thetuplespace server that stores it; this is achieved by storingwithin every tuple ID and attribute name a pointer to this“home” server, as identified by the administrative domain’sname and the tuplespace server’s IP and port. The remainderof the TID is typically a unique, opaque byte string.

Some queries, such as those that use wildcard patterns,require accessing multiple attribute/value pairs from a tuple.Since the storage location of each attribute/value pair mayreside in tuplespace servers operated by different administra-tive domains, multiple servers may need to be contacted toprocess such queries. Such queries are sent to the server thatstores the tuple, which is responsible for generating a queryexecution plan. To determine which other servers to contact,the server stores a forwarding pointer to every remote tu-plespace server that stores one or more attribute/value pairsfor the tuple.

A tuplespace server may be embedded within a networkcomponent, such as a router, that exports network informa-tion, or it can operate as a discrete server in its own right.Tuplespace servers employ lease-based storage managementfor tuples and periodically purge tuples that have not recentlybeen touched.

3. POLICIES FOR TRUST ANDACCESS CONTROL

Managing the diversity of trust relationships is a centralchallenge in disseminating network properties in a heteroge-neous network. Applications and information providers canspecify policies to control how information is (1) importedand (2) exported across trust boundaries, and (3) to controlaccess to potentially dangerous tuplespace operations.

These three different contexts all share a common pol-icy language establishing trust in network entities. In Net-Query, every network entity is associated with a principal.Every utterance, or attribute/value pair or tuplespace opera-tion issued by an entity, is attributed to that entity’s principal.The principal to which an attribute/value pair is attributed isstored in the tuplespace along with the value; this attributioninformation is returned along with the value on retrieval. InNetQuery, principals are bound to a private keys. In princi-ple, every utterance is signed by the key corresponding to thespeaker; in practice, these keys are used to establish securechannels that serve as efficient surrogates for signatures.

Every NetQuery policy evaluation reduces to determiningwhether an entity trusts some utterance from a principal. Forinstance, suppose an end host wants to characterize a route.This analysis traces through the tuplespace for routing in-

formation. The accuracy of this conclusion is contingent onthe accurate of the inputs: if some analysis were to rely on arouting table tuple that was exported from a malicious router,and that router did not faithfully copy its real routing table tothe tuplespace, then the inferred route could differ from thereal route. To protect against this, the end host would spec-ify import policy that accepts only information from trustedrouters, such as those equipped with TPMs.

A NetQuery policy is a set of statements that describesa set of trusted utterances. Policy statements are based ondelegation. In delegation, a principal A empowers anotherprincipal B to make statements on its behalf. Delegations areexpressed in policy statements of the form

A says B speaksfor A on subjects

The subjects clause is a pattern that describes the scope ofthe delegation. These patterns are expressed in terms of tu-ple IDs, attribute names, and operations, and are matchedagainst utterances. The NetQuery policy language supportsrestrictions based on a tuple’s schema, owner, and underly-ing device. Delegation restrictions add a layer of indirectionthat improves convenience and efficiency while preservingsecurity. Delegation is transitive: a principal that is empow-ered to make statements about some subject is also empow-ered to delegate this right to others. NetQuery’s policy se-mantics allows only a narrowing of subjects on each dele-gation, thus preventing principals from using delegation toescalate their privilege.

Consider the import policy for a route characterization ap-plication Client. With an empty import policy, Clientonly trusts statements from itself. To trust statements from arouter R0, Client specifies the import policy

Client says R0 speaksfor Client on

Router . ∗ [router id = R0 ]

Rather than directly vet every router in the world, whichis cumbersome, inefficient, and unrealistic, the client insteaddelegates the responsibility to the manufacturer.

(1) Client says Cisco speaksfor Client on Router .∗

The first statement asserts that the client trusts Cisco tomake statements about routers. To empower a router to makestatements about itself, Cisco ships it with a unique pub-lic/private key pair X and publishes the delegation certificate,which restricts the router to only making statements about it-self.

(2) Cisco says X speaksfor Cisco on

Router . ∗ [router id = X ]

When the client retrieves a routing table Router .routing tableattributed to X, the combination of statements (1) and (2)prove that this routing table is trusted.

6

3.1 Evaluating policiesPolicies are not necessarily static: they can depend on

changes to the network. For instance, the set of routers thatis trusted can grow as new routers are installed. Thus, it isnot possible to include all statements such as (2) in a policy,since doing so requires enumerating every possible router.

NetQuery provides mechanisms for publishing and dis-covering such statements. Statements are published to thetuplespace by the issuing principal like any other property.When a client checks a policy, it locates the relevant state-ments to use as credentials. There are several possible mech-anisms for doing so. For instance, a proof checker can auto-matically search the tuplespace for the necessary credentials.While this is convenient for the programmer, this search canbe costly to execute. Often, policies follow an establishedpattern, wherein all necessary credentials are in well-knownattribute/value pairs. For the router policy, every router storesthe delegation statement issued by Cisco in the attribute/valuepair Router.credentials. Thus, a general search is not needed.

NetQuery supports both of these mechanisms. NetQueryprovides a schema for exporting policy statements as tuples.Proof checkers can use tuple queries to search for relevantstatements, just as they do for other tuples. The metadatastored with a tuple or attached to a tuplespace operation em-beds credentials supplied by the originating principal; thesecredentials are passed to the recipient.

3.2 Increasing policy expressiveness withtrusted computing

Entities built using trusted computing primitives can ex-port detailed and trustworthy descriptions of themselves tothe tuplespace. TPMs form the basis for establishing trust inthese self-reported properties. TPMs generate certificates,called attestations, that describe a chain of trust, rooted ata TPM, where each link represents a layer of hardware orsoftware that is responsible for identifying and vetting thenext layer. For instance, the TPM vets the trusted OS, whichin turn asserts that a process with a given hash and config-uration is running. TPMs provide a Quote operation thatcryptographically binds a message to an attestation. Trustedoperating systems can use quote to export detailed informa-tion about its processes that can be trusted by remote hosts.Quoted information is translated to NetQuery attribute/valuepairs by deriving the principal from the attestation and theattribute/value pair from the message.

This information expands the expressiveness of policies.For instance, rather than identifying trusted entities by prin-cipal name, policies can instead identify trusted entities bytheir properties.

Consider the tuple representation of a TPM-equipped hostrunning a trusted Linux and a software router (Figure 4):

(1) The trustworthiness of the host hardware platform tomake statements about the OS is established by the boot-time attestation, stored in the Host.credentials field. The trust-worthiness of the OS to make statements about its processes

(1) TPMi.Linuxj = 〈[TPMi, tuple type = Host],

[TPMi, credentials = TPM HW platform attestation],

[TPMi, hash = TrustedLinuxv2.6],

[TPMi.Linuxj, process = TPMi.Linuxj.SoftRouterk]〉(2) TPMi.Linuxj.SoftRouterk = 〈

[TPMi.Linuxj, tuple type = Process],

[TPMi.Linuxj, hash = Routerv1.0],

[TPMi.Linuxj, allow only = CfgRTNetlink],

// allow only of all other processes exclude CfgRTNetlink

〉(3) R1 = 〈

[TPMi.Linuxj.SoftRouterk, tuple type = Router],

[TPMi.Linuxj.SoftRouterk, routing table =

{10.0.0.0/8 ⇒ eth0, . . .}]〉

Figure 4: Tuple representation for TPM-equipped software router. Per-attribute/value paircredentials are omitted.

is established by Host.hash. (2) Even though the client truststhe Routerv1.0 software to properly report routing informa-tion, checking the hash is not sufficient. Under Linux, anysufficiently privileged processes can modify the kernel rout-ing table by invoking the RTNetlink kernel interface, caus-ing the device state to diverge from the tuple representation.Thus, the import policy for router information (3) verifiesthat (a) the router process has the right hash (Process.hash)and (b) only the router process, and no other process, caninvoke RTNetlink (Process.allow only of all processes).

3.3 SummaryFigure 5 summarizes how policies are integrated into the

NetQuery interface and data model.

3.3.1 Import policiesClient applications check import policies for all inputs re-

ceived from the tuplespace (Figure 5-B.1). These checkscover both values retrieved from the tuplespace and triggernotification upcalls, which deliver information about valuechanges in attribute/value pairs. Import policies are typicallyspecified by the application developer, but can be modifiedby users according to local requirements. For instance, a sys-tem administrator might relax policies to trust all statementsmade by routers in the intranet.

3.3.2 Export policiesThe tuplespace server checks all operations that can re-

turn tuples and attribute/value pairs against export policies(Figure 5-B.2). In addition to query and retrieve, which im-mediately return information, trigger notification upcalls arealso checked, since they return information in response tonetwork changes. Export policies can be specified for every

7

Figure 5: Complete NetQuery architecture. (A)Extensions to data model to support attribution andcredentials. (B) Location of policy enforcement: (B.1)Import policies. (B.2) Export policies. (B.3) Modifica-tion policies.

attribute/value pair, and are typically specified by the user.For instance, an ISP that wants to protect proprietary infor-mation from escaping its network would allow NetQuery ac-cess only from entities residing on its own network.

3.3.3 Modification policiesThe tuplespace server checks against modification poli-

cies any operations, such as ask-delay and ask-veto triggerinstallation and notification upcalls, that can modify the be-havior of the network (Figure 5-B.3). Export policies canbe specified for every attribute/value pair, and are typicallyspecified by the user. For instance, system administratorswill typically restrict ask-delay and ask-veto triggers for sharednetwork devices, such as switches and routers, to only thosetrusted hosts used for network management and monitoring.

4. APPLICATIONSWe present examples of how the NetQuery programming

model enables new functionality based on reasoning aboutnetwork and client properties.

4.1 Networks checking end hostsNetQuery enables networks to make decisions based on

the properties of end hosts. Network administrators often re-strict network access to hosts meeting criteria based on suchproperties. For instance, our campus network administra-tors require end hosts to defend themselves and the networkagainst known worms by requiring Windows installations tobe up-to-date with patches and to run a virus checker. These

requirements are only loosely enforced: compliance is mon-itored only by a web page where the user checks a checkbox.

NetQuery enables the network to securely query end hostsfor the relevant properties. End hosts export tuples describ-ing their operating system, configuration of the network stack,and active processes. Network switches query this informa-tion when checking whether a new host is authorized to at-tach to the network.

Network switches specify an import policy that ensuresthat this information is genuine. The import policy requiresthe end hosts to back up their claims with execution cer-tificates, as derived from TPM attestations. These certifi-cates assert that the host is running a operating system thatis trusted to make statements concerning the set of runningprocesses. To prevent process-level spoofing, the operatingsystem describes the identifying properties of each process,such as a cryptographic hash of its program image.

Switches initially restrict new hosts to only send and re-ceive packets for NetQuery. When a host first attaches toa switch, it exports its tuples to the local NetQuery server.Upon detecting a new host, a switch checks the host’s tuplesto determine the list of processes:

NewHost says NewHost.processes = [

{HashLabel = VirusChecker}, {HashLabel = Firefox3.0}, . . .

]

If the host runs the required software, then the switch al-lows it full network access. On doing so, the switch installsa trigger on the process list to detect changes that would in-dicate a violation of the policy. For instance, the trigger willfire if the virus checker stops running, notifying the switchto revoke the host’s network access.

4.2 End hosts checking networksService providers often compete solely on price, because

the differences between a well-operated, well-provisionednetwork and a poorly-run competitor are mostly hidden fromclients when each offers the same IP forwarding interface.Compared with existing out-of-band mechanisms, exchang-ing information about differences through NetQuery is morerobust and efficient. NetQuery facilitates the use of a ser-vice description language that enables service providers todifferentiate their service by directly advertising their per-formance characteristics and empowers clients to exchangeinformation about service providers. Clients can use infor-mation self-reported by an ISP or reported by other clientsin making service selection decisions.

A high-performance ISP can advertise this fact by describ-ing how its network is operated. For instance, a wireless ISPmight perform traffic shaping or QoS to ensure that each endhost is allocated a fair share of resources. Access points ex-port statements such as

APi says wlan0.traffic shaping = true

to specify that the features are active on its wireless interface.

8

To improve efficiency and to hide proprietary details fromend hosts, the ISP can use a third party analysis service toassert that its network is well-run. Such services query thetuplespace for an ISP’s topology information to determinehow it is expected to perform at a particular access point andannotates the ISP with the result:

TPMi.Linuxj.H323 Certifierk says

GoodForVoIP(ISP.HW MAC#00 : 00 : 00 : 11 : 11 : 11)

Users can also report information about an ISP. For in-stance, a VoIP application can report a summary of the de-livered quality of service:

CiscoATAi says GoodForVoIP(10.1.2.55)

Clients use import policies to rely on only trustworthystatements during service selection. By attributing the state-ments to trustworthy providers or trusted software, NetQueryprevents malicious ISPs or users from injecting inaccurateinformation. The H.323 certifier is trusted to make accuratestatements about a network’s ability to support the H.323telephony protocol: it only bases its analysis on tuples fromtrustworthy devices. Similarly, Cisco ATA hardware is trustedto make statements based on actual observations.

4.3 Between networksWhen two networks are interconnected, network adminis-

trators and system designers often concerned about the prop-erties of each network. For instance, ASes generally requiretheir peers to be reliable and behave consistently [2].

Using NetQuery to exchange information between ASesimproves security and robustness. NetQuery enables ASesto perform static analysis on the configuration and topologyof their peers. If this analysis shows that a peer is properlyconfigured, an AS can accept routes from that peer with-out adverse performance impacts. For instance, an analy-sis can use standard techniques to verify that the peer net-work is robust to single-node failures and propagates onlyvalid routes. Such analyses operate on tuplespace encodingsof router configuration files, as expressed in the normalizedform from rcc:

Router1 says Router1.sessions = [

{ neighbor = 10.1.2.3,

importPolicy = [

{ permit = allow, ASregex = ∧65000, . . . },

. . . . . .

ASes that enter mutual transit backup relationships canuse NetQuery to verify that backup routes are used onlywhen alternate routes are not available. These analyses areimplemented using dynamic checking: triggers detect routewithdrawals and link failures by monitoring received adver-tisements and link status:

EdgeRouter1 says EdgeRouter.validAdjRibIn = [

{nextHop = eth0, prefix = 10.1.2.0/24,

ASPath = (22 1 76)},. . . ]

EdgeRouter1 says EdgeRouter.eth0.status = UP

Changes such as EdgeRouter1.eth0.status transitioning to DOWNserve as evidence of failures that can authorize the peer totransmit packets through the backup ISP.

As in preceding examples, the analysis software is sup-plied by a third party and it executes on a trustworthy plat-form under the control of the ISP. The verity of the state-ments contained in the preceding tuples stems from the useof trusted routing platforms. Other import policies can beemployed to support legacy routers. For instance, the analy-sis service can use statements from a trustworthy scraper ap-plication that extracts configuration data through router CLI.The scraper only executes on behalf of ISPs that are trustedto operate routers that respond correctly to the scraper’s CLIqueries.

4.4 Extending functionality with triggersNetQuery triggers allow the interposition of arbitrary code

on network operations. By combining triggers with analy-sis of end host information, NetQuery clients can extend thenetwork with new functionality without modification to theunderlying NetQuery devices.

For instance, a NetQuery switch can allow NetQuery clientsto manage access to the network by providing ask-veto ac-cess to port status changes. This approach provides moreflexibility than embedding a policy within the device (Sec-tion 4.1). NACAgent, a NetQuery application that enforces aprocess list-based access policy, uses the trigger

CHANGE(Switch.ports[∗].status)

to detect when a port connects to a new host. Upon detectinga new host, the switch proposes the change:

Switch says

PROPOSE CHANGE(Switch.ports[0].status = UP)

The switch does not fully activate the port until after thetuplespace accepts the proposed update, which occurs afterall matching triggers are delivered and acknowledged. Uponreceiving the trigger, NACAgent analyzes the new host’s tu-ples to determine whether it is running an approved set ofprocesses and accordingly signals acceptance or veto in theupdate’s acknowledgment. The host is subsequently allowedto send packets to the network.

A network administrator who wants to restrict potential at-tacks through unsecured wireless links can modify NACAgentto verify that all hosts have deactivated their wireless NICsbefore connecting to the network. This NACAgent′ inspectsthe host’s tuplespace description of its network stack:

NewHost says NewHost.wifi0.status = DISABLED

9

NACAgent′ allows access once it verifies that all wirelesscards are disabled. As with the process list policy, NACAgent′

adds triggers on the status attribute/value pairs to detect fu-ture enablement of the wireless card; upon receiving suchchanges, NACAgent′ revokes the host’s access to the net-work.

5. IMPLEMENTATIONWe have implemented NetQuery on real network hard-

ware and software. The core functionality is composed of astand-alone tuplespace server and a client library for writingNetQuery devices and applications. We have implemented aNetQuery switch as a Linux user process, a NetQuery routeradapted from the Quagga router, a NetQuery host that runsNexus and exports attested information about its processesand network stack, and applications that rely on the exportedproperties from these devices.

The tuplespace server, while largely unoptimized, is highlyscalable. Since the tuplespace stores representation of net-work state that already exists elsewhere in the network, al-beit not in a format that enables trustworthy disseminationand analysis, the tuplespace server does not need to pro-vide durability. In the event of server failure, the originalcopy of the network state resides within network devices; torecover from failure, the tuplespace servers instruct the de-vices to store anew its full tuplespace representation. Thus, asimple in-memory implementation suffices and can providehigh performance without compromising correctness. Theserver implements the distributed tuplespace access proto-col, allowing us to manage the load from large test networksby spreading it across multiple tuplespace servers.

The tuplespace server supports the complete set of attributetypes and operations. The tuplespace supports a basic triggerpredicate and query language. We are extending the predi-cate, query, and indexing functionality as needed by our ap-plications. Currently, patterns are specified by exact or wild-card match on a single tuple ID and attribute name. Exactmatch triggers are used during analysis to detect concurrentchanges that might invalidate the analysis. For convenienceof programming, applications can configure the library toautomatically generate such triggers during all reads. Weintend to use wildcard matches to build rendezvous mecha-nism for discovering network devices and invoking networkservices.

Our work on applications and devices has highlighted theimportance of several optimizations. Special indexes forwidely-used network state are essential to achieving goodperformance. For instance, efficiently exporting and explor-ing the IP level connectivity of a network requires good up-date and lookup performance for forwarding tables. To achievethis in a compact representation, we imported the Linux im-plementation of LPC-trie. Although NetQuery, by design,enables clients to establish trust without the assistance ofnetwork administrators, trusted tuplespace servers can sig-nificantly reduce computational overhead. By trusting a tu-

plespace server to preserve the integrity of stored statementsand to accurately report the speaker of a statement, a Net-Query client does not need to generate and verify digital cer-tificates on every statement exported to and imported fromthe tuplespace.

5.1 NetQuery devicesThe NetQuery switch provides features that enable the

derivation of strong network guarantees. The switch exportsits internal state as attribute/value pairs that support ask-vetotriggers. This provides applications with a great degree ofcontrol over network topology, including what devices areattached. The NetQuery switch provides similar function-ality to wireless 802.1x to preserve the accuracy of the re-ported topology. Peer devices are authenticated before theyare allowed to connect to the network. Once devices areconnected, transparent encryption can be activated to pre-serve link-layer integrity. These mechanisms enable Net-Query clients to trust that adversaries cannot compromisethe accuracy of the tuplespace by subverting the underlyinglink layer assumptions.

The NetQuery router is Linux-based and derived from Quagga.Only localized changes to the control plane were necessaryto convert all interface and routing table changes into Net-Query attribute/value pairs that support ask-delay triggers.NetQuery-Quagga interposes on Quagga’s interface tortnetlink, Linux’s low-level interface to its in-kernel dat-aplane, and exports all relevant requests, such as changes tothe forwarding table and NIC state, to the tuplespace. In to-tal, only 777 lines of localized changes were needed, out ofa total code base of 190,538 LOC.

The NetQuery switch and router export connectivity infor-mation about the NetQuery-enabled peers that are attachedto their local interfaces. Exporting connectivity informa-tion enables applications to explore the network by crawl-ing its tuple representation. Unlike the NetQuery-Switch,NetQuery-Quagga does not support veto triggers; doing sowould have required extensive changes to Quagga.

5.2 NetQuery applicationsWe have built several applications that use the tuplespace

to extend the functionality of these devices and to provideapplications with guarantees about the network. We usedtriggers to extend the NetQuery switch with a custom net-work access control (NAC) policy, described in Section 4.4,that allows access only to Nexus hosts that are running viruscheckers. We implemented a generic route characterizationanalysis, which determines the route between a source anddestination, and detects any changes that can modify theroute. This analysis is a building block for more complexanalyses of route characteristics; for instance, an applicationcan check whether a route respects user privacy by checkingthat none of the entities on the route perform packet inspec-tion.

10

100000 200000 300000 400000 500000 600000 700000

100000 300000 500000 700000

Ope

ratio

n th

roug

hput

(ops

/s)

Tuplespace size(# of tuples + # of attributes)

Read_Attribute()Update_Attribute()

Create_Tuple()Create_Attribute()

Delete_Tuple()Delete_Attribute()

100

120

140

160

180

200

100000 300000 500000 700000

Ope

ratio

n la

tenc

y(u

s)

Tuplespace size(# of tuples + # of attributes)

Read_Attribute()Create_Tuple()

Figure 6: NetQuery operation throughput andlatency versus tuplespace size. The performance ofall operations is independent of the tuplespace size.

6. EVALUATIONNetQuery performs well for both ISPs and applications: it

can be integrated into network devices with little overhead,efficiently deployed on large networks and provide strongguarantees and good throughput for applications. All ex-periments used a testbed consisting of Linux 2.6.23 hostsequipped with 8-core 2.5GHz Intel Xeon processors and con-nected over a Gigabit Ethernet switch. Cryptography is dis-abled in all experiments.

6.1 MicrobenchmarksWe show through microbenchmarks that NetQuery can

support large tuplestore and immensely high tuple creation,modification, and deletion rates. Thus, the unoptimized pro-totype is capable of supporting the peak needs of an ISP. Themicrobenchmark was distributed across sixteen NetQueryclient processes, running on a single machine with eight cores,and eight tuplespace server processes, running on a single,separate machine with eight cores. In the throughput exper-iment, the microbenchmark issued a sequence of pipelinedrequests, without waiting for responses from the server untilall were issued (Figures 6). No forwarding pointers were fol-lowed for these operations, since every attribute/value pairresided on the same server as its tuple.

With the exception of Delete Attribute(), the perfor-mance of all tuplespace operations is decoupled from thesize of the tuplespace. For smaller tuplespace sizes, theDelete Attribute() experiments were of such short du-ration that initialization overheads dominated the overall ex-ecution time. The reduced throughput of Delete Tuple()

0

100

200

300

400

500

600

100% at 0 ms 100% at 10 ms 50% at 10 ms50% at 100 ms

100% at 100 ms

Upd

ate

com

plet

ion

time

(ms)

NO TRIGGER

FYIASK-DELAY

ASK-DELAY+ASK-VETONO TRIGGER

Figure 7: Trigger-induced update completiontime overhead. The types of trigger and maximumserver to client RTT, rather than the number of trig-gers, dominate trigger overhead.

was due to the initial tuplespace state for this experiment,wherein every tuple was initialized with nine attribute/valuepairs. Deleting a tuple also entailed deleting its attribute/valuepairs.

The latency experiments used one server and client andonly measured operations where the client necessarily waitsfor data. For instance, Create Tuple() and ReadAttribute() returned the tuple ID of the new tuple and at-tribute value, respectively.

The latency of a tuplespace operation can also depend onany induced trigger notifications; such operations may re-quire the server to wait for a response from a trigger recipi-ent. Sending trigger notifications in parallel limits such im-pacts. Figure 7 shows the trigger-induced delays to an at-tribute update under trigger workloads that varied by triggertype and WAN delay between the tuplespace server and trig-ger recipients. RTT and trigger types were the primary de-terminants for delay. Since trigger notifications were sent inparallel, the overall completion time for the update time wasdominated by the latency of the most expensive notification,rather than the total number of trigger notifications. This ex-periment shows that NetQuery can support a large number ofWAN end hosts that use FYI triggers. While ask-delay andask-veto is less scalable, these are intended to support man-agement and control applications, which typically run on asmall number of hosts.

6.2 Performance on real-world tracesTo demonstrate that NetQuery has low device overhead

and efficiently utilizes server resources, we measured theNetQuery overheads associated with reflecting device statechanges to the tuplespace and the corresponding overheadfor tuplespace servers. Our exemplar device was a NetQueryQuagga BGP router and was subjected to a realistic work-load, derived from a RouteViews trace. The Quagga routerand tuplespace servers were executed on separate machines.The workload generator converted the RouteViews data intoa BGP stream and ran on the same machine as the router.

11

To demonstrate that NetQuery can efficiently import bulkdata from NetQuery devices, we measured the completiontime for router initialization and the tuplespace memory foot-print of a router. We used RouteViews RIB traces to generatea full BGP routing table transfer (268K prefixes) that fullyinitialized a router’s forwarding table; such transfers wereexceptionally intensive for both the router and tuplespaceserver. Upon receiving the transfer, the BGP router down-loads a full forwarding table to the IP forwarding layer. Thestandard BGP router completed in 5.70 s, while the Net-Query BGP router completed in 13.5 s.

To demonstrate that NetQuery routers perform well in stea-dy state, we evaluated the router against a workload derivedfrom RouteViews update traces. The workload generatorsent updates in 1 second batches. The experiment recordedthe time needed to completely commit the resulting changesto the IP forwarding tables and to the tuplespace. We mea-sured NetQuery’s performance on a total of 30 hours of up-date traces, composed of three hour traces from 10 randomlyselected source routers.

The standard Quagga server had a median completion timeof 62.2 ms, while the NetQuery Quagga server had a me-dian completion time of 63.4 ms. Thus, the NetQuery routercan react to network changes almost as quickly as a standardrouter, minimizing the disruption to local forwarding.

The overhead introduced by NetQuery does not increaseconvergence time even under conservative router implemen-tation assumptions. Suppose the overhead is in the criticalpath of BGP update propagation, either because RIB infor-mation is to be exported, or because the FIB update blocksBGP update propagation. Since BGP convergence is gov-erned by mandatory artificial delays to BGP update propaga-tion, NetQuery’s overheads do not impact convergence time.For instance, BGP’s MRAI timer delays the propagation ofall BGP updates by 30 seconds, sufficiently long to hide Net-Query’s initialization and steady state overheads. Even theinitialization overhead, the largest possible forwarding tablechange, is shorter.

The full 268K entry forwarding table consumed only 10.7MB of server memory. The unoptimized NetQuery wire pro-tocol required 62.8MB to transmit the full table at initializa-tion time. NetQuery required only 3 KB, 92 KB, and 480 KBto transmit the median, mean, and maximum update sizesseen during steady state.

Performance within ISP networks.Here, we show through analysis and experiments that a

small number of NetQuery servers can collect tuplespace up-dates from every router in a POP without degrading perfor-mance.

We can see this by using the results from the previousexperiment to estimate the necessary resources to deployNetQuery. The largest POP in the Sprint RocketFuel topol-ogy consists of 66 routers [19]. Storing the full forward-ing tables of all routers required 0.66 GB of memory, easily

fitting within the 16 GB memory capacity of our test ma-chine. Since tuplespace servers are local to the POP, theburst of traffic during initialization traverses high capacitylocal links. Assuming a conservative lower bound of 1 s be-tween routing updates, the median bandwidth utilization isless than 198 KB/s, or less than 0.2% of a gigabit link.

To determine the number of servers needed to process tu-plespace updates without delaying BGP convergence, con-sider the processing bottlenecks. The MRAI timer sets aconstraint of 30 s on completion time. In this worst case, allrouters are reloading at the same time, ensuring that load-balancing is close to ideal. For 66 routers and initializationoverhead of 13.5 s, 30 cores, or four machines, are neededto meet this deadline. These constraints can be relaxed if theISP and users are willing to make tradeoffs during periods ofhigh network disruption. For instance, high availability ap-plications might forgo guarantees in exchange for minimiz-ing downtime, while security conscious applications mightaccept increased downtime in exchange for preserving guar-antees.

To validate these results, we ran an experiment to mea-sure the aggregate route update overhead induced by routerswithin a real ISP topology. We used the Sprint Rocket-Fuel topology, which consists of 17,163 Sprint and customeredge routers. Although the resulting forwarding table aresmaller, IGP deployments can have stricter deadline con-straints: since routing instability is easier to manage within asingle AS, routers are often configured to propagate updatesas quickly as possible to minimize convergence time. Theworkload consisted of forwarding table updates for internalroutes, generated through simulation of hierarchical shortestpath routing. The resulting trace for a single POP was fed toa real tuplespace server over a Gigabit Ethernet link.

We measured for each POP the completion time of ini-tialization and of updates resulting from a set of correlatedlink failures, with the failure rate varying between 0.01 and0.09. For the five largest POPs, each consisting of 51 to 66routers, the median initialization time for loading all rout-ing table entries for every router was 13.3 s. For failurerates of up to 0.05, the mean and median update times ofthe POP were less than 0.24 s and 0.14 s, respectively; theselocal delays to forwarding table updates are negligible inmost networks. This delay does not impact global routingconvergence for typical link-state routing protocols such asOSPF. Such delays typically result if flooding is delayed. Ingeneral, routing protocol implementations decouple flood-ing from RIB/FIB calculations. Since NetQuery only hooksonto the critical path of forwarding table updates, no delayis introduced to the critical path for flooding.

6.3 MacrobenchmarksTo determine the cost of using NetQuery to establish guar-

antees, we measured the performance of the route charac-terization analysis, which is used by several applications toestablish path guarantees. We ran the query on a set of Net-

12

20

40

60

80

100

120

1 2 3 4 160

320

480

640

800

960B

ytes

sen

t/rec

eive

d (K

B)

Que

ry r

ate

(Que

rys

/ s)

Path length (# hops)

Sim. traceroute, kB sent/receivedSim. traceroute, Querys / s

Figure 8: Route characterization analysis. Thenetwork and processing overhead increase linearly withpath length, as does completion time.

Query Quagga servers arranged in a line topology and con-nected to a single tuplespace server process. The query costand execution time increases linearly with path length (Fig-ure 8). Route characterizations enables applications to de-termine path properties in a trustworthy fashion. Althoughstandard traceroute is less expensive than route characteriza-tion, it is not capable of supporting such applications.

7. RELATED WORKNetwork design has shifted towards using a logically cen-

tralized server to make control decisions on behalf of everydevice in a network, rather than distributing control acrossmultiple devices [5, 9, 3]. Similarly, NetQuery centrally col-lects network properties to isolate devices from the perfor-mance impact of queries and triggers.

Perhaps the closest work to NetQuery are network excep-tion handlers (NEH) and NOX, which are logically central-ized systems for enterprise networks that collect networkproperties and disseminate it to administrative applications[14, 10]. NEH collects dynamic topology, load statistics,and link costs from the network infrastructure, and exposesthese properties so that end hosts can detect and react toexceptional network conditions. NOX collects informationabout topology from infrastructure devices and user/servicebindings concerning end hosts and exposes this informationto network management applications running on a centralcontroller. NetQuery supports these applications for enter-prise networks, while also supporting applications that is-sue queries that span multiple ASes. NetQuery’s tuplespacesupports heterogeneous information sources by tracking thesource of every statement and by leveraging trusted hard-ware. While NEH and NOX are intended for applicationsspecified by the network administrator, NetQuery enablesany user application to query the network.

Secure Network Datalog (SeNDlog) is a declarative pro-gramming system for building distributed applications onuntrusted networks [22]. With declarative programming sys-tems, distributed applications are written as high-level rules,which are compiled to query execution plans that are exe-cuted at every node. SeNDlog extends declarative program-ming with logic-based access control. As with NetQuery,

SeNDlog uses says as the basis for decentralized autho-rization policies.

Network management systems are widely deployed on theInternet. Intradomain management protocols such as SNMPand CMIS/CMIP provide standard schemas and protocolsfor disseminating information about network devices [11,13]. NetQuery can be retrofitted on top of existing SNMPand CMIS/CMIP interfaces. RPSL is the standard languagefor exporting information about ASes across administrativeboundaries [1]. This information is disseminated throughthe Internet Routing Registry (IRR), a globally distributeddatabase for RPSL statements. Although network analyseshave been designed using this system [18], the questionableaccuracy of IRR information, which stems from inconsistentupdate mechanisms across the ISPs that contribute informa-tion, limits the guarantees that applications can derive fromsuch analysis. NetQuery’s rich policy language enables ap-plications to reason about the verity of tuplespace informa-tion.

Trusted Network Connect (TNC) is an emerging standardfor network access control of end hosts [21]. Client autho-rization in TNC is similar to that of NetQuery. A networkadministrator specifies a network access policy to restrict ac-cess to the network. Before an end host is allowed to join thenetwork, TNC verifies that its software and hardware config-uration satisfies the policy. Unlike NetQuery, TNC does notprovide a channel by which the end host can discover theproperties of the network before deciding to connect.

Internet measurements are used to derive properties of thenetwork such as network topology or predicted network per-formance [19, 20, 16]. In addition to providing informationfor these inference tools, NetQuery can serve as a mediumfor network information service providers to publish the re-sults.

8. CONCLUSIONNetQuery provides a new channel for disseminating the

properties of network entities. The architecture of the sys-tem is well-suited to the scale of the Internet and supportsthe hardware provisioning, confidentiality policies, and trustrelationships found in heterogeneous networks. It is admin-istratively decentralized, provides a flexible tuple-based datamodel, and supports safe analysis. These features reduce thebarrier to entry for new kinds of network information ser-vices and applications, and facilitates incremental deploy-ment in legacy networks. We have experimentally shownthat NetQuery performs well on real routers, and can sup-port the volume of network state found in large enterpriseand ISP networks. NetQuery provides a general program-ming model that appropriate for a diverse range of applica-tions, including verifying contractual obligations, enforcinguser-specified policies, and enhancing performance.

NetQuery is well-positioned to leverage the trusted hard-ware that is increasingly available within networks. We be-lieve that a general-purpose channel for disseminating se-

13

cure statements issued by trusted hardware as well as signedstatements from network operators and users will enable anew and exciting class of applications based on meta-levelreasoning about the state of the network.

9. REFERENCES[1] C. Alaettinoglu, C. Villamizar, E. Gerich, and

D. Kessens. RFC 2622: A Framework for IP BasedVirtual Private Networks. 1999.

[2] AOL. AOL Transit Data Network: Settlement-FreeInterconnection Policy, 2006.http://www.atdn.net/settlement%5Ffree%5Fint.shtml.

[3] M. Caesar, D. Caldwell, N. Feamster, J. Rexford,A. Shaikh, and J. van der Merwe. Design andImplementation of a Routing Control Platform. InUSENIX/ACM NSDI, May 2005.

[4] M. Casado, P. Cao, A. Akella, and N. Provos.Flow-Cookies: Using Bandwidth Amplification toDefend Against DDoS Flooding Attacks. In IEEEIWQoS, June 2006.

[5] M. Casado, M. J. Freedman, J. Pettit, J. Luo,N. McKeown, and S. Shenker. Ethane: Taking Controlof the Enterprise. SIGCOMM CCR, 37(4):1–12,2007.

[6] P. England, B. Lampson, J. Manferdelli, M. Peinado,and B. Willman. A Trusted Open Platform.Computer, 36(7):55–62, 2003.

[7] N. Feamster and H. Balakrishnan. Detecting BGPConfiguration Faults with Static Analysis. InUSENIX/ACM NSDI, May 2005.

[8] M. Gasser, A. Goldstein, C. Kaufman, andB. Lampson. The Digital Distributed System SecurityArchitecture. In Proc. 12th NIST-NCSC NationalComputer Security Conference, pages 305–319,1989.

[9] A. Greenberg, G. Hjalmtysson, D. A. Maltz, A. Myers,J. Rexford, G. Xie, H. Yan, J. Zhan, and H. Zhang. AClean Slate 4D Approach to Network Control andManagement. SIGCOMM CCR, 35(5):41–54, 2005.

[10] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado,N. McKeown, and S. Shenker. NOX: Towards anOperating System for Networks. SIGCOMM CCR,38(3):105–110, 2008.

[11] D. Harrington, R. Presuhn, and B. Wijnen. RFC 3411:An Architecture for Describing Simple NetworkManagement Protocol (SNMP) ManagementFrameworks. Dec. 2002.

[12] S. Incorporated. Meeting the Challenge of Today’sEvasive P2P Traffic: Service Provider Strategies forManaging P2P Filesharing, 2004. http://www.sandvine.com/solutions/pdf/Evasive%5F P2P%5FTraffic.pdf.

[13] International Organization for Standardization. ISODIS 9596-2: Information Processing Systems - OpenSystems Interconnection, Management InformationProtocol Specification - Part 2: Common Management

Information Protocol. Dec. 1988.[14] T. Karagiannis, R. Mortier, and A. Rowstron. Network

Exception Handlers: Host-network Control inEnterprise Networks. In ACM SIGCOMM, Aug.2008.

[15] T. Karygiannis and L. Owens. Wireless NetworkSecurity: 802.11, Bluetooth and Handheld Devices.Special Publication 800-48, NIST, November 2002.

[16] H. V. Madhyastha, T. Isdal, M. Piatek, C. Dixon,T. Anderson, A. Krishnamurthy, andA. Venkataramani. iPlane: An Information Plane forDistributed Services. In USENIX/ACM OSDI, Nov.2006.

[17] NebuAd. Juniper Networks partners with NebuAd toenable ISPs to participate in online advertisingrevenues on the web.http://www.juniperamspmarketing.com/NebuAD.htm.

[18] G. Siganos and M. Faloutsos. Analyzing BGPPolicies: Methodology and Tool. In IEEEINFOCOMM, Mar. 2004.

[19] N. Spring, R. Mahajan, D. Wetherall, andT. Anderson. Measuring ISP topologies withRocketfuel. IEEE/ACM Transactions onNetworking, 12(1):2–16, 2004.

[20] N. Spring, D. Wetherall, and T. Anderson. ReverseEngineering the Internet. SIGCOMM CCR,34(1):3–8, 2004.

[21] Trusted Computing Group. TCG Trusted NetworkConnect: TNC Architecture for Interoperability,Specification Version 1.3. Trusted ComputingGroup, Apr. 2008.

[22] W. Zhou, Y. Mao, B. T. Loo, and M. Abadi. UnifiedDeclarative Platform for Secure NetworkedInformation Systems. In IEEE ICDE, Apr. 2009.

14