ndn libraries progress and plans · • complex namespace queries / pattern matching (i.e., regexp,...

27
NDN Libraries Progress and Plans March 24, 2019 Jeff Thompson, NDN Team [email protected]

Upload: others

Post on 08-Sep-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

NDN Libraries

Progress and Plans

March24,2019JeffThompson,[email protected]

Page 2: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Overview

• CommonClientLibraries(CCL)• PSync• CommonNameLibrary(CNL)• NDN-RTC• Quicksummaryofrecentresearchprogress

2

Page 3: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

What are the Common Client Libraries (CCL)?

•  EnableclientapplicationstouseNDNinC++,Python,JavaScript,Java,.NET•  CommonAPIacrosslanguages:http://named-data.net/doc/ndn-ccl-api•  Interest/Data,signatures,encryption,transports,apputilities,unittests,examples•  Trackndn-cxxresearch(security,NAC,NDNprotocols,NFDinteraction)•  Backwardscompatibility,platformflexibilityfordevelopmentstability•  UsedinNDN-RTC,BMS,mHealth,neighborhoodnetwork,webpageapps,ICE-AR•  Specializedlibraries:NDN-CPPLite(Arduino),Imp,Android,browserspeedups•  Stats(total):10,771commits,277closedissues,79pullrequests,80forks

Page 4: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Example

face = Face("memoria.ndn.ucla.edu")

name = Name("/ndn/edu/ucla/remap/demo/ndn-js-test/hello.txt/%FDU%8D%9DM")

def onData(interest, data):

print(data.content.toRawStr())

face.expressInterest(name, onData)

Page 5: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

CCL Features

•  Certificatesigning/validating–RSA,ECDSA,HMAC•  Configurablecertchainpolicies,regexnamematching•  Flexiblepublic/privatekeydatabaseAPI•  SignedInterests–verifywithsameAPIascerts•  Name-baseaccesscontrol(AESencryption,RSAkeyprotection)• MemoryContentCache,SegmentFetcher•  Optionalthread-safenetworkI/O•  Configurablewireformat(seebelow)•  ChronoSync,PSync(seebelow)•  Unittests,exampleprograms

Page 6: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

CCL wire format abstraction

• APIisnothard-wiredtoonewireformat•  Enablebackwardscompatibilityifrunningwitholdforwarders

WireFormat.setDefaultWireFormat(Tlv0_1WireFormat.get())

• Canspecifyonadhocbasisifsendingtoavariousnetworksface.expressInterest(name, onData, Tlv0_1WireFormat.get())

• WasusedfortransitionfromCCN0.x• PlanstosupportotherICNwireformats

Page 7: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

CCL – Next steps

• NDNwireformatv0.3(withbackwardscompatibility)•  Typednamecomponents•  Removed(most)Interestselectors•  Interesthopcount•  Interestdefaultstoexactname(optionalCanBePrefix)•  ExtraapplicationparametersintheInterest•  Explicitfieldsforsignedinterests(insteadofusingnamecomponents)

• Newwireformats•  Supportnewnetworkautoconfigprotocols

Page 8: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

What is PSync?

• DevelopedasimprovementtoChronoSync• UsedinNLSRtosyncroutesontheNDNtestbed• PartoftheCCL•  InvertibleBloomfilterofasetofhashednames

•  SendinterestwithmyIBF,receiveinterestswithothers’IBF•  Stablestate:EveryonesendsthesameIBF–Interestaggregation,noData•  Update:IreceiveadifferentIBFwithmissingnamesandprovideinreplyData•  IBFefficientlyupdatesasetdifferenceof~275names

•  Eventualconsistencyfrompairwiseupdates–broadcastnotneeded• Optiontosubscribetopartialnamespaceupdates

Page 9: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Example PSync app face = Face() def onNamesUpdate(names): print("Got names, starting with " + names[0].toUri()) updateSize = 80 pSync = FullPSync2017(updateSize, face, Name("/sync"), onNamesUpdate)pSync.publishName(Name(”/edu/ucla/jefft/paper.txt"))

Page 10: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

PSync – Next steps

•  ImplementinPython,JavaScript,Java(currentlyinC++)• UseasnativesyncfortheCommonNameLibrary(seebelow)•  Stresstest“eventualconsistency”withoutbroadcast•  SupportpartialPSync(waitingforusecase)• NDNProject:AQuickSummaryofRecentProgress

Page 11: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

What is the Common Name Library (CNL)?

•  Libraryenablingapplicationstoworkwithhierarchical,nameddatacollections.•  Namespaceobject(rootandchildnodes)•  ApplicationinteractswithaNamespacenode(attachhandlers,receivenotifications)

•  Providesalightweightwaytointegratevarious:•  Syncmechanisms(i.e.,PSync,vectorsync)•  Dataaccesspatterns(i.e.,Consumer/ProducerAPI,fetchlatest),•  Publishingmodels(i.e.,publish/subscribe,in-memorycontentcache),•  Complexnamespacequeries/patternmatching(i.e.,regexp,wildcards),•  Triggereddatageneration(supportingsecurity)

•  CurrentlyusinginICE-ARmobileclientapplication(Nointerest-dataexchangeexposedtodevelopersofthatapp.)

•  SegmentedcontentwithaMetapacketandversioning•  Built-inencode/decode,encrypt/decrypt,sign/verifyaspartofthepipeline•  NewnamesaddedtotheNamespacetreethroughPSync,appisnotified

Page 12: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

CNL Motivation

•  Providetoolsforworkingwithnamespacesastheyrepresentcollections,inaninformation-focusedratherthancommunication-orientedway•  Assumeasynchronousnetworkoperationswillbeusedtosyncthenamespaceandconsume/publishobjectsinthecollection•  Insulatenon-networkingdevelopersfromcommunicationdetails•  MakeprogresstowardsNDNasamiddleware-replacementintermsofhigh-level,application-facingfeatures,buttrytostayasgeneralaspossible• Workwithaggregateapplication-specificobjects,not(segmented)blobsinpackets•  Asaresult,supportnamespacesynchronizationthewaythatisconceived/describedatahigh-level,andpromoteitasanapplication-levelconcepttoexplore

12

Page 13: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Example segmented content consumer app face = Face("memoria.ndn.ucla.edu") page = Namespace("/ndn/edu/ucla/remap/demo/ndn-js-test/named-data.net/project/ndn-ar2011.html/%FDX%DC5B")page.setFace(face)

def onSegmentedObject(namespace): print("Got segmented object size " + str(namespace.obj.size()))

page.setHandler(SegmentedObjectHandler(onSegmentedObject)).objectNeeded()

Page 14: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

CNL Handlers

•  Supportextensibility•  SetNamespaceforspecialfetching,publishing,objectrepresentation• UnifiedAPIfordeveloperstoo•  https://github.com/named-data/PyCNL/blob/master/python/pycnl/segmented_object_handler.py

Page 15: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Unified publisher/consumer

•  objectNeeded()–Fromapplication(producer)ornetwork(consumer)•  Producer

•  CNLreceivesInterest,addstoPIT,callsOnObjectNeeded(ifnotalreadyincache).•  Handler’sOnObjectNeededanswersTrue.•  CNLwaitsforapplicationtoproducedataasynchronously.•  ApplicationcallssetObject().•  CNLdoesserialize/encrypt/signandsatisfiesPIT.

•  Consumer•  ApplicationcallsOnObjectNeededforaNamespacenode.•  (AllhandlersanswerFalse.)•  CNLdoesFace.expressInterestandwaitsforData.•  CNLreceivesData,doesverify/decrypt/deserializeandOnStateChanged(OBJECT_READY)

Page 16: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

NAMEEXISTS

INTERESTEXPRESSED

DATARECEIVED DECRYPTING

OBJECTREADY

OBJECTTYPE/NAME

STRUCTUREKNOWN

Ifanaggregateobjectwith

versioning,forexample

OBJECTREADYBUTSTALE

Existinglisteners

VALIDATING(ALL)

VALIDATESUCCESS

(ANY)VALIDATEFAILURE

WAITINGFORDATA

(ANY)DECRYPTION

ERROR

ChilddatareceivedInteresttimeoutChildverificationerror

(ANY)INTERESTTIMEOUT

Signing/validationandencryption/decryptionmaybeperformedatboththepacketandobjectlevel,dependingontheobjecttype

(ANY)INTERESTNETWORKNACK

ToobjectNeeded

PRODUCINGOBJECT

AnOnObjectNeededanswerstrue

AllOnObjectNeededanswerfalse

ENCRYPTING

(ANY)ENCRYPTION

ERROR

SIGNING

(ANY)SIGNINGERROR

ReplytopendingincomingInterests

WheretostorechilddecryptedcontentBlobsbeforedeserializingthe

parentobject?

objectNeeded() SERIALIZING

setObject()

DESERIALIZING

NDN-CNL: Name node state diagram Integrating Interest/Data and Packet-/Prefix-level objects

Page 17: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

CNL – Next steps

• High-performancepersistentstorage• PorttoJavaandJavaScript• Moreapplications

•  Currentlyusedinaugmentedrealitymobileclientapplication

Page 18: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

NDN Data Packets

/<base-prefix>

<timestamp>

<stream-name>

<seq #> _gop

_latest _live

_meta<segment #> _manifest _meta

_parity

frame payload manifest payload

Content-Type = 'ndnrtc'

Timestamp

Content-Size

Other

gop pos, gop #

capture timestamp

frame type

complete frame

parity size

generation delay

<segment #>

FEC payload

<seq #>

start end

/<stream-prefix>/<seq#> /<stream-prefix>/<seq#>

<version #>

0

1

/<stream-prefix>/<seq#>

/<stream-prefix>/_gop/<seq#>

<version #>

timestamp

publish rate

WxH

bitrate

description

What is NDN-RTC? •  NDNC++video(HD)streaminglibraryformacOS,Ubuntu,Android•  Sub-second(~150ms)latency•  BasedonVP9videoencoder•  Pipelining,retransmission,FEC•  Unifiedconsumerforliveandstoredvideo•  Recentupdates

•  RDRprotocolforgettinglatestdata•  RemovedependencyonWebRTC(thoughnoechocancellationpipelinefornow)•  New“FastRepo”rocksdb-basedstorage•  Movingtowardsgeneralizedapproachforobjects,metadata,etc.

github.com/remap/ndnrtc

Page 19: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Applications Current

•  ICE-AR(ARbrowseraspartofIntel/NSFICN-WEN)•  NDN-RTCstreamsphonePOVvideoforedgeprocessing(object,face,poserecognition)•  processedinformationdeliveredbacktothephonetoenrichphone’senvironmentalunderstanding(deepcontext)

•  TouchNDN(integrationwithDerivative’sTouchdesigner)•  aimingtoreplaceNDIforlivevideoproduction•  leverageNDNtoefficientlydisseminatelivevideooverL2(orL3)tomultiplenodesforsimultaneousprocessing&storage•  nodesintegrate“historical”playbackfromrepodataseamlesslywithlivestreaming,forscrubbingreal-timestreams

•  Assortedcommandlinetools

Previous

•  ndncon/flume–pureP2Pconferencingapp•  notuptospeedwithlatestlibrary

Page 20: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Future Plans

•  Scalablevideocoding(VP9)•  Region-of-Interest-basedfetching(360ovideousecase)•  Volumetricvideostreaming•  Congestioncontrol,whenappsneedit(basedonSchneider2016)

=>Lookingforappusersandcodebasecollaborators

Page 21: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Recent NDN Code Release Updates • NFDandndn-cxxversion0.6.5• ndn-toolsversion0.6.3• NDNAndroidversion0.6.5-3•  BasedonthelatestversionofNFD(0.6.5)•  IncludingupdatedGUIbasedonworkatNDNhackathon

• ndnSIM2.7•  BasedonthelatestreleasedversionsofNS-3(version3.29)andNFD(version0.6.5)https://ndnsim.net/2.7/RELEASE_NOTES.html

• Mini-NDN0.40https://github.com/named-data/mini-ndn/releases/tag/v0.4.0

• Named-dataLinkStateRoutingProtocol(NLSR)version0.5.0https://named-data.net/doc/NLSR/0.5.0/RELEASE-NOTES.html

Page 22: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Recent New Code Releases / In progress • pSync,asynchronizationprotocolforNDN

https://named-data.net/doc/PSync/0.1.0/RELEASE-NOTES.html

• NDNIoTPackage• Mini-NDN-WiFi

Page 23: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

NDN IoT Package •  AnNDN-basedIoTframeworkwithtwogoals•  Localizedtrustandautomatedsecuritymanagement•  Ease-of-useIoTsoftwaredevelopmentkit

•  Features•  LightweightNDNsoftwarestackandforwarder,specificallytunedforconstraineddevices•  Seamlessintegrationofheterogenouslinklayerprotocols(BLE,WiFi,IEEE802.15.4,etc.)•  Ease-of-usehigh-levelAPIsforbootstrapping,servicediscovery,accesscontrol,andschematizedtrustmanagement

•  EasyadaptationtonewIoThardware/softwareplatforms

•  Ongoingefforts•  Furthermemory-savingNDNforwarderdesign•  DemonstrativeapplicationtoillustratehowtobuildanIoTsysteminafundamentallydifferentwayfromtoday’sIP-basedsolutions

• Moredetailedtobereported@nextIETF

Page 24: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

8th NDN Hackathon (March 8-10, 2019) http://8th-ndn-hackathon.named-data.net/hacks.html

•  FirstPrize:NFD-AndroidEnhancements•  AlexAfanasyev,JuPan,SanjeevKaushikRamani,DavidePesavento

•  SecondPrize:SigcommTutorialApp(NDN-IoTdemo)•  ZhiyiZhang,XinyuMa,EdwardLu,YuGuan,Erynn-MariePhan,LaqinFan

•  ThirdPrize•  Self-LearningforAdHocWirelessNetworks•  MdAshiqurRahman,DavidePesavento

•  SyncinMANETLibrary+Demo•  Tianxiang,Zhaoning,Spyros

•  AddressingndncatchunksPerformanceIssues•  KlausSchneider,SaurabDulal

Page 25: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

NDN Project at Google Summer of Code • https://summerofcode.withgoogle.com/organizations/6559809451589632/

Page 26: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

Publications/Presentations/Tech Reports •  “ANoteonNamingandForwardingScalabilityinNamedDataNetworking”YuZhangetal,ICC2019Workshop,May2019

•  “TheRoleofDataRepositoriesinNamedDataNetworking”LixiaZhangetal,ICC2019Workshop,May2019

•  “ProofofAuthenticationforPrivateDistributedLedger“ZhiyiZhangetal,NDSS2019workshop,Feb2019

•  “OntheGranularityofTrie-basedDataStructuresforNameLookupsandUpdates“ChavooshGhasemietal,toappearinACM/IEEETransactionsonNetworking2019

•  “PacketForwardinginNamedDataNetworkingRequirementsandSurveyofSolutions”ZhuoLietal,toappearinIEEECommunicationsSurveys&Tutorials2019

•  “AnOverviewofSecuritySupportinNamedDataNetworking”ZhiyiZhangetal,IEEECommunicationsMagazine,Nov2018.

SincelastIETF

Page 27: NDN Libraries Progress and Plans · • Complex namespace queries / pattern matching (i.e., regexp, wildcards), • Triggered data generation (supporting security) • Currently using

How to learn more

•  CommonClientLibrary(CCL)•  C++:https://github.com/named-data/ndn-cpp•  Python:https://github.com/named-data/PyNDN2•  JavaScript:https://github.com/named-data/ndn-js•  Java:https://github.com/named-data/jndn•  C#(.NETFramework):https://github.com/named-data/ndn-dot-net

•  PSync:ScalableName-basedDataSynchronizationforNamedDataNetworking•  https://named-data.net/publications/scalable_name-based_data_synchronization/

•  CommonNameLibrary(CNL)•  C++:https://github.com/named-data/cnl-cpp•  Python:https://github.com/named-data/PyCNL

•  NDN-RTC:https://github.com/remap/ndnrtc