xrootd release 4 and beyond gsi seminar stanford university/slac july15, 2015 andrew hanushevsky,...

XRootD Release 4And Beyond

GSI SeminarStanford University/SLAC

July15, 2015

Andrew Hanushevsky, SLAC

http://xrootd.org

July 15, 2014 2GSI Seminar

User Visible Changes I

New client library: libXrdCl.so First available in 3.3.0 now is default

Old deprecated client library is libXrdClient.so

Copy command: xrdcopy xrdcopy renamed to xrdcp Xrdcopy is now a symlink to xrdcp Original xrdcp renamed toxrdcp-old

All have same command line interface!


User Visible Changes II

New meta command: xrdfs Logical replacement to xrd command

Different command line interface More user friendly and intuitive

When installing Release 4 RPM You must de-install previous XRootD

release They are incompatible


Root Support

New client is the default in ROOT Since 5.34.14 (Dec 14, 2013)

Via TNetXNGFile plug-in Setting envar XNet.UseOldClient to 1

Loads TNetXFile instead (old client)

Root 6 does not support new client Due to conflicts with PROOF Expected to be resolved in release

6.1


Admin Visible Changes

IPv6 supportPublic Private NetworkingDisk Caching ProxyHTTP Plug-inRemote DebuggingOptional Client Plug-insMiscellaneous


IPv6 Support

The new client supports IPv6 It prefers IPv6 whenever possible

This corresponds to HEPIX request Connects using mapped IPv4 addr if

need be

The old client is deprecated It will not support IPv6


IPv6 Configuration

Frankly none needed Clients detect available network

stacks May force it to use various IP modes

Via configuration file or envars

Server detects interfaces and DNS settings

It’s important that DNS is setup correctly

May force server to use IPv4 only Via command line option (-I v4)


IPv6 Considerations I

While servers and redirectors can accept both IPv6 and IPv4 clients. . . IPv6 clients are assumed dual-stack

May be redirected to either IPv6 or IPv4 node

Client error recovery will resolve this if unworkable

This works if all the servers are dual-stack

Generally the preferred migration path Since redirection is via hostname…

DNS entry must have no un-routable entries


IPv6 Considerations II

Client dual-stack assumption will be lifted in release 4.1 (3Q14) Clients will be redirected to compatible

nodes If none, the client receives an error

Largely driven by public-private networks

And interactions with IPv6/4


Public-Private Network Support

Redirectors are now network cognizant Servers inform redirectors of usable

i/f’s

Clients always compatibly redirected Private to private and public to public

Subject to configured network topology Via new xrd.network directive option Applies to servers and redirectors


Public-Private Net Topologies

xrd.network routes type [use if1[,if2]] type: local | common | split

local (default) No address differentiation (i.e. pre R4 mode)

common Private incoming -> private (preferred) or public Public incoming -> only public

split Incoming address must match outgoing address

If use unspecified addresses come from DNS!


Private Addresses & DNS

Private addresses should never be in DNS Unless

It’s a site local DNS server or The address is zone registered (i.e. only locally

available)

This is the assumption used by XRootD To avoid DNS reverse translation

timeouts

Hence, the use option might be needed If a server connects using a private

address


Public-Private Considerations

The available server interfaces Must be uniform within a cluster Must be compatible with configured

topology

Restrictions relaxed in R 4.1 Redirectors will match clients & server

i/f’s May lead to inaccessible files if no match

exists Still working through external access

issues May require a separate redirector for

external access Due to IPv6/4 and public-private network

interactions


Networking Is Complex!

There are many combinations now Client and server capabilities must now

match

There are 4 basic combinationsPublic PrivateIPv6IPv4

But things are not that simple!Dual stack clients add another 4 combinations


Disk Caching Proxy

New proxy server mode Configured via pss.cachelib directive

Caches whole files or file segments Mode is configurable

Cached content available for future access Until LRU purged (configurable)

Many use cases to increase access speed


Typical Disk Caching Proxy Uses

LocalClients

CachingProxy

Remote XRootDClusters

Speed upRemote Access

CachingProxy

Speed upHD AccessSSD

XRootDCluster FS

CachingProxy

Speed upRandom Access

XRootDServer HDFS


HTTP Plug-in

Basic http, https, WebDav access Suitable for browsers, curl, wget, &

davix Provides another mode of well-known

access http is neither low latency nor high-

performance Google & Microsoft have proposed improvements

• Changes submitted to W3C and IETF as http2• Improvements are considered incremental

• They only address the most vexing problems

Configured via xrd.protocol directive And specialized http.xxx directives


XRootD Multi-Protocol Support

Always supported multiple protocols Improved architecture makes it much

easier New protocols can leverage XRootD

features Security, monitoring, file system plug-ins,

etc

XRootDServer

XRootDBridge

LoadableProtocol

ClientsMemory Based

Protocol Conversion


Remote Debugging via DigFS

XRootD provided pseudo file system Provides restricted selectable R/O

access to Configuration file Log files Core files /proc/self (Linux only)

Has authentication & authorization options

Including access control restrictions View is standardized regardless of

location Configured via xrootd.diglib directive


core/cmsdcore/xrootd

The DigFS View

/=/

logs/cmsdlogs/xrootdproc/cmsdproc/xrootd

confconf/etc (site specific)

Virtualexported

path


DigFS Authorization

DigFS consults authorization file Created by the site and specified in

config file xrootd.diglib * authfile

all [-]conf[-]core[-]logs[-]proc

allow

gsihostkrb5pwdsss unix

g=grouph=hostn=nameo=orgr=role

++ÖÖÖÖÖ

ÖÖÖ

ÖÖÖ


XRootD Client Relationships

xrdcopy

FTS3

CopyProcess xrdfs

XrdClFilesystem

PyXRootD

Dirac

root

Gaudi

Athena

CMSSW

PROOF

XrdClFile

XrdClPostMaster

XRootD - coreXRootD - add onExternal Package

XRootD – plug-ins

Plug-ins replace original implementation All calls may be replaced

All layers above benefit with any code changes


Client Plug-ins

Plug-ins are loaded at run-time Client looks for plug-in configuration

files 1st Locally:

~/.xrootd/client.plugins.d/ 2nd Globally:

/etc/xrootd/client.plugins.d/ Both locations can be over-ridden via

envar XRD_PLUGINCONFDIR

Plug-ins are strictly version checked Allows for independent development


Miscellaneous I

Readv proxy pass-through Automatic & greatly improves

performance

Manual log file rotation (e.g. logrotate) Via extended –k command line option

High precision log file timestamps New –z command line option

Log timestamp appears in microsecond format


Miscellaneous II

Special stat() plug-in for odd file systems Configured via the oss.statlib directive

Includes plug-in for GPFS with tape backend

Cancellable third party copy Used by transfer tools

Fast directory listings Stat info can now be included w/ dir

entry xrdfs uses this to speed up long listings


Miscellaneous III

New query config options Use the xrdfs command to display

Query cms - shows cmsd status Query role - shows server’s cluster

role Query sitename - shows the site’s name Query version - shows server’s version

Cluster node blacklisting Via cms.blacklist directive & blacklist

file Useful in federated environments


Miscellaneous IV

New monitoring information User login record now also includes

Name of the client’s executable Contents of client’s XRD_MONINFO envar

Useful to tie external information to actual data usage

E.g. the Panda jobid to cross-reference I/O usage


Looking Beyond Release 4

Cross Protocol RedirectionMeta-linksI/O Throttling Plug-in


Cross-Protocol Redirections I

XRootD protocol is capable of redirecting to a protocol other than xroot On file open the server may tell the

client that it’s more efficient to try something else

E.g. read the file locally from disk Open(xroot://host/filename) -> redirect

file://filename


Cross-Protocol Redirections II

New client already capable of processing out-of protocol redirections Server needs some development to do soRoot’s TFile framework needs updating to handle a change in protocols Changes already in development

Target for root 6.1 or 6.2


Meta-Link Files I

XML file that describes one or more data files available for access Meta-link file identified by dot suffix

metalink (v3) or meta4 (v4 & incompatible w/ v3)

File is read and parsed by the client Client picks one based on certain criteria

E.G. priority, location, etc If access fails, client can pick another one


Metalink Files II

XRootD client will support meta-link files But the road is not straight-forward

Incompatible meta-link formats, which one? Need to avoid encumbering installation

I.e. pre-reqs for sites that don’t care about meta-links

Targeting availability in 4Q14


I/O Throttling Plug-in

Allows site to limit I/O access to disk Useful for throttling external access

Used in federated environments via proxy server

Configured via xrootd.fslib directive And specific throttling directives

Plug-in is in code review phase Already used by CMS in production

Targeting 3Q14


AcknowledgementsCurrent Software Contributors ATLAS: Doug Benjamin, Patrick McGuigan, Ilija Vukotic CERN: Lukasz Janyst, Andreas Peters, Justin Salmon Fermi: Tony Johnson Root: Gerri Ganis, Bertrand Bellenot SLAC: Andrew Hanushevsky, Wilko Kroeger, Daniel Wang, Wei

Yang UCSD: Matevz Tadel UNL: Brian Bockelman WLCG: Mattias Ellert, Fabrizio Furano, David Smith

US Department of Energy Contract DE-AC02-76SF00515 with Stanford University

xrootd release 4 and beyond gsi seminar stanford university/slac july15, 2015 andrew hanushevsky,...

Documents

ipv6 slide

gsi seminar ipv6 configuration

ipv6 clients

public public incoming

new client library

publicprivate networks

old client root

incompatible slide