srm basic spec lbnl proposal

C O M P U T A T I O N A L R E S E A R C H D I V I S I O N

SRM Basic Spec

LBNL Proposal

Arie Shoshani, Alex Sim, Junmin Gu, Kurt StockingerScientific Data Management Group

Lawrence Berkeley National Laboratory

Summary

• LBNL proposal of SRM Basic specification based on v.1.1• Original v.1.1 methods/arguments in BLUE

Proposed modifications in Green for essential additions with + to the v.1.1 Skyblue for optional additions with ++ to the v.1.1 Red for subtractions from the v.1.1 Purple for renaming and implied updates of the v.1.1

with =>

• Notations:{} … tuple() … list (ordered collection)[] … set (unordered collection)_ … underline as mandatory (as opposed to optional)::= … definition operator| … choice operator (exclusive or)// … commentitalic … recommended

Basically

• From v.2.1.1 or in addition to v.1.1 No space management

o Automatic minimum directory management

o Ls, removeFileso No mkdir, etc.

No permission managemento Somewhat automatic

Some new functionso srmGetSystemInfo // ConfigInfoo Ping

Relationship to v.1.1 and v.2.1.1 (1)

v.1.1 This version v.2.1.1

requestID = integer requestID=string requestToken

=string

requestStatus Separated into RequestSummary and ReturnRequestStatus. FileStatuses from v.1.1 is separated out.

Split into RequestSummary and RequestFileStatus as the response to the request.

FileMetaData Enhanced FileMetaData

MetaDataPathDetail

Relationship to v.1.1 and v.2.1.1 (2)

v.1.1 This version v.2.1.1

RequestFileStatus RequestFileStatus Many different request status. E.g. GetRequestFileStatus, PutRequestFileStatus, etc.

StatusCode Enhanced and

concrete

Explanation added

extensive

automatic storagetype FileStorageType desired to be added

fileStorageType added

Relationship to v.1.1 and v.2.1.1 (3)

v.1.1 This version v.2.1.1

Get Enhanced srmPrepareToGetAdded more parameters

Put Enhanced srmPrepareToPutAdded more parameters

Copy Enhanced srmCopyAdded more parameters

Relationship to v.1.1 and v.2.1.1 (4)

v.1.1 This version v.2.1.1

MkPermanent srmChangeFileStorageType srmChangeFileStorageType

Pin Automatic when file is ready automatic

UnPin srmRelease, or automatic when file lifetime is expired

srmRelease or automatic

Additional pinning to extend lifetime ?

srmExtendedFileLifeTime added

srmExtendedFileLifeTime added

Relationship to v.1.1 and v.2.1.1 (5)

v.1.1 This version v.2.1.1

getRequestStatus srmGetRequestStatus to return RequestSummary

Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request

Together in getRequestStatus

srmGetRequestFilesStatus added to return detailed information on each file

Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request

getFileMetadata srmLS with ability to return single directory meta information

srmLS with extensive

Options Enhanced

directory management

Relationship to v.1.1 and v.2.1.1 (6)

v.1.1 This version v.2.1.1

getProtocols srmGetTransferProtocols none

getEstGetTime Absorbed into srmGetRequestStatus

Or srmStatusOf{Get,Put,Copy}Request

Absorbed into srmStatusOfGetRequest

getEstPutTime Absorbed into srmGetRequestStatus

Or srmStatusOf{Get,Put,Copy}Request

Absorbed into

srmStatusOfPutRequest

Relationship to v.1.1 and v.2.1.1 (7)

v.1.1 This version v.2.1.1

setFileStatus srmPutDone Enhanced into srmChangeFileStorageType

Or srmPutDone

AdvisoryDelete srmRemoveFiles ? srmRemoveFiles

Automatic space management

Automatic Space Management

Space Management

Permission Management

minimal Directory management

GET

• In: SURLs, ProtocolsAuthorizationID,

DesiredFileLifeTime,FileStorageType

Out: RequestStatusSummary

RequestID

• Basic assumptions: UserID

• Get()

• In: SURLs, Protocols

Out: RequestStatus

• srmPrepareToGet()

• In:authorizationID,arrayOfGetFileRequest,arrayOfTransferProtocols, userRequestDescription, storageSystemInfo,

totalRetryTime

• Out: requestToken,

returnStatus,

arrayOfFileStatus

RequestStatusSummary

• RequestIDType (Get, Put, … ) State ErrorMessage FileStatusesRetryDeltaTimeSubmitTimeStartTimeFinishTimeEstimatedTimeToStart

• RequestIDRequestTypeStatusCode // RequestErrorCodeStatusExplanation // ErrorDetailsSubmitTimeStartTimeFinishTimeEstimatedTimeToStartTotalFilesInRequestNumberOfQueuedFilesInRequestNumberOfFinishedFilesInRequestNumberOfProcessingFilesInRequest

• requestToken,requestType,

totalFilesInThisRequest,

numOfQueuedRequests,

numOfFinishedRequests,

numOfProgressingRequests,isSuspended

PUT

• Put()

• In: Src_file_names Dest_file_names, Sizes WantPermanent Protocols

• Out: RequestStatus

• In: {Dest_SURLSize }[]

FileStorageType Protocols AuthorizationID OverwriteMode DesiredLifeTime

• Out: RequestStatusSummary RequestID

• srmPrepareToPut()

• In: authorizationIDarrayOfFileRequest,

arrayOfTransferProtocols,

userRequestDescription,

overwriteOption,

storageSystemInfo,

totalRetryTime

• Out: requestToken,

returnStatus,

arrayOfPutFileStatus

COPY

• Copy()

• In: Source_SURLs, Dest_SURLs

• Out: RequestStatus

• In: { Source_SURL,Dest_SURL }[]

AuthorizationID

DesiredLifeTime

FileStorageType

OverwriteModeOut: RequestStatusSummary RequestID

• srmCopy()

• In: authorizationID,

arrayOfCopyFileRequest,

userRequestDescription,

overwriteOption,

removeSourceFilesstorageSystemInfo,

totalRetryTime

• Out: requestToken,

returnStatus,

arrayOfCopyFileStatus

MkPermanent

• MkPermanent• IN: SURLs• OUT: ReturnStatus

• srmChangeFileStorageType()

• In: authorizationID,

arrayOfPath,

desiredStorageType,

spaceToken

• Out: returnStatus,

arrayOfFileStatus

PIN

• Obsolete

• Automatic when file is ready

• In: TURLs

• Out: RequestStatus

UnPin

• UnPin()

• In: RequestIDTURLs

• Out: RequestStatus

• UnPin => Release

• In: RequestIDTURLs | SURLsAuthorizationID

• Out: { TURL | SURL

RequestStatusCode Explanation } []

• srmReleaseFile()

IN: requestToken authorizationID,

siteURLs,

keepSpace

• Out: returnStatus,

arrayOfFileStatus

getRequestStatus

• In: RequestID

• Out: RequestStatus => RequestStatusSummary

• For details: getRequestFilesStatus (previous getRequestStatus which used to include RequestFileStatus and FileMetaData)

• getRequestStatus()

• In: RequestID

• Out: RequestStatus

• srmStatus(Get,Put,Copy)Request

• IN: requestToken,

authorizationID

arrayOfFromSURLs,

• Out: returnStatus,

arrayOfFileSta

getFileMetaData

• getFileMetaData

• In: SURLs

• Out: FileMetaData

• getFileMetaData => LS

• In: SURLsAuthorizationIDIsAllLevelRecursiveListOffsetListCount

infoOnly

• Out: FileMetaData

• srmLs()

• IN: authorizationID,

path,

fileStorageType,

fullDetailedList,

allLevelRecursive,

numOfLevels,

offset,

count

• Out: MetaDataPathDetail[]

returnStatus

FileMetaData

• SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueIsPermanent IsPinned IsCached

• SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueFileStorageTypeStatusCodeStatusCodeLifetimeAssignedLifetimeLeftCreatedTimeLastModifiedTimeFileTypeOriginalSURLSubDirectoryPTR // dirEntries

• path,

status,

size, // 0 if dir

ownerPermission,

userPermission,

groupPermission,

otherPermission

createdAtTime,

lastModificationTime,

owner,

fileStorageType,

fileType, // Directory or FilelifetimeAssigned,

lifetimeLeft,

checkSumType,

checkSumValue,

originalSURL, MetaDataSubPathDetail[]

getProtocols

• In: none

• Out: list of transfer protocols

• None in v.2.1

• Let’s keep this or put into the new srmSystemInfo or srmConfigInfo

getEst[Get/PUT]Time

• Make it Obsolete

• getEstGetTime: • Out- RequestStatus

In- SURLs, Protocols

• getEstPutTime: • Out- RequestStatus

In- Src_file_name, dest_files_names, sizes, wantPermanent,Protocols

• getEstimationTime

• In: Source_SURLDest_SURLSize

AuthorizationID

TransferProtocols

FileStorageType

• Out: RequestEstimation

• None• Similar one is

getStatusRequest() after request is submitted

AdvisoryDelete

• Obsolete

• AdvisoryDelete:

• Out- noneIn- SURLs

• srmRemoveFiles

• In: SURLs authorizationID

• Out: ReturnRequestStatus

• srmRemoveFiles

• IN: requestToken,

authorizationID,

siteURLs

• Out: returnStatus,

arrayOfFileStatus

GetRequestFilesStatus

• In: RequestIDOut: RequestFileStatus []

• In: RequestIDListOffsetListCount

• Out: RequestFileStatus []

• srmStatusOf{Get, Put, Copy}Request

• IN: requestToken,

authorizationID

arrayOfFromSURLs,

• Out: returnStatus,

arrayOfFileStatus

RequestFileStatus

• SURL SizeState FileldTURLOwner, Group, PermModeCheckSumType, CheckSumValueIsPinned, IsPermanent QueueOrderEstSecondsToStart SourceFilenameDestFilenameIsCached

• Source_SURLDest_SURLSizeStatusCode Explanation TURLCheckSumType, CheckSumValueFileStorageTypeEstWaitingTimeOnQueueEstProcessingTimeSourceFilenameDestFilenameRemainingLifetime

• fromSURLInfo,fileSize,

• status,

• estimatedWaitTimeOnQueue,

• estimatedProcessingTime,

• transferURL

• remainingPinTime

• siteURL,

• toSURL,

Ping

• In: none

• Out: alive_response

GetConfig

• In: flag• Out: { value |

list of supported functions}

• e.g. GET, PUT, COPY, DYNAMIC_MKDIR, PUSH_MODE, PULL_MODE,Default_space = 5GB/user, etc.

New types

• FileStorageType VOLATILE, DURABLE, PERMANENT

• FileType FILE, DIRECTORY, LINK

• OverwriteMode NEVER, ALWAYS, WHEN_DIFFERENT

• RequestType GET, PUT, COPY

• StatusCode // RequestStatusCode, FileStatusCode,

RequestErrorCode, FileErrorCode Explanation: human-readable strings // ErrorDetails

• RequestID : from int to string

Conclusion

• Based on SRM v.1.1 and many of v.2.1.1• Included much of Olof’s issues and suggestions to

v.1.1• Operational specification would be bound to Web

Services.