ifdh tickets - fermilabhome.fnal.gov/~mengel/talks/ifdh_ticket_class/ifdh_tickets.pdf · ifdh...

Marc Mengel

On Call Class

20161126

IFDH Tickets

Overview

● Types of calls● New vs old: ifdhc v2_0_x● Field guide to error messages

Types of calls

● Why did this job crash? (see below) ● I'm getting this error, what does it mean? (error guide,

coming up...)● How can I get (different behaivior)

– see Environment Variable Reference in Redmine, esp.● IFDH_CP_MAXRETRIES● EXPERIMENT● IFDH_GRIDFTP_EXTRA =cd● IFDH_STAGE_VIA=url

● First of all, get the actual job error logs, users inevitably trim out the useful errors in favor of what they think the error message is.

● Sometimes you can also get timestamps from the FIFE logs to refer issues to DCache admins:

ssh [email protected] logsgrep $jobid joblogzgrep $jobid joblog.1

● Or, nowadays, look in kibana under fife-dh*

Field Guide to Error Messages

8/3/17 Presenter | Presentation Title or Meeting Title4

ifdh: command not found• usually a setup problem, check for:

– earlier setups error messages

– $TMPDIR full

– /cvmfs or /grid/fermiapp not mounted

terminate called after throwing an instance of 'std::logic_error' what(): no ifdhc config file environment variables foundAborted

• Also a setup problem



error: globus_ftp_client: the server responded with an error451 Operation failed: Path exists but is not of the expected type

• You forgot to use D, or there's a file where you want a directory.



error: globus_ftp_client: the server responded with an error 550 File not found

• You're trying to copy to/from a directory on the server side that is not there



Error: globus_ftp_control: gss_init_sec_context faile globus_gsi_gssapi: Error with gss credential handle globus_credential: Valid credentials could not be found in any of the possible locations specified by the credential search order....

• You don't have a proxy, you need to get one.



gfalcopy error: 70 (Communication error on send) DESTINATION OVERWRITE srmifce err: Communication error on send, err: [SE][srmRm][] httpg://fgbestman1.fnal.gov:10443/srm/v2/server: CGSIgSOAP running on belkwin ith.fnal.gov reports Could NOT load client credentials...globus_credential: Valid credentials could not be found in any of the possible locations specified by the credential search order.Valid credentials cou

• You don't have a proxy, and need to get one



error: globus_ftp_control: gss_init_sec_context failedglobus_gsi_gssapi: Error with GSI credentialglobus_gsi_gssapi: Error with gss credential handleglobus_credential: Error with credential: The proxy credential: /tmp/x509up_u1733 with subject: /DC=gov/DC=fnal/O=Fermilab/OU=People/CN=Marc W. Mengel/CN=UID:mengel/CN=811559939 expired 3720 minutes ago....

• You had a proxy, but it expired.



error: globus_xio: Unable to open file /tmp/misspeledglobus_xio: System error in open: No such file or directoryglobus_xio: A system call failed: No such file or directory

• You misspelled a local filename, or tried to copy a nonexistent file



error: globus_ftp_client: the server responded with an error451 Operation failed: Path exists but is not of the expected typeprogram: globusurlcopy rstretries 1 gridftp2 nodcau restart stalltimeout 14400 dp gsiftp://ifgridftpfermilab.fnal.gov/no/such/dir/file gsiftp://stkendca50a.fnal.gov/pnfs/fnal.gov/usr/nova/scratch/users/mengel exited status 1

• You misspelled a local directory, or specified a nonexistent one, so ifdh decide it is on bluearc, and...



... (ifdh just sits there...)

Sometimes DCache is just slow. Sometimes DCache is stuck on particular files. You can't tell from the error message, or lack thereof.

● Check Fifemon's dcache pages, look for big queues● Could your file be on tape?● If you still can't tell, open a ticket with “Storage Service”. Be

specific, wHere, wHen and How. (see earlier slide on getting timestamps, etc from logs)



ifdh tickets - fermilabhome.fnal.gov/~mengel/talks/ifdh_ticket_class/ifdh_tickets.pdf · ifdh...

Documents