condor j2 + developer apis to condor + a tutorial on condor’s web service interface
DESCRIPTION
Condor J2 + Developer APIs to Condor + A Tutorial on Condor’s Web Service Interface. CondorJ2. Quill/Quill++: Database reflects state of Condor pool Condor J2: Database is the state of Condor pool Overview of CondorJ2 - PowerPoint PPT PresentationTRANSCRIPT
Todd TannenbaumComputer Sciences DepartmentUniversity of Wisconsin-Madison
[email protected] [email protected]://www.cs.wisc.edu/condor
Condor J2+
Developer APIs to Condor+
A Tutorial on Condor’s Web Service
Interface
2http://www.cs.wisc.edu/condor
CondorJ2› Quill/Quill++: Database reflects state of Condor pool› Condor J2: Database is the state of Condor pool› Overview of CondorJ2
Use database to maintain operational data (workflow state, machine state, config policies, etc.)
Implement workflow management, resource management and resource allocation in J2EE Application Server environment
Modify master, startd and starter to be web service clients Provide web interface for all system services (workflow
submission, machine reconfiguration etc.)
3http://www.cs.wisc.edu/condor
Motivation› Flexibility› Centralized Administratibility› Attempt to leverage standard
“enterprise” technology in this space› Scalability
As big as you want if you are willing to pay the big $$$
4http://www.cs.wisc.edu/condor
Java Application Servers› Industrial strength middleware for high performance &
scalable web applications› Widely deployed systems
Oracle AS 10g, IBM WebSphere, BEA WebLogic, JBoss (open source)
› Key features Database connection pooling Support for transactions Web service interfaces Support for clustering (for scalability) Pluggable security models / role based authorization Backend database independence
CondorWeb Services
User’s WebBrowser
Condor PoolWeb Site
CondorDatabase
Web Service Clients
SOAP over HTTP
masterstartdstarter
Execute Machines
Application Server
User’s CustomTools
MachineModules
MatchmakingModules
WorkflowModules
HTTP
JDBC
JDBC
Execute Machines
SOAP over HTTP
startd
job
starter
PoolDatabase
ApplicationServer
ApplicationServer
ApplicationServer
LoadBalancer
Firewall
startd
job
starter startd
job
starter
NAT
7http://www.cs.wisc.edu/condor
What can do in CondorJ2 via browsers and web
services?› Where do we stand now?
Add and configure new machines Reconfigure machines on the fly Specify, submit, monitor and manage workflows Monitor global system state No matchmaking (yet)
› Is currently research work. When will it ship? Will it ever ship? Only time will tell.
8http://www.cs.wisc.edu/condor
Interfacing Applications w/ Condor
› Suppose you have an application which needs a lot of compute cycles
› You want this application to utilize a pool of machines
› How can this be done?
9http://www.cs.wisc.edu/condor
Some Condor APIs› MW (previous talk) › Command Line tools
condor_submit, condor_q, etc› DRMAA› Condor GAHP› Condor Perl Module› SOAP
10http://www.cs.wisc.edu/condor
Command Line Tools› Don’t underestimate them!› Your program can create a submit
file on disk and simply invoke condor_submit:system(“echo universe=VANILLA > /tmp/condor.sub”);system(“echo executable=myprog >> /tmp/condor.sub”);. . .system(“echo queue >> /tmp/condor.sub”);system(“condor_submit /tmp/condor.sub”);
11http://www.cs.wisc.edu/condor
Command Line Tools› Your program can create a submit
file and give it to condor_submit through stdin:PERL: fopen(SUBMIT, “|condor_submit”);
print SUBMIT “universe=VANILLA\n”;. . .
C/C++: int s = popen(“condor_submit”, “r+”);write(s, “universe=VANILLA\n”,
17/*len*/);. . .
12http://www.cs.wisc.edu/condor
Command Line Tools› Using the +Attribute with
condor_submit:universe = VANILLAexecutable = /bin/hostnameoutput = job.outlog = job.log+webuser = “zmiller”queue
13http://www.cs.wisc.edu/condor
Command Line Tools› Use -constraint and –format with
condor_q:% condor_q -constraint ‘webuser==“zmiller”’-- Submitter: bio.cs.wisc.edu : <128.105.147.96:37866> : bio.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 213503.0 zmiller 10/11 06:00 0+00:00:00 I 0 0.0 hostname
% condor_q -constraint 'webuser=="zmiller"' -format "%i\t" ClusterId -format "%s\n" Cmd
213503 /bin/hostname
14http://www.cs.wisc.edu/condor
Command Line Tools› condor_wait will watch a job log file
and wait for a certain (or all) jobs to complete:
system(“condor_wait job.log”);
› can specify a timeout
15http://www.cs.wisc.edu/condor
Command Line Tools› condor_q and condor_status –xml
option› So it is relatively simple to build on
top of Condor’s command line tools alone, and can be accessed from many different languages (C, PERL, python, PHP, etc).
› However…
16http://www.cs.wisc.edu/condor
DRMAA› DRMAA is a GGF standardized job-
submission API› Has C (and now Java) bindings› Is not Condor-specific -- your app could
submit to any job scheduler with minimal changes (probably just linking in a different library)
› SourceForge Project http://sourceforge.net/projects/condor-ext
17http://www.cs.wisc.edu/condor
DRMAA› Easy to use, but› Unfortunately, the DRMAA API does
not support some very important features, such as: Two-phase commit Fault tolerance Transactions
18http://www.cs.wisc.edu/condor
Condor GAHP
› The Condor GAHP is a relatively low-level protocol based on simple ASCII messages through stdin and stdout
› Supports a rich feature set including two-phase commits, transactions, and optional asynchronous notification of events
› Is available in Condor 6.7.X
19http://www.cs.wisc.edu/condor
GAHP, contExample:
R: $GahpVersion: 1.0.0 Nov 26 2001 NCSA\ CoG\ Gahpd $S: GRAM_PING 100 vulture.cs.wisc.edu/forkR: ES: RESULTSR: ES: COMMANDSR: S COMMANDS GRAM_JOB_CANCEL GRAM_JOB_REQUEST GRAM_JOB_SIGNAL
GRAM_JOB_STATUS GRAM_PING INITIALIZE_FROM_FILE QUIT RESULTS VERSIONS: VERSIONR: S $GahpVersion: 1.0.0 Nov 26 2001 NCSA\ CoG\ Gahpd $S: INITIALIZE_FROM_FILE /tmp/grid_proxy_554523.txtR: SS: GRAM_PING 100 vulture.cs.wisc.edu/forkR: SS: RESULTSR: S 0S: RESULTSR: S 1R: 100 0S: QUITR: S
20http://www.cs.wisc.edu/condor
Condor Perl Module› Perl module to parse the “job log
file”› Recommended instead of polling w/
condor_q› Call-back event model› (Note: job log can be written in
XML)
21http://www.cs.wisc.edu/condor
SOAP› Simple Object Access Protocol
Mechanism for doing RPC using XML (typically over HTTP or HTTPS)
A World Wide Web Consortium (W3C) standard
› SOAP Toolkit: Transform a WSDL to a client library
22http://www.cs.wisc.edu/condor
Benefits of a Condor SOAP API
› Condor becomes a service Can be accessed with standard web service
tools› Condor accessible from platforms
where its command-line tools are not supported
› Talk to Condor with your favorite language and SOAP toolkit
23http://www.cs.wisc.edu/condor
Condor SOAP API functionality
› Submit jobs› Retrieve job output› Remove/hold/release jobs› Query machine status› Query job status
24http://www.cs.wisc.edu/condor
Getting machine status via SOAP
Your program
SOAP library
queryStartdAds()
condor_collector
Machine List
SOAP over HTTP
25http://www.cs.wisc.edu/condor
Lets get some details…
26http://www.cs.wisc.edu/condor
The API› Core API, described with WSDL, is
designed to be as flexible as possible File transfer is done in chunks Transactions are explicit
› Wrapper libraries aim to make common tasks as simple as possible Currently in Java and C# Expose an object-oriented interface
27http://www.cs.wisc.edu/condor
Condor setup› Start with a working condor_config› The SOAP interface is off by default
Turn it on by adding ENABLE_SOAP=TRUE› Access to the SOAP interface is denied by default
Set ALLOW_SOAP and DENY_SOAP, they work like ALLOW_READ/WRITE/…
See section 3.7.4 of the v6.7 manual for a description Example: ALLOW_SOAP=*/*.cs.wisc.edu
› If using HTTP, must setQUEUE_ALL_USERS_TRUSTED=TRUE (not needed/wanted with HTTPS)
28http://www.cs.wisc.edu/condor
Necessary tools› You need a SOAP toolkit
Apache Axis (Java) - http://ws.apache.org/axis/ Microsoft .Net - http://microsoft.com/net/ gSOAP (C/C++) - http://gsoap2.sf.net/ ZSI (Python) - http://pywebsvcs.sf.net/ SOAP::Lite (Perl) - http://soaplite.com/
› You need Condor’s WSDL files Find them in lib/webservice/ in your Condor release
› Put the two together to generate a client library $ java org.apache.axis.wsdl.WSDL2Java condorSchedd.wsdl
› Compile that client library $ javac condor/*.java
All our examples are in Java using Apache Axis
29http://www.cs.wisc.edu/condor
Helpful tools› The core API has some complex spots› A wrapper library is available in Java and C#
Makes the API a bit easier to use (e.g. simpler file transfer & job ad submission)
Makes the API more OO, no need to remember and pass around transaction ids
› We are going to use the Java wrapper library for our examples You can download it from
http://www.cs.wisc.edu/condor/birdbath/birdbath.jar Will be included in Condor release
30http://www.cs.wisc.edu/condor
Submitting a job› The CLI way…
universe = vanillaexecutable = /bin/cparguments = cp.sub cp.workedshould_transfer_files = yestransfer_input_files = cp.subwhen_to_transfer_output = on_exitqueue 1
$ condor_submit cp.sub
cp.sub:
Explicit bits
clusterid = Xprocid = Yowner = mattrequirements = Z
Implicit bits
31http://www.cs.wisc.edu/condor
• The SOAP way…1.Begin transaction2.Create cluster3.Create job4.Send files5.Describe job6.Commit transaction
Repeat to submit multiple jobs in a single cluster
Submitting a job
Repeat to submit multiple clusters
32http://www.cs.wisc.edu/condor
1. Begin transaction
2. Create cluster3. Create job
4&5. Send files & describe job6. Commit transaction
Schedd schedd = new Schedd(“http://…”);Transaction xact =
schedd.createTransaction();xact.begin(30);int cluster = xact.createCluster();int job = xact.createJob(cluster);File[] files = { new File(“cp.sub”) };xact.submit(cluster, job, “owner”,
UniverseType.VANILLA, “/bin/cp”, “cp.sub cp.worked”, “requirements”, null, files);
xact.commit();
Submission from Java
33http://www.cs.wisc.edu/condor
Schedd’s location
Max time between calls (seconds)
Job owner, e.g. “matt”
Requirements, e.g. “OpSys==\“Linux\””Extra attributes, e.g. Out=“stdout.txt” or Err=“stderr.txt”
Schedd schedd = new Schedd(“http://…”);Transaction xact =
schedd.createTransaction();xact.begin(30);int cluster = xact.createCluster();int job = xact.createJob(cluster);File[] files = { new File("cp.sub") };xact.submit(cluster, job, “owner”,
UniverseType.VANILLA, “/bin/cp”, “cp.sub cp.worked”, “requirements”, null, files);
xact.commit();
Submission from Java
34http://www.cs.wisc.edu/condor
Querying jobs› The CLI way…$ condor_q
-- Submitter: localhost : <127.0.0.1:1234> : localhost ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1.0 matt 10/27 14:45 0+02:46:42 C 0 1.8 sleep 10000…
42 jobs; 1 idle, 1 running, 1 held, 1 unexpanded
35http://www.cs.wisc.edu/condor
Also, getJobAds given a constraint, e.g. “Owner==\“matt\””
String[] statusName = { “”, “Idle”, “Running”, “Removed”, “Completed”, “Held” };
int cluster = 1;int job = 0;
Schedd schedd = new Schedd(“http://…”);ClassAd ad = new ClassAd(schedd.getJobAd(cluster, job));
int status = Integer.valueOf(ad.get(“JobStatus”));System.out.println(“Job is “ + statusName[status]);
Querying jobs› The SOAP way from Java…
36http://www.cs.wisc.edu/condor
Retrieving a job› The CLI way..› Well, if you are submitting to a local
Schedd, the Schedd will have all of a job’s output written back for you
› If you are doing remote submission you need condor_transfer_data, which takes a constraint and transfers all files in spool directories of matching jobs
37http://www.cs.wisc.edu/condor
Discover available files
Remote file
Local file
Retrieving a job› The SOAP way in Java…
int cluster = 1;int job = 0;Schedd schedd = new Schedd(“http://…”);Transaction xact = schedd.createTransaction();xact.begin(30);FileInfo[] files = xact.listSpool(cluster, job);for (FileInfo file : files) {
xact.getFile(cluster, job, file.getName(), file.getSize(), new File(file.getName()));
}xact.commit();
38http://www.cs.wisc.edu/condor
Authentication for SOAP› Authentication is done via mutual SSL
authentication Both the client and server have certificates and
identify themselves› Possible in 6.7.20› It is not always necessary, e.g. in some
controlled environments (a portal) where the submitting component is trusted
› A necessity in an open environment -- remember that the submit call takes the job’s owner as a parameter
39http://www.cs.wisc.edu/condor
Questions?
40http://www.cs.wisc.edu/condor
Authentication setup› Create and sign some certificates› Use OpenSSL to create a CA
CA.sh -newca› Create a server cert and password-less key
CA.sh -newreq && CA.sh -sign mv newcert.pem server-cert.pem openssl rsa -in newreq.pem -out server-key.pem
› Create a client cert and key CA.sh -newreq && CA.sh -sign && mv
newcert.pem client-cert.pem && mv newreq.pem client-key.pem
41http://www.cs.wisc.edu/condor
Authentication config› Config options…
ENABLE_SOAP_SSL is FALSE by default <SUBSYS>_SOAP_SSL_PORT
• Set this to a different port for each SUBSYS you want to talk to over ssl, the default is a random port
• Example: SCHEDD_SOAP_SSL_PORT=1980 SOAP_SSL_SERVER_KEYFILE is required and has no
default• The file containing the server’s certificate AND
private key, i.e. “keyfile” after cat server-cert.pem server-key.pem > keyfile
42http://www.cs.wisc.edu/condor
Authentication config› Config options continue…
SOAP_SSL_CA_FILE is required• The file containing public CA certificates
used in signing client certificates, e.g. demoCA/cacert.pem
› All options except SOAP_SSL_PORT have an optional SUBSYS_* version For instance, turn on SSL for everyone
except the Collector with• ENABLE_SOAP_SSL=TRUE• COLLECTOR_ENABLE_SOAP_SSL=FALSE
43http://www.cs.wisc.edu/condor
One last bit of config› The certificates we generated have a principal name,
which is not standard across many authentication mechanisms
› Condor maps authenticated names (here, principal names) to canonical names that are authentication method independent
› This is done through mapfiles, given by SEC_CANONICAL_MAPFILE and SEC_USER_MAPFILE
› Canonical map: SSL .*emailAddress=(.*) \1
› “SSL” is the authentication method, “.*emailAddress….*” is a pattern to match against authenticated names, and “\1” is the canonical name, in this case the username on the email in the principal
44http://www.cs.wisc.edu/condor
HTTPS with Java› Setup keys…
keytool -import -keystore truststore -trustcacerts -file demoCA/cacert.pem
openssl pkcs12 -export -inkey client-key.pem -in client-cert.pem -out keystore
› All the previous code stays the same, just set some properties javax.net.ssl.trustStore, javax.net.ssl.keyStore,
javax.net.ssl.keyStoreType, javax.net.ssl.keyStorePassword
Example: java -Djavax.net.ssl.trustStore=truststore -Djavax.net.ssl.keyStore=keystore -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=pass