scripted job submission with grisu jython nesi workshop for nz eresearch 2011 aaron hicks, landcare...
TRANSCRIPT
![Page 1: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/1.jpg)
Scripted Job Submission with Grisu Jython
NeSI Workshop for NZ eResearch 2011Aaron Hicks, Landcare [email protected]
![Page 2: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/2.jpg)
Introduction
Welcome to my workshop, where we will explore scripted job submission with Grisu Jython
• Why script job submission?• Introducing Grisu Jython• Practical Exercises– Trivial HelloWorld– Submitting multiple jobs using lists– Using BatchJobObject to reduce overheads
![Page 3: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/3.jpg)
Why script job submission?
“I’ve got a bad feeling about this...”- various, the Star Wars franchise
![Page 4: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/4.jpg)
BeSTGRID Jobs
The Grisu clients are used to submit jobs to compute resources via the BeSTGRID Jobs service.
Usually with the Grisu Template Client:• Convenient GUI for job submission• File management• Templates for common applications• Detailed job settings can be exposed
![Page 5: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/5.jpg)
Limitations of Grisu Template Cilent
The Grisu Template Client is not suitable for all kinds of jobs:
• Batches of many jobs• Automated job submission• Jobs that require local processing to prepare
job items• Jobs that require complex submission
parameters
![Page 6: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/6.jpg)
Batches of Many Jobs
The BeSTGRID Jobs service can handle hundreds, or even thousands of job submissions, but doing this with the Grisu Template Client is mind numbingly repetitive and requires inhuman patience.
SOLUTION: Write a script or application that iterates through all the jobs, submits them, and then retrieves the output.
![Page 7: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/7.jpg)
Automated Job Submission
The Grisu Template Client is a human interface and can not submit jobs without human interaction. Not useful for submitting jobs from another application or service
SOLUTION: Write a script or object that can submit a job from within an application, or can be called as a scheduled task.
![Page 8: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/8.jpg)
Jobs Requiring Pre-processing
A job may require a number of steps to be performed before a job is even entered into the Grisu Template Client for submission. This may include chaining multiple jobs together.
SOLUTION: Write a script that performs pre-processing steps as part of the job submission process
![Page 9: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/9.jpg)
Complex Job Parameters
A job may require specific parameters, such as:• Specific submission sites• Specific applications and versions• Specific selection and partitioning of resourcesWhich may be difficult to repeat reliably using a
human interfaceSOLUTION: Write a script that sets the
parameters exactly every time
![Page 10: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/10.jpg)
All of the Above
Complex job submission may exhibit any and all of the features discussed previously, and the human interface provided by the Grisu Template Client becomes cumbersome and limits job submission.
These can all be solved by creating a script or application to make job submission automated, repeatable and reliable.
![Page 11: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/11.jpg)
How?
There are a number of solutions from the Grisu Project:
• Grisu Framework: for creating Java applications using the Grisu libraries
• GriCli: a command line interface for Grisu• Grisu Jython: Python-like scripting using Grisu
http://github.com/grisu
![Page 12: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/12.jpg)
Introducing Grisu Jython
“Everything should be made as simple as possible, but not simpler.”
Albert Einstein
![Page 13: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/13.jpg)
Grisu Jython
Grisu Jython allows scripting using Jython, a Python-like dynamically typed language, allowing complex job submission scripts using tools that many researchers are already familiar with, without the overheads of Java, or not easily done with GriCli.
Python, and Jython, can be used very simply, are well established in the research community, and are known for ease of use for new users.
![Page 14: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/14.jpg)
Why Jython?
Jython is Python implemented in Java, and though this seems cumbersome, it provides some advantages over a pure Python implementation:
• Java is ubiquitous, and the JVM is available on almost all platforms
• Jython allows access to native Java objects and methods to a Python-like language
• Jython can be bundled into .jar files for distribution, hence does not need installing
![Page 15: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/15.jpg)
Using Grisu Jython
1. Download the grisu-jython.jar file from:http://code.ceres.auckland.ac.nz/downloads
2. Write your script in your favourite editor or IDE
3. Execute your script in the form of:
java -jar grisu-jython.jar myscript.py
![Page 16: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/16.jpg)
Monitoring Jobs
The Grisu Template Client still has a use in Grisu Jython job submission scripts. It provides a convenient interface for monitoring and managing submitted jobs, and as a file manager between the local workstation and remote sites.
Download the Grisu Template Client from:http://code.ceres.auckland.ac.nz/downloads
Run the Grisu Template Client with:java -jar grisu-template-client.jar
![Page 17: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/17.jpg)
Practical Exercises
“Work! Work!”- Orc Peons, WarCraft
![Page 18: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/18.jpg)
Practical Exercises
Next comes the practical component of the workshop, where we will explore three use cases of Grisu Jython.
1. A trivial HelloWorld job2. Submission of multiple HelloWorld jobs using
the Python List objects3. Using the BatchJobObject to parallelise job
submission and minimise upload and download time and volume
![Page 19: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/19.jpg)
Do you have a Grid identity?
Let’s sort that out now.
![Page 20: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/20.jpg)
Get the examples
The example code is available on my personal GitHub repository:
http://github.com/aethylred/grisu-scripted-submission
This should soon be merged with the Grisu Examples repository:
http://github.com/grisu/examples
This can be retrieved via eGit within Eclipse
![Page 21: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/21.jpg)
Trivial HelloWorld
Submit a job that echoes “Hello World” to the standard output.
• This example demonstrates the absolute minimum of code required to submit a job.
• Good for repurposing to run shell commands for testing or demonstrations.
• Requires specifying the job submission site
![Page 22: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/22.jpg)
Processing jobs in a list
Using one script to run a HelloWorld script as a job multiple times.
• Attaching input files to jobs• Using loops to create multiple jobs in a list• Iterating through a list of jobs to:– Define and submit jobs– Wait for jobs to finish– Retrieve job outputs– Kill jobs and clean up
![Page 23: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/23.jpg)
BaconCount
The Bacon Counter uses random text generated by the Bacon Ipsum site: http://baconipsum.com
The Bacon Counter is actually a simple dictionary matching script that counts the incidents of the words provided in the dictionary.txt file in each file in the input directory.
![Page 24: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/24.jpg)
Using BatchJobObject
The BatchJobObject has additional features and methods that can upload files and submit jobs in parallel.
• Submission location Blacklists and Whitelists• De-duplicate file uploads by uploading files per
batch, rather than per job• Simplify staging files, submitting jobs, and killing job
within the batch fileBUT not perfect, job output has to be retrieved
individually using the BatchJobObject as list
![Page 25: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/25.jpg)
Submitting Bacon
The submitbacon.py script :• Uploads and submits 5 jobs at a time• Uses the BatchJobObject to upload common
files (countbacon.py and dictonary.txt) only once
• Create jobs per file in a directory
![Page 26: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/26.jpg)
Support for Grisu
Grisu JavaDocs on GitHub:http://grisu.github.com/grisu/javadoc
Known issues at the Grisu Project site on GitHubhttp://github.com/grisu/grisu-jython/issues
http://github.com/grisu/grisu/issuesNew issues should be submitted to NeSI support
with “[Grisu]” in the [email protected]
![Page 27: Scripted Job Submission with Grisu Jython NeSI Workshop for NZ eResearch 2011 Aaron Hicks, Landcare Research hicksa@landcareresearch.co.nz](https://reader035.vdocuments.us/reader035/viewer/2022062511/551b8c12550346d6338b5c64/html5/thumbnails/27.jpg)
Finished
I hope you enjoyed the workshop.
Aaron [email protected] 353 4954
PS. Yes, I wrote this presentation on the flight down