what’s new in condor
DESCRIPTION
What’s New in Condor. Overview. Quick ‘sound bytes’ on new functionality in recent Condor releases Condor Development Process New Features in Condor version 6.6.x New Features in Condor version 6.7.0. Condor Development Process. We maintain two different releases at all times Stable Series - PowerPoint PPT PresentationTRANSCRIPT
Todd TannenbaumComputer Sciences DepartmentUniversity of Wisconsin-Madison
[email protected]://www.cs.wisc.edu/condor
What’s New in Condor
2www.cs.wisc.edu/condor
Overview
Quick ‘sound bytes’ on new functionality in recent Condor
releases› Condor Development Process
› New Features in Condor version 6.6.x
› New Features in Condor version 6.7.0
3www.cs.wisc.edu/condor
Condor Development Process
› We maintain two different releases at all times Stable Series
• Second digit is even: e.g. 6.2.2, 6.4.7, 6.6.3
Development Series• Second digit is odd: e.g. 6.5.1, 6.7.2
4www.cs.wisc.edu/condor
Stable Series› Heavily tested› Runs on our department production pool
of nearly 1,000 CPUs (for min of 3 weeks)› No new features, only bugfixes and ports. › A given stable release is always
compatible with other releases fromthe same series 6.6.X is compatible with 6.6.Y
› Recommended for production pools
5www.cs.wisc.edu/condor
Development Series› Less heavily tested› Runs on our small(er) test pool.› New features and new technology are
added frequently› Versions from the same development
series are not guaranteed compatible with each other (although we try hard)
6www.cs.wisc.edu/condor
New in version 6.6.x
› Version 6.6.0 released in November 03.
› Current release: version 6.6.3, released in April 04.
7www.cs.wisc.edu/condor
The Struggle to Build Condor
› Condor is BIG Condor code consists of primary
source plus ‘externals’.• Externals include Kerberos, zlib,
GSI, PVM, gSOAP…• Patches to externals
8www.cs.wisc.edu/condor
The Struggle to Build Condor
› Condor is BIG Condor code consists of
primary source plus ‘externals’.
• Externals include Kerberos, zlib, GSI, PVM, gSOAP…
• Patches to externals Current shipped source +
externals: ~415MB of source, or ~9 million lines!
Building Condor outside of UW-Madison used to be very difficult.
• ““LIST OF SHAME”LIST OF SHAME”: Build pointed to packages on UW-Madison fileservers.
9www.cs.wisc.edu/condor
Now Condor Source “Self-Contained”
› Source code to externals are now bundled w/ Condor itself. Self-contained Allows version control on externals + patches
› Build w/ just “configure; make” ! Checks for existence and proper version of all
“bootstrap” requirements, such as the compiler
Applies our patches to the externals All 9 million lines built and bundled
10www.cs.wisc.edu/condor
Building Condor
Building Condor beforeVersion 6.6.0…
Building CondorPost Version 6.6.0!
11www.cs.wisc.edu/condor
› NMI = NSF Middleware Initiative
› Automated build and test infrastructure built on top of Condor Pool of 37 machines of
many architectures Scalable Runs every night, builds
several Condor source branches, then runs 114 test programs.
All results stored in RDBMS, reported on the web.
Yes, Condor builds Condor!
Condor + NMI
12www.cs.wisc.edu/condor
Ports
› New Ports w/ v6.6.x –vs- v6.4.x : Solaris 9 RedHat Linux 8.x, 9.x for x86 (+RPMs) RedHat Linux 7.x and SUSE 8.0 for
IA64 (clipped) Tru64 5.1 (clipped) AIX 5.2 (clipped) Mac OS X (clipped)
13www.cs.wisc.edu/condor
Some new components› Computing On Demand (COD)
› Integration of “Hawkeye” technology
› Condor-G Additions Matchmaking Grid Monitor Grid Shell
14www.cs.wisc.edu/condor
Computing On Demand (COD)
› Introduce effective timesharing to a distributed system Batch applications often want
sustained throughput for a long period of time
Interactive applications often want a quick burst of CPU power for small period of time
COD : Allow both to co-exist
15www.cs.wisc.edu/condor
HawkEye Technology› Dynamic Resource Monitoring, now
‘built-in’ to Condor. Allows custom dynamic attributes to be
added into machine classads. These attributes can be used for
• Queries• Scheduling
Many plugins available.• Disk space, memory used, network errors, open
files/descriptors, process monitoring, users, …
16www.cs.wisc.edu/condor
Condor-G› Condor-G Matchmaking
Condor-G can determine which grid site to utilize via ClassAd matchmaking (grid planning, meta scheduling, …)
› Condor-G Grid Monitor Reduces the load on a GT2-based
gatekeeper, greatly increasing the amount of jobs that can be submitted
› Condor-G GridShell A wrapper for the job Reports exit status, cpu utilization, more
17www.cs.wisc.edu/condor
Improvements in Condor for Windows
› Ability to run SCHEDULER universe jobs Including DAGMan
› JAVA universe support
› More Win32 flavors, incl international versions.
› Added support for encryption on disk of the job and data files on execute machine.
18www.cs.wisc.edu/condor
New Features in DAGMan
› DAGMan previously required that all jobs in a DAG share one log file
› Each job can now have it’s own log file
› Understands XML formatted logs› Can draw a graphical representation
of your DAG Uses GraphViz,
http://www.graphviz.org/
19www.cs.wisc.edu/condor
20www.cs.wisc.edu/condor
Central Manager New Features
› Central Manager daemons can now run on any portCOLLECTOR_HOST = condor.cs.wisc.edu:9019NEGOTIATOR_HOST = condor.cs.wisc.edu:9020 Useful for firewall situations Allows multiple instances on one machine
› Keeps statistics on missed updates› Can use TCP instead of UDP, if you
must
21www.cs.wisc.edu/condor
Command-line Tools› ‘condor_update_stats’ tool to display information on any
dropped central manager updates› ‘condor_q –hold’ gives you a list of held jobs and the
reason they were put on hold› ‘condor_config_val –v’ tells you where (file and line
number) an attribute is defined› ‘condor_fetch_log’ will grab a log file from a remote
machine: condor_fetch_log c2-15.cs.wisc.edu STARTD
› ‘condor_configure’ will install Condor via simple command-line switches, no questions asked
› ‘condor_vacate_job’ to release a resource by job id, and can be invoked by the job owner.
› `condor_wait’ blocks until a job or set of jobs completes
22www.cs.wisc.edu/condor
New 6.7.x Development Series
› Release of v6.7.0 in April 04.
› Can you take the suspense?!?
23www.cs.wisc.edu/condor
V6.7 Themes
› Scalability Resources, jobs,
matchmaking framework
› Accessibility APIs, more Grid
middleware, network
› Availability Failover
24www.cs.wisc.edu/condor
Thank You!
› Later this afternoon is the roadmap for future work.
› Questions?