a scheduling service oriented approach for workflow scheduling
DESCRIPTION
A Scheduling Service Oriented Approach for Workflow Scheduling. by Conan Fan Li. Supervisor: Dr. Wendy MacCaull Committee member:Dr. Man Lin Committee member:Dr. Iker Gondra. A SSO approach for workflow scheduling. Initiatives - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/1.jpg)
A Scheduling Service Oriented Approach for Workflow Scheduling
byConan Fan Li
Supervisor: Dr. Wendy MacCaullCommittee member: Dr. Man LinCommittee member: Dr. Iker Gondra
![Page 2: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/2.jpg)
A SSO approach for workflow scheduling Initiatives
AIF project: building decision-support through dynamic workflow systems, academia and industry working together for better healthcare
Workflow engines are not naturally built for scheduling problems
![Page 3: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/3.jpg)
What is a workflow? A workflow or a workflow model is a depiction
of a business process composed of a sequence of operations (tasks).
Tasks are connected in the form of a directed graph to provide an abstraction of the real work for further assessment.
![Page 4: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/4.jpg)
What is a workflow?
![Page 5: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/5.jpg)
What is a workflow?
![Page 6: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/6.jpg)
Why workflow? Abstraction & visualization Clarity & consistency Automation
![Page 7: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/7.jpg)
What is scheduling? The process of making decisions about the
allocation of resources for a number of tasks to achieve one or more objectives?
Two main applications
![Page 8: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/8.jpg)
Application of schedulingManufacturing (e.g., a car factory)
![Page 9: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/9.jpg)
Application of schedulingService industry (e.g., Gate Assignments at an
Airport)
![Page 10: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/10.jpg)
Basics of Scheduling A scheduling problem can be described by a
triplet α | β | γ.
α describes the machine environment β describes the processing characteristics and
constraints γ describes the objective
The triplet Jm | prec | Cmax describes a job shop scheduling problem with precedence constraints and an objective to minimize the makespan
![Page 11: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/11.jpg)
A Job Shop Example
![Page 12: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/12.jpg)
The SSO approach An approach that maps a scheduling problem
onto a workflow model that consists of tasks with built-in services for scheduling.
![Page 13: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/13.jpg)
Definitions A schedule has unforced idleness if some machines
idle when there are jobs waiting for processing. A schedule is non-delay if unforced idleness is prohibited.
Examples of possible objective functions to be minimized are: Makespan: completion time of the last job to leave the
system. Maximum lateness: the worst violation of the due dates. Total weighted completion time: the sum of the weighted
completion times of the n jobs. ...
A multi-instance task is a task that may have multiple distinct execution instances running concurrently within the same workflow case.
![Page 14: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/14.jpg)
Definitions We say a job is waiting when it is not
assigned to any machine or finished. We say a job is independent when it does
not have a precedence constraint or the precedence constraint is satisfied (i.e., the preceding job is finished).
We say a job is machine-ready when its required machine is free.
We say a job is enabled when it is waiting, independent and machine-ready.
![Page 15: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/15.jpg)
Definitions Schedule-flow is a framework that has a
predisposition to model scheduling processes in workflow. Schedule-flow patterns are an extension to the workflow language formalism - Workflow Patterns. By assembling and modifying the existing workflow patterns, schedule-flow introduces new patterns that carries particular responsibilities and services in scheduling systems.
![Page 16: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/16.jpg)
Definitions Schedule-flow is a framework that has a
predisposition to model scheduling processes in workflow. It is an extension to the workflow language formalism - Workflow Patterns. By assembling and modifying the existing workflow patterns, schedule-flow introduces new patterns that carries particular responsibilities and services in scheduling systems.
A* search uses a distance-plus-cost evaluation function (f(x)) to determine the order in which the search visits nodes in the fringe. The distance-plus-cost heuristic is a sum of two functions: the cost function, which is the cost from the starting node to
the current node (usually denoted g(x)) and an "heuristic estimate" of the distance to the goal (h(x)).
![Page 17: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/17.jpg)
Definitions We say an event e=(m,j,start,end) is a future
event of schedule S if e.start >= S.clock
![Page 18: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/18.jpg)
Relation Scheduling is a ___ and a workflow is to
represent a ___. Why use workflow to model scheduling?
Workflow is concise, comprehensive and high-level Scheduling is diverse, technical and low-level We want to visualize the scheduling process
Workflow has a wider range of audience than scheduling does. That is why we need to bridge
them.
![Page 19: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/19.jpg)
Attempt Case: a simple job shop scheduling problem
Objective: minimize makespan(Cmax)
![Page 20: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/20.jpg)
Attempt1. Messy
What you see is messy
What you do not see is messier
![Page 21: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/21.jpg)
Attempt
![Page 22: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/22.jpg)
Attempt
We may choose Job0 to process first. Why does this seem like a smart choice?
2. multi-instance tasks are
confusing and high-maintenance
3. Need to incorporate smart
choices
![Page 23: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/23.jpg)
Attempt
Finish Job0Clock+=7
Job0.waiting=False
![Page 24: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/24.jpg)
Attempt
We want to assign as many
jobs as possible before
processing
4. Options for
unforced idleness
![Page 25: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/25.jpg)
Attempt
We may choose Job4 to
process
![Page 26: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/26.jpg)
Attempt
Finish job1Clock=7+9=16
Job1.waiting=False(Note, job4 has
been processed for 9 time units)
![Page 27: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/27.jpg)
Attempt
No other jobs are available to
process except Job4
![Page 28: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/28.jpg)
Attempt
Finish Job4Clock=16+(15-9)=22Job4.waiting=False
![Page 29: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/29.jpg)
Attempt
Assign job2
![Page 30: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/30.jpg)
Attempt … we get this:(57, {<0,0,0,7>, <0,4,7,22>, <0,2,22,43>,<1,1,7,16>, <1,5,22,52>, <1,3,52,57>})
An event:<machine, job,start,end>
clock attribute of a schedule
![Page 31: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/31.jpg)
Attempt … we get this:(57, {<0,0,0,7>, <0,4,7,22>, <0,2,22,43>,<1,1,7,16>, <1,5,22,52>, <1,3,52,57>})
Gantt chart
Forced idleness
![Page 32: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/32.jpg)
Problems The size issue
The size of the resulting workflow grows in accordance to the size of the problem (number of jobs and number of machines). There are also too many variables to configure inside the workflow. This is neither concise or comprehensive.
The complication of multi-instance tasks Users may not understand when to use them
The lack of heuristic incorporation When several jobs are presented to a machine,
there should be a mechanism to decide which one appears to be the best option
![Page 33: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/33.jpg)
Problems The lack of options for unforced idleness
There should be an easy way of expressing that we do not want to allow unforced idleness, that is, when a machine is free, we always try to assign a job to it if possible.
The lack of comparison and sorting We do not want the workflow to stop as soon as it
finds one feasible schedule. Instead, we need it to compare all the schedules and present the best one. There should be a mechanism to easily compare and sort the schedules.
![Page 34: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/34.jpg)
Proposal The current workflow components are clearly
not sufficient for constructing sophisticated schedulers. Therefore, we need a set of new workflow patterns to provide the services we need in scheduling. We call the extension schedule-flow patterns.
![Page 35: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/35.jpg)
Schedule-flow Present the data (jobs and machines) in a single file
instead of mapping each one of them to a task in the workflow. This way, the size of the workflow will not be proportional to the size of the scheduling problem. More importantly, the same workflow can now work with different sets of data.
Eliminate the usage of multi-instance tasks. Instead, we use a data structure called “fringe”(a collection of ideas, see A*) which is implemented as a priority queue. Different execution instances (schedules) will be evaluated first and then pushed into the fringe.
Heuristics may be given by users regarding the preference of assignment. For example, we may want to assign the jobs with the least processing times first (SPT).
![Page 36: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/36.jpg)
Schedule-flow By default, we do not allow unforced idleness.
We would like to keep the machines as busy as possible. “For many the models that have regular objective
functions, there are optimal schedules that are non-delay”
The fringe provides options for its priority rule, which is the order that the schedules are sorted.
![Page 37: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/37.jpg)
Schedule-flowFor the same problem, use schedule-flow:
Built-in variables: fringe, current
![Page 38: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/38.jpg)
Schedule-flow task components Creation
Does nothing Pop
current=fringe.pop() Selection
By default, select jobs that are enabled (or independent, machine-ready…)
Allocation Generate schedules by assigning jobs to corresponding machines
Process Generate one schedule by advancing time until any job is
finished Push
fringe.push(schedules)
![Page 39: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/39.jpg)
Schedule-flow condition components Exist
Tests if the input exists Continue
By default, tests if the fringe is empty. Other options may be used such as limiting the execution time or the number of schedules generated…
![Page 40: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/40.jpg)
Use Schedule-flow in practice Design a schedule-flow with a graphic editor
We used YAWL in this case.
![Page 41: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/41.jpg)
Use Schedule-flow in practice Provide the files
Job file (e.g., “jobs.csv”) Machine file Schedule-flow XML file generate by the graphic
editor Parse the schedule-flow file and detect
schedule-flow components by matching names. We used a Python script to parse the YAWL’s XML
file.
<task id="POP_93"> <name>POP</name> <flowsInto> <nextElementRef id="SELECT_87" /> </flowsInto> <join code="xor" /> <split code="and" /> </task>
![Page 42: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/42.jpg)
Use Schedule-flow in practice Configure the components, objective and
heuristic In this case, every component uses the default
setting Objective is set to minimize the makespan.
Therefore, the cost function g(S)=S.clock Heuristic is initially 0 (f(x)=g(x), see A*). For this
case, we set the heuristic function to return the processing time of the future event (see future event) with the earliest end time
Bind: automatically connect the components according the schedule-flow
Run the schedule-flow
![Page 43: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/43.jpg)
Schedule-flow simulationChanges Values
fringe = [S0]S0 is the initial
schedule where no jobs are assigned.
S0={ }
![Page 44: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/44.jpg)
Schedule-flow simulationChanges Valuescurrent=fringe.pop() current=S0
fringe=[]
![Page 45: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/45.jpg)
Schedule-flow simulationChanges ValuesSELECT.output=[0,2,4] current=S0
fringe=[]SELECT.output=[0,2,4]
![Page 46: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/46.jpg)
Schedule-flow simulationChanges ValuesEXIST returns True current=S0
fringe=[]
SELECT.output=[0,2,4]
![Page 47: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/47.jpg)
Schedule-flow simulationChanges ValuesALLOCATE.output={S1,S2,S3} current=S0
fringe=[]S1: (0,{<0,0,0,7>})S2:(0,{<0,2,0,21>})S3:(0,{<0,4,0,15>})
![Page 48: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/48.jpg)
Schedule-flow simulationChanges Valuesfringe.push(ALLOCATE.output) current=S0
fringe=[S1,S3,S2]
f(S1)=0+7=7f(S3)=0+15f(S2)=0+21
S1: (0,{<0,0,0,7>})S3:(0,{<0,4,0,15>})S2:(0,{<0,2,0,21>})
![Page 49: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/49.jpg)
Schedule-flow simulationChanges ValuesCONTINUE returns True current=S0
fringe=[S1,S3,S2]
![Page 50: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/50.jpg)
Schedule-flow simulationChanges Valuescurrent=fringe.pop() current=S1=(0,{<0,0,0,7>})
fringe=[S3,S2]
![Page 51: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/51.jpg)
Schedule-flow simulationChanges ValuesSELECT.output=[] current=S1=(0,{<0,0,0,7>})
fringe=[S3,S2]SELECT.output=[]No jobs are enabled in
S1!
![Page 52: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/52.jpg)
Schedule-flow simulationChanges ValuesEXIST returns False current=S1=(0,{<0,0,0,7>})PROCESS.output=S4=(7,{<0,0,0,7>}) fringe=[S3,S2]
![Page 53: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/53.jpg)
Schedule-flow simulationChanges Valuesfringe.push(S4) current=S1=(0,{<0,0,0,7>})
fringe=[S4,S3,S2]
f(S4)=7+0f(S3)=15f(S2)=21
S4: (7,{<0,0,0,7>})S3:(0,{<0,4,0,15>})S2:(0,{<0,2,0,21>})
![Page 54: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/54.jpg)
Schedule-flow simulationChanges ValuesCONTINUE returns True current=S1=(0,{<0,0,0,7>})
fringe=[S4,S3,S2]
![Page 55: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/55.jpg)
Schedule-flow simulation Many more iterations… … we get this:(57, {<0,0,0,7>, <0,4,7,22>, <0,2,22,43>,<1,1,7,16>, <1,5,22,52>, <1,3,52,57>})
Gantt chart
![Page 56: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/56.jpg)
FAQ Why is schedule-flow a framework?
Like many other frameworks (e.g., openGL, Joomla...), schedule-flow is “an abstraction in which software providing generic functionality can be selectively changed by user code, thus providing application specific software. It is a collection of software libraries providing a defined application programming interface (API)”.
How to use schedule-flow with a particular workflow language (e.g., NOVA workflow). The translation between workflow languages and schedule-flow
should be left to the applications. The translation could be as simple as extracting the names and dependencies of the tasks in the workflow like what we did with YAWL. However, for a more sophisticated and user-friendly software, advanced configurations such as heuristic function may be also incorporated into the workflow graphic editor to provide better user experience.
![Page 57: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/57.jpg)
FAQ Why do we need to use workflow to model
schedulers anyway? To be concise, comprehensive and visual. If we can see
each step in the scheduling process, it is easier to optimize and make changes.
What problems does schedule-flow solve? By cutting the direct mapping between jobs and
workflow tasks, schedule-flow provides more flexibility so that the same structure can be used for more than just one problem. Moreover the size of the resulting workflow is significantly reduced.
By introducing the concepts of heuristics, schedule-flow makes “smart” choices instead of random permutation for job assignments.
![Page 58: A Scheduling Service Oriented Approach for Workflow Scheduling](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816581550346895dd8211f/html5/thumbnails/58.jpg)
Tool: PYSF We developed a tool called PYSF (Python
schedule-flow) for experiments Demo…