maestro - cs.columbia.edu · why maestro? consider an experiment that is divided into 3 consecutive...
TRANSCRIPT
![Page 1: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/1.jpg)
MaestroA language for job scheduling
![Page 2: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/2.jpg)
Team MembersVaggelis AtlidakisGeorgios KoloventzosMathias LecuyerArun SwaminathanYiren LuTA: Junde Huang
![Page 3: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/3.jpg)
Agenda❖ What is Maestro?❖ Why Maestro?❖ Who uses Maestro?❖ Example # 1❖ Syntax Explanation❖ Example # 2❖ Syntax Explanation❖ System Architecture❖ Testing❖ Demo❖ Lessons❖ Thank you!!
![Page 4: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/4.jpg)
What is Maestro?Declarative, Interpreted Scripting Language for Job Scheduling
Dynamically TypedPowerful Semantics for Job Distribution
![Page 5: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/5.jpg)
Why Maestro?Consider an experiment that is divided into 3 consecutive steps. Maestro can help:● Express each step with a script and define a Maestro
Job.● Express the correlation of steps using Maestro Job
Dependencies.● Execute each step only after its dependencies are
Resolved.
![Page 6: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/6.jpg)
Who Uses Maestro?❖ Research Labs like CERN that run large-
scale distributed jobs❖ Academics running thousands of
experiments on a strict timetable❖ Anyone who is tired of hand-holding a script
through a conditional pipeline
![Page 7: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/7.jpg)
Example # 1 (Hello World)master("systems-yellow.cs.columbia.edu:6379");
a = Job("print.rb", "Hello");
b = Job("print.rb", "World");
c = Job("print.rb", "!");
run(a -> b -> Wait(10) -> c);
Redis
![Page 8: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/8.jpg)
Example # 1 (Hello World)master("systems-yellow.cs.columbia.edu:6379");
a = Job("print.rb", "Hello");
b = Job("print.rb", "World");
c = Job("print.rb", "!");
run(a -> b -> Wait(10) -> c);
Defining Jobs
![Page 9: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/9.jpg)
Example # 1 (Hello World)master("systems-yellow.cs.columbia.edu:6379");
a = Job("print.rb", "Hello");
b = Job("print.rb", "World");
c = Job("print.rb", "!");
run(a -> b -> Wait(10) -> c);Dependencies Syntax Operators
![Page 10: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/10.jpg)
Example # 2 (MapReduce)master("systems-yellow.cs.columbia.edu:6379");
a = Job("split.rb", "/tmp/big_file_name.data");
maps = map(a, "map.rb");
red = reduce(maps, "reduce.rb");
run(red);
Redis
![Page 11: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/11.jpg)
Example # 2 (MapReduce)master("systems-yellow.cs.columbia.edu:6379");
a = Job("split.rb", "/tmp/big_file_name.data");
maps = map(a, "map.rb");
red = reduce(maps, "reduce.rb");
run(red);
Define Job
![Page 12: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/12.jpg)
Example # 2 (MapReduce)master("systems-yellow.cs.columbia.edu:6379");
a = Job("split.rb", "/tmp/big_file_name.data");
maps = map(a, "map.rb");
red = reduce(maps, "reduce.rb");
run(red);
Map
![Page 13: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/13.jpg)
Example # 2 (MapReduce)master("systems-yellow.cs.columbia.edu:6379");
a = Job("split.rb", "/tmp/big_file_name.data");
maps = map(a, "map.rb");
red = reduce(maps, "reduce.rb");
run(red);
Reduce
![Page 14: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/14.jpg)
Example # 2 (MapReduce)master("systems-yellow.cs.columbia.edu:6379");
a = Job("split.rb", "/tmp/big_file_name.data");
maps = map(a, "map.rb");
red = reduce(maps, "reduce.rb");
run(red);Run Command
![Page 15: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/15.jpg)
System Architecture
API
![Page 16: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/16.jpg)
Testing● Custom test engine -no library or module used● One framework for testing all parts of the program● Tests not only the execution of Maestro, but also the
execution of the job sent to Maestro● Supports individual or batch testing● Logs test results for deeper analysis and to help
locate/fix errors quickly
![Page 17: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/17.jpg)
Testing Batch tests Test log
![Page 18: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/18.jpg)
Demo
![Page 19: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/19.jpg)
Lessons Learned
❖ Start early❖ Pick scope of project wisely❖ Constantly reprioritize❖ Integrate continuously and often❖ Modularize intelligently
![Page 20: Maestro - cs.columbia.edu · Why Maestro? Consider an experiment that is divided into 3 consecutive steps. Maestro can help: Express each step with a script and define a Maestro Job](https://reader034.vdocuments.us/reader034/viewer/2022042805/5f5d2913606f297ebc33b5b1/html5/thumbnails/20.jpg)
Thank you!!