automatic assessment of programming assignments
DESCRIPTION
Slides from my Koli Calling 2010 presentation: Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, Otto Seppälä (2010). Review of recent systems for automatic assessment of programming assignments. In: Koli Calling ’10: Proceedings of the 10th Koli Calling International Conference on Computing Education Research. Koli, Finland: ACM, pp. 86–93. ISBN: 978-1-4503-0520-4. http://dx.doi.org/10.1145/1930464.1930480TRANSCRIPT
![Page 1: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/1.jpg)
Back to programming teaching tools…
Petri Ihantola, Koli Calling 2010
![Page 2: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/2.jpg)
![Page 3: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/3.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 4: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/4.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 5: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/5.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 6: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/6.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 7: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/7.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 8: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/8.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 9: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/9.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 10: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/10.jpg)
Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka describes what features of programming assignments are automatcally assessed. Dynamic analysis is ofen used to assess functonality, efciency, and testng skills. Statc checks are used to provide feedback from style, programming errors, sofware metrics, and even design. In additon, there are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010
![Page 11: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/11.jpg)
Our Method
Systematc literature review:We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR 'grading’) AND 'programming' from the conference proceedings and journals through ACM Digital Library and IEEE Xplore published in between 2006 and 2010. We then applied the inclusion criteria to the abstracts (a paper must have presented an system providing summatve, numerical, automatc feedback from programming assignments or described results from using such system.) and fnally read through the remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010
![Page 12: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/12.jpg)
Our Method
Systematc literature review:We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR 'grading’) AND 'programming' from the conference proceedings and journals through ACM Digital Library and IEEE Xplore published in between 2006 and 2010. We then applied the inclusion criteria to the abstracts (a paper must have presented an system providing summatve, numerical, automatc feedback from programming assignments or described results from using such system.) and fnally read through the remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010
![Page 13: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/13.jpg)
Our Method
Systematc literature review:We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR 'grading’) AND 'programming' from the conference proceedings and journals through ACM Digital Library and IEEE Xplore published in between 2006 and 2010. We then applied the inclusion criteria to the abstracts (a paper must have presented an system providing summatve, numerical, automatc feedback from programming assignments or described results from using such system.) and fnally read through the remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010
![Page 14: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/14.jpg)
Our Method
Systematc literature review:We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR 'grading’) AND 'programming' from the conference proceedings and journals through ACM Digital Library and IEEE Xplore published in between 2006 and 2010. We then applied the inclusion criteria to the abstracts (a paper must have presented an system providing summatve, numerical, automatc feedback from programming assignments or described results from using such system.) and fnally read through the remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010
![Page 15: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/15.jpg)
![Page 16: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/16.jpg)
Languages
Petri Ihantola, Koli Calling 2010
![Page 17: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/17.jpg)
LanguagesJava, Python, Pascal, C/C++,
assembler, shell scripts, don’t care, ….
Petri Ihantola, Koli Calling 2010
![Page 18: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/18.jpg)
![Page 19: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/19.jpg)
![Page 20: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/20.jpg)
![Page 21: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/21.jpg)
![Page 22: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/22.jpg)
![Page 23: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/23.jpg)
![Page 24: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/24.jpg)
![Page 25: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/25.jpg)
![Page 26: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/26.jpg)
![Page 27: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/27.jpg)
![Page 28: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/28.jpg)
![Page 29: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/29.jpg)
![Page 30: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/30.jpg)
![Page 31: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/31.jpg)
![Page 32: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/32.jpg)
![Page 33: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/33.jpg)
![Page 34: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/34.jpg)
![Page 35: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/35.jpg)
![Page 36: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/36.jpg)
Summary• Programming languages• Integraton into LMSs (e.g. Moodle)• Diferent ways to defne tests
– Xunit, acceptance testng, web testng frameworks– Something not widely applied in industry
• Manual assessment• Resubmissions
– Limitng tme, feedback– Customizaton– Interestng hybrid
• Sandboxing• Availability• Specialtes
– GUIs, data bases, concurrency, web programming, students’ own testng skills
Petri Ihantola, Koli Calling 2010
![Page 37: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/37.jpg)
![Page 38: Automatic Assessment of Programming Assignments](https://reader034.vdocuments.us/reader034/viewer/2022051819/54c46e444a79598d628b46be/html5/thumbnails/38.jpg)
Thanks to the authors of the images I used:• htp://www.fickr.com/photos/smanography/3073414449• htp://www.fickr.com/photos/stephenpof/2920198159/• htp://www.fickr.com/photos/kaptainkobold/3203311346/• htp://www.fickr.com/photos/superk8/3868056064/• htp://www.fickr.com/photos/naomi_pincher/2691787431/• htp://www.fickr.com/photos/secondtoughest/347049865/• htp://www.fickr.com/photos/terrio/4920670944/• htp://www.fickr.com/photos/pranavsingh/1946245078• htp://www.fickr.com/photos/arteurbana/496777749/• htp://www.fickr.com/photos/litle_lushie/28955776/• htp://www.fickr.com/photos/12392252@N03/2037540230/• htp://www.fickr.com/photos/markjsebastan/324425852• htp://www.fickr.com/photos/benheine/5044969981/• htp://www.fickr.com/photos/piblet/1053194618/• htp://www.fickr.com/photos/ogil/1507585665/• htp://www.fickr.com/photos/brandoncripps/2308741771/• htp://www.fickr.com/photos/clydeorama/2442172268/• htp://www.fickr.com/photos/imutoo/2631466945/• htp://www.fickr.com/photos/beatkueng/1350250361/• htp://www.fickr.com/photos/pagedooley/3227848591/• htp://www.fickr.com/photos/levork/4509401065/All these images are licensed under the creatve commons.
Petri Ihantola, Koli Calling 2010