web-based quantitative analysis and reporting of program outcome coverage and student performance

1

Web-based quantitative analysis and reporting of

program outcome coverage and student performance

Paul Van [email protected]

Paul Van Halen Quantitative Program Assessment 2

PRESENTATION GOALS• Gain/improve understanding

– Quantitative course-to-program outcome mapping philosophy

– Importance of designing course assessment tools with program assessment in mind

– Capabilities of open-source software based program assessment framework


OUTLINE• Background• Motivation• Implementation

– Course Assessment– Program Assessment

• Results• Challenges• Future Development• Conclusions


BACKGROUND• Oregon University System

– 81,000 students– 7 universities: PSU, UO, OSU, OIT and 3

regional schools– OSU and PSU offer engineering degrees

• Portland State University– Located in downtown Portland– With 24,000 students, the largest enrollment in

the state system


BACKGROUND• Maseeh College of Engineering and

Computer Science– Electrical & Computer Engineering– Mechanical & Materials Engineering– Civil & Environmental Engineering– Engineering & Technology Management– Computer Science– 70 full time faculty– 1266 undergraduate, 492 graduate students


BACKGROUND• Electrical & Computer Engineering

– 2 ABET accredited programs: Electrical Engineering and Computer Engineering

– 20 faculty– 290 bachelor students– 164 MS and MEng Students– 42 PhD students


MOTIVATION• ABET 2000 criteria• False starts

– College initiatives– University assessment initiative

• What are our program objectives?• What are our program outcomes?• Once we have program outcomes how do

we find out how well we address them and how do we perform?


MOTIVATION• How does this connect with the grades we

give our students?• We should be able to combine the grading

data to assess our stated program outcomes both in coverage and student performance

• Course outcomes and course assessment are a prerequisite

• How do we get from course assessment to program assessment?


PROGRAM ASSESSMENT• Assessment in most cases is still focused

on course assessment• Convincing faculty to participate in this

endeavor is a challenge• Good course assessment does not

necessarily compile to a meaningful program assessment, unless the course assessment was planned with program assessment in mind.


IMPLEMENTATION• Program objectives• Program outcomes• Program outcomes to objectives mapping• Course outcomes• Course outcomes to program outcomes

mapping


PROGRAM OBJECTIVESThe electrical engineering program has the

following educational objectives:

• Knowledge. To provide our students with a broad knowledge base in the fundamentals and techniques of the engineering sciences, required for engineering careers in a changing technical environment, to prepare them for successful participation in multi-disciplinary teams.

• Application. To provide our students with an in-depth knowledge of the concepts, techniques and tools of the electrical engineering discipline and to impart the ability to apply their proficiency to engineering design and problem solving.


PROGRAM OBJECTIVES• Innovation. To provide our students with the

ability and desire to continually renew their education in a rapidly developing discipline, enabling them to participate in the research and development of the discipline and to realize their full potential throughout their career.

• Community. To ensure awareness of: a) the need for personal development, both in discipline

related aspects and in terms of understanding the impact of the profession on social and environmental issues.

b) the importance and benefits of personal involvement in professional societies and local communities.


PROGRAM OUTCOMES• Added more specific EE and CpE outcomes

to the existing (a) – (k) ABET outcomes.• Created a mapping from the program

outcomes to the program objectives.• UPDATE:

– Detailed outcomes are TOO fragmented to guide the decision making process

– Automatic aggregate of details into (a) – (k) is more helpful in loop closure on program assessment


NUMERICAL COURSE TO PROGRAM OUTCOME MAPPING• Instructor in charge of a course has to

provide a list of course outcomes as part of a standard syllabus for the course

• Instructor also has to provide a weighted mapping of course to program outcomes– Provides relative importance of each course

outcome– Provides information on how a particular

course addresses program outcomes– Mapping is normalized with respect to credit

hours


COURSE TO PROGRAM OUTCOME MAPPING• Concepts and limitations of OPAMP-based circuits: 10%;

outcome (a) and (c): 50/50• Concepts of small-signal modeling and amplification: 10%;

outcome (a), (c): 50/50• Analyze and design single-transistor gain stages: 15%;

outcome (c): 100• Analyze and design multistage/differential amplifiers: 15%;

outcome (c) and (k): 50/50• Analyze and design amplifier biasing schemes: 10%;

outcome (c) and (k): 50/50• Frequency response of transistor-based gain stages: 10%;

outcome (c): 100• VCO design project: 30%;

outcome (c), (g) and (i): 40/50/10


PROGRAM OUTCOMES SUMMARY• Program outcome mapping from all

courses can be combined into a single table

• Summary table shows program outcome coverage, or lack thereof


PROGRAM OUTCOMES SUMMARY• Observations:

– 57.5 credits of required ECE courses in the program

– 25.7 credits focused on outcome category (a)– 14.7 credits focused on outcome category (c)– No coverage of outcome category (j)

• Coverage appropriate?• Conclusion: Room for improvement


COURSE GRADING AND ASSESSMENT• Excel spreadsheet• For every item that is part of the grade

– Maximum grade– Mapping to course outcome– Grade for every student


COURSE GRADING SHEETdist A 80 B 65 C 50 D 35

rubrics T1Q

1

T1Q

2

T1Q

3

T1Q

4

T1Q

5

T2Q

1to2

T2Q

3to5

T3Q

1

T3Q

2

T3Q

3

T3Q

4

T3Q

5

T4Q

1

T4Q

2

T4Q

3

T4Q

4

T4Q

5

PA

PB

PC

PD

Tota

l

mx_grade 3 3 3 3 3 6 9 3 3 3 3 3 3 3 3 3 3 12 12 8 8 100

outcome 1 1 1 1 1 2 2 3 4 3 4 4 4 6 6 5 5 5 6 6 7

S1 3 0 3 3 0 1 7 3 3 0 0 3 0 0 3 0 3 9 8 6 6 61

S2 0 0 0 3 0 6 6 0 3 2 3 3 0 2 3 0 3 10 9 7 6 67

S3 3 3 0 3 0 6 6 3 1 3 0 0 0 3 3 3 3 10 9 7 6 73

S4 0 3 0 3 0 3 6 3 3 3 3 3 0 0 3 0 3 6 7 5 6 60

S5 3 3 0 0 0 2 1 3 3 3 3 3 3 0 3 0 3 6 7 5 6 58

S6 3 3 3 3 3 5 5 0 3 3 3 0 0 0 3 3 3 8 8 5 4 68


COURSE OUTCOME ASSESSMENT

Cou

rse

Out

com

e 1

Cou

rse

Out

com

e 2

Cou

rse

Out

com

e 3

Cou

rse

Out

com

e 4

Cou

rse

Out

com

e 5

…

student 1 A B …student 2 B Astudent 3 A Astudent 4 F Dstudent 5 C Cstudent 6 D Fstudent 7 B Bstudent 8 B Cstudent 9 F D…


VISUAL DISPLAY OF RESULTS• Graphics have been updated to improve

communication of results• Use JFreeChart as graphics vehicle• Credit to “Creating Clear and Concise

Visual Display of Information”, K.C. Dee, G. Livesay, BAP IX


PROGRAM ASSESSMENT RESULTS• Summary of performance for each of the

(a) – (k) ABET outcomes• All course in the EE program for which

data was available (110 credit hours, 3744 student credit hours: Σ (# of credit hours) x (# of students)

• For each of the (a) - (k) outcomes:– Weight in the overall assessment– Grade distribution plot


PROGRAM OUTCOME COVERAGE• Table compares the coverage of program

outcomes according to course syllabi (required and electives) and assessment data

• EE required courses: 66 credit hours• EE elective courses: 132 credit hours• Is coverage appropriate ?


EE PROGRAM OUTCOME COMPARISONProgram Outcome (EE) Importance

(required)Importance (elective)

Importance (syllab % )

Importance (assess % )

(a) An ability to apply knowledge of mathematics, science and engineering 27.6 70.1 52.3% 44.8%

(b) An ability to design & conduct experiments, as well as analyze and interpret data 1.55 4.35 3.2% 2.1%

(c) An ability to design a system, component, or process to meet a range of informal to formal descriptions/specifications 14 21.3 18.9% 20.6%

(d) An ability to function on multidisciplinary teams 2.2 1.1 1.8% 0.9%(e) An ability to identify, formulate, and solve engineering

problems 1.75 11.4 7.0% 4.9%(f) An understanding of professional and ethical responsibility 0.65 1.7 1.3% 0.6%(g) An ability to communicate effectively 3.6 5.7 5.0% 3.1%(h) Have the broad education necessary to understand the impact

of engineering solutions in a global and societal context 0.25 0.8 0.6% 0.2%(i) Recognition of the need for & ability to engage in life-long

learning 1.3 2.3 1.9% 1.0%(j) Knowledge of contemporary issues 0 0.2 0.1% 0.0%(k) Ability to use the techniques, skills, modern engineering tools

necessary for engineering practice 4.85 10.2 8.1% 11.2%


TECHNICAL DETAILS• Use Apache Software Foundation tools as

a framework:– Tomcat is a servlet container/server– Cocoon is a web development framework

which provides us with a component-based ability to dynamically generate, transform, serialize web pages from a variety of data sources through “component pipelines”

– Forrest is a Cocoon-based publishing framework that transforms data from various sources into a unified presentation, including web site navigation/decoration


TECHNICAL DETAILS• Data storage:

– All data stored in a RDB, postgresql.– Data stored as a mixture of relational fields

and xml text– XML text can be accessed through XPath

functions– Every faculty member has his/her own table

which is derived from a “master” table.– Faculty has read/write permission on their own

table and read permission on the master table enables data collation


TECHNICAL DETAILS• Data stored in faculty tables for each

course: syllabus, outcome mapping, rubrics/metrics, assessment data

• Data entry and manipulation:– Web-based cocoon forms, using AJAX


TECHNICAL DETAILS• Majority of pages are dynamic, XSP pages

contain data base queries.• XSP formats query results in XML• XML results are transformed in one or

multiple stages into HTML


TECHNICAL DETAILS• Avoids problems:

– Platform independent: faculty on Windows, Mac, Linux, Unix

– XML content in database can be fixed, modified, augmented by the database owner


FLOW EXAMPLE• http://abet.cecs.pdx.edu/ece/quarterAssessmentList/EE/courseAssessListxsp.html• URL invokes pipeline in cocoon sitemap

<map:match pattern="quarterAssessmentList/*/*xsp.xml"> <map:generate type="serverpages" src="{properties:content.xdocs}/courses/quarterAssessList.xsp"/> <map:transform type="xslt" src="{properties:forrest.stylesheets}/ece/assessmentQuarterSelect.xslt"> <map:parameter name="siteurl" value="http://abet.cecs.pdx.edu"/> </map:transform> <map:serialize type="html"/></map:match>

• The first step in this pipeline is the execution of an XSP page: quarterAssessList.xsp• The resulting XML data is rendered into a form through the

assessmentQuarterSelect.xslt stylesheet• Submission of this form replaces the current page with a new URL which contains the

form results as parameters http://abet.cecs.pdx.edu/ece/programAssessmentList/courseAssessListxsp.html?prog=EE&qrts=200704,&lvl=A

http://abet.cecs.pdx.edu/ece/quarterAssessmentList/EE/courseAssessListxsp.html


FLOW EXAMPLE: quarterAssessList.xsp<xsp:page language="java" xmlns:xsp="http://apache.org/xsp" xmlns:xsp-request="http://apache.org/xsp/request/2.0"

xmlns:esql="http://apache.org/cocoon/SQL/v2"> <xsp:structure> <xsp:include>java.text.*</xsp:include> </xsp:structure> <content> <xsp:logic> String uri = <xsp-request:get-sitemap-uri/>; String splt[] = uri.split("/"); String program = splt[1]; </xsp:logic> <title>Assessment quarter selection</title> <program><xsp:expr>program</xsp:expr></program> <esql:connection> <esql:pool>dbname</esql:pool> <esql:execute-query> <esql:query>select distinct quarter from dbtable where rowtype='assessment' order by quarter;</esql:query> <esql:results> <quarters> <esql:row-results> <quarter><esql:get-string column="quarter"/></quarter> </esql:row-results> </quarters> </esql:results> </esql:execute-query> </esql:connection> </content></xsp:page>


FLOW EXAMPLE: assessmentQuarterSelect.xslt<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xalan="http://xml.apache.org/xslt"

exclude-result-prefixes="encoder xalan"> <xsl:param name="siteurl"/> <xsl:template match="/"> <document> <header> <style> .body {font: normal 10pt Arial, sans-serif} </style> <title>Program Assessment Plot detail and Quarter(s) Selection</title> </header> <body> <script> function submitted() { var cl = document.qlist; var e = cl.quarter; var m = cl.level; value = ""; var lvl = ""; for (var i = 0; i < e.options.length; i++) { if (e.options[i].selected) { value += e.options[i].value + ","; } } for (var j = 0; j < m.length; j++) { if (m[j].checked) { lvl = m[j].value; } } window.location.replace("<xsl:value-of select="$siteurl"/>/ece/programAssessmentList/" +

"courseAssessListxsp.html?prog=<xsl:value-of select="/content/program"/>&qrts=" + value + "&lvl=" + lvl);

} </script> <p style="font: bold 10pt Arial, sans-serif">Select plot detail and quarter(s) to include for program assessment</p> <form name="qlist"> <table border="0" class="ECETable" style="padding: 50px; border-collapse: collapse; width: 50%; "> <tbody>


FLOW EXAMPLE: assessmentQuarterSelect.xslt (cont.) <tr> <td width="15%" align="right" style="padding: 6px; vertical-align: top; font-weight: bold;">Plot detail</td> <td style="padding: 6px; vertical-align: top; font-family: font-weight: bold;"> <input type="radio" checked="true" name="level" value="A">Program Outcomes Category Summary

(a-k)</input> <input type="radio" name="level" value="B">Program Outcomes Detail</input> … </td> </tr> <tr><td colspan="2" style="padding: 6px; vertical-align: top; font-weight: bold;"/></tr> <tr> <td width="15%" align="right" style="padding: 6px; vertical-align: top; font-weight: bold;">Quarter(s)</td> <td style="padding: 6px; vertical-align: top; font-family: font-weight: bold;"> <select name="quarter" size="10" multiple="true"> <xsl:for-each select="/content/quarters/quarter"> <xsl:sort select="."/> <xsl:element name="option"> <xsl:attribute name="value"><xsl:value-of select="."/></xsl:attribute> <xsl:value-of select="."/> </xsl:element> </xsl:for-each> </select> </td> </tr> <tr><td colspan="2" align="center"><input type="button" value="Submit" onclick="submitted()"/></td></tr> </tbody> </table> </form> </body> </document> </xsl:template></xsl:stylesheet>


CHALLENGES• Faculty participation• How do you get faculty engaged?• Version control OR How useful is feedback

information when we change the feedback system?

• Incorporating constituents feedback


UPDATES• Convert data storage from file based in

faculty web directories (~/public_html) to a postgresql database

• Most data stored in an xml text block in the database

• Each faculty member has a table derived from a ‘faculty’ master table

• Faculty can modify/update their own table, master user has access to ALL tables


UPDATES• Use XSP to retrieve postgresql data• Replace Altova Authentic with Cocoon

forms for data entry/update to make the whole process truly web based

• Postgresql database provides improved data handling and revision control


LIVE ACCESS• http://abet.cecs.pdx.edu/ece


FUTURE• Calibrate assessment results:

– Incorporate survey and focus group results in the feedback loop


CONCLUSIONS: Closing the loop …• The quantitative program assessment

outlined in this session provides fact based, data driven feedback in the continuous program improvement cycle mandated by ABET, and facilitates a better quality management decision process

• Benefits all constituents


QUESTIONS ?

web-based quantitative analysis and reporting of program outcome coverage and student performance

Documents

meaningful program assessment

stated program outcomes

course assessment tools

engineering careers

engineering sciences

quantitative course

indepth knowledge

broad knowledge base