on-line testing center

41
1 On-Line Testing Center Database Laboratories Root Questions Automating Homeworks

Upload: connor-howard

Post on 03-Jan-2016

28 views

Category:

Documents


0 download

DESCRIPTION

On-Line Testing Center. Database Laboratories Root Questions Automating Homeworks. Tools for Increasing the Efficiency of Teaching. Laboratories that give immediate, accurate feedback for teaching SQL, e.g. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: On-Line Testing Center

1

On-Line Testing Center

Database LaboratoriesRoot Questions

Automating Homeworks

Page 2: On-Line Testing Center

2

Tools for Increasing the Efficiency of Teaching

1. Laboratories that give immediate, accurate feedback for teaching SQL, e.g.

2. Automated homeworks that simulate the effect of carefully graded “long-answer” homework.

3. Lectures consisting of PowerPoint slides with voiceover.

Page 3: On-Line Testing Center

3

Productivity in Education

The education industry has a terrible productivity-improvement record.

Using database courses as a focus, we have developed a system, OTC (On-Line Testing Center) that automates grading and allows teaching effort to be reused.

Page 4: On-Line Testing Center

4

Comparison: Versus Telecom

Tuition 3-min LD call Ratio

1959 $ 1,200 $3.00 400

2004 $30,000 $0.15 200,000

In 45 years, high-end college tuition has gotten5000 times more expensive relative to along-distance phone call!

Page 5: On-Line Testing Center

5

But Isn’t … ?

The telecom industry is arguably the best example of the use of technology to reduce costs.

How about the much-maligned US Post Office?

Page 6: On-Line Testing Center

6

Comparison: Versus Post Office

Tuition Airmail Stamp Ratio

1959 $ 1,200 $0.08 15,000

2004 $30,000 $0.37 81,000

In 45 years, high-end college tuition has gotten5.4 times more expensive relative to a stamp!

Page 7: On-Line Testing Center

7

One Thing I’d Like to Do, but Haven’t

Global, on-line TA system. Students can ask questions about

a given course, via email. They get fast email response. A TA network is guided by a

database of previous questions and answers.

Page 8: On-Line Testing Center

8

Lectures

We have PowerPoint slides with voiceover for an introductory DB course.

Intended use: play for 50-60% of the lecture; use the rest of the time for discussion.

Pace is critical --- stop for class thought after each slide.

Page 9: On-Line Testing Center

9

Solution

SELECT beer, AVG(price)FROM SellsGROUP BY beerHAVING COUNT(bar) >= 3 OR

beer IN (SELECT name FROM Beers WHERE manf = ’Pete’’s’);

Beers manu-factured byPete’s.

Beer groups with at least3 non-NULL bars and alsobeer groups where themanufacturer is Pete’s.

Page 10: On-Line Testing Center

10

Laboratory Assignments

Conventional SQL homework: “Here is a database; write these queries in SQL.”

TA’s look at SQL answers and try to figure out whether the queries do what they’re supposed to do.

Rate of regrades tells me this task is too hard to get right.

Page 11: On-Line Testing Center

11

OTC Laboratory

Students get a description of a schema and an English description of several queries to write.

Queries are sent to a database system (Oracle or mySQL) for testing on a preset, sample database.

Page 12: On-Line Testing Center

12

OTC Laboratory --- (2)

Three possible outcomes:1. Syntactically incorrect --- they get

the feedback of the DBMS.2. Semantically incorrect --- they get to

see what their query did on another sample database and what it should have done.

3. Semantically correct --- they get credit for the problem.

Page 13: On-Line Testing Center

13

Creating an SQL Lab

1. Stem to describe the schema and queries.

2. CREATE TABLE statements to define the schema.

3. INSERT statements for the test DB.4. INSERT statements for the sample DB.5. Reference queries for the correct

answers.

Page 14: On-Line Testing Center

14

Other Labs

Recently added: similar lab-creation faciltities for:

1. Relational algebra.2. JDBC.3. XQUERY.

Page 15: On-Line Testing Center

15

Policy Issues

The lab is set up so students may submit a query as many times as they like.

Once correct, a query can be stored and the next one worked on.

Page 16: On-Line Testing Center

16

Problem: Automate Construction of Sample DB’s.

Queries involve particular constants. Changing the constants in your

explanation doesn’t explain anything. Example: “find all the bars in Boston.”

The sample DB better not change ’Boston’ in tuples or you’ll be explaining: “if the DB contains (’Joe’’s Bar’, ’Miami’) you need to produce ’Joe’’s Bar’ in your answer.”

Page 17: On-Line Testing Center

17

A Harder Example

Consider query: “find all the beers Joe’s Bar sells for less than $5.”

You can’t change prices in tuples like (’Joe’’s Bar’, ’Bud’, 4.00) randomly, or you’ll give advice like “if the DB contains (’Joe’’s Bar’, ’Coors’, 6.50), you need to produce ’Coors’.”

Page 18: On-Line Testing Center

18

Example --- Continued

You need a “less than $5 – preserving” transformation.

Example: p -> 2*p – 5.

Page 19: On-Line Testing Center

19

Automating Homework

The heart of OTC is a system for automating homeworks and exams.

Goal 1: Encourage students to work “long-answer” problems for themselves.

Goal 2: Inhibit cheating. Goal 3: Eliminate the drudgery of

grading.

Page 20: On-Line Testing Center

20

Modeling “Long-Answer” Questions with Multiple-

Choice Here is a typical “long-answer”

question we might ask in a DB course:

Relation R consists of the following tuples,and relation S has the following tuples.Compute the join of R and S.

Page 21: On-Line Testing Center

21

Root Questions

A root question is a multiple-choice question with several right and many wrong answers.

Example:

Relation R consists of the following tuples,and relation S has the following tuples. Whichof these tuples is in the join of R and S ?

Page 22: On-Line Testing Center

22

Writing a Root Question

The question-designer provides several correct answers. In our example, each tuple of the join

could be one correct answer. Many wrong answers are also

provided. Here, any tuple of the correct length

that is not in the join could be used.

Page 23: On-Line Testing Center

23

Writing Root Questions --- (2)

For each wrong answer, write a choice explanation that gives student a hint or explanation of why it is wrong.

For the question as a whole, write a question explanation.

Page 24: On-Line Testing Center

24

Assigning Root Questions

The instructor develops an assignment consisting of several root questions. 4-6 seems to be the right number ---

we’ll see why. Students take the assignment as

many times as they like and are encouraged to get a perfect score.

Only the final score counts.

Page 25: On-Line Testing Center

25

Assigning Root Questions --- (2)

Each time the student opens the assignment, they are given the same questions, but with a different choice of one correct and three incorrect answers, in random order.

To prevent rapidfire guessing, the student may open an assignment only once per x minutes (instructor choice).

Page 26: On-Line Testing Center

26

Student Responses

Each root question suggests a conventional, “long-answer” question, that the student should work.

Example: for the join question, they may as well compute the entire join. With the join tuples listed on scratch

paper, they can quickly solve any instance of the root question.

Page 27: On-Line Testing Center

27

Automatic Student Help

When a student submits work, they immediately get the choice explanations for those questions they get wrong.

After the due date, students can see their assignments, with not only the choice explanations but the question explanations as well.

Page 28: On-Line Testing Center

28

How Many Questions? We recommend 4-6 questions per

assignment. Fewer than 4 encourages students

to guess; too many questions runs the risk a student will miss one for carelessness. When first given at Stanford with no

limit, some students tried hundreds of times.

Page 29: On-Line Testing Center

29

Comparison

There is a simpler scheme used in courses like physics, where questions are parametrized, and the correct answer computed by a formula.

A weight of $w kilograms is dropped from height $h. How long does it take the weight to reach the ground?

Page 30: On-Line Testing Center

30

Comparison --- (2)

Question is generated by choosing random values of the parameters, and the answer checked against the result of the formula.

Root questions simulate this question type by selecting many parameter values and asking for a correct pairing of parameters and result.

Page 31: On-Line Testing Center

31

Comparison --- (3)

Example:

A weight of w kilograms is dropped from height h. For which of the following triples (w, h, t ) is t the time it takes the weight to reach the ground?

Page 32: On-Line Testing Center

32

Comparison --- (4)

In the database domain, many kinds of questions cannot have their answer computed by arithmetic formula: “Which of these functional dependencies

follows from the given FD’s?” “Which of these schedules is serializable?” “For which relation sizes is query plan A

better than plan B?”

Page 33: On-Line Testing Center

33

Comparison --- (5)

If you are willing to write a program to (say) test serializability, you can write a program that generates a root question with lots of serializable and lots of unserializable schedules.

The output of this program can be input automatically to OTC.

Page 34: On-Line Testing Center

34

OTC Status

About 400 root questions, mostly on databases, developed. Many have explanations included. Let’s face it: writing a root question

correctly is hard. But once done and debugged, it can

be used in many courses.

Page 35: On-Line Testing Center

35

OTC History --- Spring, Fall, 2002

One assignment in Stanford CS347 (Transaction-Processing and Distributed Databases) supported, Spring 2002.

CS145 (Intro. DB course at Stanford) supported in Fall, 2002. 2 Lab assignments, 11 root-question

assignments, midterm (not root questions).

Page 36: On-Line Testing Center

36

OTC --- Winter, 2003

Supported CS245 (DB Implementation, Hector Garcia) at Stanford.

Supported a CS145/245-like course at North Carolina State (Rada Chirkova).

Page 37: On-Line Testing Center

37

OTC Status --- Spring 2003

Supported CS145, CS347, and CS345 (DB Theory) at Stanford.

Continued support at NC State. Supported CS145-like courses at

UC Santa Cruz (Arthur Keller) and Univ. of Leipzig (Erhard Rahm).

Supported a Discrete Math course at NTU Athens (Foto Afrati).

Page 38: On-Line Testing Center

38

OTC Status --- Fall 2003

New Sites: Penn State, U. Chicago, Yale, U. Alabama, U. Karlsruhe, York College/CUNY, U. Business & Econ. (Athens).

Page 39: On-Line Testing Center

39

OTC Development Team

The core software was developed by Murty Valiveti and his team at Gautami Software.

Alan Beck and Ramana Yerneni adapted the OTC core for database instruction and implemented a number of important features.

Page 40: On-Line Testing Center

40

Content Creators

Alan Beck: SQL, JDBC, and XQuery labs.

Austin Shoemaker: relational algebra lab.

Root-question developers: Foto Afrati, Rada Chirkova, Mayur Datar, Prasanna Ganesan, Wang Lam, Anand Rajaraman, Jeff Ullman, Jennifer Widom, Ramana Yerneni.

Page 41: On-Line Testing Center

41

Find Out More

A tutorial for instructors is atwww-db.stanford.edu/~ullman/pub/otc.pdf Demo site:chub.stanford.edu:8181/CS145-demo/

index.html