the portals (coolfont) workshop decided that: dlese will be a two-level collection: –the...
Post on 22-Dec-2015
215 views
TRANSCRIPT
![Page 1: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/1.jpg)
![Page 2: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/2.jpg)
The Portals (Coolfont) Workshop decided that:
• DLESE will be a two-level collection: – The unreviewed collection: a broad collection of
content which is relevant to Earth System education and meets minimum quality and technical standards
– The reviewed collection: a subset of high quality teaching and learning materials which have been rigorously evaluated
![Page 3: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/3.jpg)
• The rationale for having a reviewed collection: – For the user: guaranteed high-quality resources, even
for a teacher without expertise in the field or time to “comparison shop”
– For the creator: inclusion in the reviewed section of DLESE can become a recognized stamp of professional approval
• The rationale for having an unreviewed collection:– For the user: access to a wider range of teaching and
learning resources
– For the library builders: a pool from which to select the reviewed collection
![Page 4: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/4.jpg)
OK, so how do we decide what goes into the reviewed collection?
![Page 5: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/5.jpg)
From the Portals (Coolfont) workshop:
• Selection Criteria:– Accuracy, as evaluated by
scientists
– Importance/significance
– Pedagogical Effectiveness
– Well-documented
– Ease of use for students and faculty
– Motivational/inspirational for students
– Robustness/Sustainability
![Page 6: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/6.jpg)
Familiar review procedures:
• “Traditional peer review”
• “Traditional educational evaluation”
![Page 7: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/7.jpg)
Traditional “Peer-Review”
• Reviewers are selected for their expertise by an editor.
• Reviewers examine the material, or a description of the material, in their home or office.
• Typically two
reviews.
![Page 8: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/8.jpg)
What’s wrong with this picture?
Traditional “Peer-Review”
![Page 9: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/9.jpg)
There are no students in this picture!
Traditional “Peer-Review”
![Page 10: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/10.jpg)
“Traditional Educational Evaluation”
• Evaluator (reviewer) is selected by the developer.
• Evaluator observes students in another teacher’s classroom and/or administers evaluation instruments
• Typically one evaluator, several
classes of students.
![Page 11: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/11.jpg)
What’s wrong with this picture?
“Traditional Educational Evaluation”
![Page 12: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/12.jpg)
Evaluation by independent professionalevaluators is labor-intensive and expensive!
“Traditional Educational Evaluation”
![Page 13: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/13.jpg)
Community Review ConceptPremises
• The materials in the “inner-circle” of reviewed, DLESE-stamp-of-approval-bearing resources must be classroom-tested.
– However, testimony from the creator of a resource that learning has occurred in his or her classroom is insufficient.
– It is not realistic to pay for professional evaluators to go into classrooms to evaluate whether student learning has occurred for
every potential DLESE resource. – Experienced educators can tell whether or not their own students
are learning effectively from an educational resource. – It is easier to answer: “Did your students learn?” than “Do you
think students would learn?”
![Page 14: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/14.jpg)
![Page 15: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/15.jpg)
Community Review ConceptPremises (cont’d)
• In order to be useful, DLESE has to contain lots of resources. Therefore it must grow fast.
• In the DLESE ecosystem, teachers, classrooms and students will be abundant resources.
• The rate-limiting resources in DLESE’s growth will be money, and the time of paid librarians/editors/gatekeepers and computer wizards.
• This is a digital library; we can and should take advantage of automated information gathering
![Page 16: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/16.jpg)
![Page 17: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/17.jpg)
Community Review ConceptProcedure
Coolfont Selection Criteria How to implement
Well documented Review by library staff
Importance/Significance More than N (threshold number tobe chosen) educators from DLESEcommunity tried this resource intheir classroom.
Pedagogical Effectiveness
Ease of use for students andfaculty
Inspirational or motivational forstudents
On-line questionnaire filled out byeducators who used resource intheir classroom
Accuracy, as evaluated byscientists
Invited review by a scientist,recruited by an editor
Robustness/sustainability Testing by a quality assuranceprofessional
![Page 18: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/18.jpg)
![Page 19: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/19.jpg)
![Page 20: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/20.jpg)
Community Review ConceptProcedures (cont’d)
• What happens to the questionnaire information? – Creator receives all feedback from “YES” and “NO” respondents.
– Builders of Discovery System receive feedback from “NO” respondents.
– Suggestions typed in the teaching tips field are added to the Teachers’ section of the resource.
– “Editor” or “gate-keeper” is automatically notified and receives full packet of reviews when number of complete reviews exceeds N and the average, or weighted average, of the numerical scores exceeds Y.
![Page 21: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/21.jpg)
Community Review ConceptProcedure
Coolfont Selection Criteria How to implement
Well documented Review by library staff
Importance/Significance More than N (threshold number tobe chosen) educators from DLESEcommunity tried this resource intheir classroom.
Pedagogical Effectiveness
Ease of use for students andfaculty
Inspirational or motivational forstudents
On-line questionnaire filled out byeducators who used resource intheir classroom
Accuracy, as evaluated byscientists
Invited review by a scientist,recruited by an editor
Robustness/sustainability Testing by a quality assuranceprofessional
![Page 22: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/22.jpg)
Community Review ConceptStrengths
• Inclusive: The community builds the library.
• Scalable: Hundreds or thousands of resources can be classroom-tested.
• Thorough: All seven Coolfont/Portals selection criteria are applied.
• Economical: Scarce talents are applied at the end of the process, to the smallest number of items.
![Page 23: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/23.jpg)
Community Review ConceptIssues
• How do we get educators to send in their reviews?
• How do we ensure that reviews come from bona fide educators?
• Would creators “spin” the review process by soliciting reviews from
their friends? • Would the merely-good early arrival tend to keep out the truly
excellent later arrival?
• Some topics are inherently less inspirational/motivational than others; how do we avoid filtering out resources on such topics?
• What about off-the-wall, or erroneous, or malicious reviews?
![Page 24: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/24.jpg)
Traditional “Peer-Review” Traditional “EducationalEvaluation
Proposed “CommunityReview”
Reviewers are selected bytheir expertise by an editor.
Evaluator (reviewer) isselected by the developer.
Reviewers step forward fromthe community.
Reviewers examine thematerial, or a description ofthe material, in their homeor office.
Evaluator observes studentsin another teacher’sclassroom and/or administersevaluation instruments
Reviewers test the materialin their own classroom withtheir own students.
Typically two reviews. Typically one evaluator,several classes of students.
A large number ofreviewers/students ispossible.
Moderate cost High cost Moderate cost
Editor is supposed to filterout unfair or maliciousreviews.
Evaluator is a neutral party,neither student’s teacher norresource creator.
Unfair or malicious reviewscan be swamped out bynumerical weight of goodreviews.
![Page 25: The Portals (Coolfont) Workshop decided that: DLESE will be a two-level collection: –The unreviewed collection: a broad collection of content which is](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d785503460f94a5ac0d/html5/thumbnails/25.jpg)
How can I become part of DLESE?
• … as a resource creator/contributor
• … as a user
• … as a reviewer/tester