annual report 30 september 2004 · 2004. 11. 15. · qmark question mark perception triads...

OLAAF Annual Report, Year 2 1

Annual Report

30th September 2004

Project Number: 5/02

Project Title: OLAAF: OnLine Assessment And Feedback

Project Website: http://www.bbk.ac.uk/olaaf

Completion Date: September 2005

Project Director: Dr Richard Rayne

School of Biological and Chemical Sciences

Birkbeck College, University of London

Malet Street, London

WC1E 7HX

0207 631 6253

[email protected]


Summary

OnLine Assessment And Feedback (OLAAF) is a 3-year project funded under the HEFCE FDTL4. OLAAF’s

primary aim is to support lecturers and other authors in the design, delivery and evaluation of Computer

Based Assessment with Feedback. Deliverables from this project will be widely disseminated throughout the

UK higher education community. The project responds explicitly to several of HEFCE’s priorities as outlined

in the HEFCE Strategic Plan, 2003-2008 (see Section 1.8.3 herein, and 1.8.3 and 1.8.4 in the Year 1 report).

This report retrospectively covers Year 2 of the OLAAF project and sets out a plan for Year 3.

Key activities and main accomplishments for Year 2 include the following (with relevant report section

numbers indicated):

• Establishment of Links with other FDTL4 Projects and HE Academy Subject Centres

We have forged particularly strong links with the Formative Assessment in Science Teaching (FAST)

project (FDTL 74/02) [see Sections 1.3.3, 1.5.1, Appendix 7] with whom we have planned major

cooperative dissemination activities in Year 3. We have formed excellent working relationships with

appropriate HE Academy Subject Centres, having utilised them in dissemination events in Year 2 [see

Section 1.5.2]. They will play important roles in our planned activities for Year 3 [see 1.4.5, 1.5.1, 1.5.2,

Appendix 7].

• Continued Progress in Production of Assessment Construction Resources and Authoring of

Computer-Based Assessments

Computer-based assessments (CBA) are presently being developed in parallel at various OLAAF sites

(1.4.1, 1.4.2, 1.6.3; Appendices 1 & 2). OLAAF supports this innovation by providing: a range of

Assessment Construction Resources available online (1.4.3, 1.6.2, Appendix 3); centrally placed

expertise (i.e. at Birkbeck) in CBA and in evaluation of CBA; funding to support training and authoring

at Partner Sites; online technical support; and workshops to embed skills. The culmination of the first

year’s work was the OLAAF Conference in January 2004, which brought together all partner and

Interest Group sites to share successes and plan collaboration on future developments. The 3rd

Project Team Meeting (in Sept 2004; Appendix 5) served as a planning session for the case studies

that will report our work at the end of Year 3 and for the OLAAF 2 Conference.

The project’s financial position is excellent (see Section 3) and we foresee no impediments to future activities

owing to budgetary constraints. A plan for activities in Year 3 is described in Section 2 (see also Appendix

10). We are presently on course to achieve all key objectives/targets as set for Year 3.

There is no doubt that this has been a very difficult year. In particular, there have been a number of

personnel changes, some of our project members/contributors have suffered illness and one, sadly, passed

away earlier this year. Some of the consequent disruptions have put a large added burden of work on the

Project Director particularly. Despite these difficulties, we have managed to meet most of our key milestones,

and have not needed to alter course except in rather minor ways.

We look forward to an exciting third year of the project, when many of our evaluation activities will be collated

and our products can be rolled out to those who might benefit from them. We are confident that our

commitment to maintaining principles of good assessment practice will prove its worth through positive

impacts on student learning and retention.


CONTENTS OF OLAAF ANNUAL REPORT, YEAR 2

1. Year 2 Review 1.1 Main Objectives/Targets for Year 2

1.2 Changes to the Project Plan

1.3 Particular Successes in Year 2

1.4 Major Activities Undertaken in Year 2

1.5 Collaborations and Links

1.6 Outputs from Year 2 Activities

1.7 Evaluation of the Work

1.8 Learning from Activities and Progress Made During Year 2

2. Year 3 Plan 2.1 Project Work Plan for Year 3

2.2 Dissemination Plan for Year 3

2.3 Evaluation Plan for Year 3

2.4 Help Needed for Year 3

3. Financial

Statement 3.1 Expenditure Statement for Year 2

3.2 Anticipated Expenditure for Year 3

4. Annexes A & B

5. Appendices 1 OLAAF Partner Sites: Summary of Activities in Year 2

2 OLAAF Interest Group: Summary of Activities in Year 2

3 Notes on Plans for the Release Version of Assessment Construction Resources

4 OLAAF 1 Conference Programme and Delegate List

5 Agenda for the 3rd

Project Team (Site Leaders) Meeting (September 2004)

6 Minutes of the of the 3rd

Steering Group Meeting (September 2004)

7 Dissemination Strategy (revision 2, August 2004)

8 Evaluation Strategy (revision 2, August 2004)

9 External Assessment Evaluation: Evaluator’s Brief (August 2004)

10 Project Plan for Year 3

11 Budget (Years 1-3)

Glossary ACR Assessment Construction Resources CBA Computer Based Assessment CBAF Computer Based Assessment with Feedback CETL Centre for Excellence in Teaching and Learning CIAD Centre for Interactive Assessment Development, University of Derby FAST Formative Assessment in Science Teaching project (FDTL 74/02) FDTL Fund for the Development of Learning and Teaching HEFCE Higher Education Funding Council for England LTSN Learning and Teaching Support Network NCT HEFCE’s National Coordination Team OLAAF OnLine Assessment And Feedback QMark Question Mark Perception TRIADS TRI-partite Interactive Assessment Delivery System


1. Year 2 Review

1.1 Main Objectives/Targets for Year 2

Annex A shows the major objectives and targets for Year 2. Two targets were not met within their

original time frame: availability of draft Assessment Construction Resources for outside use, and

External Project Evaluation. There has been a delay in receipt of the External Assessment

Evaluator’s Report. In addition, return of the Annual Report (this report) was delayed until November.

None of the delays have had a consequential impact on the overall progress toward meeting our final

objectives, nor have they required any major changes to the Project Plan. All other objectives for the

Year 2 period have been achieved in a timely manner.

Explanations for the delayed outcomes are given in Annex A along with indications of new target

dates for their completion. In some cases, additional commentary is provided in Section 1.2.

1.2 Changes to the Project Plan

There were FIVE main changes to the original project plan, as listed in Annex A and described

below. These were approved by our NCT Coordinator and recorded in the quarterly reports.

Resignation of Project Manager and Appointment of Project Assistant (1.2.1)

Our Project Manager (PM), Jenny Phillips, left the project in Dec 2003. Rather

than refilling this post, it was decided that the interests of the project would be

better served by hiring a “Project Assistant” who would: a) learn TRIADS

authoring and assist the current CBAF Support Officer (Ellen McCarthy Howey),

and b) provide occasional support to the Project Director in organisational and

clerical matters. Caroline Pellet-Many was appointed to this post at the end of

March 2004. I am pleased to report that Caroline has done superb work since

joining the project. The project partners who rely on Birkbeck as a central

resource for TRIADS CBA authoring have benefited immensely from the

support provided by Ellen and Caroline.

As a consequence of the loss of the PM, the Project Director (Dr Rayne) has

taken on the PM duties. He is grateful to the Head of the School of Biological &

Chemical Sciences at Birkbeck for releasing him from some administrative

duties, giving him more time to devote to OLAAF.

Postponement of Site Leader and Steering Group Meeting (1.2.2)

Owing to the disruptions arising from the events described in Section 1.2.1, it

was decided that we should postpone the Site Leader/Steering Group Meeting,

originally scheduled for May 2004, until Sept 2004 (it was held on 23rd

Sept).


New partner site: University of Kent (1.2.3)

Dr Darren Griffin, formerly Site Leader at Brunel University (see also 1.2.4.) has

moved to the University of Kent where he will carry on his OLAAF-related

activities, some of which will continue at Brunel. Dr Griffin also will seek new

collaborators in his OLAAF work from an existing group of bio-scientists at Kent

who have strong records in innovative approaches to teaching and learning.

Changes of personnel at Partner Sites (1.2.4)

In Dec 2003, a member of the project team from Brunel University, Kavita Shah,

left the project. Kavita had been on course to carry out various end-user

evaluations of TRIADS CBA; some of these will now have to be abandoned.

She has since been replaced by Julia Stephenson, who now has moved to

University of Kent with Dr Griffin. Julia is developing computer-aided learning

materials and computer-based assessments which will contribute to genetics

modules at Brunel, Kent and at Birkbeck.

Although not a change of personnel, it should be noted here that there were

some disruptions to planned activities at University of Plymouth in winter 2003-

2004 owing to the sudden illness of the Site Leader, Dr Les Jervis. Dr Jervis

has recovered fully and since then has resumed his valued contributions to the

project.

At Birmingham University, the site co-leaders Dr Richard Freer-Hewish and Ms

Barbara Hallam have retired as of end Sept 2004. Despite having left her post

at Birmingham, Ms Hallam will be compiling materials for a Case Study based

on their CBA work. Meanwhile, we have gained an enthusiastic new Site Leader

at Birmingham, Dr Jill Ramsay, who has been developing CBA materials in

WebCT for her physiotherapy modules. Dr Ramsay recently attended the Site

Leader and Steering Group Meeting (23 Sept 2004). Another academic from

Birmingham, Dr Natalie Rowley (Chemistry), is due from Dec 2004 to undertake

a small case study on TRIADS CBA usage in her modules.

At University of Plymouth, the TRIADS authorer, Janet Carboy, left her post as

of end Aug 2004. She has since been replaced by Mark Pannell. We look

forward to meeting Mark at the OLAAF 2 Conference in November 2004.

External evaluation (1.2.5)

We were shocked and saddened by the untimely death in late June 2004 of our

External Project Evaluator, Dr Joanna Bull. Joanna had already contributed to

the development of our Evaluation Strategy and we had made plans for her to

undertake evaluative activities over the Summer 2004. We have yet to enlist a

replacement, but are working toward achieving this by end 2004. Necessarily,


the nature of the evaluation may need to alter somewhat to take account of its

belated timing.

We have had two External Assessment Evaluations, both conducted by Dr

David O’Hare (Univ. of Derby). Dr O’Hare completed an evaluation in Aug 2004,

but he has unfortunately taken ill and his report has been delayed. We wish him

a speedy recovery and look forward to receiving his report in due course.

1.3 Particular Successes in Year 2

Recovery from setbacks (1.3.1)

Despite the set-backs experienced owing to personnel changes, illness of

project participants, etc., we have stayed largely to our planned timetable. This

is due to the excellent efforts of all our team members, but I would like to

highlight two from our Birkbeck team in particular. Our CBAF Support Officer

(whose main role is to author TRIADS CBA), Ellen McCarthy Howey, has been

an outstanding contributor, working to tight deadlines and producing work of

exceptional quality. She also has helped our Project Assistant, Caroline Pellet-

Many, to quickly become highly proficient in TRIADS authoring. Together, they

have ensured that our TRIADS assessments have been delivered on time and

without technical faults. We are fortunate in having such talented staff on the

project!

OLAAF Conference 1 (1.3.2)

As described further in Section 1.4.4, we held an internal conference for all

project participants in Jan 2004. Attendees viewed this as a truly enjoyable and

productive event by all accounts. It certainly sparked a great deal of enthusiasm

and has provided avenues for collaboration subsequently. Many thanks are due

to Jenny Phillips who did the bulk of the organisation just prior to her departure

from the project in Dec 03.

Collaboration with the FAST Project (1.3.3)

We have been very pleased to become involved with another FDTL project,

Formative Assessment in Science Teaching (FAST) [FDTL 74/02]. There have

already been numerous benefits to our collaboration; these are described

further in Section 1.5.1.

Additional Interest Group Members (1.3.4)

We have gained 3 new OLAAF Interest Group members at 2 different

institutions, taking the number of institutions represented to 9. See Section

1.4.2 for more information.


1.4 Major Activities Undertaken in Year 2

OLAAF Consortium Partner Sites (1.4.1)

The OLAAF Consortium Partners, the academic units represented, and the Site

Leaders are listed:

• Birkbeck, University of London School of Biological & Chemical Sciences, Site Leader: Dr Glenn Baggott. School of Earth Sciences, Site Leader: Dr Karen Hudson-Edwards

• Brunel University Biological Sciences, Site Leader: Dr Darren Griffin [from 1 October, University of Kent. See also, Section 1.2.3]

• London Metropolitan University Human & Health Science, Site Leader: Prof. Chris Branford-White

• University of Birmingham Civil Engineering, Site Leaders: Dr Richard Freer-Hewish and Ms Barbara Hallam [from 1 October Physiotherapy; Site Leader: Dr Jill Ramsay. See also, Section 1.2.4]

• University of Brighton Institute of Nursing & Midwifery, Site Leader: Mr Patrick Saintas

• University of Plymouth Biological Sciences, Site Leader: Dr Les Jervis. Chemistry, Site Leader: Dr Simon Belt. Navigation, Neil Witt

• University of Wales College of Medicine Dental School, Site Leader: Dr John Potts

A summary of the activities undertaken in Year 2 by each OLAAF Consortium

Partner Site is contained in Appendix 1.

OLAAF Interest Group Sites (1.4.2)

The OLAAF Interest Group has been formed in order to extend the reach of the

project beyond the consortium by involving colleagues at other institutions

throughout the UK. The Interest Group has added a further element of diversity

to the project participants in terms of geography, subject area and pedagogy. It

also encompasses a range of computer-based assessment (CBA) authoring

tools. This will allow us to pilot our resources to ensure that they are truly

generic, with relevance both to TRIADS-users and to users of other CBA

authoring tools.

The Interest Group members recruited in Year 1, the subject areas represented,

and the Site Contacts are below:

• British School of Osteopathy: Osteopathy (Dr Stephen Tyreman)

• Edgehill College of Higher Education: Geology, Geography (Mr Gerry Lucas)

• Keele University: Life Sciences (Dr Peter Chevins)

• Kingston University: Life Sciences, Computing, Chemistry, Mathematics, Pharmaceutical Science, Medicinal Chemistry. (Main contact: Tim Linsey)

• Manchester Metropolitan: Biological Sciences (Dr Alan Fielding)

• University of Ulster: Chemistry and Biochemistry (Dr Keith Adams)

• Warwickshire College: Equine Distance Learning (Ms Karen Tolley)

There have been three new additions to the Interest Group since Year 1. Dr


Alan Boyle and Dr Steve Millard both are representatives from University of

Liverpool in Earth & Ocean Sciences and Civil Engineering, respectively. They

have long experience of using TRIADS to produce innovative and effective

CBA. Mr Gavin Sim from the University of Central Lancashire Department of

Computing is making use of CBA in various ways in their curriculum.

A summary of the Interest Group activities for Year 2 is contained in Appendix

2.

Assessment Construction Resources (1.4.3)

Researching and compiling material for the Assessment Construction

Resources (ACR) was a major focus of activity during the first year of the

project. Since Summer 2003, a versions of ACR materials have been available

on a private internet site for members of the project team and Interest Group to

utilise, evaluate and contribute to.

It was envisaged that ACR materials would be made publicly available from ca.

May 2004, but this was not possible, primarily due to the loss of the Project

Manager, who had been working on readying these for general release.

Meanwhile, in line with advice arising from peer review of the draft materials

and following some brainstorming within the Project Team, we have planned

some changes to the nature and content of the materials we intend to release.

These plans are sketched in Appendix 3. We are now aiming for an end of

2004 initial release via our web site and will exhibit materials in “beta” stage

through May 2005. During this time we will seek review by visitors to the web

site, by Project Consortium and Interest Group members, and other peers.

It should be noted that some useful review of the draft ACR took place in

Summer 2004 when we pleased to have input from an overseas visitor, Dr Paul

McGrath (University of Newcastle, Australia) who was spending a sabbatical

period in the UK. Dr McGrath is a physiologist who has extensive experience in

development of computer-aided learning packages. Paul has also agreed to

distribute some of these materials to colleagues in Australia, particularly to

those responsible for CAL and CBA implementation at Newcastle. It will be very

useful to obtain a non-UK perspective on our work and we look forward to

receiving this feedback.


OLAAF 1 Conference, Project Group Meeting, Steering Group Meeting (1.4.4)

OLAAF 1 Conference. The first OLAAF Conference was held at St Edmund’s

Hall, Oxford in Jan 2004. This was an opportunity for members representing the

entire OLAAF Network to meet and to discuss their work, past and planned.

Appendix 4 contains a Programme and delegate list.

3rd

Project (Site Leaders) Meeting. This took place on 23rd

September 2004, at

Birkbeck College and preceded a Steering Group Meeting. Both meetings were

very well attended with only two members being absent.

A major aspect of business at the Project (Site Leaders) Meeting was to monitor

progress on Case Studies and to identify areas where members of the project

could assist each other in their work through the final year. These activities also

served as preparations for the OLAAF 2 Conference, to be held on 19 and 20

Nov 2004. Also considered were a tentative timetable of events for OLAAF 2 as

well as plans for dissemination events in the final year. See Appendix 5 for an

agenda.

3rd

Steering Group Meeting. At the time of the Steering Group meeting, this

Annual Report was incomplete, but the Appendices and financial reports were

tabled and were accompanied by verbal reports given by the Project Director

and others. The Steering Group agreed to allow the Chair to examine the

finished report and approve it by Chair’s action, should he find it meeting his

approval. The meeting concluded with discussion of the ACR and of CBA

developed during the first year. Dr Jervis, Site Leader from Plymouth, gave a

brief, engaging presentation of approaches to the application of CBA being

employed at Plymouth which are designed to promote problem solving abilities

in biochemistry students.

It was extremely useful to have our HE Academy Senior Advisor, Dr Paul

Martin, present at this meeting. We are grateful to him for finding the time to

attend at this busy part of the year.

The Minutes of the Steering Group Meeting are found in Appendix 6.

Membership and terms of reference of the Steering Group are included in

Appendix 7 as part of the Dissemination Strategy.


Dissemination Strategy and Major Activities in Year 2 (1.4.5)

Revisions to the original Dissemination Strategy (Appendix 7) have been

made. Many of the changes are purely editorial (clarifying ambiguities and

correcting minor errors), while others have simply taken account of

developments, reflecting minor changes of plan or our engagement with new

opportunities. Key changes relating to the Dissemination Strategy are described

briefly below.

Web Log. This did not prove popular as a mechanism for internal

dissemination/communication. We have recently instated a hosted project

management system called Basecamp (http://www.basecamphq.com) as a

form of replacement. This is an inexpensive (less than £11 per month) and

well-featured solution with a calendar, messaging facility, to-do list, and other

organisational tools. It has been used internally (by the Birkbeck team), but

will soon be made available to the rest of the OLAAF partners.

Publicity. The OLAAF web site was re-launched in late August 2004 (note

that a few areas still require updating). The appearance was altered to more

closely resemble the style used by parts of the Birkbeck site (on which it is

hosted) while still retaining its distinctive branding. Most pages have been

altered to contain less text than before; visitors are encouraged to download

PDFs containing additional commentary or information, if desired. This is a

more user-friendly way of sharing materials and leverages the time already

spent in writing up-to-date strategy documents, advertising materials, etc.

The Assessment Construction Resource materials are currently being re-

formatted so that they will become (mainly) downloadable PDFs or Word

documents, using the web site as an organising container.

http://www.bbk.ac.uk/olaaf

Timeline. A number of rather minor changes have been made to the timeline

of dissemination activities, but two additions represent important

developments for the Year 3: the planned workshop at Loughborough on 1st

Dec 2004, and the planned national conference on 27th

/28th

Jun 2005 at

Warwick. For more information on these, see the OLAAF web site:

http://www.bbk.ac.uk/olaaf/events/index.html

Major dissemination activities undertaken in Year 2 are listed on the next page.


Presentations and workshops

• ‘OnLine Assessment and Feedback’ a one-day series of presentations and workshops. Held at University of Plymouth, 21 October 2003. Co-sponsored by University of Plymouth, LTSN Bioscience, LTSN GEES, and OLAAF. ca. 85 attendees. Presentations from OLAAF by G Baggott, R Rayne (Birkbeck); N Witt, S Belt (Plymouth); D Mackenzie (Derby).

• R Rayne and J Phillips (Birkbeck) gave a presentation to ca. 20 attendees at the TQEF Annual Conference, Nottingham, 3 and 4 Nov 2003. Title: Developing a multidisciplinary shared resource for computer based assessment.

• K Shah (Brunel) gave a presentation to ca. 12 attendees at meeting of the ‘Virtual University Project’ group, Brunel University. Title: Evaluating innovations in CAL and CBA.

• R Rayne, K Shah and J Phillips (Birkbeck) led a workshop with sessions covering Writing Quality Feedback for CBA and Evaluating Your CBA at Birkbeck on 6 Nov 2003. There were ca. 15 attendees from Birkbeck, Brighton, British School of Osteopathy, and University of Kingston.

• G Baggott and J Phillips (Birkbeck) gave presentations on the OLAAF project as part of Innovations in e-learning in HE a day-long symposium, held at Birkbeck, 4 December 2003. ca. 30 attendees.

• R Rayne (Birkbeck) and P Saintas (Site Leader, Brighton) gave a presentation / workshop on OLAAF to ca. 25 attendees at University of Brighton on 16 Jan 2004.

• OLAAF has committed to contributing a chapter (on strategic use of CBAF) in a multi-author volume, Innovating in Assessment (ed. G Gibbs). The chapter will include examples from across the OLAAF project.

• R Rayne (Birkbeck) led a session in May 2004 contributing to a module within

Birkbeck’s Teaching in Lifelong Learning (TiLL) certificate. This is a compulsory module undertaken by all probationary lecturers at Birkbeck. The session was entitled The OnLine Assessment and Feedback Project (OLAAF): Strategic use of computer-based assessment. ca. 15 attendees.

• R Rayne (Birkbeck) gave a presentation to approximately 15 attendees at the Northumbria/EARLI SIG Assessment Conference in Bergen, 23-25 June 2004. Title: The formative-summative interface: strategic incorporation of computer-based assessment (RC Rayne and GK Baggott).

• R Rayne (Birkbeck) presented OLAAF-related work to approximately 35 attendees at the 8th International CAA Conference in Loughborough, 6 & 7 July 2004. Titles: Student perceptions of computer-based formative assessments in a semi-distance module (GK Baggott and RC Rayne); Computer-based and computer-assisted tests to assess procedural and conceptual knowledge (RC Rayne and GK Baggott).

• P Saintas (Brighton) and D Mackenzie (Derby) presented at the University of Brighton Learning and Teaching Conference on Friday 16th of July. Title: Beyond Multiple Choice--Using TRIADS to Design Online Assessment and Feedback.

• L Jervis (Plymouth) presented OLAAF-related work at the Bioscience 2004 Conference (sponsored by the Biochemical Society) in Glasgow, 18-22 July 2004. Title: Incorporating On-Line Assessment and Feedback into an Integrated Biochemistry Course (L Jervis and L Jervis). The abstract will appear in Biochemical Society Transactions.

Flyers

• Northumbria/EARLI SIG Assessment Conference, Bergen, Norway, June 2004. Flyers and paper abstract distributed.

• CAA Conference, Loughborough, July 2004. Flyers in all 150 delegate packs.

E-mail Newsletters

• 3rd

OLAAF Newsletter (by email to approximately 300 recipients) Dec 2003.

• 4th

OLAAF Newsletter (by email to approximately 300 recipients) Sep 2004.


1.5 Collaborations and Links

Other FDTL projects (1.5.1)

We have collaborated with other FDTL projects in exchanging ideas and

refining evaluation and dissemination plans. In particular, there is much

common ground between our project and the project Formative Assessment in

Science Teaching (FAST) (FDTL 74/02).

http://www.open.ac.uk/science/fdtl/index.htm

We are members of the He Academy Centre for Bioscience Special Interest

Group established by FAST, and Dr Rayne is contributing to their work by

undertaking a FAST Development Project. OLAAF and FAST also have agreed

to a common reporting format for Case Studies/Development Projects so that

these may ultimately be easily collated into a common, searchable digital

archive.

Our collaboration has now extended to planning and holding shared

dissemination activities through national workshops/conferences. The

forthcoming events are:

Assessment in Science Teaching: Technological Solutions? To be held at Loughborough University, 1

st Dec 2004.

The Science Learning and Teaching Conference 2005. To be held at Warwick University, 27

th-28

th June 2005.

Planning of the Loughborough event has also brought us into contact with FDTL

159/02, Promoting Physics Learning and Teaching Opportunities (PPLATO),

who also will co-sponsor the Warwick event.

http://www.rdg.ac.uk/AcaDepts/sp/PPLATO/publish/index.htm

In addition, we have benefited from useful discussions with Prof. Tony Gardner-

Medwin, whose FDTL4 project, 77/02 Dissemination of Formative Confidence-

Based Exercises, has particular relevance to the work of OLAAF.

We look forward to continued collaborations with these and other FDTL projects

in Year 3.

Bioscience and other HE Academy Subject Centres (1.5.2)

We have collaborated closely with the HE Academy Centre for Bioscience

throughout the bid writing stage, and from the start of the project. Dr Steve Maw

from the Bioscience Centre is a member of our Steering Group. We have

attended all relevant events organised by the Centre for Bioscience, and

OLAAF members have made presentations at several of these. We co-


sponsored a joint LTSN [as was] Bioscience and LTSN GEES event on OnLine

Assessment and Feedback at Plymouth on 21st October 2003. We have

contributed articles to web sites, newsletters and bulletins in the following

related Subject Centres [all in their nomenclature at the time, as LTSN subject

centres]: LTSN Health Science, LTSN-01 (Med, Den, Vet), LTSN Bioscience

and LTSN Hospitality Leisure Sport Tourism.

We will continue our close working with the Centres in Year 3 of the project. In

particular we will collaborate via both of our major, planned dissemination

events as noted in Section 1.5.1, to which the Centre for Bioscience and the

Physical Sciences Centre are co-sponsors.

Other bodies (1.5.3)

As in Year 1, each Site Leader has been responsible for forming links with their

local Disability Officers, learning and teaching centres, and central C&IT support

staff. Active participation in other relevant institutional networks continues to be

widespread. For example the site leader for Brighton is also the nominated

Learning Technology Support Officer for the Institute of Nursing and Midwifery;

at Birkbeck project representatives sit on both the Cross-College Working Party

in ICT and Education and the Working Party on Teaching and Learning.

Participation in such groups affords opportunities to raise the profile of the

OLAAF project and to address its concerns through key networks, and will

ultimately facilitate potential input into policy-making within the institutions. As

an example, Dr Rayne (Project Director), has been closely involved in the

drafting of Birkbeck’s eLearning Strategy (this should come into force in 2005).

He and Dr Baggott (Birkbeck Site Leader) also have had a major impact on the

revision of Birkbeck’s Learning and Teaching Strategy (due to be fully revised

and submitted to HEFCE in 2005). Dr Rayne and Dr Baggott hope to have

continued influence on policy at Birkbeck through a CETL, should this be

funded (Dr Rayne and Dr Baggott were co-authors of Birkbeck’s CETL bid); the

CETL also would serve as a major outlet for the continuation of OLAAF

activities beyond the FDTL4 funding.


1.6 Outputs from Year 2 Activities

Re-Launched OLAAF website (1.6.1)

The site, which originally went live in Jan 2003, was thoroughly re-designed in

Aug 2004 and some work to re-vamp it further is on-going. It is a regularly

updated portal containing information about the project, links to materials we

have produced, and links to other useful sites. Apart from cosmetic changes,

the tone of the site was altered in line with the progress of the project. While

originally “invitational”, being designed to recruit new members to the OLAAF

Network, it now is more focused on organising and delivering outputs. See

Section 1.4.5 above for additional information.

http://www.bbk.ac.uk/olaaf/

Assessment Construction Resources (1.6.2)

The evolving ACR have been discussed already in Section 1.4.3 and in

Appendix 3.

Computer-Based Assessments (1.6.3)

There has been a great deal of activity in authoring and evaluating CBA

authored under the auspices of OLAAF in Year 2. A synopsis of “centrally

supported” CBA (i.e. authored by the Birkbeck team and delivered via our

server) is given below, in chronological order. (Complete reports on CBA

produced, delivered and evaluated at other Partner Sites and/or Interest Group

Sites will appear in OLAAF Case Studies, some of which will be complete as

soon as Spring 2004.)

Autumn 2003. The Field Biology module (2nd

-year, Birkbeck BSc Biology; 18

students) received a new TRIADS-based final examination which was improved

in line with student evaluations from the previous year. An in-depth evaluation of

the application of CBA in this module was carried out using questionnaires and

informal focus group sessions. The outcomes of this evaluation and our

responses to them were reported at the 8th

International Computer Assisted

Assessment Conference in July 2004 (see Section 1.4.5). New TRIADS

computer-based tests were authored for the Practical Biology 1st-year module

for Birkbeck bioscience undergraduates (90 students). This included a practice

CBA to acquaint students with the workings of TRIADS-delivered tests and a

CBA covering several subject areas, but focusing mainly on the structure and

functioning of biomolecules. A series of tutorial/tests on basic numeracy for

nurses was authored for use at University of Brighton; these were piloted in

2003. They will be used widely in 2004 and evaluated with respect to their

efficacy in supporting other assessments. A CBA for the 2nd

-year module at

Birkbeck, Cellular Metabolism (30 students), was authored and delivered in


Spring 2004. This test was designed to help students diagnose misconceptions

and lacunae in their knowledge in advance of a written theory examination. A

series of CBA covering cell biology were authored for pilot use in the Spring

term at London Metropolitan University (LMU). There were some technical

problems with the LMU network that precluded widespread use of these

assessments, but we were able to solve these over the Summer 2004. These

CBA will be administered to large numbers (>100) students in 2004-2005 and

evaluations of their impact will be carried out.

Spring 2004. The main authoring activity was centred on Molecular Cell

Biology, a first-year unit for biosciences students at Birkbeck. We have used

TRIADS CBA extensively in this module for several years, and have reported on

our work via several outlets, most recently in 2004 at the EARLI/Northumbria

SIG Assessment conference in Bergen and at the 8th

International CAA

Conference in Loughborough (see Section 1.4.5). Evaluation of students’ study

behaviour in this module and in another contemporaneous module where there

was no CBA provided for learning support was carried out using the

“Assessment Experience Questionnaire” and the “Study Process

Questionnaire”. These evaluations, carried out as a FAST Development Project

(see Section 1.5.1), have suggested that the CBA in Molecular Cell Biology

promoted a steady pace of study throughout the 14 weeks of the module. New

CBA were produced and delivered for the first time to support student learning

in Statistics for Biologists, a year 3 module for Birkbeck BSc Biology students.

The use of TRIADS CBA permitted students to practice statistical problems

repeatedly (through randomisation of variables within the question frameworks)

and receive immediate feedback on their efforts. Evaluations indicated that

students valued this support and on this basis additional supporting CBA

materials will be developed for use in Spring 2005.

Summer 2004. Assessments for the Field Biology module at Birkbeck were

extensively re-vamped and extended based on feedback form last year’s

iteration. The new materials were evaluated by students (this is ongoing, as the

module carries on into Autumn 2004) and by the External Assessment

Evaluator in August 2004 (see Section 1.7.1). A series of TRIADS CBA on

geochemistry were produced for use in 2004-2005 on Birkbeck Earth Sciences

BSc programmes. Authoring these represented a useful challenge, as the test

items were converted from existing paper-based homework assignments. The

systematic approach to this conversion effort has informed aspects of our ACR;

this should be of use to others who wish to move paper-based materials into

useful computer-based formats. Other work, over the summer and to Year 2

end, was devoted to usability analysis of the TRIADS interface and the initiation


of its re-design. An external consultant was hired to assist; her

recommendations are due in October 2004 and on the basis of these we will

prepare new question templates for future use.

1.7 Evaluation of the Work

Evaluation Strategy and Activities (1.7.1)

The Evaluation Strategy (revision 2) for the project is attached as Appendix 8.

This was developed from discussions at the first Steering Group meeting in

2003, and refined in consultation with our external evaluators, Dr Joanna Bull

(Eduology) and Dr David O’Hare (Centre for Interactive Assessment

Development, University of Derby). Although most of the originally articulated

goals remain as intentions of planned evaluation activity, the strategy has been

revised as of Aug 2004 to take account of changing circumstances within the

project. A summary of the revisions is provided on the last two pages of

Appendix 8.

The second External Assessment Evaluation took place in August 2004 and

culminated in a site visit by Dr David O’Hare (Derby) on 20th

August 2004. This

evaluation focused attention on project outputs including developments in the

ACR, and examined the qualities of selected computer-based assessments; the

brief supplied to the Evaluator is given as Appendix 9. Dr O’Hare was taken ill

somewhat after the time of his visit and has not as yet been able to complete

his written report. Nonetheless, his commentary on the day was positive,

valuable, and has informed our work since.

We had planned for external project evaluation, undertaken by Dr Joanna Bull,

to commence from Summer 2004. As discussed previously (see Section 1.2.4)

Dr Bull passed away in June 2004. We have yet to arrange a replacement.

Because of the necessary change in timing of the evaluation, it is likely to be

more retrospective in nature, and more focused on how we can ensure effective

continuation of the outcomes of the OLAAF project. We have contacted several

candidates to serve as Project Evaluators and await their replies. As indicated

in the strategy document, our aim is to engage an evaluator before the end of

2004, to receive recommendations in early Spring 2005, and to follow-up on

these by Summer 2005.


1.8 Learning from Activities and Progress made during Year 2

Changing Conceptions of the Focus of the OLAAF Project (1.8.1)

The main goals of the OLAAF project as enumerated in the bid for funds and

elsewhere remain as the key organising principles driving the project. As the

project has progressed, and our philosophy informing its direction has matured,

we have made some changes to the way in which we articulate our vision for its

success. We presently describe the main goal of the project in the following way

(e.g. see the OLAAF web site):

Computer-based assessment (CBA) offers many potential benefits to students and teachers, not least the possibility of providing instantaneous, precisely-targeted feedback to the test-taker. With the increasing availability and ease of use of CBA authoring tools (particularly through virtual learning environments), it is especially important to ensure that when CBA is used, it is applied in an educationally principled manner. Quite a few very useful CBA authoring resources exist that focus on, for example, construction of multiple-choice questions and/or on technical/logistical aspects of CBA. But few resources focus adequately on guiding the author in designing appropriate questions (especially those that are not MCQs), on combining these questions effectively, and/or on composing the feedback to accompany them. Still fewer existing resources recognise that even the best CBA may be of minimal use to the student unless it is appropriately interwoven within a course’s assessment regime.

The OLAAF project will address these deficits by developing and disseminating tools to support authors in the design, delivery and evaluation of CBA with feedback (CBAF). These Assessment Construction Resources will reflect the scholarly approaches undertaken by the project participants (reported as OLAAF Case Studies) as well as findings from the latest literature on assessment for learning.

The above, though remaining true to our original vision, differs in some subtle,

but important ways. A key element that has come to the fore is an emphasis on

assessment for learning. Whilst present in our original vision (e.g. see Section

1.8.4 of the Year 1 Annual Report), it has become clear as we implement CBAF

in different scenarios and with different types of students that the formative

elements that can be built in to quality CBA are those which have the greatest

impact on learning. Offering the opportunity for practice, providing immediate

feedback to check for learning, and using frequent CBA in a

summative/formative mix all are can have important impacts on learning and, as

our evaluations have indicated, are valued by students.

A second key change of emphasis—congruent with the emphasis on

assessment for learning—is a focus on ensuring that CBA are appropriately

interwoven into the overall assessment regime. Here, we have been inspired by

the work of Gibbs et al., and particularly of the FAST project, with respect to the

“11 conditions under which assessment promotes learning”. One can argue

over the way these 11 conditions are articulated, whether some should be

amalgamated, etc, but the overall idea remains powerful. Our experience in the


OLAAF project so far has shown us that an attempt to meet these conditions is

key to promoting student learning, and that in some cases, CBA may not be the

best way forward! Designing an effective assessment regime is a difficult, but

important business; under appropriate conditions, CBA can play an important

role. One of our goals in the remainder of the project will be to articulate such

conditions and provide evidence-informed advice on how teachers can use CBA

most effectively within a mixed assessment regime.

Embedding changes (1.8.2)

It is worth reiterating the points enumerated in the Year 1 report regarding

conditions under which computer-based assessment is likely to continue in the

partner institutions beyond the life of the project, and the accompanying support

and funding:

1. Building networks of support - opportunities for collaboration, exchange

of ideas, peer review.

2. Providing training based on highly practical resources which encourage the adoption of good models of working practice which should outlast the project and equip individuals and teams to continue developing CBA in the future.

3. Developing skills and expertise in relevant CBA authoring tools and providing software and ongoing support.

4. Involving existing support centres and institutional networks.

5. Integration of local (i.e. Departmental) plans with Institutional Learning and Teaching Strategy and e-Learning Strategy.

Evidence that we have made progress toward meeting these conditions has

been given throughout this document. In its final year, the project will continue to

use this framework to inspire its work and to evaluate the effectiveness of its

activities.

Responding to HEFCE priorities (1.8.3)

The OLAAF project remains committed to responding to HEFCE priorities,

particularly as described in the Year 1 report:

The project emphasises the importance of well-constructed formative

assessment that helps learners to understand their developing strengths and

makes visible the value added through their studies (a priority highlighted in

the HEFCE Strategic Plan, 2003-2008, under ‘Excellence in Learning and

Teaching’). Our commitment to maintaining principles of good assessment

practice and transferring these to a computer-based environment also

embraces the challenge identified in the HEFCE Strategic Plan, (under

‘Meeting customer and stakeholder needs’) of maintaining the quality of

experience offered to the student through new media.


2. Year 3 Plan

2.1 Project Work Plan for Year 3

A Project Timeline for Year 3 is attached as Appendix 10. Key milestones are identified within the

timeline, and include:

• National conference OLAAF 2, Nov 2004

• Assessment Construction Resources (ACR) available outside of project via OLAAF web site,

Dec 2004

• Alpha-stage specification of ACR CD content, Mar 2005

• Completion of Partner and Interest Group case studies and release version of ACR CD, Aug

2005

Since last year, we have added a few target statements to the plan (and or re-phrased them), having

analysed the tasks and responsibilities for Year 3 in more detail. However, the original targets remain

unchanged and we anticipate no consequential changes to the budget for Year 3 (see Section 3).

Further activity in building the ACR will follow the framework outlined in Appendix 3.

Also included following the Year 3 Timeline is a list of continuation activities. We see these activities

fully as integral parts of the project, and it is in the interest of OLAAF project members to see them

through (importantly, they are activities with potential to contribute to publications, a prime motivator

for academic staff!). We are hopeful, too, that continuation will also be afforded by the location of a

CETL at Birkbeck. Our proposed CETL in many aspects will build on the activities and outputs of the

OLAAF project and much of our putative CETL’s philosophy has indeed been informed by what we

have learned thus far in undertaking the OLAAF project.

2.2 Dissemination Plan for Year 3

The revised Dissemination Strategy (Appendix 7) will be carried forward in Year 3. We have

already planned major dissemination events (esp. national workshops and conferences) for Year 3

that leverage our collaborations with other FDTL4 projects and with appropriate HE Academy

Subject Centres. Given their remit and the audiences to whom they cater, these events are sure to

reach wide and receptive audiences. The Project Director has been closely involved in the planning

of these events and will continue to be so in coming months.

2.3 Evaluation Plan for Year 3

Plans articulated in our Evaluation Strategy (Appendix 8) will be carried forward in Year 3.


2.4 Help Needed for Year 3

Looking toward Year 3, we already have asked for the guidance of our Senior Advisor in identifying

an appropriate replacement External Project Evaluator. We will remain in contact with him regarding

this issue until it is resolved.

We would find it useful, however, to have an extended deadline for final reporting to HEFCE. We

anticipate working right up to the end of the project in Sept 2005, and it would be appreciated if we

were afforded a short period of reflection (perhaps an extra 4 to 6 weeks?) to allow us to incorporate

complete accounts of all of the outcomes in the final report.

Otherwise, we don’t foresee any additional special help being required in Year 3. We have made

useful alliances throughout the project that we anticipate will continue to serve us well. Should

circumstances change, we are confident that we will be well advised by our Senior Advisor.


3. Financial Statement

3.1 Expenditure Statement for Year 2

The Expenditure Statement for Year 2 is shown in Annex B; an alternative view of this information is

found in Appendix 11. This same form of presentation was given in the Year 1 Annual Report as its

Appendix 10. The table herein has been updated to show actual expenditure for Years 1 and 2 as

well as projected expenditure for Year 3. Each is shown in relation to the originally budgeted sums

(as enumerated in the OLAFF Stage 2 bid); this comparison has been chosen for clarity and

consistency with the method of record keeping.

The total award from HEFCE over the 3-year course of the project is £249,956. It should be

emphasised that this total sum is still required to support the project over this period and, further, that

we presently anticipate no substantive changes to the total amounts allocated to the respective

named categories.

In the subsections below, explanations for the lower-than-anticipated rates of expenditure to-date are

given, starting with a brief recap of the position at the end of Year 1 (Section 3.1.1). An account is

then given of the current position at Year 2 end (Section 3.1.2). Anticipated expenditure for Year 3 is

discussed in Section 3.2.

Financial position at Year 1 end (3.1.1)

To describe the present situation fully requires this brief digression which refers

to the Year 1 Annual Report.

Annex B (and Appendix 10) in the OLAAF Year 1 Annual Report showed an

underspend of £17,541 for Year 1 (against income from HEFCE of £96,090). As

described in the Year 1 Report, this arose almost entirely for the following two

reasons: 1) the rate of expenditure in the Staff category was less than originally

anticipated; 2) the rate of expenditure in the Dissemination category was less

than originally anticipated. Further explanation was given in the Year 1 Report.

Also in Annex B of the Year 1 Report was a Budget Statement showing

anticipated expenditure for Year 2. Overall, this showed a small overspend (ca.

£6,800), effectively “reclaiming” a portion of the Year 1 underspend. According

to this Budget Statement, the remainder of the Year 1 underspend was to be

reclaimed in Year 3.

As indicated in 3.1.2 below, accumulated funds from the Year 1 underspend

were not reclaimed in Year 2. These funds will, however, be spent on necessary

project activities by the end of its 3-year term.


Financial position at Year 2 end (3.1.2)

In Year 2, we again have spent at a slower rate than was originally anticipated.

Comparing against the originally budgeted sums, the underspend is £18,279

(Annex B, Expenditure Statement; Appendix 11). This creates a cumulative

underspend going into Year 3 of £35,354 (Annex B, Budget Statement;

Appendix 11). Reasons accounting for this reduction in spending rate are given

below.

Resignation of the Project Manager. The Project Manager resigned in Dec 2003

for personal reasons. (See 1.2.1 for further discussion of this.) There was a 3-

month gap between the resignation of the Project Manager and the appointment

of a Project Assistant which reduced the outlay of funds in the Staff category.

This “saving” will be reclaimed (i.e. spent) in Year 3, as described in Section

3.2. These alterations to the planned staffing met with the approval of our HE

Academy Senior Advisor, Paul Martin (see Year 2 Quarterly Progress Reports 1

and 2).

Additionally reducing the Staff expenditure was the fact that the Project

Assistant’s rate of pay is considerably less than that of the Project Manager.

Much of this difference will be used (as agreed with the HE Academy Advisor)

to purchase clerical support and teaching assistance to Dr Rayne, who has

taken on the role of Project Manager. However, owing to the nature of the

academic cycle and the pattern of work within the OLAAF project, a substantial

proportion of this difference will not be spent until Year 3.

Retention of Dissemination and Travel/Subsistence Funds for Specific Use in

Year 3. It became clear in ca. Spring 2004 that our collaboration with the FAST

project would happily result in our co-sponsorship of two national dissemination

events in Year 3. Because of this, it was necessary to deliberately earmark

funds form these categories toward this planned expenditure. This adjustment

rather dramatically reduced the anticipated expenditure from these categories in

Year 2, but all of this—plus the underspend from Year 1—will be allocated in

Year 3. It also was clear that a number of project partners would benefit from

attendance at conferences, etc. in 2005 to present their OLAAF work. We have

therefore shifted additional Dissemination expenditure to Year 3.

Delays in Paid Evaluation Activities. We spent less than anticipated on

Evaluation in Year 2 primarily owing to the untimely death of our external

evaluator, Dr Joanna Bull. As of Year 2 end, we have not yet been able to

arrange a replacement evaluator. Significant expenditure relating to this service

will be postponed to Year 3.


3.2 Anticipated Expenditure for Year 3

Anticipated Expenditure for Year 3 is shown in Annex B, Budget Statement; an alternative view of

this information is found in Appendix 11.

The totals in Annex B and Appendix 11 reveal a sizable cumulative underspend (£35,354) relative

to the originally budgeted sums going into Year 3. As explained in Section 3.1, this has accumulated

owing to unavoidable reductions in the rate of expenditure in Years 1 and 2. However, this

underspend is to be matched by an identical planned overspend in Year 3, balancing the figures and

resulting in the same overall expenditure as appeared in the original OLAAF budget.

As pointed out in Section 3.1, some of the expenditure originally planned for Years 1 and 2 has

simply been shifted into Year 3. These postponements of expenditure have already been explained;

in general, these represent “normal” alterations in accord with changing circumstances within the

project. Another substantive factor contributing to consumption of unspent funds in Year 3 is the

extension of contract of the CBAF Support Officer (Ellen McCarthy Howey). Originally, this post was

funded only until March 2005, but we have extended it by 6 months to keep Ms Howey in post until

the project’s end. Further regarding this post, we have increased the salary by one increment in

recognition of the stellar work Ellen has performed and to ensure that we retain her in this critical

post. Ms Howey, originally on a 0.75 FT contract, also has increased her time commitment to the

project; the appointment is now 0.85 FT. The large increase in expenditure on Dissemination in Year

3 reflects, in part, the substantial costs involved in co-sponsoring a national workshop (to take place

in Dec 2004) and a two-day national conference (to be held in June 2005). We feel that these are

excellent investments, as they will expose the OLAAF project to a large audience who are likely to

benefit from our outputs. Other additional expenditure in this category in Year 3 results primarily from

cumulative effects of “phase shift” across the 3-year period. Shifting of some evaluation activities

from Year 2 to Year 3 also contributes significantly to the increased expenditure in the final year.

In summary, our projected expenditure in Year 3—while substantially higher than anticipated at the

start of the project—will support essential project activities and will be met by recovery of the

cumulative underspend.


Declaration

I certify on behalf of the lead institution that the attached progress and financial report, and the information

contained therein is correct, and that the funds allocated have been applied to the purpose for which they

were made available.

Project Name: OnLine Assessment and Feedback (OLAAF)

Project Number: 5/02

Head of Institution or Nominated Deputy

Name (print)

Signed Date

Project Director

Name (print) Dr Richard C. Rayne

Signed Date


Annex A

Fund for the Development of Teaching and Learning (FDTL)

Activities and Progress

Project no. and name:

5/02 OLAAF: OnLine Assessment and Feedback

Reporting Period: 1 October 2003 - 30 September 2004

Phase 2 (May 2003 through Dec 2004)*

Objective/target/outcome/ milestone

Original date to be achieved

Output to demonstrate objective/target/outcome/milestone achieved

Date achieved

If applicable, reasons for slippage in date achieved

If applicable, reasons for change to original objectives

Authoring of CBAF at sites (for

semesters 1 & 2, first iteration)**

Nov 2003 See next item Nov 2003

Roll-out of CBAF’s in modules from Oct 2003– May 2004

TRIADS computer-based assessments delivered. Samples evaluated by External Assessment Evaluator (see penultimate milestone in this table). Reports on outcomes/evaluations given at national and international conferences in Summer 2004.

May 2004

Refinement and final implementation

of ACR

Oct 2003 See next item Oct 2003

ACR piloted within OLAAF Network Feb 2004 Materials available to OLAAF project

members via website; edited to take account of project members’ commentary collected at OLAAF Conference 1 in Jan 2004.

Feb 2004

Draft ACR available publicly on web

site for outside use and evaluation

May 2004 Resignation of personnel and

decision to change form in which ACR materials will be made available for public use. Also permits continued peer review by project team prior to release. New target: Dec 2004.

*We have included work of ongoing nature that may not end formally until sometime into project year 3. Note also that “Phase 2” refers to a time period as defined in the OLAAF grant proposal (Stage 2). The endpoint of Phase 2 has been redefined herein; it is now considered to be Dec 2004 rather than Feb 2005. All of the “Preparatory Phase” and “Phase 1” activities have been completed and these were reported in the OLAAF Year 1 Annual Report. **Reported as an ongoing activity in the OLAAF Year 1 Annual Report.


Phase 2, continued

Objective/target/outcome/ milestone

Original date to be achieved

Output to demonstrate objective/target/outcome/milestone achieved

Date achieved

If applicable, reasons for slippage in date achieved

If applicable, reasons for change to original objectives

Workshops at Target Sites on implementation of ACR

Dec 2003 Workshops held at Birkbeck and University of Plymouth.

Dec 2003

National conference: OLAAF1 Jan 2004 Conference held at St Edmund Hall, Oxford. Jan 2004

Reporting of developments at

International CAA Conference

July 2004 Two papers on OLAAF given by Dr Rayne;

attendance by 4 other OLAAF members.

July 2004

1st Internal Project Evaluation: to

review delivery of outcomes to-date

Jan 2004 Group work undertaken at OLAAF1

Conference.

Jan 2004

1st External Evaluation: to evaluate the project’s progress against plan, its internal functioning and to review dissemination of its resources.

by Jun 2004

Untimely death of our designated evaluator, Dr Joanna Bull. A new evaluator is being sought. New target: Dec 2004. See also 1.2.5.

2nd External Assessment Evaluation

of ACR and CBAFs

Aug 2004 External Assessment Evaluation Report

(incomplete)

Aug 2004 The evaluation was undertaken, as

scheduled; receipt of the completed report has been delayed owing to long-term illness of the evaluator.

3rd Steering Group Meeting Sep 2003 Minutes of meeting given as Appendix 6. Sep 2003 Note that the 3rd

SG

Meeting had been originally scheduled for ca. March 2004. Delays brought about by events beyond our

control led to its post-ponement. See also 1.2.2.

2nd Annual Progress Report to NCT Sep 2004 Nov 2004 Project Director was lead writer of Birkbeck’s CETL Bid; this commitment slowed compilation of the OLAAF report. Apologies for the delay.

Annex B



Expenditure Statement



Actual expenditure for year two 1 October 2003 - 30 September 2004

Income for year two

Allocation from the HEFCE: £96,553

Total for year two: £96,553 + £17,134 underspend from year 1 = £113,687 of income for year 2

Total budget for period

1

Total expenditure

Variance Underspend – Overspend +

Reasons for variance and action taken/intended

Staff2 £76,153 £65,866 -£10,287 The anticipated rate of expenditure was not met owing mainly to the Project

Manager’s resignation in Dec 2003. No direct replacement was sought, but a “Project Assistant” post was created. This appointment was made after a gap of 3 months. Some funds originally to be spent on the PM post will be allocated to purchase clerical and other support in Year 3. All of the presently unspent funds will be used for the originally intended purposes by project end. See Section 3 and

Appendix 11 for further explanation.

Travel & subsistence £5,800 £3,373 -£2,427 Underspend will be used in year 3 owing to shift of some activities to year 3. See Section 3.1.2.

Dissemination £9,300 £5,344 -£3,956 The rate of expenditure was deliberately reduced to permit OLAAF funds to support two substantial, collaborative national events in year 3. See Section 3 and

Appendix 11 for further explanation.

Equipment 0 0 0

Evaluation £2,800 £1,200 -£1,600 Owing to the untimely death of our external project evaluator, our expenditure in this category was less than anticipated. See Section 3.1.2.

Other costs (please

detail)3

£2,500 £2,491 -£9 No rectifying action required.

Total £96,553 £78,274 -£18,279 See above. To be carried forward into Year 3.

1

The originally budgeted sums—as enumerated in the original OLAAF budget—are shown here. See Section 3.1 for further comment. 2Birkbeck College paid 45% of the total cost of the Project Manager (this is additional to the budgeted sum shown) from Oct 2003 to Dec 2003, when she resigned her

post. Thereafter, Birkbeck funds were allocated to part of the salary of the new appointee, a “Project Assistant”, and to support other requirements of the project. 3

“Other costs” are primarily those incurred in purchase of software, but also include difficult to categorise computer accessories.

Annex B



Budget statement



Anticipated expenditure for year three (1 October 2004 - 30 September 2005) Income for year three

Underspend to be carried over from year two £18,279

Cumulative underspend (year 1 + year 2) £35,354 (= £17,134 + 18,279)

Allocation from the HEFCE £57,313

Total for year three £92,667

Anticipated expenditure

Variance +/– from original budget

1

Notes regarding allocation of funds

Staff2 £60,721 +£22,308 See Year 2 Expenditure Statement; also Section 3.2 and Appendix 11. Overspend

balances budget over the 3 years.

Travel & subsistence £8,060 +£2,760 Required to support activities now planned for Year 3. Overspend balances budget over the 3 years.

Dissemination £17,769 +9,069 Expenditure withheld from Year 2 will be invested in Year 3 to support dissemination events guaranteed to reach a wide audience. No change in total expenditure over the 3 years. See

Section 3.2 and Appendix 11.

Equipment 0 0

Evaluation £4,262 +1,462 Costs of a new External Evaluator are uncertain; in any case, the expenditure has been “time-shifted” to Year 3. Will be within budget over the 3 years. See Appendix X.

Other costs (please

detail)3

£1,855 -£245 Small underspend this year, but matches budget over 3 years.

Total £92,667 +35,354 See above. The year 3 overspend matches exactly the cumulative overspend. Balance at end of year 3 will be 0.

1

The “original” budget is taken here to be that which appeared in the OLAAF grant proposal. See Section 3.1 for comment on this. 2

Birkbeck College contributes to the cost of the Project Assistant. 3

“Other costs” are primarily those incurred in purchase of software, but also include difficult to categorise computer accessories.

annual report 30 september 2004 · 2004. 11. 15. · qmark question mark perception triads...

Documents