2. Evaluation Plan

Evaluation and quality assurance management is a crucial element of the Pelican project. This document sets out a plan for evaluation activities in line with project aims and objectives for the duration of the Pelican project. As the research process is non-linear, valuable feedback can be achieved at each stage in the project, which will be channeled back into technical specifications to allow for re-design and new testing of the pricing model. The evaluation will be carried out by project staff at both sites as well as incorporate the feedback of the Advisory Board where appropriate.

It needs to be emphasised that, as the project is exploratory in nature, some parts of the Pelican evaluation strategy form an integral part of the project methodology. For example, face-to-face interviews and/or focus groups with stakeholders in the HE/FE community provide the framework for good practice of the evaluation plan but, at the same time, they represent a key methodological tool to inform the development work.

The evaluation strategy will be supervised by the Project Director; however, all project staff are committed to implementing a successful evaluation plan. The Advisory Board will also contribute to the successful implementation of the evaluation plan.

The evaluation of Pelican must be closely tied to the objectives of the project. It is worth emphasising that these are to:

• To promote awareness of the different policies, and pricing strategies that are available for the delivery of commercially published electronic texts to students within the various stakeholder groups.
• To identify, document and disseminate strategic frameworks within which individual HEI libraries can develop budget strategies for electronic texts that are appropriate to their needs and resource limitations.
• To investigate, document and promote research into the most appropriate pricing strategies that satisfy the needs of all stakeholders.
• To investigate, document and promote the most appropriate hardware and software solutions to the problems that are identified for the delivery of the electronic texts.

Constant contact with the CLA, library staff, publishers, authors and users will allow the project team to:

• improve communication between all stakeholders, which, in turn, will have a positive effect on developing, and testing pricing mechanism.
• provide evidence as to the usability, cost-effectiveness and implementation of new pricing mechanisms that are being developed.
• contribute to the overall understanding of economic processes in the digitisation of academic texts, which will be useful for future projects and programmes.

Pelican’s organisational structure is intended to provide a framework for evaluation and monitoring of the project management. The split of the research team over two sites provides an obvious challenge. However, close liaison of the teams, the Independent Consultant, and the the Advisory Board are all designed to facilitate communication between different areas of project work and provide appropriate channels of communication by which feedback and evaluative results can be channeled back into project development.

The three main components of the management structure are:

A broad based Advisory Board, which will meet on three occasions during the 12-months project period. The Advisory Board represents a wide range of stakeholders that claim an interest in the development of a suitable pricing mechanism. It is anticipated that the presentation of diverse views will strengthen the evaluation process, as only this approach allows for the recognition of the complexity of the study field. The Advisory Board will be the forum where real and perceived differences are discussed, potential outcomes are optimised for different stakeholders and the wider issues of digitisation are raised and argued.

The advice of the Advisory Board is used on a daily basis by the Pelican Project Team, which operates from its two sites in Loughborough and Stirling. The project team provides a report to the Advisory Board at each meeting. It will guarantee that no stakeholder is disadvantaged in the day-to-day operations of the project, as only in this way can we ensure successful implementation of the evaluation strategy. The close liaison between Stirling and HERON is of particular importance for the evaluation plan, as HERON’s expertise will provide a substantial contribution to the evaluation of the pricing mechanism developed by Pelican. At the same time, the physical separation of the Stirling and Loughborough team will help the implementation of objective evaluation policy practices.

The Independent Consultant will also advise the project team. He will advise the project team on all matters relating to copyright licensing, and act as a representative for all parties that show an interest in the Pelican project. The Independent Consultant has an integral role in the evaluation strategy as the successful development and implementation of a pricing mechanism ultimately depends on agreement between all stakeholders.

An evaluation strategy must take into account the priorities of those involved in the research process. Their concerns and interests in the project will determine what questions they will be asking of the evaluation plan. As the development of a suitable pricing mechanism would be the ultimate goal of Pelican, it is essential that all stakeholders are constantly involved in the evaluation process and provide feedback to the project team for further refinement.

The Pelican team will need to consult all stakeholders prior to the development of any proposed pricing mechanism to identify key areas of concern. It may be necessary to provide information on digitised texts in HE/FE for those stakeholders who have only an indirect relationship with the publishing industry, in particular, the end users of those materials and/or authors. It is essential that these people are constantly informed about the development of the project and a clear line of communication is established to guarantee the long-term effectiveness of the evaluation strategy.

It is further of fundamental importance for the project team to recognise potential tensions, which may arise between different stakeholder groups. An evaluation strategy document, which demonstrates reasoning and applicability of proposed pricing mechanism, needs to be proposed to everyone involved with a realistic assessment of its chances for successful implementation.

Evaluation of the project must be based firmly within the project’s overall objectives. However, evaluation cannot be unlimited within a specific budget. Although it would be desirable to address all of the questions, which might arise during the term of the project, within the project budget and timetable, it is more appropriate to identify key areas of inquiry and focus our evaluation activities on those areas.

Key questions that have been identified for inquiry are:

1. To what extent does the project answer the needs and concerns of UK HEIs?

Suggested Methods:

• Face-to-face, telephone interviews and/or questionnaire surveys with representatives of UK HEIs to be organised by the project team. Based on the close links between HERON and some HEIs, Pelican anticipates contacting some institutions that are already linked to the HERON programme. A further list of HEIs exists at Loughborough University, which was used in another project to evaluate the information needs and practices of part-time and distance learning students. This list would provide an additional point of reference to the student body in these institutions.

 

2. To what extent does the project answer the concerns of CLA and the publishers?

Suggested Methods:

• face-to-face interviews and/or questionnaire survey with CLA and publishers (possibly through the PA)

 

3. To what extent does the project answer the concern of authors?

Suggested Methods:

• face-to-face interviews and/or questionnaire survey with ALCS and authors (possibly authors from the same HEIs as 1.)

 

4. What are the main technical implications of the recommended approaches?

Suggested Methods:

• technical reports from the research team, proposing a possible pricing mechanism
• possible simulation of economic processes to demonstrate technical implications
• incorporation of stakeholders' feedback on proposed pricing mechanism and on simulation runs

 

5. What are the main financial implications of the recommended approaches?

Suggested Method:

• technical report from the research team, drawing up the main financial implications
• possible simulation of economic processes to demonstrate financial implications
• incorporation of stakeholders' feedback on proposed pricing mechanism and on simulation runs

A summative evaluation report will be published by the end of programme. This will be distributed widely among the HE/FE community through the WWW and publication in appropriate academic journals. The summative evaluation will allow for stakeholders to engage in further discussions beyond the lifetime to the Pelican project.

Link to PELICAN's Communication & Dissemination Plan

Back to PELICAN Home Page