Natural Research Assessment: ability and new requirements for

CRIS

Sergey Parinov,

CEMI RAS, Russia, Moscow

Natural Research Assessment is …

– an inherent part of science communication, since it is a basic consequence of existed labor division and knowledge exchange between scientists

– a practical “testing” of research outputs by scientists when they are trying to use it for producing new scientific knowledge

– a statistical portrait of total trials-and-errors for some researchers, their results, etc.

• Can we practically use it? If yes, how?

Natural Assessment in Social Systems

• All social systems, where independent actors are working with labor division and products exchange between them, naturally produce trial-and-error usage statistics

• If in the system the practical testing of is doing well and trial-and-error data is available for actors, so the system provides for actors useful natural assessment mechanism and statistics

• By such assessments actors know how their activity are currently valuable for the system and what/whose activities are the most useful now

• Than better such signaling mechanism works, than the system is more efficient

Scientific Social System Features• Science is a social system with labor division and

knowledge exchange between scientists in a trial-and-error form

• Trial-and-error usage statistics are mostly latent since researchers are testing “units of thought” hidden inside publications which are currently visible form of scientific exchange

• Scientists can get trial-and-error data mostly in a form of lists of publications selected by publishers and citations, what is really not enough

• Current scientific practical testing (trial-and-error) process is doing not very well and compare with many other social systems the Science has weaker signaling system and its efficiency can be improved

Current Role of CRIS (e.g. socionet.ru CRIS)

• CRIS is an interface for scientists to give their research outputs for common usage and to take “products” from other scientists for their personal usage

• CRIS is working as a scientific circulation mechanism, since it delivers “products” from “producers” to “consumers”

• CRIS is a tool of practical testing of research outputs and virtual environment for scientific trial-and-error acts

Specification of Tasks

• Strong natural assessment and professional signaling system can be designed by:– discovering of what is a matter (object) of scientific trial-and-error

process in producing a new knowledge– efficient scientific circulation mechanism, which involves all

objects into representative trial-and-error process– registering and collecting all existed trial-and-error data with

presenting them as publicly available/visible statistics– the statistics should show both side of scientists’ activity within a

network of professional relations: as a “producer” and as a “consumer” of research outputs

• We should exploit typical motivations and scientific behavior models: a wish to get professional recognition and existed norms to provide professional reviews

Trial-and-error process within CRIS• An author registers research output as a ready for

testing scientific object-for-reuse (OfR) by– specifying which research materials were used as

roots/basements for his/her output (citation links)– specifying materials/scientist where/by whom the output

could be used/reviewed (links to possible users)

• Authors of linked materials receive a notification about created links, and– they can protest on how the materials were used– they can use suggested OfR and link it with their

materials, or can review it, or can ignore it

Scientific Communications within CRIS

• On a producer side

– specify used OfR with quality characteristics

– request on using-reviewing own OfR by linking it with other OfR or scientists

• On a consumer side

– protest against usage characteristics and/or provide comments on it, or do nothing

– ban requests from some authors, or specify personal reviewing rate, or rewrite own OfR by using/citing suggested OfR

Trial-and-error statistics within CRIS

• Accumulated data about scientist-producer activity– number of produced OfR for certain period of time– number of OfR usage for certain period of time, including quality

characteristics distribution– number of OfR reviews for certain period of time, including

quality characteristics distribution

• Accumulated data about scientist-consumer activity– number of requests from scientists on using/reviewing their OfR– number of made citations (utilized OfR), including quality

characteristics distribution– number of made reviews, including quality characteristics

distribution – number of banned authors, queue length of requests compare

with personal rate, number of rejections

Who will benefit?

• A scientist – – by data on how people used his/her outputs and by usage

request as some can be valuable

• A research institution – – by data on staff’s produced/used outputs, etc.

• A research funder – – by data on produced/used outputs, performance statistics

about researchers and organizations

• A research community – – by better scientific circulations, usage and professional

signaling system

Conclusion

• We are implementing natural research assessment scheme within Socionet CRIS (socionet.ru) as a pilot to allow a playing with it

• We suggest for euroCRIS to include a support of natural research assessment into CRIS-CERIF model

Thanks!

Necessary Role of CRIS

• Electronic registration research outputs/results focused on its usage

• Tools to request usage/review/evaluation from a community and/or its members

• A form to make usage/review/evaluation in easy way

• Taking statistics on research performance of scientist in a role of a producer and a consumer research outputs/results

Requirements for new CRIS features

• To operate with OfR as citation and artifact information objects

• To make semantic links (by usage/impact metrics) with CRIS objects (material, person, etc.)

• To run monitoring of linkages and notification services

• To give scientist an ability to specify a personal norm of reviewing efforts

• To gather data, collect and process trial-and-error statistics

Background

• Open Science as a conceptual platform– 1 open access to research– 2 open access to usage and impact data– 3 open access to basic assessment data

• CRIS Socionet as technological platform– open archives and IR– scientific circulation mechanism– monitoring and statistics services