https://www.uni.edu/chfasoa/reliabilityandvalidity.htm

http://www.measuringu.com/blog/measure-reliability.php

https://explorable.com/definition-of-reliability?gid=1579

https://explorable.com/internal-consistency-reliability?gid=1579

SUMBER 1 

Reliability is the degree to which an assessment tool produces stable and consistent results.

Reliability is a measure of the consistency of a metric or a method.

 Every metric or method we use, including things like methods for uncovering usability problems in an interface and expert judgment, must be assessed for reliability.

Imagine that a researcher discovers a new drug that she believes helps people to become more intelligent, a process measured by a series of mental exercises. After analyzing the results, she finds that the group given the drug performed the mental tests much better than the control group.

For her results to be reliable, another researcher must be able to perform exactly the same experiment on another group of people and generate results with the same statistical significance. If repeat experiments fail, then there may be something wrong with the original research.

Types of Reliability

 

1. Test-retest reliability is a measure of reliability obtained by administering the same test twice over a period of time to a group of individuals.  The scores from Time 1 and Time 2 can then be correlated in order to evaluate the test for stability over time. 

 

Example:  A test designed to assess student learning in psychology could be given to a group of students twice, with the second administration perhaps coming a week after the first.  The obtained correlation coefficient would indicate the stability of the scores.

2. Parallel forms reliability is a measure of reliability obtained by administering different versions of an assessment tool (both versions must contain items that probe the same construct, skill, knowledge base, etc.) to the same group of individuals.  The scores from the two versions can then be correlated in order to evaluate the consistency of results across alternate versions. 

 

Example:  If you wanted to evaluate the reliability of a critical thinking assessment, you might create a large set of items that all pertain to critical thinking and then randomly split the questions up into two sets, which would represent the parallel forms.

 

3. Inter-rater reliability is a measure of reliability used to assess the degree to which different judges or raters agree in their assessment decisions.  Inter-rater reliability is useful because human observers will not necessarily interpret answers the same way; raters may disagree as to how well certain responses or material demonstrate knowledge of the construct or skill being assessed. 

 

Example:  Inter-rater reliability might be employed when different judges are evaluating the degree to which art portfolios meet certain standards.  Inter-rater reliability is especially useful when judgments can be considered relatively subjective.  Thus, the use of this type of reliability would probably be more likely when evaluating artwork as opposed to math problems.

4. Internal consistency reliability is a measure of reliability used to evaluate the degree to which different test items that probe the same construct produce similar results. 

 

A. Average inter-item correlation is a subtype of internal consistency reliability.  It is obtained by taking all of the items on a test that probe the same construct (e.g., reading comprehension), determining the correlation coefficient for each pair of items, and finally taking the average of all of

these correlation coefficients.  This final step yields the average inter-item correlation. 

 

B. Split-half reliability is another subtype of internal consistency reliability.  The process of obtaining split-half reliability is begun by “splitting in half” all items of a test that are intended to probe the same area of knowledge (e.g., World War II) in order to form two “sets” of items.  The entire test is administered to a group of individuals, the total score for each “set” is computed, and finally the split-half reliability is obtained by determining the correlation between the two total “set” scores.

SUMBER 2

The test-retest reliability method is one of the simplest ways of testing the

stability and reliability of an instrument over time.

For example, if a group of students takes a test, you would expect them to show very similar results

if they take the same test a few months later. This definition relies upon there being no confounding

factor during the intervening time interval.

Instruments such as IQ tests and surveys are prime candidates for test-retest methodology, because

there is little chance of people experiencing a sudden jump in IQ or suddenly changing their

opinions.

On the other hand, educational tests are often not suitable, because students will learn much more

information over the intervening period and show better results in the second test.

  Test-Retest Reliability and the Ravages of Time

For example, if a group of students take a geography test just before the end of semester and one

when they return to school at the beginning of the next, the tests should produce broadly the same

results.

If, on the other hand, the test and retest are taken at the beginning and at the end of the semester, it

can be assumed that the intervening lessons will have improved the ability of the students. Thus,

test-retest reliability will be compromised and other methods, such as split testing, are better.

Even if a test-retest reliability process is applied with no sign of intervening factors, there will always

be some degree of error. There is a strong chance that subjects will remember some of the

questions from the previous test and perform better.

Some subjects might just have had a bad day the first time around or they may not have taken the

test seriously. For these reasons, students facing retakes of exams can expect to face different

questions and a slightly tougher standard of marking to compensate.

Even in surveys, it is quite conceivable that there may be a big change in opinion. People may have

been asked about their favourite type of bread. In the intervening period, if a bread company mounts

a long and expansive advertising campaign, this is likely to influence opinion in favour of that brand.

This will jeopardise the test-retest reliability and so the analysis that must be handled with caution.

Test-Retest Reliability and Confounding Factors

To give an element of quantification to the test-retest reliability, statistical tests factor this into the

analysis and generate a number between zero and one, with 1 being a perfect correlation between

the test and the retest.

Perfection is impossible and most researchers accept a lower level, either 0.7, 0.8 or 0.9, depending

upon the particular field of research.

However, this cannot remove confounding factors completely, and a researcher must anticipate and

address these during the research design to maintain test-retest reliability.

To dampen down the chances of a few subjects skewing the results, for whatever reason, the test

forcorrelation is much more accurate with large subject groups, drowning out the extremes and

providing a more accurate result.

 For any research program that requires qualitative rating by different

researchers, it is important to establish a good level of interrater reliability,

also known as interobserver reliability.

This ensures that the generated results meet the accepted criteria defining reliability, by

quantitatively defining the degree of agreement between two or more observers.

Interrater Reliability and the Olympics

Interrater reliability is the most easily understood form of reliability, because everybody has

encountered it.

For example, watching any sport using judges, such as Olympics ice skating or a dog show, relies

upon human observers maintaining a great degree of consistency between observers. If even one of

the judges is erratic in their scoring system, this can jeopardize the entire system and deny a

participant their rightful prize.

Outside the world of sport and hobbies, inter-rater reliability has some far more important

connotations and can directly influence your life.

Examiners marking school and university exams are assessed on a regular basis, to ensure that

they all adhere to the same standards. This is the most important example of interobserver reliability

- it would be extremely unfair to fail an exam because the observer was having a bad day.

For most examination boards, appeals are usually rare, showing that the interrater reliability process

is fairly robust.

An Example From Experience

I used to work for a bird protection charity and, every morning, we went down to the seashore and

used to estimate the number individuals for each bird species.

Obviously, you cannot count thousands of birds individually; apart from the huge numbers, they

constantly move, leaving and rejoining the group. Using experience, we estimated the numbers and

then compared our estimate.

If one person estimated 1000 dunlin, one 4000 and the other 12000, then there was something

wrong with our estimation and it was highly unreliable.

If, however, we independently came up with figures of 4000, 5000 and 6000, then that was accurate

enough for our purposes, and we knew that we could use the average with a good degree of

confidence.

Qualitative Assessments and Interrater Reliability

Any qualitative assessment using two or more researchers must establish interrater reliability to

ensure that the results generated will be useful.

One good example is Bandura's Bobo Doll experiment, which used a scale to rate the levels of

displayed aggression in young children. Apart from extensive pre-testing, the observers constantly

compared and calibrated their ratings, adjusting their scales to ensure that they were as similar as

possible.

Guidelines and Experience

Interobserver reliability is strengthened by establishing clear guidelines and thorough experience. If

the observers are given clear and concise instructions about how to rate or estimate behavior, this

increases the interobserver reliability.

Experience is also a great teacher; researchers who have worked together for a long time will be

fully aware of each other's strengths, and will be surprisingly similar in their observations.

Internal consistency reliability defines the consistency of the results

delivered in a test, ensuring that the various items measuring the different

constructs deliver consistent scores.

For example, an English test is divided into vocabulary, spelling, punctuation and grammar. The

internal consistency reliability test provides a measure that each of these particular aptitudes is

measured correctly and reliably.

One way of testing this is by using a 

test-retest method, where the same test is administered some after the initial test and the results compared.

However, this creates some problems and so many researchers prefer to measure internal

consistency by including two versions of the same instrument within the same test. Our example of

the English test might include two very similar questions about comma use, two about spelling and

so on.

The basic principle is that the student should give the same answer to both - if they do not know how

to use commas, they will get both questions wrong. A few nifty statistical manipulations will give the

internal consistency reliability and allow the researcher to evaluate the reliability of the test.

There are three main techniques for measuring the internal consistency reliability, depending upon

the degree, complexity and scope of the test.

They all check that the results and constructs measured by a test are correct, and the exact type

used is dictated by subject, size of the data set and resources.

Split-Halves Test

The split halves test for internal consistency reliability is the easiest type, and involves dividing a test

into two halves.

For example, a questionnaire to measure extroversion could be divided into odd and even questions.

The results from both halves are statistically analysed, and if there is weak correlation between the

two, then there is a reliability problem with the test.

The split halves test gives a measurement of in between zero and one, with one

meaning a perfect correlation.

The division of the question into two sets must be random. Split halves testing was a popular way to

measure reliability, because of its simplicity and speed.

However, in an age where computers can take over the laborious number crunching, scientists tend

to use much more powerful tests.

Kuder-Richardson Test

The Kuder-Richardson test for internal consistency reliability is a more advanced, and slightly more

complex, version of the split halves test.

In this version, the test works out the average correlation for all the possible split half combinations

in a test. The Kuder-Richardson test also generates a correlation of between zero and one, with a

more accurate result than the split halves test. The weakness of this approach, as with split-halves,

is that the answer for each question must be a simple right or wrong answer, zero or one.

For multi-scale responses, sophisticated techniques are needed to measure internal consistency

reliability.

Cronbach's Alpha Test

The Cronbach's Alpha test not only averages the correlation between every possible combination of

split halves, but it allows multi-level responses.

For example, a series of questions might ask the subjects to rate their response between one and

five. Cronbach's Alpha gives a score of between zero and one, with 0.7 generally accepted as a sign

of acceptable reliability.

The test also takes into account both the size of the sample and the number of potential responses.

A 40-question test with possible ratings of 1 - 5 is seen as having more accuracy than a ten-question

test with three possible levels of response.

Of course, even with Cronbach's clever methodology, which makes calculation much simpler than

crunching through every possible permutation, this is still a test best left to computers and statistics

spreadsheet programmes.

Summary

Internal consistency reliability is a measure of how well a test addresses different constructs and

delivers reliable scores. The test-retest method involves administering the same test, after a period

of time, and comparing the results.

By contrast, measuring the internal consistency reliability involves measuring two different versions

of the same item within the same test.