Reliability is an essential part of research as without it how would we know which results to trust? For the purpose of this blog I’m only going to talk about it within research terms as otherwise I will end up writing some massive essay and going off on a tangent.
There are so many different types of reliability within research that you’d have thought all published research would have high reliability trying to fit in with all of the guidelines. However, if you look into it, no piece of research is going to be perfect. I think there will always be some unreliable aspects of research, particularly when studying humans. For example, how can we possible account for every type of variable? Is the person hungry? Are they nervous? Or are they tired?
So, to define: reliability is when we are able to repeat a measure and gain the same (or similar) result time and time again. But how do we know if an experiment is reliable? Well there are several different methods that can be used to determine reliability.
First, the test-retest reliability method can be useful in determining how reliable a measure is. For example, if a class of psychology students participate in a study to test reaction time in which they have to respond to certain stimuli and then perform the same task a week later we would hope for similar results. However, one of the main flaws of this method is that it would be likely to see a testing effect on participants. For instance, if the students do the same test twice there may be an issue if practice effects. By this I mean they will be more familiar with the test and because of this their reaction time may increase. Which, may in turn, reduce the reliability of the study. This is why it is best to use this method of testing reliability with things that remain stable over time, such as intelligence or personality.
Another measure of reliability is inter-rater reliability. This is used for simultaneous measurements between more than one researcher and is often used when observing behaviour. This measure makes an observation more reliable as if two or more observers are watching then it is less likely that something will be missed. I can remember learning about one study, but I can’t remember who did it. In the study there were two observers that went out into the real world and conducted a study of children’s aggression by observing how many aggressive acts the children demonstrated. By using two observers the reliability of the study was improved as it would have provided more accurate results. Cohen’s Kappa coefficient is a measure of inter-rater agreement for qualitative data, such as observational studies, and is an effective measure as it also takes into account that an agreement between observers may be due to chance*.
There are other factors that can affect the reliability of a research study, and as I don’t want to waffle on forever I will briefly mention two of them.
The first factor is observer effect. It has been suggested by Eagly and Carli (1983)** that characteristics of the experimenter, such as age, sex or behaviours such as body language can affect the participant during a study, which can lead to a loss of reliability. For instance, Bickman (1974)*** conducted a study in which three confederate participants randomly asked people on the street to, for example, “pick up that bag”. They were all dressed differently; one confederate was a milkman, another was a civilian and the third dressed as a guard. The study found that people were more likely to obey the guard as they saw him as an authority figure. Therefore, we could suggest from this that participants in research studies may react differently than they normally would because they view the experimenter as an authority figure, particularly if they are wearing a white lab coat, so they may try extra hard to please them or may do the complete opposite, thus reducing the reliability of the study.
The second factor I want to briefly mention is environmental changes. Whilst researchers take every effort to make the conditions that same for all participants it would be extremely difficult to account for everything. Changes in the time of day or time of year can affect how a participant will respond in an experiment or study, even a slight change in temperature could affect how likely a person will be to complete a task compared to another participant. If it’s hot then the participant may feel tired or if it’s too cold a participant may not be able to concentrate#.
So, to conclude, reliability in research is always important as it helps us to ensure that our measures are consistent. Unfortunately when working with people it is difficult to account for every possible factor that could affect the reliability of a study. Most of the time researchers try to account for the most likely variables and understand that they will never have the perfect experiment.