One of the major issues that critics have had with the findings of parapsychology is the issue of nonrepeatability. The ability to replicate an experimental finding is taken as one of the hallmarks of modern science, and critics often claim that parapsychology has been unable to consistently display this hallmark throughout its history. For example, psychologist and long-time parapsychology critic Ray Hyman (1996) states that:

 

"Students in a particular field of inquiry can be assigned the task of replicating some of these classic experiments. The instructor can make this assignment with the confident expectation that each student will obtain results consistent with the original findings. The physics instructor, for example, can ask novice students to try Newton's experiments with colors or Gilbert's experiments with magnets. The students who do so will get the expected results. The psychology instructor can ask novice students to repeat Ebbinghaus' experiments on forgetting or Peterson and Peterson's classic experiment on short-term memory and know that they will observe the same relationships as reported by the original experimenters…Parapsychology is the only field of scientific inquiry that does not have even one exemplar that can be assigned to students with the expectation that they will observe the original results" (p. 49).

 

The picture of experimental replication that Hyman paints here is the ideal one of how we would probably like science to be, but at the same time, it is not 100% realistic. The ability to replicate is not as easy as it seems in Hyman's view, and the problem of reproducing experimental results is not solely limited to parapsychology as he claims. Recently, an article by science reporter Jim Giles (2006) appeared in the prominent mainstream science journal Nature that addressed the problems mainstream scientists face in attempting to replicate published experimental findings. In his article, Giles tracks the fate of the 19 papers published in the July 4, 2002 issue of Nature to see whether or not their results had been replicated. While he found that most had been independently reproduced or corroborated, there were some in which replication was proving to be very difficult, if not impossible.

 

For one paper, reporting the first-ever detection of hydrogen-3 ions (found in the atmospheres of Jupiter and Saturn, suggesting that it may be linked to gas giant formation) in the gaseous disk surrounding a young star, two authors from the University of Notre Dame in Indiana claimed that they detected infrared emissions in two of the three particular frequencies that hydrogen-3 ions emits at. Attempts to independently detect these emissions by a second group from the University of Chicago, however, were not meeting with the same observations. Even when the authors of the paper exchanged raw data with this second group, neither side was able to come to a single conclusion based on the other's data.

 

In the case of another paper, reporting on a University of Minnesota (UM) group's apparent success in extracting fully functioning stem cells from adult human bone marrow, the implications for tissue regeneration were so promising that there was a mass effort by various high-profile biomedical groups to replicate the results. These efforts to replicate did not go smoothly, however. Even after several of these groups went to the UM group's lab to learn how to extract and culture these stem cells, they found themselves unable to do so when they tried it in their own labs. The problem of replicating the results became so vast in this case that it is now difficult for any researchers reporting results using procedures similar to the UM group's to publish because journal referees have become skeptical based on the apparent inability to replicate.

 

Giles (2006) notes in cases like this: "…it is often hard to tell whether an inability to replicate a result is due to a group's failings or a flaw in the original paper. The reason is often the countless tiny details of experimental method that are omitted from the methods sections of papers but can influence results" (p. 345). He goes on to quote University of Cambridge cell biologist Gillian Murphy as saying, "Things are different in different labs for very subtle reasons…The water can be different. We're about to move labs, and my group is very concerned that delicate cells might hate something in the new pipes" (p. 345). He also quotes another physicist as saying in relation to reproducing an exact copy of a piece of an experimental set-up, "It's very difficult to make a carbon copy. You can make a near one, but if it turns out that what is critical is the way he glued his transducers, and he forgets to tell you that the technician always puts a copy of Physical Review on top of them for weight, well, it could make all the difference" (p. 346). Giles also notes that attempts at replication can be greatly hindered when the reported results are actually fraudulent (p. 346), something that occurs in the mainstream sciences, as well, not just in parapsychology.

 

Thus, it appears that replication in other fields of inquiry is often not as clear-cut as Ray Hyman (1996) makes it out to be. The lessons being taught in the examples in Giles' (2006) article should have the greatest implications for parapsychology, since replication is vital to the field and the situation is often much more complex. Unlike the physical sciences, which largely involve tangible phenomena, the behavioral and social sciences deal mostly with intangible phenomena. The brain and its resulting behaviors are not as easy to put under the microscope, and the issue of statistical replication is therefore added to the complexity. What makes it even more complex is the observation that human behavior displays a wide range of variability, probably even more so than the natural phenomena studied in the physical sciences. This make experimental findings in the behavioral and social sciences particularly difficult to replicate exactly, and that may be one reason why there are very few psychological laws (as opposed to physics, which has many).

 

The UM group example in Giles' article suggests that there are so many variables involved in the experimental procedure that attempts to replicate will prove very challenging. The same goes for parapsychology. Here, we also have experimenter effects, sheep-goat effects, the decline effect in psi test scoring, possible geophysical factors (e.g., the negative correlation between positive ESP scoring and increased geomagnetic activity), conditions relating to the psychological atmosphere of the lab (e.g., subject motivation, comfort, etc.), among several others. This should be an example to the mainstream sciences that parapsychology is often faced with many more variables, and thus has been stuck in the same boat as the UM group quite frequently. With so many variables involved, it should not be surprising that replication is difficult.

 

Statistical replication has also been an added complexity for parapsychology because it has been difficult to come to a general consensus (both by those within the field and those outside looking in) on what level of statistical significance should constitute a successful replication. One way that parapsychology has dealt with the issue of replication is through the use of meta-analysis (Utts, 1991, 1999), something also used in other areas of behavioral and social science, but not as much in the physical sciences. By systematically combining the results of many studies together, one can examine for the possibility of an overall effect rather than looking from study to study, an approach very useful in a behavioral field where the phenomena tend to fluctuate.

 

Even in meta-analysis, the complexity of multiple variables can make the interpretation of parapsychological findings difficult. One example of this comes from the recent meta-analyses of the new ganzfeld ESP experimental database collected in the years following the successful PRL autoganzfeld series (Bem & Honorton, 1994). Two independent meta-analyses of the new ganzfeld database that were published in the mainstream psychological literature (Milton & Wiseman, 1999; Storm & Ertel, 2001) both came to opposite conclusions on whether the PRL autoganzfeld results had been replicated by independent laboratories. A critical issue in these two meta-analyses, however, is that no clear distinction was made between studies that were meant to be direct replications of the PRL series, and studies that had purposely varied their procedures in order to take a process-oriented approach. Since the studies in the new database had varying approaches, they may not have been directly comparable and it may therefore not have been wise to combine their results (i.e., the database was highly heterogeneous). Only when a distinction was made in the latest meta-analysis (Bem, Palmer, & Broughton, 2001) were PRL series results shown to have been replicated through direct experimental replications. An entire electronic debate held on this matter was also published in the Journal of Parapsychology (Schmeidler & Edge, 1999), indicating the many variable issues facing replication of ganzfeld results

 

Multi-variable complexity might also in part account for the inability of three independent laboratories, working in collaboration, to replicate the results of the Princeton Engineering Anomalies Research (PEAR) Lab's longitudinal micro-PK database (Jahn et al., 2000). Although the three labs were unable to obtain overall statistical significance in the combined database they collected, they did note several internal structural anomalies in the database that have been observed in other micro-PK experiments, including the decline effect. The presence of these structural anomalies in the database may suggest an interaction of variables that influenced the overall results. Alternatively, as Giles (2006, p. 345) noted, different conditions and procedures exist for different labs, and these differences are often quite subtle. With three different labs involved, the differences could have been quite notable, despite being unapparent.

 

One value that Giles' article has for parapsychology is that it should demonstrate to the mainstream sciences that parapsychology is not the only one with replication problems, and should therefore not be singled out in this regard. It should also demonstrate to the mainstream sciences that many variables can factor into replication attempts, and once it is realized that parapsychology has to deal with this probably more than other sciences, then one possible answer as to why the nonrepeatability problem exists will be known. Giles' article also has value in that it provides a useful suggestion towards its end that scientists should disseminate information and encourage informal avenues of discussion relating to subtle variables that may contribute to replication problems. Parapsychology would greatly benefit from this, and the mediums for this are available (e.g., meeting discussions, e-mail discussion groups, newsgroups, blogs, etc.). The ganzfeld electronic debate (Schmeidler & Edge, 1999) is one clear example of the benefit, and if such discussions occurred more often, both formally in journals and informally on the Internet or other mediums, then they might lead to new ways to improving replicability in parapsychology, and perhaps one day, the nonrepeatability effect might be a thing of the past.

 

- Bryan

 

************

References (in order of text citation):

 

Hyman, R. (1996). Evaluation of a program on anomalous mental phenomena. Journal of Scientific Exploration 10(1), Spring. pp. 31 – 58.

 

Giles, J. (2006). The trouble with replication. Nature 442(7101), July 27. pp. 344 – 347.

 

Utts, J. (1991). Replication and meta-analysis in parapsychology. Statistical Science 6(4), November. pp. 363 – 403.

 

Utts, J. (1999). The significance of statistics in mind-matter research. Journal of Scientific Exploration 13(4), Winter. pp. 615 – 638.

 

Bem, D. J., & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin 115(1), January. pp. 4 – 18.

 

Milton, J., & Wiseman, R. (1999). Does psi exist? Lack of replication of an anomalous process of information transfer. Psychological Bulletin 125(4), July. pp. 387 – 391.

 

Storm, L., & Ertel, S. (2001). Does psi exist? Comments on Milton and Wiseman's (1999) meta-analysis of ganzfeld research. Psychological Bulletin 127(3), May. pp. 424 – 433.

 

Bem, D. J., Palmer, J., & Broughton, R. S. (2001). Updating the ganzfeld database: A victim of its own success? Journal of Parapsychology 65(3), September. pp. 207 – 218.

 

Schmeidler, G. R., & Edge, H. (1999). Should ganzfeld research continue to be crucial in the search for a replicable psi effect? Part II. Edited ganzfeld debate. Journal of Parapsychology 63(4), December. pp. 335 – 388.

 

Jahn, R., Dunne, B., Bradish, G., Dobyns, Y., Lettieri, A., Nelson, R., Mischo, J., Boller, E., Bosch, H., Vaitl, D., Houtkooper, J., & Walter, B. (2000). Mind/Machine Interaction Consortium: PortREG replication experiments. Journal of Scientific Exploration 14(4), Winter. pp. 499 – 455.