A systematic and critical response to Pendrous et al. (2020) replication study

Context and Objectives

[x] Ruiz, Luciano, and Sierra (2020) publish a theoretical and critical response to Pendrous et al. (2020) in the Journal of Contextual Behavioral Science (volume 17, pages 39-45). This systematic critical analysis addresses a failed replication of Sierra et al. (2016), whose original study demonstrated that metaphors with shared physical properties and appetitive augmental functions increased pain tolerance in a cold pressor task.

[x] The scientific context is fundamental to understanding the significance of this critical analysis. Sierra et al. (2016) presented innovative findings indicating that exposure to coherent metaphors sharing physical properties with the painful task and possessing appetitive augmental functions (i.e., increasing approach toward valued objectives) facilitated participants' tolerance of prolonged immersion of their hands in very cold water. This discovery carries substantial implications for Contextual Behavioral Psychology, particularly for Relational Frame Theory (RFT) and clinical applications of Acceptance and Commitment Therapy (ACT).

[x] When Pendrous et al. (2020) report inability to replicate these findings, a critical methodological dilemma emerges: do these negative results represent a valid refutation of the original effect, or do unrecognized methodological differences preclude appropriate comparison between studies? Ruiz, Luciano, and Sierra argue that the purported "extended direct replication" by Pendrous et al. contained substantial differences that disqualified it as a direct replication from the outset.

[x] The specific objectives of Ruiz et al.'s article are threefold: first, to demonstrate that fundamental methodological differences between the original and replication studies prevent classifying it as a valid direct replication; second, to indicate that Pendrous et al.'s report inadequately acknowledges the methodological strengths of Sierra et al.'s original study; and third, to identify methodological problems in the replication study that went unnoticed. These objectives aim toward improving precision and rigor in how replication studies are reported and discussed in behavioral research.

Principal Arguments

[x] Randomization and Blinding Design: The first substantial argument centers on experimental design characteristics. Pendrous et al. claimed to employ double-blind design with sex stratification. However, Ruiz et al. demonstrate that Sierra et al. also employed random assignment with sex stratification and experimenter blinding. The critical finding here is that the replication produced unequal sex ratios across experimental conditions: 77.5% female participants in the replication versus 50% in the original study. This deviation from planned stratification challenges the assertion of robust "double-blind" design and suggests randomization implementation problems that would affect group comparability.

[x] Participant Characteristics: The second argument exposes fundamental differences in sample characteristics. In Pendrous et al.'s study, 40.5% of participants possessed prior knowledge of ACT/RFT (key concepts for interpreting metaphors), whereas original study participants were naïve regarding these theories. This is a crucial difference because prior RFT knowledge could affect how participants process and respond to experimentally manipulated metaphors. Additionally, 23.8% of replication participants had prior cold pressor task experience compared with naïve participants in the original. Furthermore, 21.4% were non-native English speakers, an important factor when the intervention depends on subtle linguistic understanding of metaphors in a foreign language. Most problematically, significant differences existed in naïve status across experimental conditions, suggesting randomization either failed or was inadequately reported.

[x] Experimental Apparatus and Task Parameters: The third argument addresses substantial differences in physical study parameters. Pendrous et al. employed commercial apparatus while Sierra et al. used artisanal apparatus; water temperature differed considerably (3°C in Pendrous versus 4.5-5.5°C in original); and administration procedure varied (automatic in replication versus experimenter-present in original). These differences in painful stimulus intensity and contextual conditions could produce significant effects on the behavioral mechanisms under investigation.

[x] Procedures and Initial Task Contact: A fourth critical argument concerns experimental procedure. Pendrous et al. administered an analogical reasoning test at pretest, potentially priming relational elaboration in participants before exposure to experimental metaphors. The original study included no such reasoning test. Furthermore, the replication included no assessment of participant protocol comprehension, a relevant methodological step when working with metaphors and relational reasoning. Experimenter presence also varied between conditions in ways not fully documented.

[x] Protocols and Formulation of Metaphoric Interventions: The fifth and perhaps most critical argument concerns the metaphor protocols themselves. Specific wording of appetitive augmentals differed substantially between original and replication study. Pauses designed to permit relational elaboration were substantially shorter or absent in the replication. Detailed analysis of audio files used in both studies (presented in Table 5) reveals dramatically different pause durations across experimental conditions. These pauses are theoretically critical according to RFT because they permit participants to elaborate symbolic relationships between the metaphor and current painful experience. Their reduction or elimination would fundamentally alter the theoretical mechanism presumed to generate the effect.

[x] Synthesis of Arguments: The authors emphasize that these differences are neither trivial nor superficial. Collectively, they represent a set of methodological changes that substantially alter experimental procedure, sample characteristics, and intervention implementation. The central argument is that these cumulative changes are incompatible with Pendrous et al.'s characterization as a "direct replication."

Discussion and Conclusions

[x] The discussion in Ruiz et al. articulates three central concerns summarizing their critique. First, excessive variation exists between studies to consider them a valid direct replication. A direct replication requires holding constant all methodologically relevant aspects; when those aspects vary, the study becomes an "extended" or "conceptual" replication exploring how methodological changes affect the effect. Pendrous et al. labeled their work as "extended direct replication," but Ruiz et al. argue that even this characterization is inappropriate given the magnitude of changes.

[x] Second, Pendrous et al.'s report fails to explicitly acknowledge particular methodological strengths of the original study. This matters because readers cannot evaluate whether replication failure results from the original effect being spurious or from the replication implementing inferior procedures. Scientific transparency requires that a replication author acknowledge and justify any deviation from the original protocol.

[x] Third, Pendrous et al. committed unrecognized methodological errors that reduced the replication's methodological quality. Examples include lack of data on sex proportion before reporting findings, lack of clarity about experimenter blinding, and changes in audio protocols without explicit justification.

[x] Ruiz et al.'s conclusions are clear yet constructive. They do not argue that Pendrous et al. acted in bad faith; rather, they suggest widespread need for greater precision and transparency in how replication studies are reported. The authors call for future replications to: (a) precisely label studies as direct or extended based on clear criteria, (b) explicitly acknowledge and justify any deviation from original protocols, (c) proactively evaluate whether such deviations could affect replication capacity, and (d) clearly document all procedure characteristics to enable future researchers to fully understand what changed and why.

[x] Beyond the specific Sierra et al. versus Pendrous et al. case, Ruiz et al. contribute to broader methodological debate concerning the nature of scientific replication. In an era where replication studies increasingly occur and are considered critical for open science, precisely understanding what constitutes a "direct replication" and how to report it is fundamental. The article models a rigorous and generous manner of critically evaluating a study without questioning its authors' scientific integrity.

Importance and Contribution

[x] This article's importance transcends the particular debate between Sierra et al. and Pendrous et al. First, it carries significant theoretical implications for Relational Frame Theory. Sierra et al.'s findings, if valid, suggest symbolic-relational processing of metaphors can modulate pain experience, which would support central RFT postulates about human symbol-based cognition's power. Whether this effect is replicable fundamentally concerns theory validity.

[x] Second, the article holds evident clinical importance. If metaphors can modify pain tolerance through relational elaboration mechanisms, this would suggest new tools for therapeutic intervention in chronic pain conditions. ACT-based interventions could be refined to systematically incorporate metaphors designed according to RFT principles.

[x] Third, the methodological contribution may prove most enduring. Ruiz et al. provide detailed, well-documented models for conducting constructive criticism of replication studies. They demonstrate how to identify specific methodological differences without pejorativism, how to evaluate them systematically, and how to communicate concerns in ways contributing to collective scientific knowledge. This approach proves especially valuable given the contemporary "replication crisis" context in psychology.

[x] Fourth, the article contributes to clarifying communication standards for science. It explicitly argues for greater rigor in terminology (what constitutes "direct replication" versus "extended replication"), greater transparency in protocol documentation, and greater intellectual honesty in acknowledging when one's own methodological proposals differ from prior studies.

[x] Finally, the article carries broader epistemological contribution. It suggests replication studies should not be viewed simply as binary tests of "does it replicate" or "does it not," but as opportunities for understanding how methodological characteristics modulate observed effects. When replication fails, researchers should systematically ask which methodological differences might explain discrepancy. This constructive approach proves more informative and less prone to generating excessively skeptical interpretations of prior work.

Evaluation Checklist

• Is theoretical context clearly identified? Yes, situated in RFT/ACT
• Are critical analysis objectives explicit? Yes, three clear objectives
• Are systematic arguments presented? Yes, five principal arguments
• Are methodological differences addressed? Yes, exhaustively
• Is quantitative evidence present (tables, data)? Yes, including Table 5 durations
• Are theoretical implications discussed? Yes, for RFT and ACT
• Are clinical implications discussed? Yes, for pain intervention
• Are methodological implications discussed? Yes, for scientific replication
• Is the tone constructive? Yes, rigorous yet generous critique
• Are future improvements proposed? Yes, for replication standards
• Are conclusions clear? Yes, three central concerns
• Is unnecessary pejorativism avoided? Yes, professional academic approach

---

REFERENCE

Ruiz, F. J., Luciano, C., & Sierra, M. A. (2020). On the importance of distinguishing between extended and direct replication. Journal of Contextual Behavioral Science, 17, 39–45. https://doi.org/10.1016/j.jcbs.2020.04.011

Original Studies Referenced:

• Sierra, M. A., Luciano, C., Ruiz, F. J., & Moreno, N. (2016). Metaphors and emotional life: The case of pain. Journal of Contextual Behavioral Science, 5(2), 74–83.
• Pendrous, R., Tipton, E., & Wills, A. J. (2020). On the reproducibility of the relationship between metaphor and pain tolerance: A close analysis of Sierra et al.'s (2016) study and its replication failure. Journal of Contextual Behavioral Science, 17, 32–38.

---