Content validity and perceived utility of a self-help online ACT program focused on RNT (DAC)

Context and Objectives

This study emerges in response to the global COVID-19 pandemic and the urgent need to expand mental health services through online psychological interventions (OPIs). According to the World Health Organization, psychological disorders represent one of the leading causes of disability globally, and only one-third of the population suffering from these disorders receives intervention. The authors present the DAC program (Disentangled, Aware, and Committed), an adaptation of Acceptance and Commitment Therapy (ACT) protocols focused on repetitive negative thinking (RNT) to the online intervention format.

The primary objective was to evaluate the content validity and perceived utility of the DAC program through two complementary studies. Study 1 involved evaluation by four ACT experts of the clarity, utility, pertinence, adjustment to the ACT model, and therapeutic goal fulfillment of each program component. Study 2 employed 41 undergraduate clinical psychology trainees who evaluated the program components in terms of clarity and perceived utility during their participation in a randomized controlled trial examining the efficacy of the DAC program in preventing the typical increase in emotional symptoms and RNT in this population.

Method

Participants

Study 1: Four reviewers with formal training in ACT: (1) Ph.D. in Psychology with a master's degree in contextual therapies with main focus on ACT, with approximately ten published scientific papers about ACT; (2) Ph.D. in Psychology who had conducted clinical studies on child depression using RNT-focused ACT protocols; (3) Ph.D. in Psychology who had developed a psychometric instrument and tested the efficacy of an RNT-focused ACT intervention for psychological distress; (4) licensed clinical psychologist with extensive ACT training and research experience conducting an institutional internship. These experts were not involved in program development.

Study 2: Forty-one undergraduate psychology students in the ninth semester (of ten) of mandatory clinical training at a Colombian university. All were participating in a 4-month clinical practice with 28 weekly hours, handling up to 5 patients simultaneously under weekly supervision of 1.5 hours with a clinical psychologist. These participants were recruited for a randomized controlled trial evaluating the effect of DAC components on emotional symptoms and valued living.

Design

Study 1: Content validity evaluation design through expert assessment. Each reviewer accessed the complete DAC program through the Moodle platform and evaluated the content at the end of each video, audio, or activity using the EVE (Experts Validity Evaluation) instrument at 23 evaluation points.

Study 2: Cross-sectional design of perceived utility evaluation integrated within a randomized controlled trial. The 41 participants (of 43 randomly assigned to the RNT-focused ACT condition) completed evaluations of the 23 program components.

Intervention/Conditions

The DAC program is a self-help online intervention consisting of three modules with 15 videos, 9 audios, and 15 written activities and experiential exercises, with average duration of 6 hours (approximately 2 hours per module), although duration depends on participant pace.

Module I: Knowing the problem and finding solutions (7 videos, 7 activities, 1 audio). Objectives: (a) conceptualize the participant's case by assessing hierarchical triggers for RNT, experiential avoidance strategies, valued directions, and committed actions; (b) help participants identify the ineffectiveness of previous coping strategies; (c) propose an alternative behavioral pattern. Contents: introduction of intervention rationale using three hypothetical characters (Pedro, Andrea, Mariana), description of how RNT works, concept of values and valued directions, committed actions, and flexible choice repertoire.

Module II: Developing an observation point for thoughts (3 videos, 2 audios, 2 activities). Objectives: train participants to discriminate the start of RNT, understand its characteristics, and disengage from it. Contents: cognitive defusion exercises, discrimination between judging an external event versus adopting a nonjudgmental stance, observation of thoughts under voluntary control, focused attention practice, and modified observer exercise.

Module III: Focusing on what really matters (3 videos, 3 audios, 6 activities). Objectives: help participants clarify their values. Contents: autobiography of dreams exercise, exploration of long-term consequences of rigid RNT versus flexible behavior, establishment of value-related goals, exploration of time management, identification of psychological barriers, and planning of daily activities oriented toward values.

A self-clinical conceptualization diagram was used as graphical support throughout the intervention, allowing participants to complete four sections with their own information about RNT triggers, experiential avoidance strategies, valued directions, and valued actions.

In Study 2, the intervention was administered in six weekly 1-hour sessions in groups of approximately 15 participants. Sessions were led by two psychologists with a facilitator role consisting solely of presenting program materials. Video and audio files were played for the whole group, and participants individually responded to program exercises. No group discussion or interaction with facilitators was provided.

Instruments

Experts Validity Evaluation (EVE): Customized instrument developed for this study evaluating DAC content validity according to five criteria on a 5-point Likert scale (5 = completely agree; 1 = completely disagree):

• Clarity: "The content of this part of the program is clear and easy to understand"
• Utility: "The content of this part of the program will be useful for the participants"
• Pertinence: "The presentation of this content is pertinent at this moment of the intervention program"
• Adjustment to ACT model: "The content of this part of the program was conceptually coherent with the ACT model"
• Therapeutic goal fulfillment: "The content of this part of the program accomplished the therapeutic aims planned by the authors"

User Quality Survey (UQS): Customized instrument developed for this study evaluating clarity and perceived utility of DAC components on a 5-point Likert scale:

• Clarity: "The content of this part of the program has been clear and easy to understand"
• Utility: "The content of this part of the program will be useful in my daily life"

Analysis

Descriptive statistical analyses were conducted on EVE and UQS responses, including mean (M) and standard deviation (SD). To evaluate DAC content validity, Aiken's V (Aiken, 1980, 1985) and 95% confidence intervals (CI) were calculated for each variable. According to Charter's (2003) guideline, V values should be statistically significantly higher than 0.70, verified by observing whether the 95% CI included values below 0.70. Raw data are available at https://osf.io/54qpn/.

Results

Study 1: Expert Content Validity

The four reviewers evaluated the 23 DAC program components on five criteria. Results were highly positive:

Clarity: Mean scores ranged from 4.75 to 5.0, with Aiken's V values ranging from V = 0.93 (95% CI [0.71, 0.98]) to V = 1 (95% CI [0.80, 1.00]).

Utility: Mean scores ranged from 4.75 to 5.0, with V values ranging from V = 0.93 (95% CI [0.71, 0.98]) to V = 1 (95% CI [0.80, 1.00]).

Pertinence: Mean scores were 5.0 in all cases, with perfect agreement of V = 1 (95% CI [0.80, 1.00]) in all evaluations.

Conceptual Adjustment to ACT Model: Mean scores were 5.0 in all cases, with perfect agreement of V = 1 (95% CI [0.80, 1.00]).

Therapeutic Goal Fulfillment: Mean scores ranged from 4.75 to 5.0, with V values ranging from V = 0.93 (95% CI [0.71, 0.98]) to V = 1 (95% CI [0.80, 1.00]).

Study 1 Conclusion: The 23 DAC program components demonstrated a high degree of clarity, utility, pertinence, conceptual adjustment, and therapeutic goal fulfillment according to the four ACT experts. All V values were statistically significantly higher than the strict cutoff of 0.70.

Study 2: Clarity and Perceived Utility Assessment by Users

The 41 participants evaluated all program aspects with mean scores higher than 4 out of 5 points.

Clarity: Mean scores ranged from 4.15 to 4.79, with Aiken's V values ranging from V = 0.79 (95% CI [0.72, 0.85]) to V = 0.95 (95% CI [0.90, 0.97]).

Utility: Mean scores ranged from 4.23 to 4.76, with V values ranging from V = 0.81 (95% CI [0.74, 0.86]) to V = 0.94 (95% CI [0.89, 0.97]).

All V values were statistically significantly higher than the V = 0.70 cutoff. Components with lower scores were UQS-15 (Clarity: M = 4.15, V = 0.79; Utility: M = 4.23, V = 0.81), corresponding to "Audio 5" from Module II, although all values remained well above the acceptable threshold.

Study 2 Conclusion: The 23 DAC program components demonstrated a high degree of clarity and perceived utility according to user participants, with V values ranging from 0.79 to 0.95.

Discussion and Conclusions

The authors highlight that the study presents two complementary evaluation studies (expert validation and user evaluation) similar to the approach used in developing other online psychological interventions. The DAC program represents an adaptation to the online intervention format of RNT-focused ACT protocols previously tested in face-to-face format that have demonstrated efficacy in multiple emotional disorders in Latino/Colombian participants.

Relevant limitations are noted: (1) The EVE and UQS lacked reliability and validity evidence, being developed specifically for this study due to the lack of standardized instruments appropriate for evaluating individual components; (2) Content validity and perceived utility data were obtained in a different format than originally intended (weekly face-to-face group sessions, rather than through the online platform), although this was intentional to avoid contaminating judgments with usability difficulties of the preliminary Moodle platform; (3) Content validity, perceived utility, and efficacy data were obtained only from undergraduate psychology students, which limits generalizability to the general population, as these students might evaluate psychological interventions more positively and better understand components due to prior training.

The authors cite that Dereix-Calonge et al. (2019) presented preliminary efficacy data of the DAC program in preventing and reducing emotional symptoms in novice clinical psychology trainees, randomly assigning 85 students to the RNT-focused ACT condition or waitlist control. In the intervention condition, the program was efficacious in reducing emotional symptoms (d = 0.75), depression (d = 0.79), behaviors obstructing valued living (d = 0.51), RNT focused on clinical practice (d = 0.89), and general RNT (d = 0.62), with effects longitudinally mediated by reduction in RNT. In the subgroup of participants with high emotional symptom levels, effects were larger (d = 0.97 to 2.52 for emotional symptoms), with 73.33% showing reliable change and 66.67% clinically significant change, compared to only 7.14% in control.

A recent feasibility study has explored the efficacy of the DAC program implemented in its final web platform in reducing emotional symptoms, RNT, and psychological inflexibility in Colombian young adults with promising results (Sierra & Ortiz, 2022).

Importance and Contribution

This study contributes significantly to the literature on online psychological interventions. It presents rigorous evaluation of a self-help digital program (DAC) that adapts brief therapy protocols to web format, with validation by both experts and potential users. Findings demonstrate high levels of clarity and perceived utility (Aiken's V > 0.79 in users, V > 0.93 in experts), supporting that a briefly structured intervention grounded in contextual theory can be effectively implemented online. The research is particularly relevant as a response to urgent needs for expanding mental health services, especially in Spanish-speaking populations where previously no ACT adaptations existed for broad treatment and prevention purposes. The DAC program represents a methodological advance in developing transdiagnostic digital interventions, opening opportunities for future research on efficacy across different populations and implementation contexts.

---

---