Psychometric properties of the Valuing Questionnaire in a Spaniard sample and factorial equivalence with a Colombian sample

Background and objectives

The Valuing Questionnaire (VQ) is an instrument designed to measure the experience of living according to one's values in daily life according to Acceptance and Commitment Therapy (ACT) theory. Within ACT theory, values are verbally constructed positive reinforcers that provide direction and meaning to human behavior. The VQ is considered one of the most psychometrically robust instruments for measuring valued living according to the acceptance and commitment model. This study aimed to examine the psychometric properties of the VQ in a large Spanish sample (N = 846) and to analyze measurement invariance across countries by comparing with a Colombian sample (N = 724).

Method

Participants:

Sample 1 (Spain): Comprised 846 Spanish adult participants (75.7% female), with mean age 35.40 years (SD = 11.39) and age range 18–72 years. Educational level: 0.1% no formal studies, 33% primary education or mid-level studies, 65.6% university graduates or college educated (1.3% did not indicate educational level). Approximately 44.6% reported having received psychological or psychiatric treatment in the past, but only 12.8% were currently in treatment. 12.9% reported using psychotropic medication. Participants were recruited through snowball sampling via social media.

Sample 2 (Colombia): Comprised 724 Colombian participants (74.4% female), with mean age 26.11 years (SD = 8.93) and age range 18–88 years. Educational level: 17.8% primary or mid-level studies, 82.2% university graduates or college educated. 45% reported having received psychological or psychiatric treatment at some point, but only 8.4% were currently in treatment. 5.4% reported using psychotropic medication.

Main instrument:

Valuing Questionnaire (VQ): A 10-item self-report instrument with 7-point Likert scale responses (6 = completely true; 0 = not at all true). Designed to assess the general experience of living according to values during the past week, without specifying particular life domains. The original VQ by Smout et al. (2014) showed a two-factor structure: Progress (enactment of values; 5 items reflecting clear awareness of what is personally meaningful and perseverance) and Obstruction (disruption of valued living; 5 items reflecting disruption of valued living due to avoidance of unwanted experiences and distraction from values). The Spanish version was translated by Ruiz et al. (2021).

Other instruments (convergent and discriminant validity):

1. Acceptance and Action Questionnaire-II (AAQ-II): A 7-item instrument on a 7-point Likert scale measuring experiential avoidance. The Spanish version showed good psychometric properties (α = .91).

2. Cognitive Fusion Questionnaire (CFQ): A 7-item scale on a 7-point Likert scale assessing general cognitive fusion. The Spanish version showed excellent internal consistency (α = .93).

3. Depression, Anxiety, and Stress Scales-21 (DASS-21): A 21-item instrument on a 4-point Likert scale assessing negative emotional symptoms (depression, anxiety, stress). In this study it obtained alphas of .95 (total scale), .92 (depression), .87 (anxiety), and .86 (stress).

4. Satisfaction with Life Scale (SWLS): A 5-item instrument on a 7-point Likert scale assessing perceived well-being. The Spanish version showed α = .89 and convergent validity in a previous Spanish sample.

Procedure: Both samples completed an anonymous online survey distributed through social media using snowball sampling procedures. Researchers shared the survey publication to reach more people. Participants provided informed consent by accepting the conditions (only exclusion criterion: being under 18 years). Anonymity was emphasized and participants were told they could stop participation at any time. Median completion time was approximately 15 minutes. The survey was available for Sample 1 for one year until at least 200 participants of both genders were reached.

Statistical analyses:

1. Confirmatory Factor Analysis (CFA): CFA was conducted to test the two-factor model of the VQ using Sample 1 (Spanish sample). The procedure used in the previous Colombian validation (Ruiz et al., 2021) was followed. Given previous research (Smout et al., 2014), error terms between items 5 and 7 were allowed to correlate to compare fit with the model without correlations. LISREL (version 8.71) was used with robust maximum likelihood (MLR) estimation method because data did not meet multivariate normality assumptions (Mardia kurtosis test = 610.954; p < .001).

2. Goodness-of-fit indices evaluated:

   • Comparative Fit Index (CFI): values > .90 acceptable, > .95 good
   • Non-Normed Fit Index (NNFI): similar criteria to CFI
   • Root Mean Square Error of Approximation (RMSEA): values < .08 acceptable, < .05 good
   • Normalized Parsimony Fit Index (PNFI): higher values indicate greater parsimony
   • Standardized Root Mean Square Residual (SRMR): values < .08 acceptable, < .05 very good
   • Test of Close Fit (PCLOSE): one-sided test that RMSEA ≤ .03

3. Construct reliability: Composite Reliability Coefficient (CRC) was calculated considering values > .70 as adequate construct reliability and appropriate internal consistency. Average Variance Extracted (AVE) was calculated considering values ≥ .50 as indicators of satisfactory reliability.

4. Convergent and discriminant validity: Brown (2015) and Fornell & Larcker (1981) criteria were applied: (a) factor loadings should be significant (standardized estimates ≥ .40), (b) CRC > .70, (c) AVE ≥ .50 for each construct. For discriminant validity: (a) inter-construct correlation < .80 and (b) square roots of AVE should be greater than inter-construct correlations with other factors.

5. Measurement invariance: Metric, scalar, and strict invariance across countries (Spain and Colombia) and gender was analyzed following procedures by Jöreskog (2005) and Millsap & Yun-Tein (2004). Sequentially more restrictive models were tested: multigroup baseline model, metric invariance (equal factor loadings), scalar invariance (equal factor loadings and intercepts), strict invariance (equal loadings, intercepts, and residuals). For model comparison, Chen (2007) and Cheung & Rensvold (2002) criteria were used: (a) RMSEA difference (ΔRMSEA) < .01, (b) NNFI difference (ΔNNFI) and CFI difference (ΔCFI) ≥ −.01.

6. Internal consistency: The MBESS package in R (Kelley & Lai, 2012; Kelley & Pornprasertmanit, 2016) was used to calculate Cronbach's alphas and McDonald's omegas with 95% confidence intervals via bootstrap. Values > .70 considered acceptable, > .80 considered good according to Nunnally & Bernstein (1994).

7. Correlations with other variables: Pearson correlations were calculated between the VQ and remaining scales (AAQ-II, CFQ, DASS-21, SWLS) using SPSS 25. Interpreted according to Lenhard & Lenhard (2016) guidelines: small correlation .10–.20, medium .21–.36, large > .36.

8. Descriptive analyses: Differences in VQ scores across countries and gender were explored using independent samples t-tests.

Results

Validity evidence based on internal structure:

Dimensionality: The two-factor VQ model showed good fit to the data in Sample 1 according to goodness-of-fit indices: CFI = .98, NNFI = .97, SRMR = .053. The RMSEA value = .073 indicated acceptable fit. However, the test of close fit (PCLOSE) showed that the RMSEA value was significantly higher than .05 (p < .01). The alternative two-factor model with correlated error terms in items 5 and 7 did not significantly improve fit. Therefore, the standard two-factor model was selected due to its greater parsimony (PNFI = .74 vs .71).

Construct reliability and convergent and discriminant validity: The VQ showed high construct reliability with CRC values greater than .70 for both factors: VQ-Progress (CRC = .85) and VQ-Obstruction (CRC = .84). AVE values were: VQ-Progress = .54 and VQ-Obstruction = .52, both greater than .50. Regarding discriminant validity, the inter-construct correlation was r = −.60 (absolute value less than .80), indicating adequate discriminant validity. Square roots of AVE were greater than inter-construct correlations with other factors: √AVE = .73 for VQ-Progress and √AVE = .72 for VQ-Obstruction.

Psychometric quality of items: All VQ items in Sample 1 showed high corrected item-total correlations. For VQ-Progress, correlations ranged from .52 to .71; for VQ-Obstruction, from .51 to .70. Cronbach's alpha was .85 (95% confidence interval: [.83–.86]) for VQ-Progress and .84 (95% CI: [.82–.86]) for VQ-Obstruction. McDonald's omega values were virtually identical: VQ-Progress = .85 (95% CI: [.83–.87]) and VQ-Obstruction = .84 (95% CI: [.82–.86]).

Measurement invariance: Measurement invariance was demonstrated at metric, scalar, and strict levels across countries (Spain and Colombia) and gender. Differences in RMSEA, CFI, and NNFI were less than .01 in all models compared, indicating that the instrument maintains equivalence in its structure and scores across groups.

Validity evidence based on relationships with other variables:

The VQ showed correlations in expected directions and of magnitude coherent with assessed constructs.

VQ-Progress:

• Negative correlation with VQ-Obstruction (r = −.51)
• Medium-magnitude negative correlation with experiential avoidance (AAQ-II: r = −.49)
• Medium-magnitude negative correlation with cognitive fusion (CFQ: r = −.44)
• Medium-magnitude negative correlation with total emotional symptoms (DASS-21: r = −.36)
• Small to medium-magnitude negative correlations with depression (r = −.48), anxiety (r = −.23), and stress (r = −.24) specifically
• Strong positive correlation with life satisfaction (SWLS: r = .64)

VQ-Obstruction:

• Strong positive correlation with experiential avoidance (AAQ-II: r = .66)
• Strong positive correlation with cognitive fusion (CFQ: r = .68)
• Strong positive correlation with emotional symptoms (DASS-21 total: r = .65)
• Strong positive correlations with depression (r = .66), anxiety (r = .53), and stress (r = .54) specifically
• Strong negative correlation with life satisfaction (SWLS: r = −.53)

VQ scores across countries and gender:

No statistically significant differences were found in VQ scores between Spain and Colombia:

• VQ-Progress: Spain M = 19.08, SD = 6.08; Colombia M = 19.50, SD = 6.43; t(1568) = −1.35, p = .18
• VQ-Obstruction: Spain M = 11.31, SD = 6.63; Colombia M = 11.70, SD = 6.88; t(1568) = −1.16, p = .25

Regarding gender in Spanish participants, no significant differences were found:

• VQ-Progress: Men M = 18.72, SD = 6.13; Women M = 19.14, SD = 6.08; t(811) = −0.80, p = .43
• VQ-Obstruction: Men M = 11.14, SD = 6.60; Women M = 11.39, SD = 6.69; t(811) = −0.43, p = .67

Discussion and Conclusions

Summary of findings: The VQ is one of the most widely used and psychometrically robust instruments for assessing valued living according to the ACT model. Although a Spanish version of the VQ exists (Ruiz et al., 2021), this was the first study to analyze its psychometric properties in large Spanish samples. Additionally, it was the first to examine factorial equivalence of the VQ across different cultures.

Results demonstrated that the VQ obtains good psychometric properties in the Spanish sample. Regarding internal consistency, the VQ showed appropriate Cronbach's alphas and McDonald's omegas for both factors (Progress = .85; Obstruction = .84), consistent with previous validation studies.

The two-factor VQ model obtained good fit to the data. Goodness-of-fit was very similar to the two-factor model with correlated error terms in items 5 and 7, which was the factor model considered most appropriate in the original validation study (Smout et al., 2014). As in Ruiz et al. (2021), the standard two-factor model was selected due to its greater parsimony. Additionally, the measurement model showed adequate construct reliability and convergent and discriminant validity.

The VQ also demonstrated measurement invariance at metric, scalar, and strict levels across countries and gender. Measurement invariance is relevant because it allows comparison of scores across Spanish and Colombian samples with similar characteristics. This finding supports the utility of comparing valued living across Spanish-speaking countries.

Correlations of the VQ with other instruments were theoretically coherent and equivalent to those found in other studies. VQ-Progress correlated positively with life satisfaction and negatively with emotional symptoms, experiential avoidance, and cognitive fusion. VQ-Obstruction showed the opposite pattern of correlations, providing additional support for the adaptive role of valued living according to the ACT model.

Similarity with previous studies: The Spanish version of the VQ showed similar psychometric properties in this Spanish sample to those found in the previous study with Colombian samples (Ruiz et al., 2021). Accordingly, this study provides additional evidence for the robust psychometric properties of the VQ. To the authors' best knowledge, this is the first analysis of factorial equivalence of the VQ across different cultures.

Limitations:

1. The VQ was administered only to a nonclinical sample. Future studies should analyze the psychometric properties of the VQ in Spanish clinical participants and measurement invariance across clinical and nonclinical samples, particularly given that Ruiz et al. (2021) found strict measurement invariance of the VQ across Colombian clinical and nonclinical samples.

2. The percentage of women was significantly higher than men in the Spanish sample. However, this limitation was partially reduced by the results obtained in the measurement invariance analyses across gender.

3. The VQ was correlated only with other self-reports, which could have inflated the correlations found. Self-reports can present measurement biases.

4. Mean age was relatively low, possibly a consequence of more frequent internet and social media use by young people in Spain. Future studies should explore the psychometric properties of the VQ in samples of older participants.

5. Treatment sensitivity of the VQ was not evaluated in Spanish samples. Subsequent studies should analyze whether the VQ demonstrates sensitivity to change in Spanish samples according to previous research using the Spanish version of the VQ in Colombia.

Practical implications: Despite these limitations, the current study has practical implications for researchers and mental health professionals in Spain:

First, the VQ instrument could be used in research on values in basic and clinical research contexts, as well as in survey studies that analyze the role of values in mental health and clinical interventions analyzing the efficacy and change processes in ACT interventions.

Second, the VQ can be adopted in routine assessment conducted by ACT practitioners.

Third, the VQ can be used by researchers aiming to compare valued living across gender and between Spain and Colombia.

Significance and contribution

This study contributes significantly to psychological research in Spanish-speaking contexts by demonstrating that the Valuing Questionnaire maintains robust psychometric properties in Spanish samples, with confirmed internal structure validity, adequate construct reliability, and convergent and discriminant validity with theoretically related constructs. Additionally, the study provides evidence of cross-cultural factorial equivalence between Spain and Colombia, enabling valid comparisons of valued living across these contexts and facilitating the instrument's use in research and clinical practice in Spanish-speaking populations.

---