Testing process hypotheses using more than just mediation analysis

Testing process hypotheses using more than
just mediation analysis
Johann Jacoby
Leibniz-Institut für Wissensmedien / Knowledge Media Research Center
Tübingen
20130903
14. Tagung der Fachggruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Public version 20130905
Disclaimer: Even though carefully checked, these slides may contain
errors. If you identify them, please let me know: [email protected]
1
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Outline
• What
do we test using mediation analysis?
• Challenges
• Different approaches
‒
‒
‒
‒
Causal chain (Spencer, Zanna, & Fong, 2005)
Instrumentation (Angrist, Imbens, & Rubin, 1996)
Controlling previous covariance of M and Y in
(quasi-)longitudinal designs (Maxwell & Cole, 2007)
TPIS (Jacoby & Sassenberg, 2011)
• Conclusion:
2
Call for strategy diversity
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Preliminaries
• Experimentally
minded research, no large scale
longitudinal observation studies
• Manipulated X – mediation analysis without any
manipulation not at focus
• Mediation analysis may yield accurate estimates of the
direct and the indirect effect if the model holds –
mediation analysis cannot discover causal structure
and indirect effects
• → Mediation analysis tests hypotheses, it is not an
exploratory technique
3
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypotheses
• "X
affects Y via M"
• = There is variance that X and Y share that is also
shared by M
‒
‒
‒
4
If M is to explain an effect of X, it surely must share
variance with X
If M is to explain an effect on Y, it surely must share
variance with Y
If M is to explain an effect of X on Y, it surely must
share variance with both X and Y and the variances
shared must be the same → triply shared variance
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypotheses – an analogy
Hagen
Wuppertal
map from http://www.openstreetmap.org
5
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Lüdenscheid
Process hypotheses – an analogy
• How
many of the people going from Hagen to
Lüdenscheid also came from Wuppertal? → How much
variance is shared by X, M, and Y?
• Necessary: Path a from M = aX
~ Wuppertal → Hagen
6
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypotheses – an analogy
a
map from http://www.openstreetmap.org
7
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypotheses – an analogy
• How
many of the people going from Hagen to
Lüdenscheid also came from Wuppertal? → How much
variance is shared by X, M, and Y?
• People going from Hagen to Lüdenscheid:
Path b.full from Y = b.full M
8
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypotheses – an analogy
b.full
map from http://www.openstreetmap.org
9
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypotheses – an analogy
• People
going from Hagen to Lüdenscheid in general
are of limited interest for the question of the process
• Necessary: People who came from Wuppertal to Hagen
and went on to Lüdenscheid: portion P
• Portion P is represented by
a × b, where b is the
semipartial association
of M and Y
10
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypothesis test – what we need
• An
accurate estimate of
path a
‒
‒
problematic with
measured X
unproblematic with
manipulated X
• An
accurate estimate of
the partial association b
11
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Process hypothesis test – what we need
• Fundamental
meaning of causality: "If it were not for the
cause, the effect would not have happened"
• Counterfactual
‒
‒
‒
with process: c (actually observed, factual)
without process: c' (statistically simulated,
counterfactual)
indirect effect is the difference between factual and
counterfactual states of affairs:
c – c' = ab
12
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
• factual
13
• counterfactual
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Baron & Kenny (1986) Mediation analysis
• Two
regression models
‒
‒
‒
M = aX
Y = c'X + bM
[Y = cX]
• Indirect
effect = a×b
• Testing a×b against 0:
‒
‒
14
a
M
X
b
Y
c' (c)
Sobel, Aroian, Goodman tests (normal distribution =
necessarly false assumption)
Bootstrap-based confidence intervals (Preacher &
Hayes, 2004, etc.)
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Challenges: Ambiguity of path b
•M
and Y are measured
• The association of two measured variables cannot be
causally attributed in an unambiguous way
‒
‒
‒
‒
Source: X → (M, Y) – common cause X
Source: M → Y
Source: Y → M
Source: Unknown third variable → (M, Y)
• Things
get worse if we do not look for path b.full, but for
the semipartial association path b
15
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Challenges: Ambiguity of path b
• Isolating
‒
b and identifying ab is additionally problematic when
a and b are moderated differently (see Bullock et al., 2010)
Which conditional a and which conditional b should be used to
asses the indirect effect ab?
• There could be no single mediation case, but the test would
detect mediation
• There could be mediation for every single case, but the test
would detect no mediation
•
‒
b is moderated by X (sometimes incorrectly called "moderated
mediation")
a
X
‒
16
M
c' (c)
b
Y
M contains more than one single variance component (in
addition to truely unsystematic error)
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Challenges: Uncontrollable interference with
the process
• Mediation
analysis requires measurement of the
mediator for all observations
• If the measurement administration itself alters the
process we will never know about it
• No comparison "with measurement of mediator" vs.
"without measurement of mediator" possible
17
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Alternatives to Mediation analysis
• Causal
Chain Design (Spencer, Zanna, & Fong, 2005)
• Instrumentation (Angrist, Imbens, & Rubin, 1996)
• Controlling previous covariance of M and Y in
(quasi-)longitudinal designs (Maxwell & Cole, 2007)
• TPIS (Jacoby & Sassenberg, 2011)
18
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Causal Chain Design (Spencer, Zann, & Fong, 2005)
• Obtain
estimate of the effect of M on Y in a separate
experiments:
‒
‒
Experiment 1: estimate M = aX
(manipulated X, measured M)
Experiment 2: estimate Y= b.full M (manipulated M,
measured Y)
• Requirement:
M measurable and manipulable!
• Variance in the measure of M must be the same variance
as in manipulation of M
• If the effect of Mmanipulated is not exactly the difference
measured by Mmeasured, the Causal Chain Design will yield
systematically incorrect conclusions.
• Prototype: Word, Zanna, & Cooper (1974)
19
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Causal Chain Design
• Gains
‒
‒
Unambiguous estimate of path b.full
Measurement of M cannot interfere with an assessment
of the causal path
• Challenges:
‒
‒
‒
‒
20
a measurement instrument M and a manipulation of M
where M is exactly the same are required
2 × N observations
Is the difference in Y reflecting the manipulation of M
(i.e., the effect of M on Y) the same difference as the
difference in X reflected in measured M? (M can be
generally be influenced by many others variables, not
only X)
No straight forward representation of the indirect effect
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Instrumentation
• Identifying
the unconfounded effect of M on Y
Instrument Z
z
a
X {X=1, X=2}
21
M
c' (c)
b
Y
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Instrumentation
Instrument Z to M(X2) – Z increases M
Y
• Apply
X=1
22
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Instrumentation
• Apply
Instrument Z to M(X2) – Z increases M
Y
Effekt b.full * z
X=1
23
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
X=2 & Z
Instrumentation
Instrument Z to M(X2) / Z decreases M
Y
• Apply
X=1
24
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Instrumentation
• Apply
Instrument Z to M(X2) / Z decreases M
Y
Effect b.full * z
X=1
25
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
X=2 & Z
Instrumentation
Instrument Z to M(X1) – Z increases M
Y
• Apply
X=1
26
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Instrumentation
Instrument Z to M(X1) – Z increases M
Y
• Apply
Effekt b.full * z
X=1
27
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
X=1 & Z
Instrumentation
Instrument Z to M(X1) / Z decreases M
Y
• Apply
X=1
28
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
X=2 & Z
Instrumentation
Instrument Z to M(X1) / Z decreases M
Y
• Apply
Effect b.full * z
X=1
29
X=2
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
X=1 & Z
Instrumentation
Hello again, Mediation design and analysis. This time more restrictive.
Instrument Z
a
M
b
X
{X1, X2}
30
Y
c' (c)
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Instrumentation
• Gains
‒
‒
Unambiguous estimate of path b.full
Measurement of mediator not strictly necessary (albeit
adding considerably to validity)
• Challenges
‒
‒
‒
‒
31
Find Z with known size of effect on M (or measure M to
assess z)
Find Z ⊥ X that only influences Y via M, not directly and
not via additional mediators
No clear representation of the indirect effect
1.5 × N
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Longitudinal design with manipulated X
t0
t1
M
M
a
cov (M, Y)
b
"pure"
X
c' (c)
Y
32
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Y
cov(M,Y) ⊥ b
Longitudinal design with manipulated X
• Gains
‒
‒
Estimate of the association between M and Y is
adjusted for third pre-manipulation variables
Traditional mediation design is preserved to a large
degree
• Challenges
‒
‒
‒
33
M is featured as two different variables: trait vs. state
Measure M and Y twice without introducing additional
validity threats (e.g., consistency effects, changes of
psychological meaning etc.)
Measure M and Y twice while still avoiding
interference
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
TPIS (Jacoby & Sassenberg, 2011)
• Very
old idea, but not much recognized as a test of process
hypotheses anymore (Prototype: Zanna & Cooper, 1974)
• Fundamental statement of a process hypothesis is that of
counterfactuality: "If it were not for the hypothesized process,
the effect of X on Y would be smaller."
• TPIS: Create conditions B=0 and B=1, such that
‒
‒
under B = 0: the intermediate process from X to Y can occur
undisturbed
under B = 1: the intermediate process between X and Y is
broken
• If
the process hypothesis is viable, the effect of X on Y should
be smaller under B=1 than under B=0
34
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
TPIS
•B
= 1:
‒
‒
‒
‒
35
Force M(X2, B=1) = M(X1, B=1)
or at least
M(X2, B=1) – M(X1, B=1) < M(X2, B=0) – M(X1, B=0)
B is not the same as M, but a manipulated context
variable independent of X
Then even though X varies as in B=0, no process via
M can occur
Works best if M(X1) and M(X2) are near the end of the
natural range of M (e.g., cognitive load, attribution
errors)
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
TPIS
• Gains
‒
‒
‒
‒
Actual observation of the formerly counterfactual clause of the
hypothesis ("If it were not for the process...")
Differential moderation is a lesser problem
No interference of measurement activity
M needs not necessarly be measured (even though independent
eveidence for the X → M link adds credibility)
• Challenges
‒
‒
‒
‒
36
Manipulation that fixes M independently of X required
Estimate of the magnitude of the indirect effect requires solid
knowledge about the manipulation of B and its effects
Sample size doubled
Evidence for process by interaction ('moderation') – (seemingly)
calls into question the wide spread strict distinction of mediation
and moderation
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Y
TPIS: Prototypical pattern – Evidence for
hypothesized process by an interaction effect
X=1
X=2
B=0
37
X=1
X=2
B=1
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Conclusion
• Relying
on mediation analysis exclusively will not do
justice to psychological processes.
• There are a number of viable and feasible alternatives.
• None of the alternatives are entirely unproblematic.
38
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Conclusion
• Observation
of processes instead of experimentation
on them often does not give solid evidence in noisy
data environments as the social and human sciences
are
• Manipulation is not the easy answer, the process may
easily be changed through intervention (even through
observation)
• Special challenges:
‒
‒
‒
39
Asynchronous events and observations
Strong requirements for definition of events and
observations (variance partitionning)
Mixed events
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Conclusion
• The
measurement of variable states and the events
creating these states are two different things.
• Processes
measured.
always need to be inferred, cannot be
• Inference
is rooted in design, not in analysis techniques
or ways to derive confidence intervals.
• Viable
and solid understanding of processes will be
grounded in good and diverse designs.
40
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Conclusion
• Good
‒
‒
‒
‒
‒
designs comprise
well defined,
well distinguished,
psychometrically sound variables (systematic
variance + error),
designing ahead,
stating hypotheses ahead of time
• No
one single design of process testing will serve as a
panacea, rather their versatile and systematically
developed combination for each research domain will
yield robust results.
41
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
Thank you!
42
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen
References
43
•
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the
American Statistical Association, 91, 444-455.
•
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual,
strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173.
•
Jacoby, J., & Sassenberg, K. (2011). Interactions do not only tell us when, but can also tell us how: Testing process hypotheses by
interaction. European Journal of Social Psychology, 41, 180-190.
•
Maxwell, S. E., & Cole, D. A. (2007). Bias in cross-sectional analyses of longitudinal mediation. Psychological Methods, 12, 23.
•
Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models.
Behavior Research Methods, Instruments, & Computers, 36, 717-731.
•
Spencer, S. J., Zanna, M. P., & Fong, G. T. (2005). Establishing a causal chain: why experiments are often more effective than
mediational analyses in examining psychological processes. Journal of Personality and Social Psychology, 89, 845.
•
Word, C. O., Zanna, M. P., & Cooper, J. (1974). The nonverbal mediation of self-fulfilling prophecies in interracial interaction.
Journal of Experimental Social Psychology, 10, 109-120.
•
Zanna, M. P., & Cooper, J. (1974). Dissonance and the pill: an attribution approach to studying the arousal properties of
dissonance. Journal of Personality and Social Psychology, 29, 703.
14. Tagung der Fachgruppe Sozialpsychologie in der DGPs, 1.-4. September 2013, Hagen