Experiment 1
Experiment 1 was an initial, simplified design with just two between-subjects conditions: a fake specific condition, in which participants were shown a deepfake video and specifically warned beforehand that it was fake, and a control condition, in which participants were shown the control video, as described above, with no warning beforehand. Participants in the control condition were therefore provided with no specific information to assess John Carter’s guilt or innocence. Data for Experiment 1 were collected on 24–25 November 2022.
Perception of guilt ratings by condition are shown in Fig. 4, and analysis of associated hypotheses is shown in Table 1. Perception of guilt was greater in the fake specific condition (0.87; close to maybe yes) than in the control condition (–0.40; near impossible to say), t(134.17) = 6.16, p <0.001, d = 0.96, 95% CI [0.64, 1.28] (H1).
Error bars show 95% confidence intervals. Violin plots and data points show the distribution of responses. Experiment 1 was an initial, simplified design with just two conditions: control (n = 100), and fake specific (n = 75). Experiment 2 employed a 2 ×3 factorial design, plus a control condition, thus: control (n = 48), real none (n = 39), fake none (n = 37), real generic (n = 39), fake generic (n = 40), real specific (n = 38), and fake specific (n = 34). Experiment 3 employed a 2 ×2 between-subjects factorial design, thus: real none (n = 60), fake none (n = 56), real specific (n = 57), and fake specific (n = 50).
Qualitative responses for perception of guilt are summarized in Table 2. This shows that 53.3% (n = 40, coded G/V or G/WV) of the 75 participants in the fake specific condition believed John Carter was guilty based (entirely or partially) on the content of the video, compared with 12.0% (n = 12) of the 100 participants in the control condition. Only 13.3% (n = 10, coded N/W or I/W) of participants in the fake specific condition cited the warning they had been shown beforehand, stating that the video was a deepfake, as their sole reason for answering not guilty or impossible to say.
Perception of fakeness ratings by condition are shown in Fig. 5, and analysis of associated hypotheses is shown in Table 1. Perception of fakeness was greater in the fake specific condition (1.27; maybe yes) than in the control condition (–0.07; impossible to say), t(156.16) = 6.07, p <0.001, d = 0.93, 95% CI [0.61, 1.25] (H2).
Error bars show 95% confidence intervals. Violin plots and data points show the distribution of responses. Experiment 1 was an initial, simplified design with just two conditions: control (n = 100), and fake specific (n = 75). Experiment 2 employed a 2 × 3 factorial design, plus a control condition, thus: control (n = 48), real none (n = 39), fake none (n = 37), real generic (n = 39), fake generic (n = 40), real specific (n = 38), and fake specific (n = 34). Experiment 3 employed a 2 × 2 between-subjects factorial design, thus: real none (n = 60), fake none (n = 56), real specific (n = 57), and fake specific (n = 50).
Qualitative responses for perception of fakeness indicated that, of the 75 participants in the fake specific condition, 69.3% (n = 52) believed the video was a deepfake. Only 42.3% (n = 22) of these 52 participants mentioned the warning they were given before watching the video as a reason for believing it was a deepfake, whereas 61.5% (n = 32) referred to some aspect of the video itself (e.g., visual defects).
We also conducted a conditional analysis for the subset of 52 participants who were shown a specific warning followed by the deepfake video, and subsequently indicated that they believed the warning and therefore knew the video to be fake. Mean perception of guilt for this subset of participants who accepted the warning (0.87) was still greater than in the control condition (–0.40), t(84.74) = 5.50, p <0.001, d = 0.97, 95% CI [0.61, 1.34]. Qualitative responses indicated that, despite believing the warning, 53.8% (n = 28) of these participants nevertheless relied on the content of the video to conclude that John Carter was guilty, compared with 12.0% (n = 12) of 100 participants in the control condition.
Two additional variables were measured in Experiment 1: Perception of suitability was measured with the question “Do you think John Carter is a suitable person to hold a public position (e.g., planning department, tax office)?”, using the same 7-point Likert scale, reverse-coded to show unsuitability. Mean perception of unsuitability was greater in the fake specific condition (1.45; maybe yes) than in the control condition (0.13; impossible to say), t(151.57) = 6.61, p <0.001, d = 1.02, 95% CI [0.70, 1.34]. Perception of authenticity was measured with the question “Do you think the video shows what actually happened?”, using the same Likert scale, reverse-coded to show inauthenticity. There was no statistically significant difference between conditions for mean perception of inauthenticity, t(155.54) = 0.68, p = .0496, d = 0.11, 95% CI [–0.40, 0.20].
Experiment 2
Experiment 2 employed a 2 (real, fake video) × 3 (none, generic, specific warning) between-subjects factorial design, plus a control condition as in Experiment 1, making seven conditions in total. Data for Experiment 2 were collected on 13–14 April 2023.
Perception of guilt ratings by condition are shown in Fig. 4, and analysis of associated hypotheses is shown in Table 1. There was a significant main effect for both the video variable, F(1,221) = 6.55, p = 0.011, η²p = 0.029, 95% CI [0.001, 0.085], and the warning variable, F(2,221) = 15.41, p <0.001, η²p = 0.122, 95% CI [0.050, 0.203], with no significant interaction, F(2,221) = 1.39, p = 0.252, η²p = 0.012, 95% CI [0.000, 0.050]. As in Experiment 1, mean perception of guilt was greater in the fake specific condition (0.62; maybe yes) than in the control condition (–0.29; impossible to say), t(55.41) = 2.60, p = 0.012, d = 0.60, 95% CI [0.14, 1.05] (H1). We found no statistically significant difference between fake generic and fake specific conditions, t(70.98) = 0.26, p = 0.799, d = 0.06, 95% CI [–0.40, 0.52] (H1a), but mean perception of guilt was greater in the fake none condition (2.14; probably yes) than in the fake generic condition (0.73; maybe yes), t(61.30) = 4.22, p <0.001, d = 0.95, 95% CI [0.47, 1.43] (H1b), and similarly, mean perception of guilt was greater in the real none condition (2.18; probably yes) than in the real generic condition (1.49, maybe yes), t(62.55) = 2.62, p = 0.011, d = 0.60, 95% CI [0.14, 1.05] (H3a). We found no statistically significant difference between real generic and real specific conditions, t(73.86) = 0.58, p = 0.562, d = 0.13, 95% CI [–0.32, 0.58] (H3b).
Qualitative responses for perception of guilt are summarized in Table 2. This shows that 89.7% (n = 35) of the 39 participants in the real none condition, who were shown a real video of John Carter admitting to committing a crime without any prior warning, believed him to be guilty based on the video, compared with 10.4% (n = 5) of 48 participants in the control condition. In the fake generic condition, 62.5% (n = 25) of 40 participants believed John Carter to be guilty based on the video, reducing further to 47.1% (n = 16) of 34 participants in the fake specific condition.
Perception of fakeness ratings by condition are shown in Fig. 5, and analysis of associated hypotheses is shown in Table 1. There was a significant main effect for both the video variable, F(1,221) = 4.04, p = 0.046, η²p = .018, 95% CI [0.000, 0.067] and the warning variable, F(2,221) = 18.74, p <0.001, η²p = 0.145, 95% CI [0.067, 0.228], with no significant interaction, F(2,221) = 1.95, p = 0.145, η²p = 0.017, 95% CI [0.000, 0.060]. As in Experiment 1, mean perception of fakeness was greater in the fake specific condition (1.88; probably yes) than in the control condition (0.00; impossible to say), t(76.48) = 5.44, p <0.001, d = 1.21, 95% CI [0.72, 1.68] (H2). Mean perception of fakeness was greater in the fake specific condition (1.88; probably yes) than in the fake generic condition (0.18; impossible to say), t(71.67), = 4.81, p <0.001, d = 1.12, 95% CI [0.62, 1.61] (H2a), but we found no statistically significant difference between fake generic and fake none conditions, t(73.64) = –0.25, p = 0.801, d = –0.06, 95% CI [–0.51, 0.39] (H2b), nor between real generic and real none conditions, t(75.18) = –1.65, p = 0.102, d = –0.38, 95% CI [–0.82, 0.07] (H4a). Mean perception of fakeness was greater in the real specific condition (0.97; maybe yes) than in the real generic condition (–0.18, impossible to say), t(73.17) = 3.64, p <0.001, d = 0.83, 95% CI [0.36, 1.29] (H4b).
Qualitative responses for perception of fakeness indicated that, whilst 82% (n = 28) of the 34 participants in the fake specific condition believed the video was a deepfake, only 43% (n = 12) of these 28 participants mentioned the warning they were given before watching the video, whereas 71% (n = 20) referred to some aspect of the video itself (e.g., visual defects).
We also conducted a conditional analysis for the subset of 47 participants who were shown a warning (either generic or specific), followed by the deepfake video, and subsequently indicated that they believed the warning and therefore knew the video to be fake. Mean perception of guilt for this subset of participants who accepted the warning (0.43) was still greater than in the control condition (–0.29), t(77.52) = 2.13, p = 0.036, d = 0.44, 95% CI [0.03, 0.85]. Qualitative responses indicated that, despite believing the warning, 44.7% (n = 21) of this subset of participants nevertheless relied on the content of the video to conclude that John Carter was guilty, compared with 10.4% (n = 5) of 48 participants in the control condition.
Experiment 3
Experiment 3 employed a 2 (real, fake video) × 2 (none, specific warning) between-subjects factorial design, based on a new AI-generated deepfake video. Data for Experiment 3 were collected on 30 September 2024.
Perception of guilt ratings by condition are shown in Fig. 4, and analysis of associated hypotheses is shown in Table 1. There was a significant main effect for the warning variable, F(1,219) = 14.26, p <0.001, η²p = 0.061, 95% CI [0.014, 0.132], but not the video variable, F(1,219) = 0.68, p = 0.413, η²p = 0.003, 95% CI [0.000, 0.034]. As in Experiments 1 and 2, mean perception of guilt was greater than zero in the fake specific condition (1.04; maybe yes), but lower than in the real none condition (2.00; probably yes), t(88.65) = 2.74, p = .007, d = 0.53, 95% CI [0.15, 0.91] (H1).
Qualitative responses for perception of guilt are summarized in Table 2. This shows that 83.3% (n = 50) of 60 participants in the real none condition, and 87.5% (n = 49) of 56 participants in the fake none condition, believed Amelia Palmer to be guilty based on the video, reducing to 56.0% (n = 28) of 50 participants in the fake specific condition.
Perception of fakeness ratings by condition are shown in Fig. 5, and analysis of associated hypotheses is shown in Table 1. There was a significant main effect for both the video variable, F(1,219) = 8.94, p = 0.003, η²p = 0.039, 95% CI [0.005, 0.101], and the warning variable, F(1,219) = 8.21, p = 0.005, η²p = 0.036, 95% CI [0.004, 0.097], with no significant interaction, F(1,219) = 3.78, p = 0.053, η²p = 0.017, 95% CI [0.000, 0.065]. As in Experiments 1 and 2, mean perception of fakeness was greater in the fake specific condition (0.76; maybe yes) than in the real none condition (–0.60; maybe not), t(97.90) = 4.13, p <0.001, d = 0.80, 95% CI [0.40, 1.19] (H2).
Qualitative responses for perception of fakeness indicated that, whilst 60% (n = 30) of the 50 participants in the fake specific condition believed the video was a deepfake, only 20% (n = 6) of these 30 participants mentioned the warning they were given before watching the video, whereas 80% (n = 24) referred to some aspect of the video itself (e.g., visual defects).
We also conducted a conditional analysis for the subset of 30 participants who were shown a specific warning followed by the deepfake video, and subsequently indicated that they believed the warning and therefore knew the video to be fake. Mean perception of guilt for this subset of participants who accepted the warning was still greater than zero (0.67), but lower than in the real none condition (2.00), t(42.83) = 2.95, p = 0.005, d = 0.69, 95% CI [0.23, 1.16]. Qualitative responses indicated that, despite believing the warning, 50.0% (n = 15) of these participants nevertheless relied on the content of the video to conclude that Amelia Palmer was guilty, compared with 83.3% (n = 50) of 60 participants in the real none condition.
