A preliminary comparison of real focus group themes against simulated personas across three runner variants (Baseline, Think Talk, and Think Talk Verify) using theme overlap and semantic distance.
A note on this series: This is the second entry in the Synthetic Studio Research Log, a multi-part blog series sharing details around our work simulating people with language models. The first entry, LLMs Are People Simulators, made the case that the ability to simulate people is a byproduct of how language models are trained, and that nobody is measuring whether it holds. This entry takes up the qualitative half of that measurement problem: whether a synthetic focus group can reproduce the themes that emerge from a real one. As with every entry, the findings are provisional by design. Models change, methods improve, and we will revise or retract as the evidence moves.
01
The preliminary answer is encouraging: simulated focus groups recovered a meaningful portion of the themes found in real conversations. Across three research domains—Work from home Sedentary Behaviour (WFH), Scholarly Git Experience (IASGE), and Software Engineering Creativity (SEC)—all three runners surfaced relevant thematic signal, providing preliminary evidence that synthetic persona generation can reproduce part of the thematic pattern observed in real focus groups.
The three runner variants show domain- and metric-dependent strengths. Think Talk Verify performs best on several recovery metrics in WFH and IASGE, Think Talk leads several semantic and recovery metrics across IASGE and SEC, and Baseline remains competitive on selected SEC measures. The expanded evidence supports a trade-off rather than a single overall winner.
02
Focus groups are expensive, time-consuming, and operationally difficult to run repeatedly. If simulated focus groups can approximate the thematic conclusions of real conversations, they could become a useful tool for early-stage research, scenario testing, and rapid iteration.
The early evidence is promising. Simulated sessions usually identify relevant themes, especially when they do detect a theme, and the higher precision scores suggest they tend to stay close to the real codebook. Recall remains the main area for improvement, which gives the next phase a clear target: broaden coverage while preserving fidelity.
These preliminary results justify investigating which level of simulation is most appropriate for different research objectives. Even imperfect simulations may support early-stage hypothesis generation, scenario testing, and research planning, but whether the observed alignment generalizes across topics remains an open question.
03
The experiment compares three simulation runners. All three use the same moderator question list extracted from the real transcript and the same downstream theme-scoring process. What changes is how simulated participant responses are generated.
Baseline Runner is the simplest implemented runner. Each round has an interviewer turn, then each participant responds once with a direct LiteLLM call. Participant prompts are wrapped with shared conversation rules that enforce persona grounding, concise answers, no AI self-reference, and no speaker labels.
The participant response is a single call: transcript context plus an instruction to answer the interviewer from the persona’s lived perspective. There is no private reasoning step and no verification pass.
Think-Talk Runner adds a two-step participant turn. First, THINK: a private call asks the persona to reason about emotional reaction, stance, personal fit, main contribution, character fit, and whether silence or a short answer is more realistic. Then TALK: a second call receives the transcript plus the private thinking and produces the spoken answer.
Only the TALK output is appended to the transcript. The private THINK output is stored in a tracing field for analysis and debugging, but it does not appear in the simulated conversation.
Think-Talk-Verify Runner extends Think Talk with a fidelity judge. Each participant turn does three steps: THINK, TALK, and VERIFY. The VERIFY step scores the candidate response from 0–100 against the persona profile, conversation stimulus, drafted response, and private reasoning.
If the normalized score is below 0.7, the runner regenerates the answer with a corrective hint. This 0.7 threshold is a provisional engineering default, not a validated fidelity cutoff: it is intended to catch clearly weak or generic responses while avoiding excessive regeneration on borderline-but-usable turns. It allows up to two regeneration attempts and keeps the highest-scoring candidate.
The final transcript receives only the selected final answer. The raw output records verifying calls, swapping count, a fidelity regeneration count, thinking trace, and a detailed verifying trace containing scores, reasons, original and corrected text, and attempts.
04
The scored results cover three domains. WFH Sedentary Behaviour has 30 evaluation samples per runner, based on six standardized focus group transcripts with five runs per transcript. Scholarly Git Experience (IASGE) has 15 evaluation samples per runner, based on three transcripts with five runs per transcript. Software Engineering Creativity (SEC) has 20 evaluation samples per runner, based on four transcripts with five runs per transcript. The WFH dataset therefore produces 90 scored simulation runs in total across the three runners.
The main objective of experimenting with multiple focus groups is to make our results domain-agnostic. IASGE focuses on the scholarly Git experience, while SEC focuses on perceived creativity in software engineering. Together with WFH, they test whether runner patterns generalize beyond a single research domain.
05
The pipeline starts with raw focus group transcripts. Moderator turns are segmented, interviewer questions are extracted, and each transcript is standardized so the same question flow can be replayed in simulation.
Grounded personas are then generated for the real participants, and each standardized transcript is simulated with the selected runner: Baseline, Think Talk, or Think Talk Verify. The simulations use the model OpenAI/GPT-5.2, with the same moderator questions and the same downstream scoring process across runners.
After each simulated session is generated, both real and simulated transcripts are scored against the shared codebook. The resulting theme detections and supporting utterances are used to compute Jaccard, precision, recall, F1, cosine similarity, and Jensen-Shannon divergence. Metrics are then logged for comparison across runners and runs.
06
The evaluation starts with a shared codebook: a structured list of themes researchers want to track across transcripts. The same codebook is applied to both the real focus group transcript and each simulated session, creating a common basis for comparison.
This theme extraction approach is based on a common focus group analysis strategy created by Braun and Clarke. Thematic analysis is a method for identifying, analyzing, and reporting patterns (themes) within data. It minimally organizes and describes your data in detail.
The study then computes six metrics across two dimensions. Set-overlap metrics measure whether the same themes appear in real and simulated outputs. Semantic and distributional metrics measure whether the content and salience of those themes are close to the real transcript.
The six metrics compare real and simulated transcripts at the theme level:
07
The results presented here are preliminary and cover three domains with unequal evaluation-set sizes: WFH Sedentary Behaviour (30 evaluation samples per runner), Scholarly Git Experience (IASGE; 15 per runner), and Software Engineering Creativity (SEC; 20 per runner). Rankings vary by domain and metric, so small differences should not be treated as meaningful without uncertainty analysis. The results support comparison across runners, not a universal winner.
WFH Sedentary Behavior Results
Scholarly Git Experience Results
Software Engineering Creativity Results
Jaccard scores should be interpreted in relation to the dataset and evaluation design. In the WFH Sedentary Behaviour dataset, scores ranged from 0.449 to 0.495. These values may appear modest, but perfect thematic overlap should not be assumed even when humans analyze the same qualitative material.
McFarlane, Ancker, and Kukafka (2008) examined independent researchers coding the same focus-group transcripts with a shared 70-code scheme. For two coders, their agreement statistic reduces to the Jaccard coefficient. Reported agreement varied with coder training and the unit of analysis, with illustrative values ranging from approximately 0.32 to 0.58 and higher values when analysis was restricted to mutually coded passages.
The comparison with McFarlane et al. is contextual rather than direct. Their study measured agreement over codes assigned to transcript lines or text blocks, whereas this study compares sets of themes detected across complete transcripts. A study-specific human-to-human benchmark using the same codebook and transcript-level procedure is still needed before making claims about human-equivalent performance.
The cost data helps frame the practical trade-off between runner complexity and performance. We extracted a subsample of runs containing recorded usage and pricing data and calculated the average cost per runner from that subsample. The figures should therefore be interpreted as indicative estimates rather than definitive operational costs across the entire experiment.
Think Talk Verify was strongest on several WFH recovery measures, but its 3× cost increase was much larger than its improvement in theme overlap. Think Talk may therefore be a provisional practical default when balancing performance and cost.
08
The most encouraging observation is not that one runner outperforms the others. It is that all three runners recover measurable thematic signal from real focus group conversations. McFarlane et al. provide useful context for interpreting moderate overlap, but they do not establish a universal threshold or demonstrate that synthetic sessions are equivalent to human research.
Across the three domains, runner performance is domain- and objective-dependent. IASGE: Think Talk Verify leads Jaccard, F1, and recall; Think Talk leads cosine similarity and precision, while also having the lowest JSD. SEC: Think Talk leads Jaccard, cosine similarity, and recall; Think Talk Verify has the highest precision; Baseline has the highest F1 and lowest JSD. These results do not establish that simulation can replace real qualitative research, but support synthetic persona generation as a complementary research tool.
Baseline Establishes the Reference Point
Baseline remains a useful reference point and is competitive on selected Software Engineering Creativity measures. Its relative strengths differ by domain, so it should not be characterized as the universal semantic leader.
Think Talk Captures Most of the Gains
Adding private reasoning delivers most of the measurable improvement, increasing Jaccard, F1, precision, and recall without introducing the full cost of verification. This makes Think Talk an attractive balance between quality and cost.
Think Talk Verify Has the Highest Observed Recovery
Think Talk Verify performs strongly on several recovery measures, especially in WFH and IASGE, but the broader comparison shows that verification is not uniformly best across domains or metrics. The current analysis does not establish statistical significance or causality.
Considerations on Score Differences between Focus Groups
The higher scores in IASGE and SEC should not be interpreted as a universal improvement in simulation quality. The domains use different codebooks and have different thematic structures: WFH includes a broader set of 77 codes, while IASGE and Software Engineering Creativity use 40 and 39 codes respectively. The WFH simulations therefore face a more diverse thematic target and show both lower recall and lower precision. These results suggest that synthetic persona performance depends not only on runner design, but also on the coherence and granularity of the research domain’s thematic framework.
09
These findings should be interpreted as preliminary. They cover three domains but lack uncertainty intervals and significance tests and should not be pooled into a single overall ranking. Small differences, such as IASGE JSD values of 0.144 versus 0.145, should not be treated as meaningful rankings without uncertainty analysis.
The study also lacks a directly comparable human-to-human baseline. Although McFarlane et al. reported Jaccard-equivalent agreement in human coding of focus-group transcripts, their analysis operated at the transcript-line and text-block level rather than the whole-transcript theme level used here. Their findings therefore provide interpretive context but should not be treated as an acceptability threshold or as evidence that the simulations have reached human-equivalent agreement.
10
Across three research domains, synthetic persona-based focus groups recovered meaningful thematic signal from real conversations. The results suggest that synthetic personas can be a useful complementary research tool for exploring hypotheses, testing scenarios, and iterating study designs.
There is no universal best runner: performance varies by domain, metric, and cost. Simulated focus groups should not replace human qualitative research, but they can help researchers move faster and make better-informed decisions alongside it.
*A Synthetic Studio field note on the research behind synthetic personas. Synthetic persona responses are simulated, not collected from real people, and should always be validated against human data before informing a decision.
A preliminary comparison of real focus group themes against simulated personas across three runner variants (Baseline, Think Talk, and Think Talk Verify) using theme overlap and semantic distance.
A note on this series: This is the second entry in the Synthetic Studio Research Log, a multi-part blog series sharing details around our work simulating people with language models. The first entry, LLMs Are People Simulators, made the case that the ability to simulate people is a byproduct of how language models are trained, and that nobody is measuring whether it holds. This entry takes up the qualitative half of that measurement problem: whether a synthetic focus group can reproduce the themes that emerge from a real one. As with every entry, the findings are provisional by design. Models change, methods improve, and we will revise or retract as the evidence moves.
01
The preliminary answer is encouraging: simulated focus groups recovered a meaningful portion of the themes found in real conversations. Across three research domains—Work from home Sedentary Behaviour (WFH), Scholarly Git Experience (IASGE), and Software Engineering Creativity (SEC)—all three runners surfaced relevant thematic signal, providing preliminary evidence that synthetic persona generation can reproduce part of the thematic pattern observed in real focus groups.
The three runner variants show domain- and metric-dependent strengths. Think Talk Verify performs best on several recovery metrics in WFH and IASGE, Think Talk leads several semantic and recovery metrics across IASGE and SEC, and Baseline remains competitive on selected SEC measures. The expanded evidence supports a trade-off rather than a single overall winner.
02
Focus groups are expensive, time-consuming, and operationally difficult to run repeatedly. If simulated focus groups can approximate the thematic conclusions of real conversations, they could become a useful tool for early-stage research, scenario testing, and rapid iteration.
The early evidence is promising. Simulated sessions usually identify relevant themes, especially when they do detect a theme, and the higher precision scores suggest they tend to stay close to the real codebook. Recall remains the main area for improvement, which gives the next phase a clear target: broaden coverage while preserving fidelity.
These preliminary results justify investigating which level of simulation is most appropriate for different research objectives. Even imperfect simulations may support early-stage hypothesis generation, scenario testing, and research planning, but whether the observed alignment generalizes across topics remains an open question.
03
The experiment compares three simulation runners. All three use the same moderator question list extracted from the real transcript and the same downstream theme-scoring process. What changes is how simulated participant responses are generated.
Baseline Runner is the simplest implemented runner. Each round has an interviewer turn, then each participant responds once with a direct LiteLLM call. Participant prompts are wrapped with shared conversation rules that enforce persona grounding, concise answers, no AI self-reference, and no speaker labels.
The participant response is a single call: transcript context plus an instruction to answer the interviewer from the persona’s lived perspective. There is no private reasoning step and no verification pass.
Think-Talk Runner adds a two-step participant turn. First, THINK: a private call asks the persona to reason about emotional reaction, stance, personal fit, main contribution, character fit, and whether silence or a short answer is more realistic. Then TALK: a second call receives the transcript plus the private thinking and produces the spoken answer.
Only the TALK output is appended to the transcript. The private THINK output is stored in a tracing field for analysis and debugging, but it does not appear in the simulated conversation.
Think-Talk-Verify Runner extends Think Talk with a fidelity judge. Each participant turn does three steps: THINK, TALK, and VERIFY. The VERIFY step scores the candidate response from 0–100 against the persona profile, conversation stimulus, drafted response, and private reasoning.
If the normalized score is below 0.7, the runner regenerates the answer with a corrective hint. This 0.7 threshold is a provisional engineering default, not a validated fidelity cutoff: it is intended to catch clearly weak or generic responses while avoiding excessive regeneration on borderline-but-usable turns. It allows up to two regeneration attempts and keeps the highest-scoring candidate.
The final transcript receives only the selected final answer. The raw output records verifying calls, swapping count, a fidelity regeneration count, thinking trace, and a detailed verifying trace containing scores, reasons, original and corrected text, and attempts.
04
The scored results cover three domains. WFH Sedentary Behaviour has 30 evaluation samples per runner, based on six standardized focus group transcripts with five runs per transcript. Scholarly Git Experience (IASGE) has 15 evaluation samples per runner, based on three transcripts with five runs per transcript. Software Engineering Creativity (SEC) has 20 evaluation samples per runner, based on four transcripts with five runs per transcript. The WFH dataset therefore produces 90 scored simulation runs in total across the three runners.
The main objective of experimenting with multiple focus groups is to make our results domain-agnostic. IASGE focuses on the scholarly Git experience, while SEC focuses on perceived creativity in software engineering. Together with WFH, they test whether runner patterns generalize beyond a single research domain.
05
The pipeline starts with raw focus group transcripts. Moderator turns are segmented, interviewer questions are extracted, and each transcript is standardized so the same question flow can be replayed in simulation.
Grounded personas are then generated for the real participants, and each standardized transcript is simulated with the selected runner: Baseline, Think Talk, or Think Talk Verify. The simulations use the model OpenAI/GPT-5.2, with the same moderator questions and the same downstream scoring process across runners.
After each simulated session is generated, both real and simulated transcripts are scored against the shared codebook. The resulting theme detections and supporting utterances are used to compute Jaccard, precision, recall, F1, cosine similarity, and Jensen-Shannon divergence. Metrics are then logged for comparison across runners and runs.
06
The evaluation starts with a shared codebook: a structured list of themes researchers want to track across transcripts. The same codebook is applied to both the real focus group transcript and each simulated session, creating a common basis for comparison.
This theme extraction approach is based on a common focus group analysis strategy created by Braun and Clarke. Thematic analysis is a method for identifying, analyzing, and reporting patterns (themes) within data. It minimally organizes and describes your data in detail.
The study then computes six metrics across two dimensions. Set-overlap metrics measure whether the same themes appear in real and simulated outputs. Semantic and distributional metrics measure whether the content and salience of those themes are close to the real transcript.
The six metrics compare real and simulated transcripts at the theme level:
07
The results presented here are preliminary and cover three domains with unequal evaluation-set sizes: WFH Sedentary Behaviour (30 evaluation samples per runner), Scholarly Git Experience (IASGE; 15 per runner), and Software Engineering Creativity (SEC; 20 per runner). Rankings vary by domain and metric, so small differences should not be treated as meaningful without uncertainty analysis. The results support comparison across runners, not a universal winner.
WFH Sedentary Behavior Results
Scholarly Git Experience Results
Software Engineering Creativity Results
Jaccard scores should be interpreted in relation to the dataset and evaluation design. In the WFH Sedentary Behaviour dataset, scores ranged from 0.449 to 0.495. These values may appear modest, but perfect thematic overlap should not be assumed even when humans analyze the same qualitative material.
McFarlane, Ancker, and Kukafka (2008) examined independent researchers coding the same focus-group transcripts with a shared 70-code scheme. For two coders, their agreement statistic reduces to the Jaccard coefficient. Reported agreement varied with coder training and the unit of analysis, with illustrative values ranging from approximately 0.32 to 0.58 and higher values when analysis was restricted to mutually coded passages.
The comparison with McFarlane et al. is contextual rather than direct. Their study measured agreement over codes assigned to transcript lines or text blocks, whereas this study compares sets of themes detected across complete transcripts. A study-specific human-to-human benchmark using the same codebook and transcript-level procedure is still needed before making claims about human-equivalent performance.
The cost data helps frame the practical trade-off between runner complexity and performance. We extracted a subsample of runs containing recorded usage and pricing data and calculated the average cost per runner from that subsample. The figures should therefore be interpreted as indicative estimates rather than definitive operational costs across the entire experiment.
Think Talk Verify was strongest on several WFH recovery measures, but its 3× cost increase was much larger than its improvement in theme overlap. Think Talk may therefore be a provisional practical default when balancing performance and cost.
08
The most encouraging observation is not that one runner outperforms the others. It is that all three runners recover measurable thematic signal from real focus group conversations. McFarlane et al. provide useful context for interpreting moderate overlap, but they do not establish a universal threshold or demonstrate that synthetic sessions are equivalent to human research.
Across the three domains, runner performance is domain- and objective-dependent. IASGE: Think Talk Verify leads Jaccard, F1, and recall; Think Talk leads cosine similarity and precision, while also having the lowest JSD. SEC: Think Talk leads Jaccard, cosine similarity, and recall; Think Talk Verify has the highest precision; Baseline has the highest F1 and lowest JSD. These results do not establish that simulation can replace real qualitative research, but support synthetic persona generation as a complementary research tool.
Baseline Establishes the Reference Point
Baseline remains a useful reference point and is competitive on selected Software Engineering Creativity measures. Its relative strengths differ by domain, so it should not be characterized as the universal semantic leader.
Think Talk Captures Most of the Gains
Adding private reasoning delivers most of the measurable improvement, increasing Jaccard, F1, precision, and recall without introducing the full cost of verification. This makes Think Talk an attractive balance between quality and cost.
Think Talk Verify Has the Highest Observed Recovery
Think Talk Verify performs strongly on several recovery measures, especially in WFH and IASGE, but the broader comparison shows that verification is not uniformly best across domains or metrics. The current analysis does not establish statistical significance or causality.
Considerations on Score Differences between Focus Groups
The higher scores in IASGE and SEC should not be interpreted as a universal improvement in simulation quality. The domains use different codebooks and have different thematic structures: WFH includes a broader set of 77 codes, while IASGE and Software Engineering Creativity use 40 and 39 codes respectively. The WFH simulations therefore face a more diverse thematic target and show both lower recall and lower precision. These results suggest that synthetic persona performance depends not only on runner design, but also on the coherence and granularity of the research domain’s thematic framework.
09
These findings should be interpreted as preliminary. They cover three domains but lack uncertainty intervals and significance tests and should not be pooled into a single overall ranking. Small differences, such as IASGE JSD values of 0.144 versus 0.145, should not be treated as meaningful rankings without uncertainty analysis.
The study also lacks a directly comparable human-to-human baseline. Although McFarlane et al. reported Jaccard-equivalent agreement in human coding of focus-group transcripts, their analysis operated at the transcript-line and text-block level rather than the whole-transcript theme level used here. Their findings therefore provide interpretive context but should not be treated as an acceptability threshold or as evidence that the simulations have reached human-equivalent agreement.
10
Across three research domains, synthetic persona-based focus groups recovered meaningful thematic signal from real conversations. The results suggest that synthetic personas can be a useful complementary research tool for exploring hypotheses, testing scenarios, and iterating study designs.
There is no universal best runner: performance varies by domain, metric, and cost. Simulated focus groups should not replace human qualitative research, but they can help researchers move faster and make better-informed decisions alongside it.
*A Synthetic Studio field note on the research behind synthetic personas. Synthetic persona responses are simulated, not collected from real people, and should always be validated against human data before informing a decision.
