Dialogue-based CALL: a multilevel meta-analysis
Dialogue-based CALL
Dialogue-based CALL systems involve
- a dialogue (i.e., sequence of conversational turns)
- with an automated agent (chatbot, robot, voice assistant, non player character…)
- as a language learning task (≠scaffolding).
See Bibauw, François & Desmet, 2015 and 2019 for a full discussion.
Methods
Formulas for effect size calculation, on a single “raw” metric (aligned to between-groups effects) across experimental designs, from Morris & DeShon (2002):
\[d\_{ \text{PP} } = J ( df\_{\text{PP}} ) \left( \frac { M\_{ \text{post,E} } - M\_{ \text{pre,E} } } { \mathit{SD}\_{ \text{pre,E}} } \right)\] \[d\_{\text{ECPP}} = J (df\_{\text{ECPP}}) \left(\frac{M\_{\text{post,E}}-M\_{\text{pre,E}}}{\mathit{SD}\_{\text{pre,E}}}-\frac{M\_{\text{post,C}}-M\_{\text{pre,C}}}{\mathit{SD}\_{\text{pre,C}}}\right)\]In previous Equations, we use Hedges’ $J$ as a correction function for small sample size bias (the original formula rather than the commonly used approximation) in order to obtain a more accurate estimate of the effect size (Hedges & Olkin, 1985):
\[J(df)=\frac{\Gamma{\left(df/2\right)}}{\sqrt{df/2}\ \Gamma{\left[\left(df-1\right)/2\right]}}\]where $df$ corresponds to the degrees of freedom, calculated from the subsample sizes ($n$) in each study as $df_{\text{PP}}=n_{\text{E}}-1$ and $df_{\text{ECPP}}=n_{\text{E}}-1 + n_{\text{C}}-1$.
Multilevel meta-analysis modeling
| Level | Number of clusters/items | Source of variance |
|---|---|---|
| Subjects | $k=96$ ($n=804$) | Random sampling variance |
| Effect sizes | $k=96$ | Variation within study |
| Studies | $k_{studies}=17$ | Variation between studies |
All computations were done in R, with the metafor package, using the rma.mv() function (Viechtbauer, 2010):
rma.mv(di, vi, data = dataset, random = ~1|Paper/Effect)
See Van den Noortgate, López-López, Marín-Martínez & Sánchez-Meca (2013) for a discussion of multilevel modeling in meta-analyses.
Results
Mean effect of use of dialogue-based CALL for L2 development: $d = .61$ (95% CI: $[.373, .831]$).
Moderator analyses
Significant moderators:
- L2 proficiency level
- Outcome variable type (production vs. comprehension) and dimension (accuracy, complexity, fluency…)
- …
Many interesting exploratory results needing to be confirmed in future research.
References
- Bibauw, S., François, T., & Desmet, P. (2015). Dialogue-based CALL: an overview of existing research. In F. Helm, L. Bradley, M. Guarda, & S. Thouësny (Eds.), Critical CALL – Proceedings of the 2015 EUROCALL Conference, Padova, Italy (pp. 57–64). Dublin: Research-publishing.net.
- Bibauw, S., François, T., & Desmet, P. (2019). Discussing with a computer to practice a foreign language: from a conceptual framework to a research agenda for dialogue-based CALL. Computer Assisted Language Learning.
- Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407–452). Oxford: Oxford University Press.
- Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105‑125. doi:10.1037//1082-989X.7.1.105
- Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. doi:10.3758/s13428-012-0261-6
- Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3). doi:10.18637/jss.v036.i03