Dialogue-based CALL

Dialogue-based CALL systems involve

a dialogue (i.e., sequence of conversational turns)
with an automated agent (chatbot, robot, voice assistant, non player character…)
as a language learning task (≠scaffolding).

See Bibauw, François & Desmet, 2015 and 2019 for a full discussion.

Methods

Formulas for effect size calculation, on a single “raw” metric (aligned to between-groups effects) across experimental designs, from Morris & DeShon (2002):

\[d\_{ \text{PP} } = J ( df\_{\text{PP}} ) \left( \frac { M\_{ \text{post,E} } - M\_{ \text{pre,E} } } { \mathit{SD}\_{ \text{pre,E}} } \right)\] \[d\_{\text{ECPP}} = J (df\_{\text{ECPP}}) \left(\frac{M\_{\text{post,E}}-M\_{\text{pre,E}}}{\mathit{SD}\_{\text{pre,E}}}-\frac{M\_{\text{post,C}}-M\_{\text{pre,C}}}{\mathit{SD}\_{\text{pre,C}}}\right)\]

In previous Equations, we use Hedges’ $J$ as a correction function for small sample size bias (the original formula rather than the commonly used approximation) in order to obtain a more accurate estimate of the effect size (Hedges & Olkin, 1985):

\[J(df)=\frac{\Gamma{\left(df/2\right)}}{\sqrt{df/2}\ \Gamma{\left[\left(df-1\right)/2\right]}}\]

where $df$ corresponds to the degrees of freedom, calculated from the subsample sizes ($n$) in each study as $df_{\text{PP}}=n_{\text{E}}-1$ and $df_{\text{ECPP}}=n_{\text{E}}-1 + n_{\text{C}}-1$.

Multilevel meta-analysis modeling

Level	Number of clusters/items	Source of variance
Subjects	$k=96$ ($n=804$)	Random sampling variance
Effect sizes	$k=96$	Variation within study
Studies	$k_{studies}=17$	Variation between studies

All computations were done in R, with the metafor package, using the rma.mv() function (Viechtbauer, 2010):

rma.mv(di, vi, data = dataset, random = ~1|Paper/Effect)

See Van den Noortgate, López-López, Marín-Martínez & Sánchez-Meca (2013) for a discussion of multilevel modeling in meta-analyses.

Results

Mean effect of use of dialogue-based CALL for L2 development: $d = .61$ (95% CI: $[.373, .831]$).

Moderator analyses

Significant moderators:

L2 proficiency level
Outcome variable type (production vs. comprehension) and dimension (accuracy, complexity, fluency…)
…

Many interesting exploratory results needing to be confirmed in future research.

References

Bibauw, S., François, T., & Desmet, P. (2015). Dialogue-based CALL: an overview of existing research. In F. Helm, L. Bradley, M. Guarda, & S. Thouësny (Eds.), Critical CALL – Proceedings of the 2015 EUROCALL Conference, Padova, Italy (pp. 57–64). Dublin: Research-publishing.net.
Bibauw, S., François, T., & Desmet, P. (2019). Discussing with a computer to practice a foreign language: from a conceptual framework to a research agenda for dialogue-based CALL. Computer Assisted Language Learning.
Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407–452). Oxford: Oxford University Press.
Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105‑125. doi:10.1037//1082-989X.7.1.105
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. doi:10.3758/s13428-012-0261-6
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3). doi:10.18637/jss.v036.i03