Collecting data

Collecting data

3.1. Research designs in education

Choosing a study design

Overview of qualitative (yellow) and quantitative (red) research designs

When envisaging a research project in applied linguistics/second language acquisition, various study designs are possible. We have to answer the following questions:

  1. You want to
  2. If empirical, you want to
  3. If non-interventionist, do you intend to
    • observe the reality qualitatively, to understand or explain it:
      ⇒ Qualitative observational study
    • obtain a quantitative diagnostic of the reality, to measure it precisely:
      ⇒ Survey study (descriptive)
    • quantify the relationship between two aspects of the reality, by statistically measuring both:
      ⇒ Correlational study (inferential)
  4. If interventionist,
    • you are the teacher and want to try out some pedagogical intervention(s) in one class to address specific problems you have encountered:
      ⇒ Action-research
    • you want to test if a specific intervention (technique, tool, ressource…) has an impact in one group of students:
      ⇒ Pre-experimental (repeated measures/pre-post, single-group design)
    • you want to test if a specific intervention (technique, tool, ressource…) has more impact than another one (different technique or ‘business-as-usual’ control group) in at least two groups of students:
      ⇒ Experimental design (pre-post or posttest only, between-groups design)

Interventionist designs

Summary of interventionist research designs

  • Action-research
    • Teacher = researcher → subjective, reflexive process of a practitioner trying to implement change in their teaching practice
    • Predominantly qualitative 💬
    • Focus on the process: describing the process and reception of the intervention
  • Pre-experimental: single-group pretest-posttest design
    • Predominantly quantitative 📊
    • Focus on the effectiveness of the intervention on the outcome variable (pre/posttest)
  • Experimental design
    • Predominantly quantitative 📊
    • Focus on the effectiveness of the intervention on the outcome variable (pre/posttest)
    • Requirements:
      • 2+ groups/conditions: experimental vs. control
      • (pretest-)posttest measurement of the outcome variable
      • ideally: randomized group assignment
        • but often using existing groups in education (→ quasi-experimental design)

Non-interventionist designs

Quantitative designs 📊
  • Survey (descriptive, diagnostic: 1 variable)
  • Correlational (2 variables)
Qualitative designs 💬
  • Case study
  • Ethnography
  • Conversation analysis
  • Mixed methods

Loewen, S., & Philp, J. (2012). Instructed second language acquisition. In A. Mackey & S. M. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 53–73). Wiley-Blackwell.

Some model studies in SLA/analysis

Bibliographic studies: see examples of research syntheses and meta-analyses

Qualitative observational studies

  • Wesely, P. M., Vyn, R., & Neubauer, D. (2021). Teacher beliefs about instructional approaches: Interrogating the notion of teaching methods. Language Teaching Research, 136216882199218. https://doi.org/10.1177/1362168821992180
  • Timpe-Laughlin, V., Sydorenko, T., & Daurio, P. (2020). Using spoken dialogue technology for L2 speaking practice: What do teachers think? Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2020.1774904

Correlational studies

  • de Jong, N. H., Steinel, M. P., Florijn, A. F., Schoonen, R., & Hulstijn, J. H. (2012). Facets of speaking proficiency. Studies in Second Language Acquisition, 34(1), 5–34. https://doi.org/10.1017/S0272263111000489
  • De Wilde, V., Brysbaert, M., & Eyckmans, J. (2021). Formal versus informal L2 learning: How do individual differences and word-related variables influence French and English L2 vocabulary learning in Dutch-speaking children? Studies in Second Language Acquisition. https://doi.org/10.1017/S0272263121000097
  • Huensch, A., & Nagle, C. (2021). The effect of speaker proficiency on intelligibility, comprehensibility, and accentedness in L2 Spanish : A conceptual replication and extension of Munro and Derwing (1995a). Language Learning, 71(3), 626-668. https://doi.org/10.1111/lang.12451
  • Jones, L. C., Murphy, C. A., & Holland, A. (2015). The more things change, the more they stay the same, or do they? Revisiting classroom interaction approaches and their effects on quantity and characteristics of language production. CALICO Journal, 32(2), 245–272. https://doi.org/10.1558/cj.v32i2.24541

Experimental studies

  • Graham, S., Courtney, L., Marinis, T., & Tonkyn, A. (2017). Early language learning: The impact of teaching and teacher factors. Language Learning, 67(4), 922–958. https://doi.org/10.1111/lang.12251
  • Kasprowicz, R. E., Marsden, E., & Sephton, N. (2019). Investigating distribution of practice effects for the learning of foreign language verb morphology in the young learner classroom. The Modern Language Journal, 103(3), 580–606. https://doi.org/10.1111/modl.12586
  • Loewen, S., Crowther, D., Isbell, D. R., Kim, K. M., Maloney, J., Miller, Z. F., & Rawal, H. (2019). Mobile-assisted language learning: A Duolingo case study. ReCALL, 31(3), 293–311. https://doi.org/10.1017/S0958344019000065
  • Loewen, S., Isbell, D. R., & Sporn, Z. (2020). The effectiveness of app-based language instruction for developing receptive linguistic knowledge and oral communicative ability. Foreign Language Annals, 53(2), 209–233. https://doi.org/10.1111/flan.12454
  • Michel, M., Révész, A., Lu, X., Kourtali, N.-E., Lee, M., & Borges, L. (2020). Investigating L2 writing processes across independent and integrated tasks: A mixed-methods study. Second Language Research, 36(3), 307-334. https://doi.org/10.1177/0267658320915501
  • Puimège, E., & Peters, E. (2020). Learning formulaic sequences through viewing L2 television and factors that affect learning. Studies in Second Language Acquisition, 42(3), 525–549. https://doi.org/10.1017/S027226311900055X
  • Satar, M., & Özdener, N. (2008). The effects of synchronous CMC on speaking proficiency and anxiety: Text versus voice chat. The Modern Language Journal, 92(4), 595–613. https://doi.org/10.1111/j.1540-4781.2008.00789.x
  • Sauro, S., & Smith, B. (2010). Investigating L2 performance in text chat. Applied Linguistics, 31(4), 554–577. https://doi.org/10.1093/applin/amq007
  • Shintani, N., Ellis, R., & Suzuki, W. (2014). Effects of written feedback and revision on learners’ accuracy in using two English grammatical structures. Language Learning, 64(1), 103–131. https://doi.org/10.1111/lang.12029
  • Ziegler, N., Meurers, D., Rebuschat, P., Ruiz, S., Moreno-Vega, J. L., Chinkina, M., Li, W., & Grey, S. (2017). Interdisciplinary research at the intersection of CALL, NLP, and SLA: Methodological implications from an input enhancement project. Language Learning, 67(S1), 209–231. https://doi.org/10.1111/lang.12227
  • Zalbidea, J. (2021). On the scope of output in SLA: Task modality, salience, L2 grammar noticing, and development. Studies in Second Language Acquisition, 43(1), 50–82. https://doi.org/10.1017/S0272263120000261

3.2. Qualitative techniques

Friedman, D. A. (2012). How to collect and analyze qualitative data. In A. Mackey & S. M. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 180–200). Wiley-Blackwell.

Interview

Types

  • Semi-structured
  • Structured
  • Unstructured

Tools and processes

  • Interview guide/Set of questions
    • Designing interview questions
      • Open-ended questions vs. closed ended
      • Avoiding leading questions
      • Avoiding complex questions
      • Verify comprehensibility
  • Recording
    • Audio is almost always enough
  • Note-taking

Resources & references

Focus group and discussion group

≃ Collective interview

Observation

  • Observation principles
    • Participant observation
  • Recordings
  • Field notes
  • Checklists
  • Rubrics

Documentary research

  • Teaching & learning materials
    • Textbooks
    • Handouts
  • Teacher management materials
    • Assessment records and notes
    • Lesson plans
  • Official and administrative materials
    • Regulation, legal or formal instructions from multiple management levels

3.3. Quantitative techniques

3.4. Data collection processes

Piloting and validating instruments

Processes, workflows and timelines

Ethics