1. Overview

• Case study with educational data

2. Research Goal

• build detectors that can infer behaviors associated with self-regulation
• Grounded in SMART model of self-regulated learning (Winne & Hadwin, 1998)
  • searching
  • monitoring
  • assembling
  • rehearsing
  • translating

3. Why?

• embed in interventions that can make recommendations to students about strategies they are not currently using
• reports for teachers on strategies to emphasize to the class
• asset-based reports for teachers on what strategies their students are using

4. Context

• CueThink

  

  • CueThink’s Phases: designed using theory form

    

    • understand phase
      • what do you notice?
      • what do you wonder?
      • estimate your answer
    • plan phase
      • choose your strategies
      • planning journal
    • solve phase
      • video creation tools
    • review phase
      • check your math
      • check your recording
      • review your estimate
      • final answer

• current study

  • built automated detectors of SRL constructs from a theory-driven lens

5. Methods

• building detectors

  • construct operationalization

  

  

  

  • coding the data

    

  • feature distillation

    • 100 features

      

  • training models

    • used the ground truth data from 182 Thinklets and features distilled from the same set of data
    • trained models using XGBoost to predict the presence of the 5 SRL indicators
    • evaluated the model performance using 10-fold student-level cross-validation
    • computed the average AUC ROC

6. Results

• model performance

  

• feature importance

  • calculated SHAP value (SHAP?)

  • reported the most important 5 features in each model

    

  • most of the features are from the Understand and the Plan phase

    • Numerical Representation
      • use of numerical values
      • similarity measure
    • Contextual Representation
      • use of keywords
      • length of the responses
    • Outcome Orientation
      • use of keywords
      • use of numerical values
    • Data Transformation
      • strategy selection
      • length of the responses

  • Small differences in AUC ROC across gender and racial/ethnic groups

    

  • algorithmic bias

    • no student group consistently had the best-performing detectors
    • hard to discuss an example in detail without getting ahead of ourselves!

[Our Discussion]