Analyzing Small Sample Sizes After Disaggregation: Exploring Observation Oriented Modeling for Assessing Learning Outcomes

J. Vincent Nix, Yi-Chin Wu, Lan Misty Song, & Joseph D. Levy   |    Volume 19 Issue 1  |   

Traditionally, assessment professionals use analyses relying upon null hypothesis significance testing (NHST), but those tools have limitations when analyzing small samples or disaggregated data. This study used common NHST analytical techniques, compared their results, and then explored an alternative technique that perhaps allows for a more precise understanding of learning outcomes; to that end, it juxtaposed results from NHST techniques to Observation Oriented Modeling (OOM) results using data from an eight-module, massive online open course (MOOC). The OOM analysis effectively analyzed small sample sizes after disaggregation, revealing relationships between learners’ locations and the disproportionate outcomes among two groups: non-native English-speakers and persons identifying as non-binary. These findings show how meaningful disaggregation and interrogation of collected data can help instructors improve pedagogy and materials, as well as serve the field at large by suggesting alternative analytical tools to glean meaningful results from difficult-to-analyze data.

Click here to download the full article

« Back to Archive