longer.blue

First of all, I totally agree that it is difficult to reconcile the low take-up and the big effects, which bothered our team for a long time. For people who are interested in the speculation about "a preferred story", I want to direct you to this Inside Higher Ed article that discusses the origin of the study based on interviews with me and my coauthor and Provost Jennifer King Rice: https://www.insidehighered.com/news/tech-innovation/artificial-intelligence/2026/10/07/maryland-ai-research-raised-more-questions. There is another article from UMD that provides more context: https://today.umd.edu/study-suggests-more-guidance-needed-in-use-of-ai-course-assistants Below are my responses to main points made by Brendan. About mechanism speculation: Observing student use of other AI tools would be the strongest evidence if we have it, but saying this is purely speculation is also not right. We presented survey evidence that suggests 41% of treatment students (respondents) said their instructors encouraged generative AI, versus 11% of control students. The response rate is low, but that difference is too huge to be fully dismissed. The blogpost didn't engage with this evidence. About conversion of the Canvas scores and letter grades: There is no linear conversion of Canvas scores to letter grade. In our learning management system, Canvas scores are the "raw" scores at the end of a course, which combine different components and are often a weighted average. Instructors then might grade on a curve or do manual adjustments, say, once a student submits a make-up assignment. So direct comparisons are difficult for these two measures without observing each instructor's syllabi or directly documenting the grading practice. As 0-4 point course grades are commonly used in the higher education literature and there is no linear conversion, we chose to include it to be transparent. Both measures are not perfect measures of learning as there are no standardized assessments in higher education. Brendan then argued that our paper highlights the effects on the Canvas scores too much. However, in the exact match sample, the effects are also negative and significant for the letter grade outcome (0.11 points, p<0.01), so we didn't just cherry pick the Canvas scores to report. About the significant effects on engagement being driven by chance: Brendan wrote that the negative impact on engagement is driven by chance. Our footnote 5 shows that we recomputed p-values by re-randomizing treatment within blocks and the results hold. As mentioned above, in the exact-match sample, the letter-grade effect is also significant and negative. If one believes the letter grade is what matters the most, the results do hold up despite a smaller magnitude. About drops in participation driven by instructor assigned opportunities: He is totally right the Canvas measures are not great. One of our coauthors is actually a university staff member who oversees the entire Canvas system, so we already had lots of discussions internally when looking at the findings. Yes we do see some significant differences between treatment and control groups on number of assignments for the exact-match sample (8.8, p=0.07), for example. However, assignment imbalance is much smaller in the full sample (4.5, p = 0.13), yet participation still falls by about 0.89 SD there—very consistent with the coefficient from the exact-match sample. Thus, looking across all the results, the decline in engagement is quite consistent and is unlikely be explained away by differences in opportunities. About highlighting results from the exact-match sample or full sample: We went back and forth on this during the development of the manuscript. I agree we could have been more careful and consistent. The exact-match sample compares sections within the same course, with the same content, materials, assessments and grading scales. It is preferable in many ways, but sticking with it also cuts the sample size quite a bit. The abstract and the intro are transparent and we are not hiding anything. I agree that some people might not move beyond the intro when reading a paper like this. I agree we should add the 15% usage data point to the abstract, which we did when submitting the paper to a journal (before seeing this blogpost). About the first-generation results: Brendan argued this finding could be due to chance, given the number of interactions in Table 4. That concern is reasonable for exploratory subgroup analyses, but our pre-registered analysis plan specifically named first-generation students as a subgroup of interest, because of longstanding concerns that new technologies may widen gaps for students with less familiarity with college. The pattern is also consistent in direction: first-generation students had larger estimated declines in final grades, homework scores, and page views, in both the full and exact-match samples. That said, the letter-grade difference is not statistically significant. We view this result as an important signal worth testing in future studies rather than a definitive finding, and we will make that framing clearer in the abstract. As with any working paper, there are many things we can and should have done better. For example, saying "tutoring access" rather than "tutoring"; or use a different term if people perceive tutoring in a specific way. Doing real-world research on AI's impact on learning, especially in the postsecondary setting, is increasingly hard because of the ubiquity of the technology. I am glad this research is helping to take a small step in building a stronger evidence base.