Assessing evidence of academic recovery: Slight progress in math, hardly any in ELA

Are U.S. students recovering from the enormous schooling interruptions of the COVID-19 pandemic? The evidence based on test scores has been frustratingly mixed. While a highly publicized study conducted by researchers at Stanford and Harvard documented a “surprising” rebound from pandemic closures between 2022 and 2023, interim assessment providers such as NWEA and Curriculum Associates indicated little recovery during that period. Comparisons of proficiency rates on state summative assessments indicate a mixed bag in spring 2023, with some states fully recovered¹ while others have made little or no progress.

How do we reconcile these different stories? It requires a close look across these studies. The basic challenge is that the studies differ from one another in several ways: the samples and timepoints included, the metrics reported, and the ways of quantifying recovery.

To address this challenge, we apply the same measure of recovery across four multistate data sources as well as summative assessments from most U.S. states. Given that scores from many spring 2024 state assessments have not yet been released, we primarily focus on comparing recovery trends between 2022 and 2023 and then briefly summarize the available data from spring 2024. Our aim with this brief is not to declare a definitive estimate of recovery or identify the most accurate study. Rather, it is to zoom out and summarize the broader landscape of available evidence about COVID recovery across available data sources. 

Overall, we find that most data sources indicate small but promising signs of recovery in math and no widespread evidence of recovery in English/language arts (ELA)—though a few states do buck this trend. We note some reasons why data from existing assessments come to different conclusions, along with some reasons why we might expect recovery to be stalled in some places.

What do various assessments show regarding academic recovery?

We have two main forms of data available to answer this question: (a) scale score differences from interim assessments (NWEA’s MAP Growth, Curriculum Associates’ i-Ready, and Renaissance’s STAR) and the Education Recovery Scorecard (ERS); and (b) grade-level proficiency rates from state summative assessments.

Our first analysis looks at the interim assessments. To deal with differences across assessments, we collected the means and standard deviations (SDs) from the public reports for each data source for springs 2019, 2022, 2023, and, where available, 2024.² We applied a consistent standardization approach to each data source (subtracting off the 2019 mean and dividing by the spring 2019 SD). We then pooled data across grades 3-8 to give a single standardized estimate per subject and data source.³

Figure 1 displays the recovery trends across the various interim assessments between 2022 and 2024. All data sources indicate that some recovery occurred in math between 2022 and 2023, though the assessments differ both in the magnitude of the initial drop and the size of the improvement. In ELA, results are more mixed, with NWEA and Curriculum Associates data pointing to continued declines, while Renaissance data showed no change and ERS showing small improvements.

Additionally, we quantified recovery as the proportion improvement from spring 2022 and 2023 relative to the initial drop. For example, Figure 1 shows that ELA scores on the ERS scale dropped 0.10 SDs from 2019 and 2022 and then increased by 0.02 SD between spring 2022 and 2023, and so the recovery rate is 20%. Consistent with Figure 1, Table 1 shows that there is a pattern of small but consistent progress towards recovery in math and mixed results in ELA. The ERS shows the most optimistic recovery patterns, with test score drops that have recovered by spring 2023 by 30% in math and 21% in ELA. In math, the three interim assessments pointed to a smaller but still positive recovery trend (between five percent to 18% of the drop recovered).

Our second analysis looks at trends in the percentage of students classified as proficient (or better) on their state summative assessment.⁴ States use different assessments and different definitions of proficiency, which makes direct empirical comparisons across states difficult.⁵ Therefore, we simply compare high-level trends in whether percent proficiency went up or down between 2022 and 2023, excluding states known to have changed their test or their proficiency thresholds over this period.

Specifically, we categorized each state into one of four bins:

• Recovered or Exceeding: Spring 2023 proficiency rates are at or above spring 2019 rates.
• Making Progress: Spring 2023 proficiency rates are at least two percentage points higher than spring 2022 but not yet reaching spring 2019 rates.
• Minimal Change: Spring 2023 proficiency rates are within +/- two percentage points of spring 2022 and still below spring 2019 rates.
• Continued Decline: Spring 2023 proficiency rates are at least two percentage points lower than spring 2022 and still below spring 2019 rates.

Figure 2 summarizes the trends across all states with available data. With the caveat that state proficiency rates offer a blunt metric, we see some parallels with the interim assessment results. In math, 45% of states were making progress toward or achieved recovery in 2023, compared to just 31% of states in ELA. No states demonstrate continued declines in math, whereas seven percent of states show ongoing declines in ELA. Thus, consistent with trends from interim assessments, state proficiency rates suggest recovery in math has been more robust than recovery in ELA (though more states are Recovered or Exceeding in ELA than math).

Why do trends seem to differ by data source?

Figure 1 reveals some fairly large disparities in recovery patterns across datasets, particularly for ELA. There are a few key differences across the data sources that may contribute to the different patterns.

• First, each data source has its own sample inclusion rules, resulting in different sets of schools participating across studies. The ERS used data from almost all districts in 29 states (28 states in math; 26 states in ELA), while the interim assessment data sets contained most/all states, but with varying levels of representativeness within a state.
• Second, the type of assessment may matter. ERS used test score data from summative assessments, which are used in state accountability systems and therefore can have higher stakes attached compared to interim assessments that are typically used for progress monitoring. Additionally, summative assessments typically prioritize measuring grade-level skills, while interim assessments measure a broader range of achievement.
• Finally, state summative assessments more regularly go through changes in test format, content assessed, and proficiency thresholds, which can result in non-comparable test scores across years. The ERS data only included states that reported that their assessments had not changed between 2019 and 2023, but there may still have been unreported changes in the assessments or the scoring that could lead to inaccurate reported trends in the ERS results.

To try to reconcile the different 2022 to 2023 recovery stories between ERS and NWEA specifically, researchers at NWEA, Stanford, and Harvard replicated the ERS sample and methodology within the NWEA sample. In math, it appeared that reported differences were primarily due to sample inclusion rules.  When limited to a common set of districts across the two samples, the NWEA and ERS estimates of recovery in math are quite similar (about 0.03 SDs). However, in reading, it appears that two states (Illinois and Ohio) were in large part driving the ERS recovery estimates. When the data from these two states were excluded, both ERS and NWEA point to essentially zero recovery in ELA between 2022 and 2023.

Why might math be recovering more quickly than ELA?

Across data sources, we generally observe that math is showing a clearer rebound pattern than ELA. The reason why is less clear. One theory is that standardized tests may be better at measuring discrete skills that students are taught in math compared to ELA. It is also possible that schools focused recovery efforts more heavily on math skills coming out of the pandemic, since math is where the largest initial drops were reported. Additionally, progress in ELA may be more dependent on local and state decisions around the adoption of research-aligned reading practices.

Why do we see inconsistent patterns of recovery?

As noted above, one reason we are seeing differences in recovery patterns across data sources is that those data sources rely on different samples, and all the evidence suggests recovery patterns have been uneven across different communities in the US. For example, we know that districts that stayed remote/hybrid for longer periods in 2020-21 (districts that also serve higher rates of low-income students and students of color) experienced the greatest losses during the pandemic. Students in these districts have the largest lift in order to fully catch up. While there is some promising evidence of above-average recovery rates for Black students, it appears that Hispanic students and economically disadvantaged students are recovering at lower rates than white and non-economically disadvantaged students. There are also clear differences in recovery patterns across states, which likely reflect not just variation in demographics but also the unequal impact of the pandemic on different student groups, as well as differences in state policies and curricular choices.

Funding to support recovery programs also appears to be driving some of the variation in recovery. Recent research indicates that the federal government’s major investments in pandemic aid helped to support recovery during the 2022-23 school year, but additional resources would be needed to close the large gaps that formed during the pandemic. Districts that heavily invested in learning opportunities that go beyond a normal school day (such as high-dosage tutoring, summer programs, and/or additional instruction during school breaks) have reported further progress toward recovery, though scaling up these types of programs remains extremely challenging and costly.

What do we know about recovery between spring 2023 and 2024?

Two interim assessment providers, Curriculum Associates and NWEA, have released data on test score trends for the 2023-24 school year. Both organizations described inconsistent recovery patterns between 2023 and 2024, though they differed in which age groups are the furthest behind. Curriculum Associates found that the youngest students (current 2nd graders) are the furthest behind, while NWEA data indicated that 8th graders remain the furthest from recovery. Both reports did call attention to students from schools in lower-income or racially minoritized communities who remain the furthest from recovery.

Additionally, a handful of states have now released spring 2024 proficiency trends: Texas showed mixed progress in ELA and declines in math; Tennessee reached recovery in both subjects; and Indiana showed minimal change for another year. In total, recovery still appears to be fairly local and not uniform across the board.

Conclusions

As the millions of dollars of ESSER funds run out in fall 2024, it is important to take stock of where we are in terms of recovery from the largest interruption to the U.S. education system in generations. Our review of the available test score data indicates that although there may be bright spots, we still have a long road to full recovery. Progress is far from consistent, with some states fully recovered by spring 2023 while others continue to fall further behind. While math shows slow-but-promising progress to recovery, there’s little evidence of much recovery in ELA. Each individual data source has limitations, but the totality of evidence points towards the need for intensive supports to get students back on track in ELA.

In this critical juncture, we must weigh the risks of various policy responses carefully. Underreacting and prematurely declaring recovery could lead to the withdrawal of essential supports, leaving countless students without the help they need to succeed. On the other hand, overreacting by continuing intensive supports might divert resources that could be used for other important needs, such as new technology or additional school counselors. However, it’s hard to see a downside to striving for achievement levels that exceed pre-pandemic standards, especially given the educational inequalities that predate this crisis. Ensuring that every student has the resources they need to thrive and reach their full potential should be our priority. The stakes are too high to risk falling short.