All the ways ED’s latest admissions data collection went wrong

Last December, the U.S. Department of Education (ED) added the Admissions and Consumer Transparency Supplement (ACTS) to the longstanding Integrated Postsecondary Education Data System (IPEDS). As ED describes, ACTS is designed to understand if colleges are, “using race-based preferencing in their admissions processes.” From our view, while these data can offer insights into college admissions inequities, they cannot prove whether race was a determining factor in an admissions decision. But putting ED’s stated purpose for ACTS aside, submitting data has become a complicated and monumental task for institutions.

Unlike prior IPEDS updates, the ACTS collection is massive, with numerous variables and years of retrospective data. ED’s own burden estimates state institutions will spend 200 hours on ACTS reporting, 2.5 times longer than the time required for the other 12 IPEDS surveys combined. ED also implemented the ACTS data collection on an uncharacteristically compressed timeline. This meant ED could not meaningfully engage the field before finalizing the collection, resulting in unclear data definitions and guidance as well as confusion among campus-based data professionals.

In October and December, the IHEP-led Postsecondary Data Collaborative (PostsecData) warned ED about the ramifications of ACTS’ hasty implementation. Unfortunately, our concerns have come to fruition. Institutions have faced significant hurdles in attempting to submit their ACTS data.

A myriad of data reporting challenges

Since the data collection opened in December, ED changed the ACTS reporting templates for institutions multiple times, which is unprecedented for an IPEDS survey. For example, income and sex reporting formats were revised midway into the collection. These changes meant institutions that were already compiling data had to redo work, further adding to the time burden of the collection.

Throughout the data reporting period, institutions have received insufficient guidance from ED, leaving many struggling to understand definitions for requested information. In a recent survey, 74 percent of institution data professionals said interpreting ACTS definitions and guidance was a moderate or major challenge. Take, for instance, the requirement to report applicants’ unweighted high school GPAs. Institutions have been told to develop their own methodology for how to unweight these numbers. And it’s not always clear whether applicants’ high school GPAs are weighted or unweighted, adding another layer of complexity and ambiguity. ED’s guidance has also evolved or been unclear about reporting test scores if students provide multiple, avoiding double-counting students who defer enrollment, and handling students who enroll but leave before receiving a GPA. An institution data professional told the Association for Institutional Research, “definitions and guidance coming directly from the IPEDS Help Desk keep changing.” When institutions don’t receive clear and consistent definitions, data are reported differently and data quality suffers.   

Yet another set of challenges stem from processing times and technical issues with a new data aggregator tool. For the first time in IPEDS history, institutions must submit student-level data files—uploaded to an online tool or processed through a Python script provided by ED—instead of submitting aggregated data tables. The aggregator tool and Python script were developed to streamline data reporting by automatically generating thousands of required tables. But this new submission process has created challenges because it was not tested before rollout. The Python script was not available until early February, halfway through the data collection period. And the Python script file that eventually became available has presented technical challenges for institutions.

Meanwhile, processing data via the aggregator tool can be a lengthy task. One year of data can take up to several days to process, and institutions must submit seven years of data. Even after generating tables with the aggregator tool or Python script, institutions’ files must be reviewed by ED’s IPEDS contractor, a process that has taken multiple weeks for some. If errors are detected during the contractor’s review, institutions may need to restart the entire submission process.

An exceedingly crunched implementation timeline 

Changes to IPEDS are usually announced one year before the data collection period opens. Preview screens with the exact wording of requested data elements are also usually posted one year in advance. With enough time, institutions can surface questions about definitions and prepare the data, which may be stored in different campus systems and offices. For example, graduate student admissions data necessary for the ACTS submission are stored in the academic department for each graduate program, not a central admissions office. Additionally, the typical year of preparation time allows for testing the submission process and developing necessary quality checks at each stage.

For ACTS, the guidance necessary for institutions to understand required data elements was not released until November 13, the beginning of the 30-day comment period. That comment period closed on December 15, and the ACTS data collection opened December 18. This accelerated rollout left almost no time for ED to revise the collection based on public comments, and gave institutions several weeks, not a full year, to prepare.

A rushed process means questionable data quality

Hundreds of public comments, including PostsecData’s, warned about the risks of the compressed ACTS implementation timeline. Yet ED pushed ahead with a March 18 deadline for institutions to submit all seven years of ACTS data unless they request and are awarded a limited three-week extension.

The ACTS data collection requires institutions to submit an unprecedented amount of data using a new process with very little preparation time and unclear guidance. The result? Data of questionable quality for an immensely time-consuming task. That’s time data professionals could have used to support student success.

On March 11, attorneys general from 17 states sued ED over the ACTS collection alleging it’s an unlawful expansion of IPEDS and citing the “onerous burden” placed on institutions. A few days later, a federal judge issued a temporary restraining order on the collection, extending the deadline through March 25 to allow time for an, “orderly resolution of the issues.” Now the courts will hear from the states and ED about the matter.

Let the ACTS data collection be a lesson for the future. Clear data definitions, adequate preparation time, and well-tested data submission protocols are not optional.