Please visit our new home and follow us on social media: Facebook & Twitter
Sign up as a Citizen Journalist and get involved in Information Activism.
Sign Up for Watchdog Updates!
A $74,000 commissioned report by Augenblick Palaich and Associates (APA), detailing the costs and time of statewide school assessments is coming under scrutiny for data analysis, key omissions, and potential conflicts of interest.
The APA Assessment Study Report analyzing the cost and time of Colorado assessments, was formally presented to Colorado’s HB14-1202 Standards and Assessments Task Force on Nov. 17, but critics charge it omitted outlying data, failed to account for necessary capital expenses, and is unclear in its calculation of student- and district-level averages.
Report Omits Computer, Infrastructure Costs
There are 179 school districts in Colorado and 1,825 schools. While APA conducted a broad online survey to assess test preparation time and test usefulness, it chose to interview just five districts on testing costs, claiming “the responses were representative of statewide opinion.”
Additionally, APA’s HB1202 report does not include costs incurred by schools for computers, infrastructure, and bandwidth necessary to take the state-mandated online PARCC and CMAS tests. APA was asked to include these costs to schools and districts. Indeed, capital expenses are explicitly included in the Task Force Definitions, in Section 3(c):
3. Cost Definitions
c. Capacity Costs
i. Cost of purchasing any additional technology to administer assessments, if not already owned by the district or school.
ii. Cost of ensuring sufficient bandwidth to allow for online assessments.
iii. Cost of having staff needed to manage assessments and assessment data.
Nevertheless, APA decided not to include capital expenses, claiming that such information would be too difficult to obtain, and that technology acquisitions could be used for instruction as well. However, while the districts’ financial reporting would not be required to break the costs down by usage, it is likely that internal managerial, or cost accounting, would do so. Such cost accounting would be considered public records, available by CORA request.
Technology costs associated with online testing are steep. This Pioneer Institute report shows average testing costs $1.24 billion pale in comparison to technology costs $6.27 billion, nationally. Many Colorado districts have already spent millions just to meet the technological demands, and although the HB1202 APA survey did collect “some information” on technology costs to schools, again, they refused to show it. Task force members have repeatedly asked to see the quantitative data collected by the APA survey both on reported testing time and cost.
Different Reports, Different Costs
APA’s private Draft report records significantly different numbers from its public report. The “Private Draft” reports testing costs for state, federal, and local tests to range from $55 million to $130 million while the study that the public sees reports the weighted average cost of testing as $61 million, and doesn’t explain that the range was double that.
Using a weighted average allows APA to report costs on a statewide per-student basis. Since overhead costs tend to be higher on a per-student basis for smaller districts than for larger ones, a statewide figure understates the burden on smaller districts. Task Force members have expressed concern that such a statewide average is inappropriate, since districts, not the state, will be picking up the district-level compliance costs. Indeed, the Documented Quotation presented to bidders repeatedly discusses costs at the district and charter school level, while referring to statewide costs only in regards to the Colorado Department of Education (CDE) itself.
To further show how the choice of averaging method can change results, using APA’s numbers, Chalkbeat reported an average testing cost of $78 million.
Although members of the task force have asked to see the raw cost data collected, APA has refused to produce it, citing confidentiality promises made to the districts. In an email to the Task Force members, APA stated that:
For the five districts that we spoke with, we explicitly stated to participants (and shared in writing) that all cost collection information would not be specifically attributed to their district but instead be used to calculate per pupil cost estimates. As such, we see no way that we could share this individual level information without violating this agreement which we are unwilling to do. Even removing district names, the size of the district could easily be inferred by the resources identified and then that would essentially identify the specific district. As such, we will not be sharing the raw cost collection data.
Task Force members have questioned APA about the validity and propriety of those promises, inasmuch as data that is public record – such as costs and expenditures – is excluded from confidentiality in the contract. Task Force members have also asked APA who at CDE permitted both the confidentiality agreements with the districts, and the statewide per student weighted averaging of costs. As of yet, APA has not replied to those inquiries.
Testing Time Data Also Mysterious
APA’s handling of data related to testing time is also open to question. It admits to eliminating outliers from its analysis, and has similarly refused to release the raw data collected unless permitted by the individual districts, citing privacy concerns. Because the survey was voluntary, the smaller districts with few schools might only have a few respondents, making them easy to identify even if specific, record-level identifying data were removed. So far, a number of the larger districts have permitted their data to be released, while smaller districts have demurred.
When presenting the data, APA said they got a wide range of reported test prep time and rather than skew their data, or follow up /investigate further, they assumed those higher time points were typos and chose to not include them in the report. Without raw data, it is impossible for outside observers, of even the 1202 Committee itself, to evaluate that judgment.
Despite that, attendees at the Nov. 17 presentation confirm that some members of the Committee supported APA’s decision not to include outliers.
Concerns have been raised about APA’s mixing test prep time of TCAP and the PARCC field test. The shorter PARCC field test was taken by a small portion of Colorado schools. In reality, the longer PARCC test will be given twice per year, requiring more seat time and test prep time, which the APA study did not include.
APA Sole Applicant
Despite the competition placed by the CDE for study, APA’s was the sole proposal received. While 109 other bidders expressed interest, some demurred, commenting that the $74,000 budget was too small for a study of proper scope.
In addition, according to this CDE document, the task force itself expressed many concerns on APA’s proposal, including conflict of interests stemming from APA’s previous work with the Bill Gates-funded Colorado Education Initiative (CEI). CEI paid APA to do a similar assesment study just two years ago. The task force worried this prior work with CEI “could slant the focus and, consequently, the results of the HB1202 study”. They also cited APA’s tendency to not use quantitative data, resulting in reports based mostly on “perceptions and opinions, rather than actual school and district budgets and expenditures.”
Those familiar with the task force say they were encouraged to go ahead with the APA bid because restarting the bidding process would most likely result in the task force not completing its duties. The task force was assured that modifications to the bid could be made.
Tags: 1202 Commission, Cost of Statewide Testing, Education Assessment, Education Testing, PARCC, Student Testing
- Gaps in Colorado report on school assessment data under scrutiny
- Colorado taxpayers finance Gruber-Gate
- Colorado Gruber-Gate Update: Open Records Request Pending
- Colorado may cut public pension benefits
- Stimulus award to The Denver Post’s $50K bike tour goes unreported