Massive interruptions during the online portion of the ISTEP exam last spring had no noticeable effect on statewide scores, though it’s impossible to know what impact the interruptions had on individual students.
Those are the findings of a company that spent the past month conducting an independent review of this year’s ISTEP results.
More than 495,000 Indiana students, including about 12,000 students in Johnson County, took the Indiana Statewide Testing for Educational Progress-Plus exam this year. This year, for the first time, about 95 percent of those students took the multiple-choice portion of the test online. And more than 142,000 students statewide and more than 1,700 students locally experienced delays or interruptions, including frozen computer screens and being knocked offline.
Data from the company that conducted the ISTEP shows that about 14 percent of Johnson County test-takers experienced interruptions. Schools in many cases reported additional students as having delays.
What is ISTEP: An exam used to assess how well Indiana students understand lessons in language arts and math classes.
Who takes the exam: All third- through eighth-graders
How schools use the results: Schools use the results to see how well students understand what they’ve been taught. Students who don’t pass the exam often receive extra lessons in language arts and math.
How the state uses the results: The state bases the grades schools receive on the number of students who take and pass ISTEP. The results also are used as part of teacher evaluations, which impacts their pay.
Here are the details on the ISTEP review that was conducted:
Who conducted the review
Richard Hill, co-founder of the National Center for the Improvement of Educational
What he did
Analyzed this year’s ISTEP results, comparing them with previous scores and projections to see if disruptions during the online test affected the results
What he found
No measurable difference was found on the overall statewide scores. But it’s impossible to know the impact of the disruptions on individual students.
The Indiana Department of Education is preparing to make students’ scores available to parents and schools. The department also will begin using the scores to calculate the annual letter grade from assessments given to schools each year. CTB/McGraw-Hill is conducting its own review of the ISTEP results, which will be reviewed by the National Center for the Improvement of Educational Assessment.
That data could reveal more about how individual students were impacted by the disruptions.
The number of students taking the multiple-choice portion of ISTEP has been rising. Here is a look at the rate of students taking that section of the test online:
Tests in Johnson County
Tests in Indiana
More than 495,000
Parents and schools have not received individual results yet.
Now school officials’ biggest priority is to get the test results from the state. Schools typically have the results in May, but school officials haven’t been told when this year’s scores will be released, Center Grove Superintendent Richard Arkanoff and Clark-Pleasant Superintendent Patrick Spray said.
Schools need the results because they use those and other assessments to determine how well students have mastered math, language arts and other core subjects. Students who don’t understand the subjects then can receive more specific, individualized help.
“I’d want them yesterday. So as soon as we can get them,” Arkanoff said.
The data show that while students in all grade levels were affected by the interruptions, students in Grades 3-6 experienced more interruptions than students in the upper grades. Most of the delays were during the math exams, according to the study conducted by Richard Hill, co-founder of the National Center for the Improvement of Educational Assessment.
To gauge the statewide impact of the interruptions, Hill reviewed Indiana’s past overall ISTEP scores as well as what the predictions were for the future scores. Overall, Indiana showed a four-point gain in math and a one-point gain in English and language arts.
That’s compared with a four-point gain in math and a three-point gain in English and language arts each year between 2010 and 2012.
If the interruptions had a serious impact on students’ scores, the 2013 scores wouldn’t have compared as similarly with the scores of previous years, the report said.
CTB/McGraw-Hill is conducting its own study of the results, which Hill plans to review as well, and that study could help identify specific students whose results were affected by the disruptions, the report said.
The problems during testing were caused because the servers at CTB/McGraw-Hill didn’t have enough memory to handle the number of students being tested. Students, teachers and parents have been wondering whether the ISTEP results — which the state uses to gauge whether students understand what they’ve been taught in a year and are used as part of schools’ and teachers’ annual evaluations — were even usable.
Hill compared the 2013 scores with scores from previous years and found that students who experienced interruptions did not have results that were lower than students who weren’t interrupted.
That doesn’t mean that individual student scores weren’t affected by the problems, and there’s no way to know what the scores would have been without the disruptions, Hill’s report said.
The Indiana Department of Education is now working to release students’ results to parents and schools and using the results to calculate the A-F grades schools will receive later this year.
The department also is continuing to work on reaching a settlement with CTB/McGraw-Hill, which will conduct ISTEP again next year, a department news release said.
The company could be asked to pay the cost of Hill’s review, which is about $53,600, and other fines and fees because of the interruptions.
State Superintendent of Public Instruction Glenda Ritz said in a news release:
“I have spent the last several months talking with Hoosiers about the impact these interruptions had in the classroom.
“Although Dr. Hill’s report found that the statewide average score was not affected by the interruptions, there is no doubt that thousands of Hoosier students were affected.
“As Dr. Hill stated in his report, ‘We cannot know definitively how students would have scored this spring if the interruptions had not happened.’ Because of this, I have given local schools the flexibility they need to minimize the effect these tests have on various matters, such as teacher evaluation and compensation.”
Hill had two sources of data to use while reviewing the interruptions. CTB/McGraw-Hill recorded data as students completed the tests.
Hill could see how much time a student spent on the test before an interruption, how many items a student saw before an interruption and how long a student waited before answering another question after having a computer problem.
School districts also were allowed to submit their own interruption records, the report said.
Of the more than 495,000 students tested, CTB/McGraw Hill reported 79,442 students were interrupted, while schools reported more than 63,000 more. That means about 29 percent of the students who took ISTEP had some kind of disruption, the report said.
School officials are waiting to review their ISTEP scores and for further direction from the state on how the results will be used.
At Center Grove, students and staff stayed focused and completed the exams as soon as they could despite the disruptions, Arkanoff said, adding he isn’t worried about this year’s results lowering the grades the school district receives from the state.
If any Center Grove schools receive an annual assessment, based partly on ISTEP, that is drastically different from what was expected, school officials will review whether the ISTEP disruptions were a factor.
“I think that the department of education did a really good job in trying to ensure that the tests weren’t impacted,” he said.
“At this point, I just trust their judgment on it.”