Data is supposed to drive instruction. Data is supposed to be used to inform decisions. Data should do many things, but does it?
Take for example the simple decision to wear a jacket to go to school. It is March, and the thermometer reads 59 degrees. You have all the data you need right? Maybe. If you live in California like I do, I am going to grab a jacket. 59 degrees is cold in my view. If, like some of my family, you live in Minnesota, you might come to school in a T-shirt. The same data, but interpreted very differently.Frederick M. Hess had an article in ASCD’s Educational Leadership. The article titled “The New Stupid” discusses how many leaders in education are making mistakes in how they use and apply student achievement data and systematic research.
The “old stupid” was failing to use or dismissing data and research as having little value. Now, though, we live in the era of data directed decision-making. The problem is we are not carefully considering the type of data or research without asking common sense questions. According to Hess, the “new stupid” has 3 key elements.
1. Using Data in Half-Baked Ways
Many educational leaders fail to ask simple questions about the effect or impact of using data or research. For example, educational leaders who were quick to move high performing teacher to low performing schools without asking questions about the effectiveness of moving teachers, the practicality of the idea, and if there might be other ways of encouraging quality teachers to move without forced relocation.
“Then as now, the key is not to retreat from data but to truly embrace the data by asking hard questions, considering organizational realities, and contemplating unintended consequences. Absent sensible restraint, it is not difficult to envision a raft of poor judgments governing staffing, operations, and instruction—all in the name of "data-driven decision making."
2. Translating Research Simplistically
One study in one specific situation shows a promising result and educational leaders jump on the data or results and use it to put entirely new programs in place. For example, taking the STAR (Student Teacher Achievement Ratio) Program’s results (showed positive achievement for small class sizes in kindergarten and first grade) and using it as a basis to implement programs state, county, district, or school-wide.
California implemented a class size reduction program through the entire state for every school. The data it was based on did not support such a move, and student achievement was not raised in significant ways. Again, educational leaders failed to ask some probing questions about the data such as, “What would be the impact of hiring thousands of new teachers?” or “Can this data be extrapolated to mean that all students in all situation will benefit from lower class sizes?” The result in California is billions spent with very little in return.
3. Giving Short Shrift to Management Data
Educational leaders have not spent much energy or effort using data or research and applying it to the functions of the school or district. Most school district employees work outside the classroom, yet data is not collected about their performance.
“Data-driven management should not simply identify effective teachers or struggling students but should also help render schools and school systems more supportive of effective teaching and learning. Doing so requires tracking an array of indicators, such as how long it takes books and materials to be shipped to classrooms, whether schools provide students with accurate and appropriate schedules in a timely fashion, how quickly assessment data are returned to schools, and how often the data are used. A system in which leaders possess that kind of data is far better equipped to boost school performance than one in which leaders have a pallette of achievement data and little else.”
How do we avoid the “New Stupid?” Hess has some advice.
1. Good Judgment
“…educators should be wary of allowing data or research to substitute for good judgment. When presented with persuasive findings or promising new programs, it is still vital to ask the simple questions: What are the presumed benefits of adopting this program or reform? What are the costs? How confident are we that the promised results are replicable? What contextual factors might complicate projections? Data-driven decision making does not simply require good data; it also requires good decisions.”
2. The Data You Need and The Data They Need
“…schools must actively seek out the kind of data they need as well as the achievement data external stakeholders need. Despite quantum leaps in state assessment systems and continuing investment in longitudinal data systems, school and district leaders are a long way from having the data they require. Creating the conditions for high-performing schools and systems requires operational metrics beyond student achievement. In practice, there is a rarely acknowledged tension between collecting data with an eye toward external accountability (measurement of performance) and doing so for internal management (measurement for performance).
“The data most useful to parents and policymakers focus on how well students and schools are doing; this is the kind of data required by No Child Left Behind and collected by state accountability systems. Although enormously useful, these assessments have also exacerbated a tendency of school and district leaders to focus on the data they have rather than on the data they need.”
3. Know What The Research Can Tell Us and What It Can’t Tell Us
“…we must understand the limitations of research as well as its uses. Especially when crafting policy, we should not expect research to dictate outcomes but should instead ensure that decisions are informed by the facts and insights that science can provide. Researchers can upend conventional wisdom, examine design features, and help gauge the effect of proposed measures. But education leaders should not expect research to ultimately resolve thorny policy disputes over school choice or teacher pay any more than medical research has ended contentious debates over health insurance or tort reform.”
4. Reward Leaders Who Lead With Data
“…school systems should reward education leaders and administrators for pursuing more efficient ways to deliver services. Indeed, superintendents who use data to eliminate personnel or programs—even if these superintendents are successful and vindicated by the results—are often more likely to ignite political conflict than to reap professional rewards. So long as leaders are revered only for their success at consensus building and gathering stakeholder input, moving from the rhetorical embrace of data to truly data-driven decision making will remain an elusive goal in many communities.”
Data is not right, data is not wrong, it just is. It is a thing. It does not define a teacher, nor does it define a student. Most importantly, for my school, a test score does not define them and their work. It is a thing, a tool for understanding and improving their own practice.
“We are a society that is data rich, but information poor.” Robert H. Waterman
“Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everybody gets busy on the proof.” John Kenneth Galbrith
Using data and research are key to improving student achievement and creating high performing schools. But equally key, is asking great questions. It is through great questions that data and research become meaningful to each school district and school. What kinds of questions are you asking?
"Great results begin with great questions." That is Education Innovation