Why Good Data Still Gets Misused

Over the last two posts, I have talked about what statewide assessment results cannot tell us and what they actually can tell us when they are used appropriately.

So this naturally leads to the next question:

If the limitations are fairly well known, and the intended uses are fairly clear, why do these results still get misused so often?

Because this is not just a data problem, it is a human problem. This is in addition to a systems problem, because systems problems are expressions of those aforementioned human problems.

Honestly, at this point, it would be more surprising if the data wasn’t getting misused.

People Need a Winnable Game

There is a concept in The Four Disciplines of Execution by McChesney, Covey, and Huling that has always stuck with me: people need a compelling scoreboard because people will not continue playing games they believe are unwinnable.

This is an innate human impulse to give up on games we see as rigged.

I think we have lost that sense of a winnable game in K-12 statewide assessment systems.

Teachers are expected to improve results, often on timelines they do not control. School leaders are pressured to show growth quickly. District leaders are expected to explain scores publicly and respond immediately. State agencies face political pressure tied to accountability systems and public perception.

All of that creates urgency, and urgency creates fear.

Once fear enters the system, people start searching desperately for anything that feels actionable.

That is where many common misuses begin.

Not because educators are unintelligent or do not care. It’s quite the opposite, in my experience.

It is because they are trying to function inside systems that expect immediate proof from work that often takes years to fully take hold.

The System Quietly Trains Misuse

We also need to acknowledge something uncomfortable:

Many of the routines we now call “data-driven decision-making” were built inside systems that reward visible reaction more than sustained direction.

It starts with things that sound reasonable enough on their own.

• Public report cards.
• Accountability systems.
• Funding decisions.
• Board presentations.

Then the pressure expands to:

• Community comparisons.
• School rankings.
• Improvement plans.
• Questions from district leadership.
• Questions from parents wondering why scores did not improve this year.

Then it gets louder.

• Local newspaper op-eds about “failing schools.”
• Public comment at board meetings.
• Social media posts declaring public education is broken.
• Real estate websites rating neighborhoods by schools.
• Leadership turnover tied to scores.
• Political talking points.
• Headlines.
• Fear.

No wonder we are misusing this test data. All of it sits on top of systems asking educators to produce visible movement as quickly as possible. 

We have built systems where adults are expected to react to every new data point as proof of responsiveness and leadership.

In addition, once results become high-stakes targets, behavior changes around the target. That is not new. We have known this for decades. I talked about this previously through the lens of Goodhart’s Law: when a measure becomes the target, it stops functioning well as a measure.

The irony is that statewide assessment systems are built on incredibly sophisticated psychometric models with carefully constructed validity arguments and statistical assumptions. Yet those models exist inside human systems full of shifting priorities, uneven implementation, fear, political pressure, and constantly changing timelines.

The statistical models are not broken.
The human systems surrounding them often are.

Predictable Outcomes of Systems Under Pressure

This shows up in ways that are so normalized we rarely stop to question them anymore.

For example, districts are expected to establish or renew contracts with local assessment vendors before statewide summative testing is even completed, let alone before results are available.

Think about that for a moment.

We say statewide assessment results should inform major instructional and assessment decisions with massive price tags. Yet the contractual timelines for those decisions often happen before the full picture of information even exists.

So decisions get made anyway.

Not because people are careless, it is the system requiring action before the evidence arrives.

It also helps explain why systems often overreact to partial information, chase short-term indicators, or rely too heavily on trends that may already be a year behind current students.

When System-Level Data Gets Used for Individual Decisions

I have personally sat in staffing meetings where we were estimating how many advanced mathematics sections to offer the following year at a comprehensive high school.

Those decisions had to be made before current statewide assessment results were available.

So naturally, people turned to the previous year’s results.

So now we are using a broad, system-level indicator of academic performance that is already more than a year old to estimate opportunities for a completely different group of students.

If you have ever worked with middle school students, as I have, you know how quickly those students change. They walk into junior high as children and leave as teenagers.

Using old statewide assessment results to determine how many advanced opportunities will even exist in a high school master schedule quietly closes doors for students before anyone intends to.

Again, not because people do not care. It is because the system normalized using the wrong tool for the wrong decision.

This is where it becomes incredibly important to remember what statewide assessment results actually are system-level indicators.

They were designed to help identify broad patterns of academic performance across groups of students over time. That information is important. State education agencies need credible indicators to help guide funding, support, long-term planning, and accountability decisions at the system level.

That is a very different decision than determining whether an individual student should have access to advanced coursework next year.

Those decisions require evidence much closer to the student.

• Teacher input.
• Current grades.
• Student interest.
• Classroom evidence.

Those measures are much more aligned to the actual decision being made (Oh, look at me calling out multiple measures that actually matter for such a decision!)

Sometimes What We Call “Data Use” Is Really Anxiety

This is also why so many data meetings feel frustrating and unproductive.

• Charts get printed.
• Spreadsheets get color coded.
• People are asked to “explore the data together.”

Then teams spend hours trying to explain one- or two-point fluctuations that may simply be noise.

Sometimes what gets labeled as “data-driven decision-making” is really just anxiety with spreadsheets.

People feel pressure to “do something,” even when the data does not yet support a meaningful conclusion.

Since the systems often lack sustained direction, the ideas brainstormed in those meetings are frequently abandoned before they are ever fully implemented anyway.

Then the cycle starts over.

The Assessments Are Not the Problem

It is important to say this clearly: The existence of misuse does not mean the assessments themselves are useless.

These assessments are generally doing exactly what they were designed to, provide broad, system-level indicators over time.

The problem is the mismatch between:

• timelines we expect results to operate on
• kinds of decisions we are trying to make
• human pressure layered onto the system

We keep expecting statewide assessments to provide immediate proof that complex, system-wide change is working.

That urgency is shaping decisions far more than the data itself.

And honestly, I think urgency may be one of the biggest reasons meaningful improvement struggles to take hold in education systems at all.