Marketing Is Not Evidence

Educational products make some impressive claims.

Students accelerate their learning. Achievement increases. Gaps close. Instruction becomes personalized. Teachers receive actionable insights. Students get immediate feedback. The platform adapts in real time and can be accessed anywhere, at school or at home.

While some of these claims may be true, that does not make them all evidence.

That distinction between true and evidence matters when schools and districts are making decisions about products that cost money, require staff time, consume instructional minutes, collect student information, and ultimately promise to improve something for our students.

Before we go any farther, I would like to define the two main terms here, marketing and evidence.

According to the Cambridge Dictionary, marketing is “a job that involves encouraging people to buy a product or service“.

It is important to note that throughout this post, I use the term product broadly to include the tools, technologies, curricula, programs, and services schools and districts purchase or adopt in pursuit of better outcomes.

Marketing has a legitimate job. It tells us what a product does and why we might want to consider it.

However, when it comes to educational products, there is an important distinction between those marketing claims and real evidence of that product’s efficacy.

Again according to the Cambridge Dictionary, evidence refers to “facts, information, documents, etc. that give reason to believe that something is true“.

While marketing tells us all of the things we hope a product will provide us, evidence that actually supports those claims can sometimes be a little trickier to gather.

Our job is to discern what the evidence actually supports.

That discernment starts with a deceptively simple question:

Which of these statements describes what the product can do, and which actually provides evidence of what students can do because they used it?

Features Are Not Outcomes

Educational technology has become remarkably sophisticated in recent years.

A product may adapt to student responses, provide personalized lessons, generate immediate feedback, identify areas of strength and need, produce beautiful dashboards, and allow students to work from virtually anywhere.

Those features may matter. They may improve accessibility, make implementation easier, help teachers use information more efficiently, or provide students with learning opportunities they otherwise would not have.

Technological sophistication is a product feature, not evidence of effectiveness.

A product can be adaptive, personalized, engaging, available 24/7, and beautifully designed. None of those features, individually or collectively, demonstrate that students learned more because they used it.

The same is true of implementation metrics. Minutes spent in a program, lessons completed, questions answered, and frequency of use can tell us whether students used the product. They cannot, by themselves, tell us whether students learned because of it.

Getting Better at the Product

This distinction between whether a student’s learning increased because of a product or not gets harder when a product provides both the learning experience and the evidence used to demonstrate its effectiveness.

Imagine students use a digital program several times each week. The program provides lessons, practice activities, hints, feedback, and quizzes. Over time, students’ scores within the program improve.

That’s useful information.

It is also exactly what we should expect to happen.

People tend to get better at things they practice. I’ve written before about the essential role practice plays in developing knowledge, skill, and judgment. That is, after all, one of the primary reasons we give students opportunities to practice in the first place.

This should raise an important question when we are evaluating evidence of a product’s effectiveness: What, exactly, have students been practicing?

Students who spend significant time within a product become familiar with its item types, prompts, supports, routines, feedback, and ways of representing content. They may also be developing the underlying academic knowledge and skills the product is intended to support.

The product’s own measures may not be sufficient to distinguish between the two.

If the claim is that students became more successful within the product, internal measures may provide perfectly reasonable evidence.

If the claim is that students became better readers, writers, mathematicians, scientists, or thinkers, we should demand evidence that they can demonstrate that learning outside of the product as well.

We do this by asking: Can students use what they learned outside the product that taught and measured it?

After all, that’s what we’re looking for! If the skills only apply inside that product, we’ve simply made our students master users of a product they will likely never encounter again outside of school.

Where’s the value in that?

Match the Evidence to the Claim

This does not mean internal product data are bad or useless, and it also does not mean external measures automatically provide good evidence.

Students might also demonstrate improvement on an independent measure of academic performance. Now we have evidence that warrants asking whether learning transferred beyond the product.

If the marketing claim is that the product caused the improvement, our questions need to get harder still.

• Where did those students start relative to the learning being measured?
• Who was included in the analysis?
• Compared with whom?
• How long did students use the product?
• How consistently was it implemented?
• What did teachers do with the information it provided?

Finally, and perhaps most importantly: what else was happening? Context always matters.

Maybe the school district introduced a new curriculum at the same time. Maybe teachers received substantial professional learning in the area being targeted, such as foundational reading skills, in addition to the new product. Maybe schools added instructional coaches, changed intervention schedules, created additional collaborative planning time, or increased tutoring. Maybe this has been a focus for years, and the results are finally coming in.

Once we understand that context, the apparent power of the product may become much less clear. And it should.

The students may have indeed improved, but that does not automatically tell us why.

The Test Didn’t Teach Them

This concept is especially important when we are evaluating assessment products.

A test does not increase student learning. Instruction does.

An assessment product can absolutely contribute to better instruction. It might provide teachers with evidence they did not previously have. That evidence might reveal a misconception or identify a learning need. A teacher might interpret that information and change instruction, provide targeted support, or give a student a different learning opportunity.

That’s a plausible pathway to improved student learning:

Relevant, defensible evidence → Appropriate interpretation → Focused, purposeful adjustments to instruction or supports → Increased student learning

Notice how much happens between administering the test and students learning more.

If a company claims that use of its assessment product improves student achievement, we should be very interested in those middle steps:

• What changed because teachers and students had access to the product? 
• How was the information used? 
• What instructional actions followed? 
• What implementation conditions made those actions possible?

Not every product makes this leap in its marketing, but it is worth watching out for. When a product claims improved student learning without showing how its evidence connects to changes in instruction, supports, or other learning experiences, consider it a marketing red flag.

The claim may ultimately be supported, but the evidence should allow us to trace a plausible pathway from using the product to what changed for students and, ultimately, to increased student learning.

What Did It Take to Get Those Results?

Even strong evidence of improved student performance leaves school and district leaders with another important question: What did implementation actually require?

A study may demonstrate meaningful improvement under particular conditions, and those conditions matter greatly.

To figure out what conditions implementation actually occurred under, start by asking:

• How much time did students spend in the program? 
• How much professional learning did teachers receive? 
• Was instructional coaching provided? 
• Did schools alter schedules or other structures impacting students?
• Were additional staff involved? 
• How closely did implementation resemble what the vendor recommends?
• Can your school or district reasonably replicate and sustain those conditions?

These are not minor implementation details. All of this information is part of determining whether the evidence is useful for your local decision. 

A program that produces strong results with extensive training, coaching, additional staffing, and 45 minutes of daily student use may be an excellent program.

It may also be a poor fit for a district that can sustainably provide none of those things.

There Is Always an Opportunity Cost

Schools operate with finite resources and infinite expectations. 

Money is finite. Staff capacity is finite. Professional learning time is finite. Student time is especially finite.

Every additional 20 or 30 minutes students spend using one product is time they are not spending doing something else. That does not necessarily make the product a poor investment. However, the investment should be worth what we are asking students and educators to put into it.

The same is true of money. Purchasing a program demonstrates that a district has done something in response to a problem. That is not the same as demonstrating that what it did worked, because evidence that we did something is not evidence that what we did was effective. 

That can be an uncomfortable distinction when leaders are under tremendous pressure to respond to low achievement, persistent gaps, or other urgent needs. I’ve addressed the problem with urgency in education decision-making before. The pressure to do something can make buying and implementing a new product feel like progress because it is visible, immediate, and easy to point to. Meanwhile, the sustained, coherent work that actually improves instruction and student learning is slower, more complicated, and much harder to package into a board presentation.

When the problems are urgent, the consequences of getting that distinction wrong are even greater. Urgency should sharpen our questions, not lower the bar for answering them.

Be Optimistically Skeptical, Then Ask Better Questions

The bigger the marketing claim, the more important it is to understand what supports it.

When a product makes an impressive claim, don’t start by asking whether you believe it. Ask instead:

• What exactly are they claiming their product will do?
• What evidence is being offered to support it?
• Does that evidence measure performance inside the product or learning beyond it?
• Is there evidence that the learning transfers?
• What else happened while those gains occurred?
• What implementation conditions produced the results?
• Could we reasonably reproduce those conditions here?
• Which statements describe what the product can do, and which actually provide evidence of what students can do because they used it?

Good marketing should make you interested enough to look more closely.

Good evidence should help you decide whether the claim deserves your confidence.

The ultimate question isn’t whether the product has an impressive list of features or a graph with a lot of green arrows going up. 

The question is whether students know and can do more because of what happened as a result of using that product.