Who’s Afraid of Data in Schools?

It’s hard to have a school-wide faculty meeting, or open an education magazine, without encountering the word “data.”  Once highly idiosyncratic, teaching has rightly become more and more a skill to be learned with outcomes to be measured.  I could not be happier about this move from teaching-as-witchcraft (or worse, merely bad theatre) to teaching-as-skill.

Those opposed to the use of data in schools are legion.  Some teachers say gathering data gets in the way of instruction.  Lots of stakeholders argue that it couldn’t possibly capture the complexity of good schools and teaching.  The most troubling objection is that data isn’t necessary to improvement; it’s somehow just a sideshow for accounting fetishists.

A recent post by Robert Wachter in New York Times puts it like this: “the objections [to using data] became harder to dismiss as evidence mounted that even superb and motivated professionals had come to believe that the boatloads of measures, and the incentives to ‘look good,’ had led them to turn away from the essence of their work…. The focus on numbers has gone too far. We’re hitting the targets, but missing the point.”

Then, quoting a “hard-nosed scientist” and measurement expert about how to improve the helping professions, Avedis Donabedian, Wachter claims he said on his deathbed: “The secret of quality is love.”

There you have it: love.  Retool, everyone!  Make loving schools and the rest falls into place.

Hardly.  Under the rubric of “love,” lots of pretty awful things happened: low expectations, damning “ability” streaming, corporal punishment, whole-language instruction, and on and on. Yes, teachers should be loving: but we know that because ample meta-analyses suggest that teacher-student relationships have a huge impact on student outcomes.  That’s data.

Why We Should Embrace Data

“Count something.” That’s the advice of Atul Gawande, surgeon, writer, health care reformer.  I am fond of borrowing his ideas, and this is no exception.  We ought to be counting more, not less; we ought to be asking research questions in our own schools and classrooms; we ought to be finding the best evidence about the effectiveness of our individual schools and making decisions based on it.

Ask Yourself: Without “Data” or “Information” or “Evidence”…

… How would you know if your classroom or school is functioning at its best?  Rather than gathering data, would it be better to merely guess?  Or perhaps listen to the loudest teacher complaining in the faculty lounge?

… How can you find improvements?  How do you know if your students are being well-served? How do you know if program changes are aiding student learning or the opposite?

Models of Using Data That Are Likely to Work:

– Validity: Measure what is valued instead of valuing only what can easily be measured

– Balance: Create a balanced scorecard

– Insist on credible, high quality data that are stable and accurate.

– Design and select data that are usable in real time.

– Develop shared decision-making and responsibility for data analysis and improvement.

(This list is taken from Andy Hargreaves, sometimes quoted as being opposed to the use of data in schools.)

Imagine This

Using the above criteria, I suggest that each school (or department within) embrace as their operating principles – something they just do as a matter of course – a set of measures.

ImagineColleagues getting together to decide, informed by stable accepted research (yes, it does exist) and professional judgment, what things really matter.  A few possibilities to illustrate: student engagement; preparation for the next step; effective instruction; and so on.

Then, consciously asking: how can we collect data – qualitative and quantitative, in valid and practical ways – to figure out how we are collectively doing across these domains?

Then, embracing an ongoing effort to track and discuss the data to improve programming or approaches.

The Scientist in the Room

I’m fond of saying that the best teachers are scientists in the room, looking for data about how the students are doing. That data could come from a lot of different sources: it could come from examining exams after students have written them; it could come from moderated or parity marking with colleagues; it could come from student surveys; it could come from student exit interviews; it could be routine (brief “stop/start/continue” surveys for students at the end of units). The more conscious we are of the information we base decisions on the better our decisions are likely to be.

The important thing is that we count, measure, and gather information, from a variety of sources to paint the most accurate picture of what we’re doing and how well it’s working.  We need to start now.  And we need to make it part of our regular practice.

We need more data, not less.  It needs to be collected efficiently and routinely, used consciously and conscientiously, for the perpetual betterment of our schools.  People have said the use of data harms children; I argue the careful use of data is the best defence against a complacent or dysfunctional approach to schooling.

If we won’t tolerate the random best guesses of heart surgeons, why would we demand any less from those that teach?

(Photo: JD Hancock)

If High Expectations Can Get the Blind to See, What Could They Do in Our Schools?

Arnie Boldt must know the value of high expectations.  Injured in an auger accident at the age of three, he lost his right leg.  This loss did not stop him from becoming one of the greatest athletes in Canada’s history, in a seemingly improbable event no less: high jump.  Over his career, he won seven gold medals and one silver in high jump, setting more records than this space has room for.

Invisibilia, NPR’s newest science-themed podcast, recently explored the remarkable case of Daniel Kish.  Kish, blind since the age of 13 months, trained himself to use the clicking of his mouth to echolocate objects in his environment.  Kish sometimes rides a bicycle to show the amazing capacity of his talent.  He has dedicated his life to teaching the blind to, as some might say, “see” through echolocation and lead lives with more possibilities.

These examples are remarkable outliers, of course, but they establish something of critical and generalized importance: our expectations shape our possibilities.  Expectations we place on ourselves and those placed on us by others.  When we convince people they cannot do something, it acts as a barrier to their achievement.  When we act as though great things are possible, they often are.

Nowhere is this more important than in schools, where our youngest members of society often decide what constellations of possibilities to explore.  Schools have a key role to play in fostering a mindset of possibility, not unlike the growth mindset for which Carol Dweck has argued persuasively.  It is in schools where young people decide to push themselves – or, too often, not.

High expectations are communicated in many ways.  Shopworn posters line many school hallways and libraries, exhorting students to reach their potential.  Good as far as they go, but likely more powerful are the underlying messages broadcast to young people in our language, in our actions, and in our subtle overtures.  A student who thinks a teacher does not care is likely to find reason to disengage, too.  But teachers can also act as Chief Encouragement Officers of their classrooms, raising possibilities for lifelong engagement.

The importance of expectation-raising is probably especially true for groups of students who graduate at lesser rates than average, enter university at lower rates, and find fewer doors open as adults.  Even if talent is distributed equally across all races, cultures, genders, and other categories, the experience of life shows some groups do better than others.  The difference in outcomes, both in school and in life, are surely sensitive to the kinds of expectations students have of the limits of their possibilities.

I don’t know Arnie Boldt, but as one of my professors used to say, I would love to meet the physical education teacher who first worked with him.  Even if the drive to jump came from Boldt himself, it would have taken an act of pure faith to foster his natural talent.

The lesson: if determination and grit and, yes, high expectations, can allow Boldt to high jump 2.08m (!) and Kish to echolocate his way to blind bicycling, what could be done with students in our schools?  And more troubling, what might not be done, what achievements and positive life outcomes might be missed, when we fail to set expectations beyond our students’ view of what is possible?

While more – much more – than high expectations are necessary to improve schooling outcomes, the most vulnerable groups of students in our schools desperately need adults to encourage the visualization of a future that right now seems improbable.  The capacity for human ingenuity cannot be known in advance; we shouldn’t behave as though it is.

Soldier-Scholar-Statesman: The Emotional Lives of Schools

Schools are emotional places.  Any teacher knows that happy and confident students learn more effectively.  Might the same be true for teachers?  

Most reform efforts meant to improve schooling outcomes have focused on technical reforms.  Assessment and evaluation practices come to mind.  Rubrics.  Professional Learning Communities.  And I am a big fan of some of these – I’m particularly fond of the technical improvements in teaching practice – but we do ourselves a favour if we realize these changes go on in an emotional environment.  That is, for teachers to adopt new practice, they need to feel safe, nurtured, celebrated.  Improvements in the technical practices of teaching and learning are necessary to make better the experiences of students; given that teachers are the mechanism by which principals work, it makes sense for school leaders to have front-of-mind the emotional needs of teachers.

Or, put another way…

Another one of the challenges in our large-scale reform efforts right now, which focus on capacity building, is to bring emotions out of the shadows and say ‘this is part of capacity building too’. We have to nurture the way our staff is feeling about their work if we expect them to be resilient and sustain their efforts. The work has to be something that’s meaningful. It needs to feel like we’re making progress, and it needs to be something teachers are confident about being able to do. – Ken Leithwood

The ideal of US military leadership has sometimes been described as soldier-scholar-statesman.  Principals would do well to have something like that model in mind.  They need to be all three (they need to understand what is best; act decisively; and with diplomacy, tact and empathy), but knowing when which role is required takes intelligence, grace, and courage.

Why You Might Not Want An Innovative School for Your Kids

If Hospitals Were Like Schools…

Imagine a visit to the emergency room that went something like this.  Worried you might be having a heart attack, you complain of chest pains.  Instead of using the usual protocols, the attending physician says, “Yes, thousands of other doctors have had good results using what’s tried and true, but it’s not my style.  I’ve developed my own way.”  A savvy patient would be worried; while this doctor’s approach might be better than the existing protocol, it is far more likely to be inferior.  Adopting a new approach in the absence of evidence is dangerous.  Yet, this is exactly what we force teachers to do in our schools – adopt idiosyncratic and untested ideas.

Or rather, it is what many would have us do.  If we are to keep improving our schooling outcomes, we need to keep what is working.  The imperative to innovate in our schools is seductive, but we run the risk of changing what already works in favour of untested hypotheses.  Looking back over the history of schooling, we can see a lot of under-evidenced reforms.  Multiple intelligences come to mind.  As do learning styles.  The history of schooling is too often the frequent adoption of fads unsupported by evidence.  This amounts to the worst of all worlds: ignoring the systematic evidence in favour of the gleaming one-off study.  Or adopting teaching models that have no empirical evidence at all.

Teachers who resist these ideas are often labelled as lazy or troublesome, or worse of all, acting without their students’  best interest in mind.  But educational reformers, consultants, and political actors often label experimental ideas as certain truths.  In fact, teachers are often wise to approach educational innovation with some skepticism.

Isn’t Educational Research Bad? Hardly.

Part of the blame rests in the common wisdom that educational research is bad.  Any field has its share of studies with spurious conclusions and shoddy methodology.  But we should not allow the bad to obscure the research that is productive and helpful.  The best of it examines studies in the same way medical researchers do; while achieving a double-blind study in schooling research is likely impossible, thorough meta-analyses of decades of research approaches the kind of systematic understanding we rely on in health sciences.

This is the view of Canadian-trained New Zealander John Hattie, whose work – most notably, his Visible Learning project – systematically analyzes educational research and provides welcome insight into the sometimes confusing results of studies.  Not unlike nutrition research, individual studies need to be understood as part of a longer story.  Some studies say one thing; others disagree.  Meta-analyses like Hattie’s bring together thousands of research results involving millions of students to provide a good, if provisional, answer to the question: what works best in the classroom?

That language is important.  The question is not: will students learn if a teacher uses a particular model of teaching, but will they learn more using this model than if we had used another?  If there are no gains to be made over existing practice, we harm our students.  By way of small example: Hattie’s work suggests direct instruction – a teacher-centred, traditional version of teaching –  is more effective than most other teaching practices. And yet teachers have been told for at least a generation or two that direct instruction does not work.

Schools Are Already Good, But They Can Get Better

Canadian schooling has excellent outcomes – the OECD PISA results are a testament to that.  This is not to say that improvements cannot be made; they can and should.  (Schooling outcomes for aboriginal Canadians, for example, demand urgent reform.)  But if schooling is to improve further, the answer cannot lie in adopting just any notion, no matter how interesting it seems.  We can hardly afford to scale up ideas that, while different, are not improvements over current practice.  In schooling, as in medicine, what is different should only be adopted if it is demonstrably better.

(Photo: Robb North)

Making Water Flow Uphill?

“In engineering, control exists in adjustment to natural law. It does not mean making nature do our bidding. We do not, for example, dig channels in the expectation that water will flow uphill; we do not put kerosene to put out a fire… with respect to physical phenomena, control involves the selection of means which are appropriate to the nature of the phenomena with which we are concerned. In the human field, the situation is the same, but we often dig channels to make water flow uphill. Many of our attempts to control behaviour, far from representing selective adaptations, are in direct violation of human nature. They consist in trying to make people behave as we wish without concern for natural law. Yet we can no more expect to achieve desired results through inappropriate action in this field than in engineering.” –  Douglas McGregor (1960), The Human Side of Enterprisep. 8-9.

How many school administrators, in their interactions with teachers, are acting as if they can make water flow uphill?  

How many teachers, in their interactions with students, do the same?

What would a science of performance, that took into account the human side of schooling, look like?

The Power of ‘No’

“Unweeded soil undoubtedly grows wondrous things that nobody can predict. Such things we have in abundance, but it would be a rash man who would call it a harvest.” – Jacques Barzun, Begin Here: The Forgotten Conditions of Teaching and Learning, p.27.

Schools can be breeding grounds for well-intentioned distractions.  Everyday it seems, a new task is added to the list of what schools are supposed to achieve collectively, and what individual teachers are asked to do in their classrooms.  Each new problem of adolescence, or new technological product, brings a call to build new programming.

We act as though adding is more important than perfecting; a school that boasts more ought to be better, we think.

The reason for confusing more with better is not obvious.  My guess: it seems disloyal or lazy (or worse, opposed to the success of children) if one suggests a reduction in programming.  But I think those most loyal to the core principles of a school are often very careful about cluttering up our practice.

It’s likely we could achieve more of what matters by doing less of what doesn’t.

We do ourselves a favour if when we walk around our campus each day, we ask ourselves:

– Do we need to do X?

– Is X advancing our mission, or is it merely a well-intentioned distraction?

– What does it take to do X, and what could we be doing instead?

– And maybe most important: How can we tell the difference between core practices and distractions? (Sadly, I think even in the best schools, faculty and leadership would not share a coherent idea of what the school’s core mission is.)

The Power of ‘No’

Sometimes we need to say “no”; doing so is often the most loyal, committed thing a person can do.  We would do well to heed the lesson of opportunity cost, and ensure that we don’t succumb to conspicuous program inflation that sometimes grips our schools.

We need to beware our voice finding expression only in the celebration of mere increase rather than the slow and steady perfection of teaching and learning.

If you could drop three things from your schools, what would they be? And what could we spend that energy on instead?

Why Changing the Carpets Will Not Improve Schools

Abbey Lanes https://www.flickr.com/photos/abbylanes/3335173723/

We all want (even) better schools.  But anyone close to the challenge of improving schools knows that often part of the problem is not too little inspiration but too much.  Attempts to improve bring a flood of well-intentioned ideas.

“We should have a later start to the school day”

“We have to teach 21st century skills”

“We need more creativity in our classrooms”

… And so on.  Some, or all, might be valuable.  None might be.  But more important than the question “would any given idea work?” is “would this idea work better than any other possibility?”

In other words, to use the phrase of John Hattie, which change has the best impact?

Elizabeth Green recounts one experience from Doug Lemov, a prime mover in US Charter Schools.  Lemov took a group of highly motivated teachers to…

one of the crown jewels of the no-excuses world: KIPP Academy in New York City. Created by David Levin and Mike Feinberg, two early Teach For America corps members, KIPP was a perfect model of both the zero-tolerance discipline approach and the sermonizing school-as-pep-talk culture.

 

Yet when Doug asked the Syracuse teachers about the trip, he found that the visit had not proved instructive.  The teachers had seen plenty of things – the arrangement of the reading rugs, the colours of the uniforms.  But… they had not seen the things that they needed to learn.  “I just remember thinking, ‘Holy shit.  That’s what you took away?’  The things they took away were so random, and if you ranked the most important things about a high-performing school from 1 to 100, they had seen number 63, number 84, and number 47.  As opposed to numbers 1, 2 and 3.” (Green, Building a Better Teacher, p. 172)

I think teaching is where medicine was several generations ago: our dreariest days are behind us.  Our best systematic research efforts are underway.  The better districts and systems are using the best evidence, and are thriving on the best international measures of student success.  And at the level of the classroom, the mindset of the practitioner is (hopefully) no longer that mere enthusiasm for children is sufficient to be a good or great teacher.  These are all welcome improvements over previous eras.

But for profound changes, we need to stop thinking at the level of changing the colours of the carpet when making the thousands of day-to-day decisions that impact our schools.  We need to be sure we waste no time implementing changes that actually work, and politely ignore those that have a smaller effect size.  We have only so much time.

I’m Curious:

How many meetings have you been in where number 63 on the Most Important Initiative list wins out over numbers 1, 2, and 3?  

Do we have the capacity in our schools to know what are the most important factors to drive improvement? How would we tell the difference between numbers 84 and 1? 

And how do you build a culture of insisting that changes are improvements, not just changing the carpets?

Servant Leadership

Since at least Jim Collins’s Good to Great, the world of management – and educational administration – has been intrigued by the notion of humble leadership. I’ve written before about it here, and it captivates my thinking to this day.

Some researchers, J. Andrew Morris, Céleste M. Brotheridge and John C. Urbanski, developed the line of thinking in “Bringing humility to leadership: Antecedents and consequences of leader humility.” (Human Relations 2005 58: 1323.)

The paper is a truly interesting read. While the researchers don’t develop a suitable measurement tool, nor take the further step of validating the concept through empirical data, they do provide “a clear conceptualization of humility”, “ (offer) several potential predictors of humility and indicated the specific ways in which humility may impact the leadership process.”

Their propositions:
1) Higher levels of narcissism predict lower levels of humility.
2) Machiavellianism predicts lower levels of humility.
3) Low self-esteem predicts lower levels of humility.
4) Defensively high self-esteem predicts lower levels of humility.
5) Higher levels of emotional awareness and management predict higher levels of humility.
6) Leader humility predicts supportiveness toward others.
7) Leader humility predicts a socialized power motivation.
8) Leader humility predicts participative leadership.

And their conclusion:
“… individuals whose personal traits include narcissism, Machiavellianism, low self-esteem, or defensively high self-esteem are likely to have low levels of self-awareness, openness, and transcendence, the dimensions of humility; and that the latter characteristics are more likely to be found in individuals with high levels of emotional awareness and control, components of emotional intelligence. Humility, in turn, is expected to generate servant leader-type behaviors such as engaging in supportive relationships, presenting a socialized power motivation, and leading through participation.”

Priestly Advisors and Warrior Kings – Susan Cain, Again

We need “priestly advisors” in addition to “warrior kings” – Susan Cain

One of the most popular non-fiction books of this year has certainly been Susan Cain’s Quiet: The Power of Introverts in a World That Can’t Stop Talking. I’ve written before about her ideas here, but there are a few quotations I still find worthy of remembrance. (And as we’ll see below, others might benefit from her words, too.)

Don’t Extroverts Rule the World – and Rightly So?
Despite the assumptions made by schools like Harvard’s Business School, extroverts don’t always make for better leaders: “If we assume that quiet and loud people have roughly the same number of good (and bad) ideas, then we should worry if the louder and more forceful people always carry the day… yet studies in group dynamics suggest this is exactly what happens.” (51)

Does this sound like your last faculty meeting?
“Groups are like mind-altering substances. If the group think the answer is A, you’re much more likely to believe that A is correct, too… (The study subjects) were utterly blind, in other words, to how much their peers had influenced them.” (92)

Ever feel for the quiet, capable students who wish to work alone?
“Our schools should teach children the skills to work with others – cooperative learning can be effective when practiced well and in moderation – but also the time and training they need to deliberately practice on their own.” (94)

Ever wonder if school leadership might be more subtle, more thoughtful, more reflective?
Cain argues, examining the soft power examples from other cultures, most obviously those like like Gandhi, show the value of those who lead “by water rather than fire” (197)

And why repeat a posting on Cain?
And evidence Cain’s warnings have been in the news recently – indirectly. A large section of her book is devoted to critiquing Tony Robbins’s Unleash the Power Within seminar, including the extroverted-mad ending where session participants are exhorted to walk across burning coals.

The extrovert bravado can literally defy the forces of physics! Believe in your own gregariousness and tower over the laws of thermodynamics!

Except not. From a July 23rd news article in the Vancouver Sun – apparently there are limits to the Power Within:

21 injured in hot coal walk at Tony Robbins motivational event

Fire officials in California say at least 21 people were treated for burns after attendees of an event for motivational speaker Tony Robbins tried to walk on hot coals.

The San Jose Mercury News reported at least three people went to a hospital and most suffered second-or third-degree burns.

Robbins was hosting a four-day gathering called Unleash the Power Within at the San Jose Convention Center.

Witnesses said a crowd went to a park on Thursday where 12 lanes of hot coals were laid out on the grass.

Witness Jonathan Correll said he heard “screams of agony.”

Robbins’ website promotes “The Firewalk Experience” in which people are encouraged to walk on superheated coals.

Fire Capt. Reggie Williams said organizers had an open fire permit and emergency personnel were on standby.

On Knowing, Not Just Saying

Readers of this space know of my obsession with philosophical (really, epistemological) issues. Mostly, I am concerned with separating the wheat from the chaff in ideas – isolating the truly silly, from the potentially true, from the probably true, from the certain. In education, these categories tend to get tossed around together, without much reference to the important question: How Would We Know?

I recently had the pleasure of seeing John Hattie speak at the University of Toronto. To say it was refreshing is an understatement. I’ve been drawn to his work for a while now, and before I found it, was convinced such a project was possible.

(Here’s Hattie giving a similar talk.)

His interest is in measuring the effects of various inputs in the education system – inputs and interventions like team teaching, outdoor education, whole language versus phonetics, and practically every question in pedagogy. His premise: we can determine the effect size of all of the things we do in schools – how well they work. His conclusion: as a profession we should move towards those things that work well (have a large effect size), and away from those that do not.

Simple enough. I’m not able to provide a census of his detractors, but I know lots of people who critique his philosophical assumptions. They say that teaching practice cannot be reduced to such certainties. They argue that to strive to capture the subtle human interactions and nuances in teaching quantitatively is absurd. They sometimes argue that to gather such data is to take aim at low-performing schools and groups within them – they sometimes argue it’s culturally imperialist.

Obviously, they have warm hearts. All of us want students to do well, and most of us root for the disadvantaged. But until we gather reliable data on what works and what does not, we are continuing to impoverish our students. And what better way to ensure fairness in our society than by providing all students with the best possible teaching techniques and the best possible practices? And how else to do that then by measuring in the most precise way possible the effect size of what we do in schools?

Like Hattie, I am hostile to the idea that we are not professionals with something special to give. I reject outright the notion that “all teachers have their own way.” If an old-timer said, “Well, I hit the students; that’s how I get them to learn,” we would be outraged. I don’t see how it’s much different to say we are all equally successful using whatever techniques and approaches we “feel” are right. (Also, if teaching were so much based on whim, we would let absolutely anyone walk in and teach our classes; we do not, evidence we think it matters who teaches and how it is done.)

But perhaps the most satisfying element of the whole thing is the humility its adoption would bring to our field. Teaching suffers from a strange paradox of ego: on the one hand, most teachers feel like imposters, and denigrate the value of their practice; on the other, many teachers act the role of Superteacher, where everything he or she does is magical. What teacher hasn’t bristled in the staff meeting where one of their colleagues bellows, “Well, in my class, students love doing X,” or “I’ve never had that problem in my class…”, or “My students learn best when…”? I always want to ask, “How would we know?” and get a response more fulsome than “Because I’ve been teaching for 19 years, and I just know.”

Measurement projects like Hattie’s sweep all that nonsense away by asking, “What is the effect of our labours?” We can know, within some margin of error, what works and what doesn’t. At least, if we can ever know at all, it will be with an approach like Hattie’s, not our gut feelings and egotistical rantings. And in that kind of regime is comfort – it can depersonalize teaching somewhat, diminish the notion that teaching is a cult of personality, or a kind of mystical alchemy. Some approaches work better than others; let’s determine those, reliably, use them more often than not, and continue to measure the effects of our work – forever.

I dream of a rigorous measurement approach in a school setting, a unit of organization too small to hide in. You would probably need to set up long-term measurement indicators – perhaps a few basic assessments used for years within each grade or course, evaluated with clear rubrics and exemplars, and copies of old student work kept for years – and determine if students are improving by virtue of our efforts, and of course, by how much. (It isn’t enough to merely improve: as Hattie points out, we need to know the magnitude of the improvement. A student in any class will achieve some level of improvement over the year just through maturation.) Other indicators I like: success after high school, student feedback,

We could finally, without reference to our own whims, begin to address genuine “best practices” in our schools. Does team teaching work at our school? Let’s check this year’s assessment and see. Are students benefitting from the Advanced Placement regime? Let’s see how our graduates have done over the past 10 years and compare that number against our graduates from the pre-AP days. Is our program rigorous enough? Let’s gather data from 1st- and 2nd-year students in post-secondary studies. It is for these reasons I’m, in principle, a fan of large-scale assessments like the EQAO.

A proper measurement regime would provide some justification for the claims we make about our schools, our classes, our practice. In fact, it’s the only thing that ever has, and ever will. Without it, merely the loudest voice in the room wins.