Challenge the AI: Why the Best Observer in the Room Has No Memory of You

A few weeks ago I was on a screen-share with a principal and her assistant principal — two of the most thoughtful administrators I get to work with — looking at an AI report run across a stack of their classroom walkthrough videos. We'd asked the platform to rate classroom climate across the building. The numbers came back, the principal got quiet for a second, and then she said something that made me want to reach through the screen and high-five her:

"I'm not sure I'd be comfortable with this being the only data source."

She thought she was pushing back on me. She was actually telling me she understands AI better than most people who talk about it for a living.

The gift isn't intelligence. It's impartiality.

Here's what I told her. The reason to bring AI into a walkthrough isn't that it's smarter than you about teaching. You've forgotten more about instruction than most tools will ever "know." The reason is much simpler, and much harder for any human to pull off: the AI is a clean slate every single time.

It didn't have an argument with a teacher in the parking lot this morning. It doesn't remember that one tense conversation three years ago. Nobody brought it coffee. I say this to administrators all the time — I'm a human. If I'd had a disagreement with you before first period and then walked into your classroom, I am not seeing that lesson the same way I'd see it from the teacher who made me laugh at the copier ten minutes earlier. We carry our relationships into every doorway we walk through. It isn't a character flaw. It's just being a person.

The AI doesn't carry any of that. It hears what was said, it counts what happened, and it reports it — same teacher, same lesson, same read, whether you adore them or you're at your wit's end with them.

That's the gift. Not intelligence. Impartiality.

The principal who turned the mirror on herself

There was another moment, in a separate conversation, that I keep coming back to. We were looking at the read the AI had given one classroom, and the principal said something — about herself, not about the tool. She wondered out loud: Could I have come up with that on my own? Probably. But am I skewing what I see, based on what I already believe about this teacher? And then she answered her own question. Honestly, she said — maybe.

That is one of the most self-aware things I have ever heard a leader say. She wasn't doubting the AI; she was being honest that she walks into a classroom already holding an opinion of a teacher — a warm one, a worried one, a years-of-history one — and that the opinion quietly bends what she notices. What she came to love is the very thing that gives this its power: the AI holds no opinion of that teacher to protect. So when she set her own read next to the AI's, she finally had a way to catch herself.

But impartial is not the same as infallible.

This is exactly where my principal was right.

An AI can tell you that one voice was talking for three minutes straight. It can't always tell you whether that was a teacher losing the room or a teacher holding it spellbound. It can flag that a piece of feedback was "generic." It can completely miss the warmth in a teacher's voice that made that plain "nice job" land like a hug. Rapport is real, and it is hard to catch in a two-minute clip.

So the move is never to take the report as a verdict. The move is to treat it as the start of a conversation. When the AI surfaces something, the best question in the room is still a human one: Does that match what I know? And if it doesn't — say so. Push back. Ask it to look again. The administrators I work with who get the most out of this are the ones who question the tool, not the ones who salute it.

One data source, never the only one.

Which brings me back to that principal's instinct. She's right that it shouldn't stand alone — and the fix isn't to trust the AI less, it's to surround it with other evidence.

One report is a snapshot. A snapshot can lie. But a snapshot lined up next to a student work sample, next to three months of your own walkthroughs, next to what you already know about that classroom — now you've got a story. Now you can ask the question that actually matters: do these sources agree? When they do, you can act with confidence. When they don't, you've just found the exact spot worth a closer look.

We say it in coaching constantly: you have to triangulate. No single measure gets to be the whole truth. AI doesn't change that rule. It just hands you one more clean, consistent data point to triangulate with.

Same question, same yardstick.

There's a second trap, and her AP named it almost as well as I could have. The minute you start comparing classrooms — or worse, comparing buildings at a district meeting — the questions you ask the AI matter enormously. If I write my own clever prompt and you write yours, we are not measuring the same thing. We are each, without quite meaning to, nudging the tool toward the answer we already wanted. As he put it: you can manipulate the data with your questions.

That's why the vetted, calibrated prompts matter so much more than the ones we improvise in the moment — the ones that have actually been tested for reliability. If a district decides, "this year we're looking hard at engagement," then everyone needs to ask engagement the same way, with the same prompt. Otherwise you walk into that principals' meeting comparing apples to oranges and calling it data. Explore freely on your own time — but when you're holding people side by side, you need a shared yardstick.

And here's the quiet gift in that consistency: when you're using AI to support regular walkthroughs, you finally have something steady to calibrate against. The tool reads every lesson by the same measure, so it becomes a fixed point your own eye can line up to — your fifth walkthrough of the week read the same way as your first.

But the best version of this I have ever watched wasn't one leader and the AI. It was these two, in the same conversation, calibrating with each other. They'd look at the same lesson and the same AI read, and then talk it through: I saw that as strong — what did you see? The AI gave them a common, neutral starting point that neither of them had a stake in, and from there they could line their judgment up with each other and with the tool. That is how a building arrives at a shared sense of what "good" actually looks like. One leader plus AI is useful. Two leaders plus AI, calibrating out loud together, is how the standard stops living in one person's head and starts being something the whole team can see the same way.

The bias you didn't see coming.

But there is one bias the AI cannot touch, and it caught all of us a little off guard when it surfaced. Its impartiality can protect you from how you read a teacher you already have feelings about. It cannot protect you from which classrooms you choose to walk into in the first place.

I've worked in plenty of buildings with that one leader who brings glorious data to every meeting — collected entirely from their three favorite classrooms. That's not a building's story. That's a highlight reel.

If we mean it when we say all students deserve strong instruction, then all means all — the music room, the art room, the class nobody likes to visit because it's hard. A tool that reads every video the same way only helps you if you're actually feeding it every kind of classroom. The technology can be impartial. Whether your sample is — that part is still on you.

You stay in the driver's seat.

So no, AI is not going to replace your judgment as an instructional leader. If anything, it does the opposite. It protects your judgment from the parts of being human that quietly skew it — the coffee, the grudge, the favorite, the Tuesday you were just in a bad mood.

But it only works if you stay in the driver's seat: question what it tells you, surround it with other evidence, calibrate with it and with each other, ask everyone the same question, and look in every room.

The tool can tell you what it saw. Knowing what you were looking for — that part is still, gloriously, yours.

Dr. Tammy Thompson Kapp is a career educator — a former central office administrator, elementary principal, and classroom teacher — who now serves as Director of Virtual Coaching at Sibme, where she partners with school leaders learning to use video and AI to strengthen instruction in their buildings. Share this with a colleague who's rethinking what their walkthroughs are really telling them.

‍