لخّصلي

dr.

dr. William Penn well is professor of
learning sciences and human development
at the University of Colorado Boulder
School of Education before joining the
faculty in Boulder in 2011 dr. pendel
worked for more than a decade as the
director of evaluation research at the
Center for technology and learning at SR
I international prior to that he worked
in the research and evaluation
departments of the San Francisco Unified
School District in the Nashville public
schools in Tennessee he is a fellow of
the American educational research
Association and his research focuses on
teacher learning and organizational
processes that shape the implementation
of educational policies such as the next
generation science standards we invited
dr. Penn weld to join us today because
he is the co-developer of an emerging
research approach called design based
implementation research or DB IR Barry
Fishman my co-instructor is also one of
the co collaborators and developers of
DB IR and so he's joined us for this
conversation
DB IR was created to serve a deep need
for educational innovations that are
scalable sustainable and equitable at
the heart of DB IR is an approach to
designing and developing innovation
called design based research and a model
for collaboration called research
practice partnerships as you consider
how gainful learning might work in your
school I want to encourage you to think
about this as a design research activity
how is gameful learning being
implemented how will you know if it is
working what does working even mean for
gameful learning in your school thank
you Bill thank you for Barry for joining
us so where did DB I are developed from
where does it come from thanks for
having me and I think there are too many
different sources in participatory forms
of research and design but there are two
main ones that Dvir comes from one is a
tradition in the learning sciences
called design based research and another
word for design based research is
building the plane while flying it - and
oh we're going to study it while we're
doing that it's basically engineering
new conditions for learning in the real
world
not in the lab but in the classroom and
studying how how that turns out
traditionally design based research is
done in a kind of hothouse environment
I think that's Barry's term actually for
DBR with just a handful of classrooms
with lots of researchers around maybe
even leading some of the classes so Dvir
pushes on DBR by saying what happens
when we do this under much more routine
kinds of conditions and in lots of
different classrooms we also draw on
implementation research to help us think
about that problem there's a long
tradition in education research focused
on the implementation of programs at
scale on organizational change and
things like that and that's a tradition
that we try to draw and theoretically to
think about well what are the kinds of
supports that we need to design to put
into place so that's something that we
really like or has a lot of promise can
be implemented at scale I would add to
that that you know when when Bill and I
were starting to talk about the work
that we were engaged in in schools and
that includes gameful learning for me
and for him a range of different
interventions around teacher learning
and science education and another
context you know we realized that a lot
of the things that we're getting built
were temporary they only lasted as long
as somebody was really taking good care
of them or as long as there was external
funding or or as long as a champion in
the school was there to care for it and
then they would mostly just vanish and
it's hard to think about a really
well-designed innovation that was
developed with the intent of being used
in schools that actually made it very
far
into real-world practice so the idea was
to rican Sep xual eyes that we went
about doing this work and if you're a
learner in this MOOC and you've been
participating with us up till now you
should have lots of good ideas about
things you want to try out in your
school and really the whole purpose of
talking with you about Dvir today is to
convince you that you should be thinking
about what you're doing around gainful
learning not just from a development or
an implementation or an experimentation
perspective
but from a research perspective with a
goal to creating something that is
designed to be sustainable from the very
beginning and for us D BIR has been a
really useful tool for doing that for
thinking about that so we thought it
would make sense to bring it to this
MOOC so tell me what kinds of problems
is dbi are best suited to solve or to
work toward a solution for there's a
couple that I think of I mean think
about if you're a teacher think about
your own situation and how many
different messages from different
sources are coming your way about what
you should be doing in your classroom
what you should be teaching how you
should be supporting student learning
how you should be assessing it well
that's a problem of incoherence right so
many of those messages don't point in
the same direction when we work in
partnership with teachers on the one
hand but also say district leaders on
the other then we can actually begin to
bring some of those elements into
coordination we can ask ourselves what
would make them cohere a little bit
better the other kind of problem we have
is an equity of implementation issue so
it often were in the case of the rich
getting richer the poor getting poorer
in terms of really cool stuff happening
in classrooms Dvir asks the question
well does everybody get access to this
where is it
where is this thing being implemented
where is it not who's benefiting from it
who is not and so Dvir in terms of the
research side collects data on them so
in order to say what do we need to do to
make this really cool innovation
powerful and effective and accessible to
everybody can either of you articulate
one example of how DB IR has made a
difference for a district or a school
yeah one of the ways that I've been
working in partnership with Denver
Public Schools since about 2007 and one
of the things that we've had that's like
I would say a really a lasting effect is
in some of our work in middle school
formative assessment what we did in this
project was funded by the National
Science Foundation is that we adapted
some curriculum materials with teachers
to improve the quality of information
they were getting back about how their
students were thinking we supported the
teachers with a lot of set of resources
for helping them orchestrate better
discussion
their students and we looked at that how
that worked and we found that it had a
positive impact and though a lot of the
parts of the inner the intervention and
the stuff that was co.design they're not
being implemented today one of the
things that really stuck was we help
teachers develop and we actually
developed it with teachers this big
poster that had a set of norms for
classroom discussion and that norms
poster was something that a lot of them
put up on the wall and they continued to
use after the project ended in fact just
last fall I talked to a teacher but I
was doing professional development in
the district with and she says I still
have my norms closer I'd use that every
day and I refer to it in my science
classroom and you know that's really the
result of a partnership that's long term
that involve teachers as Co designers
and I'm really happy that we have that
impact even if a lot of those specific
details of that intervention are no
longer there so I want to add another
part of that story and to go back to the
four key principles of B bi are that I
described in the introductory video to
this segment of the MOOC the first
principle is negotiating a shared
problem of practice and that means
having practitioners researchers
different kinds of partners around the
table classroom teachers school
administrators who's responsible really
for seeing this problem all the way
through and even more importantly what
problem are you trying to solve because
it could well be that the problem that a
school administrator thinks they're
trying to solve is different than the
problem the classroom teacher thinks
they're trying to solve and if you bring
in a researcher as a partner they may
have a completely different set of
problems that they think they're trying
to solve and that conversation helps you
get a good way down the road to
designing something that people want and
having something that people want is a
key to having something that is going to
be sustainable and scalable so even when
you're talking about gameful design and
you're thinking about it from an
administrative or school leadership
standpoint and you're thinking about
from a classroom standpoint it's
important to have that negotiation to
make sure you're starting on the same
page with what the goals are so bill you
talked about how teachers are hearing
all sorts of conversation around them
about what practices
they should be implementing one of those
buzz words or terms that we hear is
personalized learning so for those
institutions or schools or teachers that
are interested in personalized learning
how might a DB IR approach be useful for
them well one of the things that we
really emphasize in DB IR is thinking
about the learner as embedded in this
big ecosystem of supports so decentering
the classroom but actually saying what
if we were to design a whole ecology of
learning supports for a young person
around their interests that included
school opportunities but also we were
worried a little bit about well what
about what's happening in the community
and what's happening at home and how do
we help that young person level up over
time meaning get better and challenge
themselves in an area of their own
interest and there's a Dvir project that
I'm involved in with my colleague Nicole
Pinkard from DePaul University called
the Chicago city of learning and this is
a huge partnership that includes the
schools and over a hundred and fifty
different community agencies in Chicago
that have agreed to put all of the
opportunities that they provide young
people onto a digital platform where
young people can find them and also they
can earn digital badges for
accomplishments with them and part of
what the goal of that one that the Dvir
effort is is to try to over time reduce
the friction that's involved in between
like an organization getting onto this
platform and also make it easier for
young people to find accessible
opportunities and the research that
we're doing and cu-boulder for Cecil is
really focused on mapping those
opportunities the diversity of those
opportunities and the accessibility
given what neighborhood in Chicago that
you live in and and so that's one of
those things around personalized
learning that I think is a good Dvir
problem but it really begins with
re-centering thinking about learning
from a person in this broader ecology
you glad you brought up that example
bill because it turns out we'll be
talking with Nicole about the Chicago
city of learning as a case around
connected learning and badges which I
think are important ways you can frame
and think about gainful learning
that's great so thinking about the
research component and the the
partnership component how do districts
or individual instructors go about
forming those research practice
partnerships finding research partners
to engage in a Dvir project that's a
great question so the first thing is
don't just look at universities like I
Barry found me at a research institute
called SSRI and international there's
lots of research and nonprofits that
might be of help there's a read there
you're every region in the country has a
Regional Education lab that's funded
through the US Department of Education
and you can of course go to University
websites to the School of Education
faculty but they also don't just look at
School of Education faculty part of Dvir
problems or problems of that
organizational change in innovation you
might find faculty to and your computer
science or information Sciences
Department so you can find these folks
around the university when the first
thing is is look at their webpages and
say see do they study anything remotely
to the problems that I care about and
don't hesitate to email them and say
look you know I was on your website I've
got this you know got this problem a
practice or this thing that I'm trying
to do and I'm really interested in
hearing more about what you've done and
you know researchers are really
interested when people show interest in
their work and part of that though is
that begins this dance of negotiation
finding something of mutual interest
going to a researcher as opposed to an
evaluator when you already know exactly
what you want maybe that's not the time
for a partnership but if you're actually
want a partner this long term maybe
you're willing to be shaped by their
ideas that's when you go and approach a
researcher about you know hey maybe we
could do some work together there's a
subtext there I think which is as a
school leader or as a school teacher
don't feel like you have to make it all
happen on your own they're actually
you're lots of useful resources you can
draw from to help you think about this
problem and help you both with the
implementation and the documentation of
the implementation but we're suggesting
something different than the truth
additional relationship between schools
and people like Bill and I which would
be a consulting relationship that's not
deep enough we think you can go for a
partnership which is something else
entirely with a very different set of
ground rules for what that what that
relationship looks like and we have some
great resources on a learn Dvir website
for starting partnerships and that is if
you're coming at it from being a school
teacher or a school leader and we also
have something on the research and
practice org website called the RPP
toolkit and I'd really recommend that
folks who are interested in this check
out those resources because they're
really intended to help with the
practical guidance of beginning and
forming a research practice partnership
and both of those websites will be in
the resources section for this module
within the MOOC so thanks for
referencing them both um so what's the
role of evaluation in all of this in
particular in innovation it's hugely
important we got to remember that when
we innovate it's really a bet on the
future and it's often not a bet on our
future but on some group of students
future and so one of the ways that we
really test our bets is to do research
we need to see is this really an
improvement a thing that we think is so
great is it really working or not and
for whom is it working and under what
conditions does it work and what kinds
of supports do we need to make it work
those are and that latter question is
really the quintessential Dvir question
which is what do we need to have in
place to make this work for lots and
lots of people lots of different
classrooms and different conditions so
to me research is hugely important and
it's really our obligation as innovators
to engage in research to sort of put our
own theories in harm's way so when
thinking about evaluation what level of
change or degree of change warrants an
evaluation if a school or district is
just adopting a tool or a particular
practice does it make sense to establish
an evaluation plan yeah but it really
depends on where you are in the phase of
development of this thing so let's say
you're adopting a
it's been tested it's well developed and
you're trying it for the first time in
your home district or school that that
one probably has some research
instruments that you could use and
borrow it's got some expectations
regarding what good implementation looks
like by all means evaluate and try to
see which of those instruments you can
use but when we're designing our own
homegrown innovation one should move
pretty carefully into the evaluation
process the first thing that you kit you
want to see is some changes in classroom
practice along the lines that you hope
and that means that you've got to have
some ideas about what you hope and often
times when we really get started we're
still kind of fumbling in the dark we
have some broad goals but we actually
need to get pretty refined over time
about what is what are the active
ingredients in our innovation what's
really making a difference for student
learning and so doing a big full-blown
evaluation before you know what that is
is actually not a good investment in my
view so you want to start building those
in those instruments an evaluation plan
along with the innovation as it starts
to get a little bit more specified the
one footnote on these ideas is that you
can't improve what you can't measure
that's right yeah and so there's a whole
model of improvement science that says
well actually we ought to be really
clear and right from the start about
what our outcomes are and what our
benchmarks are and start measuring those
right away but even an improvement
science you only start with one
classroom and typically we're stuck with
okay we want to do this district wide or
school wide in our school districts and
in schools and so if those are your
conditions you got to think about being
cautious about you know getting a big
evaluation going before you're ready
ready so I was very I think essentially
asked the question I was is that
evaluation itself is a design process
not something that just happens at the
end or that you start doing right away
but you have to be thoughtful about how
you are going to approach that question
of evaluating an innovation yeah one of
the things that you focus on
a lot early in the phases of
implementation is this eye part so what
is it that's going on what have we
notice about how implementation varies
from the Gecko and what do we notice
about maybe the undocumented work that
the research team or somebody on the
district is actually doing just to pull
this thing off because guess what if you
want this thing to sustain itself
somebody else is going to have to do
that work ultimately so you better know
what the work is and that's something
you can be studying right from the
get-go in terms of implementation is
what is the work that is required for us
to pull this thing off even in a handful
of classrooms right so there's another
concept in DB IR that I want to hear
your thoughts on we often think of
infrastructure as plumbing or you know
the bricks and mortar literally of a
school how does infrastructure fit in to
technological innovations in the
education space
well it's huge but I think I want to
turn the tables on my colleague Barry
and ask him that question like so you're
trying to MIT bring about gainful
learning at the scale of a university
pretty much what are the kinds of
infrastructure is you're building and
thinking about for that scale of the
university the scale of a k12 school
system the scale of the education system
what we're getting at with this notion
of infrastructures or infrastructure
ring in DB IR is a deeper more social
science way of thinking about
infrastructures which are the the
cultural norms and practices the the
policies the rules the the technological
support systems that undergird our
everyday practice so in a very simple
way you can think about the
infrastructure that something like the
gradebook provides for a classroom where
the way that gradebook is set up
establishes some norms about how it is
we might go about doing assessment so
typical grade books in a typical
classroom might have a 100 percent total
and you are having assignments that are
graded and the students grade keeps
changing based on what percentage they
got on this latest assignment and what
percent of that
Simon counts toward the final grade you
might think about a different
infrastructure where things are outcomes
or mastery oriented and so what you have
is a list of criteria that you want to
see students demonstrate proficiency or
mastery on and as students make progress
and do different things you're checking
off and entering the evidence people
might have that show a student has met
that level of mastery on that particular
criteria or standard that you care about
in gainful learning we're talking about
a really broad change in how we think
about assessment so in order for that
assessment thinking to take hold
you've got to change a lot of the ways
the classroom the learner the learners
parents the school system the
administrators the teachers around you
think about what it means to learn and
what the goals of the classroom are so a
very concrete example many many classes
especially in higher education but often
in secondary schools are graded on the
curve so the notion of a curve is the
idea that we're gonna define rigor in
this learning environment by by limiting
the number of people that can get a's
and sometimes that is brings down the
class so maybe too many people would
have gotten a so we apply a curve
sometimes it raises the entire class
average up well nobody would have gotten
an A but we're going to make sure that
people get a's and we think about how
rigorous or difficult of learning
experiences by the great outcome so when
you hear people complaining for instance
about grade inflation what they're often
saying is p it seems to me that college
is too easy these days but if we change
the frame and we make it conceivable
that everybody could earn an A but
really earn it it was a rigorous
difficult learning experience you had to
show some serious
perseverance and demonstrate your
knowledge and your learning of that set
of skills or practices or knowledge in a
very rigorous way then we would say that
if everybody got a is we must have done
a great job of being a teacher and a
supporter and of designing our learning
environment so it's a it's a change of
the frame that it's really a change of
infrastructure and how we think about
regulate measure
decree what it means for people to be
engaged in a certain kind of learning
environment I think very you're talking
about change and when I think about
infrastructure
you've got to wrestle with what's
already there so your kids are used to
this great economy where there's a curve
they're not really necessarily going to
embrace mastery learning off off the bat
so what is infrastructure around that
stuff look like isn't that actually part
of the infrastructure you have to do as
well absolutely so it it would include
professional development for teachers to
help them think about what this change
looks like it includes thinking about
lots of ways to help teachers change
their language the way they communicate
in class to make it clear that progress
toward a goal is the way this class is
organized not just giving you a set of
problems and seeing how many you get
right that you know this is a real
perspective change but it's also in this
language where we're asking you think
about it as a kind of infrastructure a
social infrastructure for thinking about
assessment what assessment means how
assessment gets communicated so for
instance what we're talking about this
at the university level we think about
what's the role of the Registrar because
the Registrar is the policy implementer
of grapes they are the ones who say to
everybody else in the world this student
got these grades and I validate these
grades this is what they I can say that
they're they're trustworthy
well admissions offices do the same
thing with high school grades so if
we're starting to talk about changing
what a grade might mean then we need to
talk about changing the infrastructure
that would make it possible for students
who succeed in this new infrastructure
to being able to communicate that
success that in a good way to people at
the next level and really assessment
systems are just trust chains and the
infrastructure change we're talking
about the gameful learning is a pretty
big shift in that trust economy there
are also mistrust chains I would add to
in the k-12 setting you know and just to
add in a k-12 perspective we work in
Denver right now a lot with building
curriculum with teachers so we're build
a set of biology and it's a line of the
next gen science standards and what
infrastructure alike in there is that
we're also working with teachers to
redesign and the district leaders to
redesign the district and of course
assessments we're also working on the
pacing guides with them all those things
that buzzing confusion of things that
affect what a teacher does in the
classroom we're trying to get those
things in to alignment with one another
so that our curriculum has a chance of
being well and being implemented by
teachers across a wide range of schools
so if you were to have one piece of
advice that you wanted to share with a
change agent in a school whether that's
an administrator or an instructor who
wanted to pursue Dvir what's your one
key piece of advice or the first thing
that you might say to them take your
time in finding a good partner finding a
good partner is a long-term endeavor and
a good research partner can both help
you with your work but also help you
with work that you didn't know was
possible for improving teaching and
learning in your schools
anything else Barry I think I would add
that it's important to have a mindset
towards research and development that
this is not something you're going to
take in and plug into your school that
is a process that's a design process as
you said in the beginning Rachel and vbi
are is we feel a very healthy way to
approach that design process in a way
that is more likely to lead to something
that's sustainable and that will be
useful to all the members of your school
community great thank you bill Thank You
Barry for sharing your insights and
expertise on how school leaders and
change agents can go about thinking how
to structure their work most effectively
take care thank you for having me thank
you thank you thank you Rachel