(Corrinne) Alrighty. It is about 2:00, so we can get started with the event.
So, good afternoon. I am Corrinne Stull with the Center for Distributed Learning,
and I would like to welcome you to "Showcasing Student Learning Gains,"
part two of UCF's Dziuban Award Workshop Series.
The Chuck D. Dzuiban Award for Excellence in Online Teaching
recognizes one outstanding UCF faculty member who has recently taught
a high quality online course.
This workshop series is designed to assist eligible faculty in preparing a strong
award submission packet.
We also hope to provide online learning professionals with tools and information
that will help them build quality online courses.
I would like to acknowledge the onsite participants with us here—
thank you for coming—as well as those participating online via YouTube Live.
Our online participants are in good hands with Amanda Major and Kathleen Bastedo
as our online moderators, as well as Arianna and Bren ensuring
technical quality.
After the presentation, we will have a short question and answer session,
and with that, please join me in welcoming today's presenters Dr. Amanda Groff
and Richard Brunson.
Take it away!
(Amanda) Yay! (laughter)
(applause)
So, thank you, Corrinne, for the introduction.
I'm Amanda Groff, and what this all is about today is trying to demonstrate
for you guys the changes that are occurring with the Chuck Dziuban Award.
I was an award winner back in 2012, and I've served on the committee
ever since then, and we decided that we needed to make some changes
to the award, and one of the things that we're going to be looking at is how
faculty members showcase student learning gains in their courses.
So, we want to go through and make sure that everybody understands
what student learning gains are, and then Rick will take over and he's going
to give you a few examples of some of the ways in which he actually assesses
student learning gains and some of the different varieties or examples
of ways we can do this in our courses.
So, in doing research on student learning gains, I found a couple of definitions
that I felt were pretty good at defining what a learning gain is.
Both of these definitions come from educational groups in the UK.
So, they're really at the forefront of trying to define student learning gains
in the ways in which we can define them in our own courses.
So, the first definition is
Another definition, which personally is my favorite, is
I feel that this second definition makes it a little bit more easy to conceive
what a student learning gain is because there's a beginning and there's an end.
So, it helps us to know what's happening at the start and what happened at the end
of a particular point in time or period of time within a student's education
or beyond.
So, again, what are student learning gains?
It's the actual change in performance between two points in time.
Now, the scale of these points in times can vary.
So, you might be fond of looking at a semester.
So, the beginning of a semester and what a student learns throughout a semester
and at the end, what they gained over that period of time.
Or it could even be on a larger scale.
This could be looking at those same kinds of things at a semester
but comparing semesters.
It can be a variety of different ways in which you can actually learn
and understand what students are learning in your courses.
The main purpose of it is to identify improvement and to identify what
you're doing in your class that actually creates that improvement
and taking what you've learned from that and using it and applying it
to other courses or other areas of study that you might be working on,
but the thing I want to point out— and Rick will also reiterate this as well—
is that learning gains are inevitably discipline-specific.
We are all from different departments. We'e all going to have different ways
in which we assess our students.
So, just because maybe we emphasize a particular way of doing an assessment
in order to identify learning gains does not necessarily mean
it's going to work for you.
We're giving you some ideas and you can use those as you see fit
and what might be applicable to your courses.
So, keep in mind as we go through the presentation that there are ways
to think about how to make it fit within your particular discipline.
So, importantly, what do student learning gains tell us?
Well, it tells us how students benefit from their time in classes rather than
their experiences, backgrounds, or preexisting skills and abilities.
So, it's literally just looking at what a student is learning in a class.
This can be in two different ways. You can look at it at the individual level.
So, if you're concerning yourself with one particular student in a class,
you can see how they were at the beginning of the semester, how they changed through
the semester, and what they learned by the end.
Or you can do it on a course-wide level. You can see the overall course average
at a particular beginning point and then how students adapted, learned, and applied
those skills by the end of the semester.
So, it really is up to you how you want to demonstrate those student learning gains.
So, again, you can look individually or course-wide, but it allows
for comparative judgements.
So, the first time you look at student learning gains, it's going to be
that piece of information, but then you have that in your back pocket
so when you try it again in a future class, you have that ability to make those
comparative judgements to see, "Oh yeah! Look! I'm seeing a trend here.
"This looks like students are responding to this kind of activity.
"I'm going to continue with this forward for future classes."
So, it helps you to eliminate what's working and what's not working.
So, effectively what a learning gain is is the effectiveness of pedagogy
and quality of education that a student is getting from your course,
and that's basically what it is.
Now, when we talk about student learning gains, inevitably there are some
misconceptions that come with them, and so we have to set some of these
aside in order to better understand what our students are learning.
Pardon me.
Thanks.
So, the first misconception is
I'm going to tell you right now they do not.
You need to go beyond that because student evaluations are basically not measurable.
They are not a measure of you as a teacher.
Sure, they're giving their opinion of what they think of you and how you taught
the class, but it is not a way to measure what they've learned because—
I'm sure we've all been there—students like to use that as a platform to complain
about us or make some sort of commentary.
We've all been there.
So, they're not really a good source in that way for measuring.
Now, the amount of what a student has learned in the class and the student's
opinion of that particular teacher and their effectiveness of teaching
do correlate.
They do go hand-in-hand, but they're not identical in any sort of way.
So, you have to think of them separately. So, you have to separate yourself
from that idea that my evaluations are going to suffice to demonstrate
student learning gains because they don't.
The second misconception is that
That's not true.
Now, using pretests and posttests should not be overstated but they should not be
understated as well.
They are valuable in the way that they can...
Basically, you can look at a student. You can see how they're performing
at the beginning of the semester, see how they've performed by the end,
and if there's an increase, that's still telling you valuable information.
So, we have to also set aside this idea that pretests and posttests
aren't that useful when they actually are.
They're quite useful and obviously the easiest way to do it.
The third misconception is that
Now, this is true and untrue. It depends on you and what you want to do
and what tools and assessments you want to use in your course.
So, it inevitably is up to you and how you want to dedicate your time to assessing
student learning gains in a class.
It's nothing that we're going to push on you.
It's what you feel would fit with your course.
So, that's as much time as you want to put into it or not.
So, we also have to remember that it's not as much time as you might think it is,
and it is very, very valuable to actually assess in your class.
So, what we're going to do now is make a transition
to the different types of assessments that students can participate in for us
to see these learning gains.
So, I'm going to pass it off to Rick.
(Rick) Thank you, Amanda. Thank you everybody for being here.
Those of you who are watching online, thanks for joining us. So...
First of all, Amanda's the best person in the world to be on a project on.
So, any time you see the words Amanda Groff and you can be involved,
jump in because she's a great partner.
So, how do we define and design assessments that show learning gains?
This is not the face of digital innovation.
I'm a 57-year-old grandfather who came to academia—if you'll pardon the
sports metaphor—on the backside of life. Okay?
The back nine. I came to teaching. So, I was a reporter
and an editor for twenty years,
and got to come back to the University where I got a degree studying journalism.
So, I was very, very fortunate, but I had to learn to teach what I knew.
I had subject knowledge but didn't necessarily have the knowledge
for how to deliver that in a way where students actually learned.
So, a person that I really found helpful in her books was Robyn Jackson who teaches
at Towson University, and her area of concentration is primarily K-12,
but a lot of what she has to say about assessment and designing them
I found applicable to what I was teaching in journalism.
So, Robyn puts forward that you have to figure out how you're going to define
a learning gain and then you have to design an assessment that actually
measures that and shows that.
What do we want students to know? How do we assess that they know it?
And so, I'm going to show you some examples of that today that we use
in our program at the Nicholson School in the journalism program.
So, as Amanda said, assessing student learning gains is highly
discipline-specific.
So, what I hope is that what I share today, even if they don't seamlessly
apply to your particular program, you can go back with your colleagues
and figure out some ways to do this.
That picture, by the way, is called a dronie.
You've heard of a selfie?
So, that's a dronie. (laughter)
We teach teaching and reporting news across platforms and one day we did
a drone workshop, so we decided to take a dronie.
Two courses that I'm going to look at today and present to you that are both online—
JOU2100 News Reporting is a W course. It's our cornerstone course in the major
where we teach our students the basic skills of news reporting.
Everybody has to take it.
We offer three to four sections of it a semester, and then the other course,
and because I think it's important and I want to hark back to what Amanda said,
measuring learning gains is about fixed points in time.
Right? So, 2100 News Reporting, the basic course happens at the beginning
of their sojourn into the major, and then JOU4941 is our internship course.
It's fully online. It's a W course.
2100 is a prerequisite for the internship course.
80% of our students complete at least one internship.
Most of them complete two to three before they graduate.
The record is by a young woman who did seven internships, and today right now
she's working at the Washington Post .
So, it comes up to the ambition of the student, but we're going to look at
these two courses because they're a great place to check that fixed-in time:
beginning and farther down the line.
So, what are we looking for?
One of the pieces that Amanda said was knowledge.
So, how do we measure knowledge?
In the journalism program, one of the key areas of knowledge is our Bible,
the Associated Press Stylebook, and it's not just a book about language conventions
such as spelling and capitalization and punctuation, although that's in here,
but it's a reference point for all the institutions that we end up covering
in journalism: law, government, business, the economy, food, and fashion.
You learn in here why French in the word french fry should be lower-cased
because it's a style of cut.
It is not the country where french fries originate.
So, students have to immerse themselves in this guidebook, and we measure
their knowledge of it.
So, we have internal assessments, and we have external assessments.
Internally, we use class exercises, right?
Externally, we use a 15-point survey that we send to every internship supervisor
that measures student learning and performance in the areas that
we're teaching in JOU2100.
So, internally, we can see how they're doing in class.
Externally, we can see how they're doing out there in the field once they leave
class, measuring what they learned and how they're applying it.
So, on AP Style, we look for 90% of our students achieving a passing score
of 75% on a post-test.
So, in 2100, we pretest and posttest on Associated Press Style.
Externally, we want 90% of our internship supervisors to express agreement
with the statement, "The intern demonstrates a good command
of grammar, punctuation, and AP Style," and we don't always meet that 90%.
This last year in assessment period, our students came up a little bit short.
So, what does that tell us? That tells us we need to go back and look at how
we're teaching AP Style in 2100 and what do we need to do to meet that standard
that we've set for ourselves?
Skill.
Another area where we want to measure learning gains.
So, journalists have to conduct research.
They interview eye-witnesses/experts.
They gather data from public records.
They go to a scene and make direct observation, right?
So, how do we measure that?
We have an external assessment.
When they complete their internship where the supervisor, the producer, or the editor
measures their ability to conduct research as a skill and that's how we measure that.
Another skill is accuracy, getting your facts right.
Internally, we require that students submit a portfolio of fifteen pieces
of published work.
The faculty reviews that and uses a rubric to measure for accuracy.
We also include accuracy as part of the external assessment that
internship supervisors do.
Work readiness.
In JOU2100, we complete lots of timed exercises that are 25 minutes or less
where they're given a press release or a fact sheet, and then they have to
turn around on the spot in the lab or online and turn in a story,
be it for online, be it for radio, be it for television, in less than 25 minutes
to a high degree of accuracy and completion.
So, internally, we have that exercise in 2100, and then, externally, internship
supervisors out there in the field measure on the same thing.
So, we can see how they did in class, and then how well did that translate
into how they did in their internship?
The young woman in this photo is highly appropriate.
Her name is Anna Espinosa. I had her in 2100 in the spring and then that
following summer, she got an internship at WDBO 96.5 here in town.
So, in the spring she's having to write radio wraps, that summer she's doing it
for real and voicing them on air.
So, I was able, in her case particularly, to see how she did in the spring
with this area of skill and work readiness and then how she did in her internship.
Another way we measure work readiness are reflection essays and site visits.
So, students how take the internship course are required to every week—
or every other week depending on the professor—file a reflection paper
where they're specifically talking about newly acquired skills and how they are
applying skills that they were taught in 2100, and then we also visit the site
where they're interning and ask their supervisors the same thing.
We want to know.
What are we teaching them in class and how is it translating,
the way they're applying it and using it in the field?
Professional and personal development.
So, we assess this internally.
In 2100, there's a part of their score in their overall course grade
that measures their professional development.
Things like the quality, the quantity of their work. Are they productive?
Did they turn in all of their assignments? Initiative, collegiality, creativity,
attitude, adaptability, and flexibility.
Really hugely important in the news business where things happen not
on schedule, right?
That you have to run out and be able to cover.
Externally, we use the internship supervisor assessment to rate
these same things, these same qualities.
And that's all that I have.
Any questions?
(Corrinne) So, thank you very much, Amanda and Richard.
So, we can move into the question and answer part of this seminar.
I guess if any of our onsite participants have any questions, we can address
those first, and then we can move to our online audience.
(Participant) You mentioned reflective essays. Can you just comment on length,
level of detail, timing?
(Rick) Great. So, we normally ask them to target it to about 300 words and include
links to any of their work that they produced that week onsite,
and then we read it, and then we review their work that they produced,
and then we give them feedback on it.
Typically, we shoot for within a week.
(Amanda) And in the anthropology department we often also
use reflective essays.
For example, for myself, in my courses, ours are more geared to what do you know.
As opposed to giving them a pre- and posttest quiz, we give them a pre-
and post-reflective essay. So, what did they know?
I throw terms at them that I know they won't be familiar with.
I ask them not to research them online or anything of the sort and tell me what
they think they know about it, and by the end of the semester, I ask them
the same questions, maybe a little bit more detailed, as a way of measuring
what they've learned through the semester, and we do it in class or I attempt to do
it online with the hopes that they're actually not looking at a book
or anything, but it works really well in the face-to-face class and I don't prepare
them for it.
I surprise them with it so that they don't study or they don't try to learn
those things beforehand.
(Corrinne) Other questions in the room here?
(Melody) So, how much do you distinguish looking at the learning gains of one
individual student at the beginning to the end versus a kind of aggregated
approach, and how do you differentiate the impact that has on your programming
and teaching?
(Rick) So, in 2100, that's a pretty high bar when we say 90% will meet
a score of 75% or higher.
75% is like a mid-range C, so anybody in the room can reach that with fairly
common rigorous personal study habits, right?
(laughter)
If a student doesn't meet that measure, especially in the pretest, it usually
requires calling them in to some office hours or some online...not remediation,
but trying to figure out, help them spot- check where their weakness are.
If it's AP Style, are you struggling with the language side of it or just
your background frankly in civics isn't that good, so when we talk about
the federal court system and the difference between a municipal
police department and a county sheriff's office and that draws a blank,
we may have to do some extra coaching and mentoring there.
(Amanda) And Melody, what I tend to do is not look at an individual because my online
classes tend to be quite large so it gets a little overwhelming when I would have
to look at individual level, so I tend to parcel them out into my transfer students
vs. my completely 100% online students vs. those traditional students
that come to campus or not, and then it's easier to identify because I know
that transfer students tend to struggle a little bit more in online courses,
so I tend to look at those more closely to see what they're struggling with
and then reach out to them.
(Rick) That's an excellent point. I mean, we're at UCF which is super
growth university, right?
So, 2100 caps at 20, and so, it's a lot easier.
(laughter)
That's why we teach four sections of it. It caps at 20.
It's a lot easier for me to intervene earlier than maybe in a class
with 350 students.
(Amanda) Yeah, it's worth considering the size of your course and how
you're going to look at student learning gains, for sure.
That's going to have a huge impact on what you do.
(Corrinne) Okay, I think we have one more onsite question and then we will
address the online questions.
(Participant) Thank you. I just have a quick follow-up question on the
reflective essays. Have either of you attempted to quantify those gains
through a rubric or something or is it more just a reflective exercise?
(Amanda) For me, it's hard to do.
My teaching assistants will help me. I'm very fortunate that I have their assistance.
Where it's looking at what they originally did and it's making my own judgement call
in that way.
A rubric would definitely be a positive way to be able to quantify that
a little bit better so that we can actually use it in such a way.
So, it's one of those things where it's what you're asking I think will be
very dependent on how you evaluate what they're learning.
So, I think producing some way of evaluating those student learning gains
through reflective essays because they are free-write would be extremely useful.
(Rick) Yeah, we don't quantify the reflective essays, and that kind of segues
to another point I like to make.
You know, all of these measures that you saw today weren't anything that
I designed myself.
It was a collaborative process among my faculty colleagues.
So, my program coordinator, Dr. Steve Collins, and my other colleagues and I
tried to identify specific things that we can measure.
We offer way more than two courses.
In the journalism major, these are two that have vehicles and mechanisms
that we can measure over time and tease out those learning gains.
So, working with your colleagues and the faculty to do this together rather than
try to figure out how to just do this yourself can be really helpful.
(Corrinne) Okay, so we have a question. It's kind of a multi-part question here
for you guys from online.
So, Paula Prior asks,
(laughter) (Rick) Oh wow.
(Amanda) Could you read the first part of the question again?
(Corrinne) How might you apply something like this to a Research Methods course?
And the other explicit question is how can I assess them better for the students
who missed the formative steps earlier? (Amanda) Oh geez.
So, does she identify what [course] it is?
Is it an upper-level graduate course or just simply undergrad?
I'm assuming graduate.
(Corrinne) She did not say but she may answer.
(Amanda) It could go either way.
(Rick) So, if I'm hearing the question right, the whole idea of measuring
two points in time doesn't work if the students don't really kick it into gear
until mid-semester. (Amanda) Right.
(Corrinne) She has followed up with it is a senior-level course.
(Amanda) Okay, it's almost worthwhile, because, you know, with Research Methods
they're developing the skills through the beginning part of the semester
and then they're starting their projects.
It might be worthwhile having something earlier on in the semester that is due
that can help be the beginning point of a measure, and a reflective essay
might work in a situation like that where at the beginning of the semester,
you can address things like what research skills do you bring to this class?
Have you ever participated in research? That way they're actively starting to
engage and think about it even though they might not physically start working
on it until mid-semester. That's always a problem in any class.
So, that might be a way to at least get them going.
Start at the beginning, have something due.
Would you say, Rick?
(Rick) Yeah, and beyond reflective essay, I'm wondering if there's things
in a Research Methods class you could pretest and posttest.
Is there a discrete set of skills or knowledge related to Research
Methods that you could pretest on early and then test them again on the backend?
It might even—knowing that there's some pretesting—make them a little more
accountable on the front end and maybe motivate them to start work a little earlier.
(Amanda) You could even gear it towards whatever the research final project is
that they wait to work on.
Gear that initial beginning point towards the end goal of having them finish something.
It could even be time management questions, too, you know?
You could find some sort of way to kick them into gear.
(Corrinne) So, awesome. Thank you.
I'm not sure if we have time for one more question.
I think Kathleen has another one who may be online.
(Kathleen) Hopefully it's a quick question.
So, I notice Rick that your department does a lot of internships,
which is really nice and gives them real world experience.
Depending on what you're teaching, you may not have that option.
Do you have another idea if you can't do real-world internships?
(Rick) A practicum! Can you develop a practicum of some type where
they're applying skills, knowledge, work readiness, personal development
in some kind of way?
Or an independent study of some kind?
(Amanda) I mean, this could even be a lab.
You can have some sort of lab. You know, science-y courses, you could have a lab
that builds up some of those seeing what they know for skills and things like that
because our department doesn't have internships, so we find different
ways to adapt and try to figure out that kind of information in different ways,
and we all do things a little differently.
Even though we're the same discipline, every class does something a little bit
different and we all have different ways of trying to measure and understand
these kinds of things.
(Corrinne) So, we are just about out of time, but if you would like to stick
around for a few more questions, feel free and we will keep the live stream going,
or you are welcome to stay onsite, as well.
But if there are no additional questions, we can conclude this workshop.
Keep an eye out for a follow-up email with a link to the session,
a recording of the session, as well as additional resources, and there will also
be a survey feedback link in there.
So, thank you again to our wonderful presenters.
(applause) (Rick) Thank you. Thank you. Thank you very much.
(Corrinne) So, thank you and have a wonderful day.