Schlechte Unterrichtsbeurteilungen für innovativen Unterricht?
“Mit viel Engagement habe ich meine Lehrveranstaltung nach neuesten didaktischen Erkenntnissen umgestaltet. Sei es im Flipped Classroom oder mit erhöhtem Einsatz von Clicker-Fragen, die Studierenden waren während der Präsenz aktiv gefordert und haben auch gerne mitgemacht. Doch mit der Unterrichtsbeurteilung kam die grosse Enttäuschung. Die Studierenden bewerten mich und meinen Unterricht deutlich schlechter als vorher. Sie bevorzugen sogar Frontalunterricht, denn damit würden sie besser lernen. Habe ich etwas falsch gemacht? Soll ich wieder zurück zu meiner altbewährten Vorlesung?”
Was hier wie ein
Einzelfall klingt, ist doch sehr verbreitet. Zahlreiche Untersuchungen
dokumentieren, dass Unterrichtsformen mit Lerner zentrierten Methoden häufig zu
schlechten Evaluationsergebnissen führen (z.B. Seidel, 2013). In einer kürzlich publizierten Studie haben
Louis Deslauriers und Kollegen/innen der Harvard University genau diese
Problematik untersucht (Deslauriers,
2019). Sie gingen der Frage nach, ob es bei aktiv involvierten Studierenden
eine Diskrepanz zwischen dem tatsächlichen und dem empfundenen Lerngewinn gibt.
Falls Studierende den Eindruck haben, weniger als in einer Vorlesung gelernt zu
haben, dann führt dies zwangsläufig zu schlechteren Evaluationsergebnissen. Die
experimentell angelegte Untersuchung bestätigte die negative Korrelation
zwischen der Selbsteinschätzung und dem effektivem Lerngewinn. Folgende drei Gründe
sind dafür verantwortlich:
Interaktive Unterrichtsmethoden verlangen eine erhöhte kognitive Leistung, die von Studierenden dann nicht unbedingt mit dem Lernen in Verbindung gesetzt wird.
Insbesondere Studienanfänger/innen verfügen noch nicht über die Fähigkeit, ihr eigenes Wissen in einem neuen Fachgebiet korrekt einzuschätzen.
Die klare und sprachgewandte Präsentation beim Frontalunterricht verleitet Studierende dazu, ihr eigenes Verständnis in Vorlesungen deutlich zu überschätzen.
Aufgrund der Ergebnisse einer Nachuntersuchung schlagen Delauriers und Kollegen/innen einige recht simple Massnahmen vor, um dieses Missverhältnis zwischen gefühltem und tatsächlichem Lerngewinn zu unterbinden. Zentral dabei ist, die Befürchtungen und Ängste der Studierenden ernst zu nehmen und sie zu thematisieren. So kann eine kurze Darlegung der Lernvorteile von aktivem Unterricht (z.B. Freeman, 2014) während der ersten Unterrichtsstunde bereits erste Befürchtungen auffangen. Aber auch im weiteren Verlauf der Veranstaltung sollte immer wieder auf den erzielten Lernfortschritt hingewiesen werden. Damit erlangen die Studierende eine bessere Einschätzung ihres eigenen Lerngewinns. Hilfreich ist zudem, auf die Gefahr der Lernillusion bei eloquenter Redegewandtheit des Dozierenden in Vorlesungen hinzuweisen (z.B. Toftness, 2018).
Auch an der ETH konnten wir den Einfluss dieser Massnahmen bestätigen. In einer Studie am Departement Physik verglichen wir den Lerngewinn zwischen interaktivem Unterricht und Vorlesung. Bereits in der ersten Lerneinheit wiesen wir die interaktive Gruppe ausführlich auf die positiven Auswirkungen des interaktiven Unterrichts hin. Zusätzlich wurden Unsicherheiten bezüglich des eigenen Lerngewinns im Semester kontinuierlich thematisiert. Bei der Unterrichtsevaluation konnten wir daraufhin kein Missverhältnis zwischen effektiver und geschätzter Lernleistung feststellen. Studierende im interaktiven Unterrichtsformat erzielten einen höheren Lerngewinn und gaben signifikant bessere Werte bezüglich ihres eigenen Lernens an als jene in der parallel durchgeführten Vorlesung (Schiltz, 2018).
Tipp: Was hier jetzt speziell für interaktive Unterrichtsformen gilt, lässt sich sicher auch auf jeden anderen Wechsel der Lernform übertragen. Insbesondere wenn die neue Lernform noch nicht geläufig ist, sollte man die anfänglichen Bedenken der Studierenden ernst nehmen und ihnen klar vermitteln, welchen Nutzen sie vom Wechsel zu erwarten haben. Daneben ist es wichtig, Ergebnisse der Unterrichtsbeurteilung (ob gute oder schlechte) kritisch zu hinterfragen. Nicht immer ist der kausale Zusammenhang zwischen studentischer Zufriedenheit und tatsächlichem Lernerfolg gegeben (z.B. Carpenter, 2020).
Case Study – Peer Review Mastering Digital Business Models
As part of a series of case studies, staff
at LET sat down to have a conversation with Prof. Elgar Fleisch, Johannes
Hübner and Dominik Bilgeri from the Department of Management, Technology, and
Economics (D-MTEC) to discuss their Mastering Digital Business Model (MDBM)
course.
What is the project about?
In this Mastering Digital Business Model
(MDBM) course, Prof. Elgar Fleisch, Dominik Bilgeri, George Boateng and Johannes
Huebner teach Master’s level students a theory- and practice-based
understanding of how today’s information technologies enable new digital
business models and transform existing ones. The course contains a novel examination
mode, a video group project is introduced as a core element contributing to the
overall course grade. In addition, students are asked to participate in a
peer-to-peer review of the videos produced by other student groups, which is
independent of the grading and is geared towards giving students insights in
how other groups solved the challenge. The best-rated videos are then shared
with the entire class in the end of the semester.
As part of this newly created examination element, course participants (in teams of two to three students) explain one of the major lecture topics (theoretical lenses) in the first half of their video.Then they apply the same lens by analysing a company, aiming to better understand its underlying business model. Companies are pre-selected and allocated to students for fairness reasons. Every year, we choose a pool of interesting companies in the context of digital transformation, the Internet of Things, Blockchain, e-health, etc.
What
motivated you to initiate the project?
The core idea was to improve students’
learning success by using an examination format that not only requires learners
to reiterate theoretical contents, but also apply the theory in a practical
context. The students have different backgrounds, and do not necessarily have a
strong business focus, which means that many of the concepts taught in class
may be rather abstract. We used the video format and specific companies as case
studies, because we think this is a good way to trigger curiosity, show
concrete examples of modern companies in a compact form, and force students to
reflect deeply upon theoretical frameworks compared to other examination
formats.
How
did you do it?
Aside from the weekly input lectures, we
ask students to form groups in the beginning of the semester. We then provide a
list of theoretical core topics from which each group can choose one. In
addition, we randomly assign each group to a case company. The theoretical
topic then first needs to be explained in the first half of the video, and then
be applied to the case company in the second half. Here we thus used a prosumer
approach, where students become part of the course because they create a small
section of the content. The best videos are shared with the class, and can be
reused as additional learning materials for future cohorts. This set-up generally
resulted in high-quality videos, perhaps also since students knew their videos
will be used again.
Students also had to review the video
projects of five other groups. They had to clearly describe whether and how
their peers used certain perspectives (called “lenses” in the course) which played
a role in the video and in their feedback. In this way they analysed once more how
the newly learned concepts were visible in other companies – a positive side
effect being that they also honed their reflection and feedback skills.
Did
you have the support you needed for the project? Is there additional support
you wish you had had to help you to achieve your goals?
We asked two students from previous cohorts
to join us as tutors, and support this year’s groups primarily with technical
questions about video-making (e.g. tools, quality considerations etc.). In
addition, we designed one of the lecture slots as a coaching session during
which we would further support student groups with their questions. In total,
this approach allowed us to provide the students with high-quality supervision
with reasonable effort.
Please
describe some of the key outcomes of the project
To most students, the task of creating a
video was new. We received feedback that while the initial effort for learning
how to make a video was higher compared to other examination formats, it was also
fun and very helpful to really understand and apply the new concepts. They said
that they learned things more deeply and more sustainably because they had to
consider all details and aspects – compared to the practical exercises they are
familiar with in other courses. By carefully phrasing their arguments in giving
feedback on peer videos, students became more aware of their own thinking and
argumentation.
We observed that the questions asked by
students once they start creating videos were different and went deeper, i.e. their
reflections were based on many concrete examples of companies, and the concepts
were put into perspective. The same sub-concepts have a different meaning in
another context, and students now see the overarching principles better and can
argue more precisely about theoretical aspects. Without these concrete
examples, it would have been harder to concretely grasp the theoretical
aspects.
How
did the project impact learners or the way in which you teach?
We were surprised by the high quality of the
best student videos. The teaching team is now really motivated to continue
innovating on our approaches in other courses. We saw clearly that when
students are very active we get better results, deeper learning and better
reflection.
What
lessons learned do you want to share with your colleagues?
It can really pay off to try things and to
experiment. We think that nowadays the classic format of passive lectures and
final exams may not always be the best choice. We believe the improved outcomes
through students who were actively engaged by the video assignment justified
the investment in developing new approaches and tools.
When considering videos as an examination
format, you should define the entire course/project very clearly. When
describing what production options students have for videos, you should be very
precise. Offering too many options can be counterproductive. It is better to
present 3-4 crystal-clear examples and stick to them.
Also, we would recommend managing students’
expectations clearly in the beginning of the semester, and highlighting both
the benefits and challenges of this examination format. Of course, this becomes
easier after the first year, when you can draw from the experience of the first
cohort, and also provide examples of prior videos to illustrate what is
expected of the groups. Because the students are co-creators you get new and
relevant content which enriches the course and can serve to motivate both
students and teachers.
What
are the future plans for this work? How do you plan to sustain what you have
created through the project?
We plan to optimize some details of this
course, and to go even more in the direction of a flipped classroom to use this
teaching approach in other courses. We will create a library of the student
videos to provide it as additional learning materials in future editions of the
course.
Student feedback
By MDBM Student Cristina Mercandetti (mercandc@student.ethz.ch)
Your opinion about this course and the peer
review & video production process – how has it influenced your learning
process? Cristina Mercandetti: I really enjoyed both the
course and the video production process. I think they complemented each other
very well and we were able to directly apply the theoretical knowledge learned
in the course to work on our project. It helped me to think more critically
about the course content, and really dive into some of the lenses and models
presented. I don’t think this would have been possible without the video
production, so it definitely improved my learning process.
Do
you think this approach could be used in other courses?
Cristina Mercandetti: Yes, I think this approach could easily be used in other
classes. However, I think part of the fun in this class was that the video
production was something very new and refreshing (a side effect was that I
learned how to cut a short movie). I imagine that if several classes introduced
this it would lose some of its novelty and could be stressful, as it took a lot
of time.
Final remarks about the course Cristina Mercandetti: I really enjoyed the whole
class, and heard a lot of good things from other students too.
Flipped learning helps evaluation of health delivery systems
When it comes to medical device and health services development, it is essential to conduct a rigorous analysis and propose fitting solutions. These should cover the specific situations that impact the user and the country’s system within which they are delivered. This not only requires knowledge in medical device evaluation and regulation, but also “soft” skills such as empathy. In order to maximise opportunities for his students to develop these competences, Prof. Dr. Walter Karlen flipped his class so that his students could practice the skills he intends for them to learn. The course in question is called Appropriate Health System Design, an elective course open to all master students of ETH.
What triggered this approach?
Comprehensive analysis and human centred design are very important competences in Dr. Karlen’s field. Encouraged and informed by what he learned in the didactic course Teaching at ETH: Committed and skilled, he adjusted his course design to become almost entirely flipped.
What exactly did he do?
Students work in groups to work their way through the challenges he sets for them every week. Readings assigned as homework prepare students to solve problems together during class time, while he is there to advise and support them. Using the principles learned during the didactic course, he ensures that he consolidates the results of their group activities and are shared in an online book so all students can benefit. This also means he can provide feedback on their progress towards the learning objectives.
Students attempt to improve existing devices (such as a standard asthma inhaler or an x-ray machine) or systems within which these devices are delivered. They do this according to the unique needs of fictional personas they collectively developed in the first weeks of the course. Then, they develop prototypes and evaluate them for appropriateness. They present their results in a final graded poster presentation to their peers and a panel of experts.
What were the results?
The response of the students has been very positive. They complimented the interactive course design and rated the course highly in the most recent evaluation. “I believe the students are inspired more. They will certainly remember this course next time they see a medical device. Even though adjusting the course plan has taken time, the effort was well worth it.”
In June 2018 ETH Zürich will host the biggest Moodle BarCamp in Europe: Moodle-DACH. Dozens of Moodle representatives from universities and business will meet to discuss hot topics like learning analytics, GDPR or eAssessment with Moodle. After one and a half days of deep discussions, a DevCamp will follow where the most interesting issues will be translated into code.
More information and registration (no attendance fee):