Schlechte Unterrichtsbeurteilungen für innovativen Unterricht?
“Mit viel Engagement habe ich meine Lehrveranstaltung nach neuesten didaktischen Erkenntnissen umgestaltet. Sei es im Flipped Classroom oder mit erhöhtem Einsatz von Clicker-Fragen, die Studierenden waren während der Präsenz aktiv gefordert und haben auch gerne mitgemacht. Doch mit der Unterrichtsbeurteilung kam die grosse Enttäuschung. Die Studierenden bewerten mich und meinen Unterricht deutlich schlechter als vorher. Sie bevorzugen sogar Frontalunterricht, denn damit würden sie besser lernen. Habe ich etwas falsch gemacht? Soll ich wieder zurück zu meiner altbewährten Vorlesung?”
Was hier wie ein
Einzelfall klingt, ist doch sehr verbreitet. Zahlreiche Untersuchungen
dokumentieren, dass Unterrichtsformen mit Lerner zentrierten Methoden häufig zu
schlechten Evaluationsergebnissen führen (z.B. Seidel, 2013). In einer kürzlich publizierten Studie haben
Louis Deslauriers und Kollegen/innen der Harvard University genau diese
Problematik untersucht (Deslauriers,
2019). Sie gingen der Frage nach, ob es bei aktiv involvierten Studierenden
eine Diskrepanz zwischen dem tatsächlichen und dem empfundenen Lerngewinn gibt.
Falls Studierende den Eindruck haben, weniger als in einer Vorlesung gelernt zu
haben, dann führt dies zwangsläufig zu schlechteren Evaluationsergebnissen. Die
experimentell angelegte Untersuchung bestätigte die negative Korrelation
zwischen der Selbsteinschätzung und dem effektivem Lerngewinn. Folgende drei Gründe
sind dafür verantwortlich:
Interaktive Unterrichtsmethoden verlangen eine erhöhte kognitive Leistung, die von Studierenden dann nicht unbedingt mit dem Lernen in Verbindung gesetzt wird.
Insbesondere Studienanfänger/innen verfügen noch nicht über die Fähigkeit, ihr eigenes Wissen in einem neuen Fachgebiet korrekt einzuschätzen.
Die klare und sprachgewandte Präsentation beim Frontalunterricht verleitet Studierende dazu, ihr eigenes Verständnis in Vorlesungen deutlich zu überschätzen.
Aufgrund der Ergebnisse einer Nachuntersuchung schlagen Delauriers und Kollegen/innen einige recht simple Massnahmen vor, um dieses Missverhältnis zwischen gefühltem und tatsächlichem Lerngewinn zu unterbinden. Zentral dabei ist, die Befürchtungen und Ängste der Studierenden ernst zu nehmen und sie zu thematisieren. So kann eine kurze Darlegung der Lernvorteile von aktivem Unterricht (z.B. Freeman, 2014) während der ersten Unterrichtsstunde bereits erste Befürchtungen auffangen. Aber auch im weiteren Verlauf der Veranstaltung sollte immer wieder auf den erzielten Lernfortschritt hingewiesen werden. Damit erlangen die Studierende eine bessere Einschätzung ihres eigenen Lerngewinns. Hilfreich ist zudem, auf die Gefahr der Lernillusion bei eloquenter Redegewandtheit des Dozierenden in Vorlesungen hinzuweisen (z.B. Toftness, 2018).
Auch an der ETH konnten wir den Einfluss dieser Massnahmen bestätigen. In einer Studie am Departement Physik verglichen wir den Lerngewinn zwischen interaktivem Unterricht und Vorlesung. Bereits in der ersten Lerneinheit wiesen wir die interaktive Gruppe ausführlich auf die positiven Auswirkungen des interaktiven Unterrichts hin. Zusätzlich wurden Unsicherheiten bezüglich des eigenen Lerngewinns im Semester kontinuierlich thematisiert. Bei der Unterrichtsevaluation konnten wir daraufhin kein Missverhältnis zwischen effektiver und geschätzter Lernleistung feststellen. Studierende im interaktiven Unterrichtsformat erzielten einen höheren Lerngewinn und gaben signifikant bessere Werte bezüglich ihres eigenen Lernens an als jene in der parallel durchgeführten Vorlesung (Schiltz, 2018).
Tipp: Was hier jetzt speziell für interaktive Unterrichtsformen gilt, lässt sich sicher auch auf jeden anderen Wechsel der Lernform übertragen. Insbesondere wenn die neue Lernform noch nicht geläufig ist, sollte man die anfänglichen Bedenken der Studierenden ernst nehmen und ihnen klar vermitteln, welchen Nutzen sie vom Wechsel zu erwarten haben. Daneben ist es wichtig, Ergebnisse der Unterrichtsbeurteilung (ob gute oder schlechte) kritisch zu hinterfragen. Nicht immer ist der kausale Zusammenhang zwischen studentischer Zufriedenheit und tatsächlichem Lernerfolg gegeben (z.B. Carpenter, 2020).
Does anything ever happen after those teaching evaluation surveys?
Maybe you know
the problem. You want feedback on your teaching from your students. You want to
know what they think went well, and what didn’t. Maybe you need their
evaluations for future job applications. In whatever case, in your evaluation a
more or less representative amount of feedback and number of ratings would come
in handy. But your students are sick of evaluations! They wonder why they have
to fill something out which will be of no use to them, and nothing ever comes
of evaluations anyway… .
So the
muttering of students. However, something actually does happen with student
evaluations – even if most students aren’t aware of it. It is rare that someone
attends a course twice, and there is little opportunity to find out whether
lecturers have implemented their students’ wishes. Therefore, to let students
and others know what happens after a questionnaire is submitted and why
evaluations are important to teaching quality, we have created a 3-minute video
with the help of Youknow (specialists in explainer videos). Please show this
video to your students and motivate them to take part in the survey! This is
especially useful if you have the opportunity to conduct the evaluation in
class.[1]
The challenge for us was to explain the entire comprehensive, stringent evaluation process from survey via publication of findings to deduction of appropriate measures briefly and appealingly. A bigger challenge was to be responsive to students and take their criticisms seriously, while also dignifying the engagement of most lecturers. Whether and how well we have achieved this in three minutes of moving images is yours to decide!
[1] By in class evaluation we mean that you programmed the time your evaluation survey will be send out to students, and you ask them during your lecture time to fill in the survey, via their laptop or smartphone.
Case Study – Peer Review Mastering Digital Business Models
As part of a series of case studies, staff
at LET sat down to have a conversation with Prof. Elgar Fleisch, Johannes
Hübner and Dominik Bilgeri from the Department of Management, Technology, and
Economics (D-MTEC) to discuss their Mastering Digital Business Model (MDBM)
course.
What is the project about?
In this Mastering Digital Business Model
(MDBM) course, Prof. Elgar Fleisch, Dominik Bilgeri, George Boateng and Johannes
Huebner teach Master’s level students a theory- and practice-based
understanding of how today’s information technologies enable new digital
business models and transform existing ones. The course contains a novel examination
mode, a video group project is introduced as a core element contributing to the
overall course grade. In addition, students are asked to participate in a
peer-to-peer review of the videos produced by other student groups, which is
independent of the grading and is geared towards giving students insights in
how other groups solved the challenge. The best-rated videos are then shared
with the entire class in the end of the semester.
As part of this newly created examination element, course participants (in teams of two to three students) explain one of the major lecture topics (theoretical lenses) in the first half of their video.Then they apply the same lens by analysing a company, aiming to better understand its underlying business model. Companies are pre-selected and allocated to students for fairness reasons. Every year, we choose a pool of interesting companies in the context of digital transformation, the Internet of Things, Blockchain, e-health, etc.
What
motivated you to initiate the project?
The core idea was to improve students’
learning success by using an examination format that not only requires learners
to reiterate theoretical contents, but also apply the theory in a practical
context. The students have different backgrounds, and do not necessarily have a
strong business focus, which means that many of the concepts taught in class
may be rather abstract. We used the video format and specific companies as case
studies, because we think this is a good way to trigger curiosity, show
concrete examples of modern companies in a compact form, and force students to
reflect deeply upon theoretical frameworks compared to other examination
formats.
How
did you do it?
Aside from the weekly input lectures, we
ask students to form groups in the beginning of the semester. We then provide a
list of theoretical core topics from which each group can choose one. In
addition, we randomly assign each group to a case company. The theoretical
topic then first needs to be explained in the first half of the video, and then
be applied to the case company in the second half. Here we thus used a prosumer
approach, where students become part of the course because they create a small
section of the content. The best videos are shared with the class, and can be
reused as additional learning materials for future cohorts. This set-up generally
resulted in high-quality videos, perhaps also since students knew their videos
will be used again.
Students also had to review the video
projects of five other groups. They had to clearly describe whether and how
their peers used certain perspectives (called “lenses” in the course) which played
a role in the video and in their feedback. In this way they analysed once more how
the newly learned concepts were visible in other companies – a positive side
effect being that they also honed their reflection and feedback skills.
Did
you have the support you needed for the project? Is there additional support
you wish you had had to help you to achieve your goals?
We asked two students from previous cohorts
to join us as tutors, and support this year’s groups primarily with technical
questions about video-making (e.g. tools, quality considerations etc.). In
addition, we designed one of the lecture slots as a coaching session during
which we would further support student groups with their questions. In total,
this approach allowed us to provide the students with high-quality supervision
with reasonable effort.
Please
describe some of the key outcomes of the project
To most students, the task of creating a
video was new. We received feedback that while the initial effort for learning
how to make a video was higher compared to other examination formats, it was also
fun and very helpful to really understand and apply the new concepts. They said
that they learned things more deeply and more sustainably because they had to
consider all details and aspects – compared to the practical exercises they are
familiar with in other courses. By carefully phrasing their arguments in giving
feedback on peer videos, students became more aware of their own thinking and
argumentation.
We observed that the questions asked by
students once they start creating videos were different and went deeper, i.e. their
reflections were based on many concrete examples of companies, and the concepts
were put into perspective. The same sub-concepts have a different meaning in
another context, and students now see the overarching principles better and can
argue more precisely about theoretical aspects. Without these concrete
examples, it would have been harder to concretely grasp the theoretical
aspects.
How
did the project impact learners or the way in which you teach?
We were surprised by the high quality of the
best student videos. The teaching team is now really motivated to continue
innovating on our approaches in other courses. We saw clearly that when
students are very active we get better results, deeper learning and better
reflection.
What
lessons learned do you want to share with your colleagues?
It can really pay off to try things and to
experiment. We think that nowadays the classic format of passive lectures and
final exams may not always be the best choice. We believe the improved outcomes
through students who were actively engaged by the video assignment justified
the investment in developing new approaches and tools.
When considering videos as an examination
format, you should define the entire course/project very clearly. When
describing what production options students have for videos, you should be very
precise. Offering too many options can be counterproductive. It is better to
present 3-4 crystal-clear examples and stick to them.
Also, we would recommend managing students’
expectations clearly in the beginning of the semester, and highlighting both
the benefits and challenges of this examination format. Of course, this becomes
easier after the first year, when you can draw from the experience of the first
cohort, and also provide examples of prior videos to illustrate what is
expected of the groups. Because the students are co-creators you get new and
relevant content which enriches the course and can serve to motivate both
students and teachers.
What
are the future plans for this work? How do you plan to sustain what you have
created through the project?
We plan to optimize some details of this
course, and to go even more in the direction of a flipped classroom to use this
teaching approach in other courses. We will create a library of the student
videos to provide it as additional learning materials in future editions of the
course.
Student feedback
By MDBM Student Cristina Mercandetti (mercandc@student.ethz.ch)
Your opinion about this course and the peer
review & video production process – how has it influenced your learning
process? Cristina Mercandetti: I really enjoyed both the
course and the video production process. I think they complemented each other
very well and we were able to directly apply the theoretical knowledge learned
in the course to work on our project. It helped me to think more critically
about the course content, and really dive into some of the lenses and models
presented. I don’t think this would have been possible without the video
production, so it definitely improved my learning process.
Do
you think this approach could be used in other courses?
Cristina Mercandetti: Yes, I think this approach could easily be used in other
classes. However, I think part of the fun in this class was that the video
production was something very new and refreshing (a side effect was that I
learned how to cut a short movie). I imagine that if several classes introduced
this it would lose some of its novelty and could be stressful, as it took a lot
of time.
Final remarks about the course Cristina Mercandetti: I really enjoyed the whole
class, and heard a lot of good things from other students too.
As part of a series of case studies, staff
at LET sat down to have a conversation with Prof. Volker Hoffmann (SusTec, the Group for
Sustainability and Technology) and Erik Jentges (Educational
Developer) from the Department of Management, Technology and Economics (D-MTEC)
to discuss their corporate sustainability project.
What
is the project about?
The course “Corporate Sustainability” aims to enable students to become advocates of sustainable business practices in their later careers. Each year it attracts 150-200 students with diverse disciplinary backgrounds and different educational levels (BSc, MSc, and MAS). We adapted the Six Sentence Argument (6SA) method for this course. The method focuses on enhancing critical thinking skills through structured writing and guided, double-blind peer-review.
What
motivated you to initiate the project?
We wanted students to get a clearer picture
of what sustainability really is. In the course, they develop not only a deeper
understanding of corporate sustainability but also the skills to give and
receive feedback.
How
did you do it?
At the core are four topics that relate to
the sustainability of corporations. These are assessment, strategy, technology,
and finance. We developed digital learning modules (videos, some with
interactive elements) that explain key concepts to support the most relevant and difficult
parts of the lecture. Also, we want to develop students’
critical thinking skills. In e-modules, students learn to formulate concise and
short arguments with the 6SA method. The core idea builds on the
assumption that writing is thinking.
In the e-modules, students face a decision
(a micro case based on the lecture content) and argue for their preferred
course of action using a logical structure of exactly six sentences. Each
sentence fulfils a specific function in the overall argument and has a 20-word
limit. A clear grading rubric enables students to assess 6SAs in double-blind
peer reviews. These have been continuously adapted and improved since 2015. The
specialized online tool “peergrade” also helped us to conduct a smooth process
– for both students and teachers.
Through the peer assessment, students engage
critically with their peers’ arguments and receive constructive feedback on
their own arguments. With the 6SA exercise, students learn to argue with
clarity, and it helps them to reflect on the way they and others think.
During the second half of the semester,
students work in diverse teams to prepare mock debates, consulting strategies,
economic models and campaign videos. In this phase, they are coached by several
postdoctoral and doctoral researchers from SusTec, the Group for Sustainability and Technology. The students then present their projects and display their skills in
a group puzzle session and are debriefed in the following final lecture
session. Students receive grades for both individual and group performance and
can earn a bonus on their exam grade when completing the critical thinking
exercises.
Did
you have the support you needed for the project? Is there additional support
you wish you had had to help you to achieve your goals?
The project received funding from different
sources. This helped us to hire academic staff to assist the development of new
teaching approaches and the production of high-quality videos. In addition, we
received specialist guidance in the instructional design and production of
videos.
Please
describe some of the key outcomes of the project
With regard to our feedback modules, we think
that the quality of the argumentation and peer reviews has increased over the
years. For example, we learned that the effective design of such peer
assessment exercises for students requires training on how to give constructive
feedback and that it should involve several feedback loops to support the
development and refinement of critical thinking skills. Overall, the course now
integrates many innovative teaching elements and was a finalist in the 2018 ETH
KITE award.
How
did the project impact learners or the way in which you teach?
When students are able to write better and concise
arguments that convince critical readers, and if they can give constructive
feedback to arguments that are being made to justify strategic decisions, then
they are able to actively shape good decisions in a company setting – they can
be change-makers for corporate sustainability. The students were motivated by
the new teaching approaches such as the supporting videos, interactive
questions inside the videos, and the critical thinking exercises. Peer
assessment is “homework” for the students, but they know that they can earn a
bonus on their exam grade – and they are already rehearsing for some parts of
the final exam.
With regard to students’ learning, the peer
review process itself is convincing. What is unique to our teaching situation is
the incredible diversity in the classroom. A 19-year-old Swiss environmental
science student may be sitting next to a 25-year-old Chinese student who is pursuing
a master’s degree in management, who in turn sits next to a 35-year-old
American part-time student with a PhD in chemistry and a management position
with responsibilities for 20 employees in a multinational company. Peer
feedback is a powerful solution to bridge these gaps of different levels of
experience and cultural backgrounds. It allows younger students to write a
creative and brilliant argument without being intimidated by more senior
students. It allows a shy and quiet student to gain confidence by formulating a
convincing argument whose strengths are recognized in their peers’ feedback. It
creates a space for older students to learn how to coach younger classmates with
constructive feedback to improve their reasoning.
That is why at D-MTEC, we use peer feedback
in other courses as well. Students learn more when actually giving feedback
compared to when only submitting an assignment.
What
lessons learned do you want to share with your colleagues?
At the beginning, it was a lot of work and many
people were involved, but it was worth it. Today, with regard to the critical
thinking exercises, we have continuously refined our processes. Every student writes
three reviews, thereby ensuring that everyone also receives much more feedback
than a single lecturer could provide. The main work for lecturers is providing
an overview of the themes in the arguments and summarizing the activity for all
students. This lets them know that their individual contribution becomes part
of a collective intelligence. There are always truly smart and innovative
solutions that need to be shared with the whole class. Also, there is little
effort involved in re-grading/moderating student questions about feedback,
because we train students to write helpful and considerate feedback and make them
aware of that they also have to learn how to receive feedback, especially if it
is feedback that they don’t want to, but need to hear.
For the production of videos, we recommend planning enough time and engaging with video experts and instructional designers early on. Especially writing a concise script for a short video requires a surprising amount of time until it effectively conveys your key points.
If you are interested in applying these concepts in your own courses please contact LET.
Note: The project received funding from different sources (Innovedum, Emil Halter Foundation, ETH Critical Thinking Initiative).
Additional
resources and comments
Article: Kölbel, J., & Jentges, E. (2018). The Six-Sentence Argument: Training Critical Thinking Skills Using Peer Review. Management Teaching Review, 3(2), 118–128. https://doi.org/10.1177/2379298117739856
Case study – Peer Review Food Chemistry Laboratory – Writing reports
As part of a series of case studies, staff at LET sat down to have a conversation with Prof. Laura Nyström and Dr. Melanie Erzinger from the Department of Health Sciences and Technology to discuss their food chemistry laboratory project.
What
is the project about?
We introduced a new way to write lab reports,
combined with a peer review method to foster collaboration and critical
thinking skills among students. In the past students did not have clear
criteria as to what makes a good report. Assistants also needed too much time
to read the reports and give repeated feedback. Thus we looked for a way to
help assistants spend less time on the review process.
We transformed the format of our Food
Chemistry Laboratory Course (Food Science, BSc level, 4th semester)
from a classical lecture format with lab exercises to a blended learning
format. With new videos, we can achieve better coverage of basic knowledge
(i.e. security, handling of equipment).
What
motivated you to initiate the project?
Student numbers have increased over the
past 10 years and we have been losing too much time in covering basic knowledge
repeatedly. Using concept videos, students will be able to review key topics on
their own. Overall, we also wanted to make the entire course more attractive. A
key intention was to develop student skills in report writing and improve
report quality.
How
did you do it?
We defined additional, clear quality
criteria for a good report. During a first round students give each other
feedback, such that final review by teaching assistants and lecturer approval involve
less effort. For each experiment, every student has to review another student’s
report. In total, each reviews four reports over the semester.
Students don’t get a grade for the peer review (semester performance in the lab course is also ungraded). They have one week for each of the four peer reviews, and must complete each by the respective set deadline. They answer various questions related to the quality of the respective report (these involve five aspects plus overall feedback; see the annex at the end of this case study). Students do not “grade” the reports, but give feedback in their own words.
Assistants are aware of what is asked in
the reports and are therefore able to provide targeted and helpful feedback in
the lab which addresses the quality criteria for reports.
We provide the students with online material on how to write reports (short videos, documents etc.). Previously we had a short lecture with examples. Until now, however, we did not train them in conducting proper peer reviews. We have now realised that we need to do this (especially for Bachelor’s degree students), and will include peer review training with the short lecture next year.
Did
you have the support you needed for the project? Is there additional support
you wish you had had to help you to achieve your goals?
We learned about a module inside our LMS for
administering the peer-review process (“workshop module” in Moodle). It would
have been helpful to have had practical tips from others, but apparently not
many lecturers have used this tool. Although the general instructions were
useful, it took quite some time to learn all the aspects of the tool.
Please
describe some of the key outcomes of the project.
Various things changed for the better. Students learned a lot by reading and reviewing the reports of their peers. They gained important input for their own reports. For many it was the first time they had had to give feedback in such a structured way. They also had to find a way to critique something in a good, constructive manner. Overall, students were introduced to a new way of critical thinking and took important first steps in this skill, which is important for their later careers.
We can say clearly that through the new
review method we were able to improve the quality of reports and reduce the
time needed by lecturers to grade them.
How
did the project impact learners or the way in which you teach?
In general the peer review method was well received in the BSc course, and we used the same approach in an MSc-level course. We therefore realised that Bachelor’s degree students need more help and training in peer review than Master’s students.
Overall we saw that the blended learning
approach and the peer review methods work to improve our courses, addressing
the above-mentioned challenges of lack of student preparation and the need to
constantly repeat basic knowledge. Students themselves clearly realised the
value and potential of better collaboration, peer feedback and critical
thinking skills.
What
lessons learned do you want to share with your colleagues?
Not every cohort is the same. While things worked quite well in 2017, in 2018 fewer students adhered to the schedule and deadlines – even though everything was communicated and documented in the same way as in the previous year.
What
are your future plans for this work? How do you plan to sustain what you have
created through the project?
More and more assistants will become competent in providing full reviews of the already peer-reviewed reports. Currently lecturers still have to do this. Lecturers will thus gain more time to be present in the labs and to give 1:1 feedback to students in the lab and online.
We will definitely create some training
material for assistants for this purpose, but it is not available yet. We also want
to create a short, ready-to-use document about giving feedback in our specific
context: what is constructive feedback, what are the do’s and don’ts? Students,
assistants and fellow instructors can use it.
We are interested in learning whether other
lab courses at ETH do something similar, and how. We also need to improve the
support situation with the Moodle review tool “workshop module”. We will
continue to work with it, but it is a bit tricky sometimes.
One additional idea is to make the videos
interactive. Students will see in-built questions in the videos which they have
to answer right away.
Additional
notes regarding resources and tools used.
We used a programme called Labster to create virtual labs in some cases to extend the experience to experiments which were not doable in reality in our labs.
We learned from other courses and departments regarding effective feedback (ETH “Foundations of Teaching and Learning” course).
To conduct the peer review we work with the “workshop module” in Moodle.
To make the videos interactive we will work with the new Moodle “interactive video suite” plugin.
Student voices:
What is your opinion about this course and the peer review
process (lab reports). How has it influenced your learning process?
Robert Spiess: I think peer-reviewing was a great way to see other students’ work. It gave me the opportunity to experience and compare different ways of writing. I could always detect things that I wanted to include in my reports. At the same time, I could see in which points my reports were better, where my advantages were.
I think this procedure is particularly useful when writing.
But the reports should not be too long, because, otherwise, students have to
spend too much time on their own report and neglect, as a consequence, the
peer-reviewing of someone else’s report. Other courses usually required longer
reports. But if the reports were shortened, the method could also be applied in
other laboratory courses (such as in the food processing or in the
biotechnology lab course).
Aline Candrian: I’m glad I did the course, I think it gives students a first impression of laboratory work. The lab report writing is an essential part of the course to understand the experiment and the obtained data. The peer review approach was fine, even though nobody was eager to do them. VERY little time was invested into peer review by most groups, as far as I’m concerned. Of course, sometimes you could benefit from your peer’s feedback but most of the times we didn’t act on them. That’s probably because it was our first time writing (semi-) real reports. We didn’t really know what we were doing and you mostly think you know better than others, especially if someone reviewed your report who you rate less familiar with chemistry.
At the same time you’re well aware that you know nothing
about report writing, so how can you evaluate someone else’s work?
Additionally, motivation was minimal since you were just
glad to be done with writing your report. Having to assess another report and
then correct your own report again was just another ‘burden’. So, altogether, I
would say report-writing was a crucial part of the course but peer review not
so much since we had no experience at all. I think peer review makes more sense
in the courses in our last semester.
Making the students just do a peer review on the last report
might work. They’ll see how it works, they’ll have had written a few reports
(and got more familiar with it) and might be more confident in providing
feedback. But I’m not an expert, it may not work the way I envision it, what do
I know 🙂
Impartial group assessment. Using peer review and economic theory to grade groups fairly.
In a clear case of practicing what he preaches, Dr. Heinrich Nax has applied game theory to his teaching practice. After lecturing on game theory for several years, he realised that his methods for teaching, more specifically, for assessing did not follow the very theories he was espousing and so he set out to correct this incongruence.
In his course «Controversies in Game Theory» students work in groups and are assessed based on a group project. Social tensions can develop between individual and collective interests in group interactions. One such tension, free-riding, when one person rides the coat-tails of other hard-working group members is well known. There are however additional potential problems when assessing group work such as collusion on grades in cases of peer review. To eliminate these tensions, Dr. Nax decided to implement a mechanism from economic theory to his assessments.
What triggered this approach?
Previously Dr. Nax gave the same final mark to everybody in a particular group regardless of their individual efforts as these could not reliably be assessed. From a game theory perspective this constituted a big temptation for free-riding and Dr. Nax decided to devise something that would incentivize individual efforts but without giving up the benefits of group work altogether.
What exactly did he do?
Influenced by the article Impartial division of a dollar by Clippel, G., Moulin, H., and Tideman, N. (2008), Dr. Nax and his colleague Sven Seuken implemented the article’s mechanism in a blockchain start-up company. The mechanism enables a group to split their financial earnings through peer review between the group members. Group members decide internally what a fair allocation of earnings should be. So he decided to try the mechanism in an educational setting where the “earnings” become the finite amount of points the group works towards, the total of which is determined by the grade he allocates to the group’s total project.
The key idea of the mechanism is that individual group members don’t evaluate their own performance and therefore don’t decide how many points they themselves have “earned”. Instead they allocate relative contributions to the other group members. So in a group of three, if student A thinks group member B did twice as much work as fellow group member C, then B should receive twice as many points as C. Using a specific formula (described in the paper) all three group members reports are then aggregated anonymously to make sure the resulting grades cannot be manipulated. In other words, student A only receives the (aggregated) amount of points, that their colleagues think student A deserves.
Courtesy of Spliddit
What were the results (for student learning)?
Not only was Dr. Nax convinced that the quality of the group projects improved, but the students were happier as well. They believed that the marking was much more fair. It is unclear if this grading method decreased free-riding, however students felt that freeriders did receive lower grades, thus increasing student satisfaction in comparison to grading methods where all members of the group receive the same grade, regardless of effort or contribution.
To see this mechanism in action, visit the Spliddit website which features a demonstration tool. Those interested in learning more should read the original paper or this second (less math-based) follow-up paper or contact Dr. Nax for further information. Dr. Nax is working on a tool to make his grading plan available to other professors.
Case study: Molegram Explorer – A mixed-reality framework for teaching drug design
As part of a series of case studies, staff at LET sat down to have a conversation with Prof. Gisbert Schneider and Dr. Jan Hiss from the Institute of Pharmaceutical Sciences in the Department of Chemistry and Applied Biosciences to discuss their mixed-reality project.
What is the project about?
The Molegram Explorer project provides a mixed-reality framework to facilitate and broaden students’ understanding of molecular structure. It is part of the Computer-Assisted Drug Design course run by Gisbert Schneider, professor in the field of the same name.
At the core of the project is the innovative hardware device “HoloLens” (comprising special glasses with 3D projection, motion sensor and environment scanning, produced by Microsoft). Users of the HoloLens see not only (real) furniture and people present in the room, but also a hologram. In our case, the virtual object is a protein that the students can explore, investigate and even walk through. They literally immerse themselves in the world of molecules.
This innovative concept presents a new way of perceiving molecular structure and facilitates new approaches to chemical structure analysis and design via human-machine interaction.
What motivated you to initiate the project?
This innovative project originated in answer to a call for projects on the use of the HoloLens in ETH teaching. Because we faced a teaching challenge where a 3D representation gave a very good use case, it was a perfect opportunity to apply for a pilot project using HoloLenses.
The idea underpinning our work is that this new technology can help our students to understand certain important principles of molecular structure which traditional teaching methods and media struggle to clarify. To identify or design a suitable drug molecule (the ligand) students must understand the protein’s surface, and particularly the cavities suitable for accommodating the ligand. The HoloLens device helps them visualise the regions of a protein that are accessible to the ligand.
How did you do it? Three groups collaborated on this project. First, Gisbert Schneider and Jan Hiss delivered the scientific content. Guided by specific learning objectives, ETH Zurich’s Educational Development and Technology (LET) unit helped organise and facilitate the development process and provided the required hardware. Finally, a specialised software company (afca) implemented the learning app software.
The HoloLenses are used in a two-week practical course in which students experience a condensed version of early-stage drug discovery. They learn how to computationally screen a catalogue of millions of molecules to identify those that might favourably interact with a particular protein. The students perform a computational analysis and select one or two molecules from the top-ranking candidates. Then they synthesise these compounds and test their activity in the laboratory.
An important basic aspect of protein-ligand interaction is the “solvent-accessible surface”. For beginners, this molecular representation often remains an abstract concept without suitable visualization. By using the HoloLens students can now create surface representations of a protein, interact with the holographic model, and simultaneously discuss it with peers and instructors.
Did you have the support you needed for the project? Is there any additional help you wish you had had?
We had excellent help from the company afca who designed a user-friendly app with an elegant interface. LET helped us with the legal aspects and provided the necessary contacts. The 12 ETH HoloLenses are stored at LET. Although we understand that HoloLenses are not easily available due to their comparably high acquisition cost, it would have been helpful to have faster and easier access to this hardware, especially when we needed to try out and check something quickly.
Please describe some of the key outcomes of the project.
The new tool proved to be a valuable addition to our course. It certainly does not replace traditional teaching and discussion, but it is an example of how technology can enhance the understanding of abstract scientific concepts which are otherwise hard to teach. Because students can virtually navigate the molecular hologram they gain a better understanding of the concept of protein structures and surfaces. In the learning sciences this effect is described by the principle of “embodied cognition”. We were also able to increase the attractiveness of our subject matter with this concrete visual experience. It was a kind of scientific marketing. We received several suggestions for additional projects in the context of other practical exercises. The positive feedback and the success of the pilot has driven us to expand the project with enhanced content and to reach out to other disciplines.
How did the project affect learners or the way in which you teach?
We observed that students became more curious, not only about the specific topic of the learning app but in general about many questions related to protein-structure-based drug design. Students certainly appreciate the new tool. The value of technology-enhanced learning apps for teaching of specific aspects in our field is obvious, and we intend to stay on this route.
What lessons learned would you share with your colleagues?
It is not always realistic or meaningful for scientists and teachers to address app programming and didactic concepts. Therefore, it is important to have experts from complementary fields working hand in hand. Experts on the subject matter can contribute the scientific content, and software developers can create user-friendly and visually appealing interfaces and functions. Learning professionals can then connect content with technological functions. They can also advise on how to transpose learning objectives into an appropriate and technology-enhanced learning process.
Overall, we encourage teachers to try out new methods in teaching, and there is much potential for combining proven learning approaches with new technology. In particular, teachers and students should not fear experiments that do not produce immediate success. “Productive failure” should be regarded as a natural part of the development process; it is a great way to learn.
What are your future plans for this work? How do you plan to sustain what you created through the project?
Based on the many positive outcomes, we plan to develop further apps. The ultimate goal is to adapt the work to a professional context by adding scientific content from our subject matter, together with advanced analysis tools. It would also make sense to develop HoloLens learning apps for selected (teaching) topics in medicine, chemistry and biology. HoloLenses are increasingly employed not only by the entertainment industry, but in business and education generally. We would welcome new, broader applications of this technology at ETH – but always with a critical double-check as to whether it actually provides added value for students compared to conventional teaching methods. In the case of Molegram Explorer, we are extremely satisfied with the learning success achieved.
Feedback from PhD students (Tutors)
What is your opinion about this course and the HoloLens process? How has it influenced your learning process?
Cyrill Brunner: Though it has been clear to me before since I have been doing research in that area before, the visualization of proteins by the HoloLens helped in getting a better feeling for 3D structure of a protein. My personal gain was clearly not so much as for the course participants, but that has nothing to do with the process itself, but the chosen protein (carbonic anhydrase II) which I’ve done research in before. I’m positive that application of the HoloLens process on a novel protein would have helped me clearly to get a better understanding of the 3D confirmation.
Dominique Bruns: The HoloLens is a useful next step in the visualisation of molecules. This hands-on experience allows the understanding of molecular properties, their definition and dependencies. In this regard, the application of the surface area visualisation and determination is an ideal showcase.
Do you think this approach could also be used in other courses?
Cyrill Brunner: Yes, indeed. The HoloLens should clearly be put into use in the lab course of medicinal chemistry as students there are already working with a 2D visualization program. A 3D full insight into what they have been studying beforehand would strengthen their understanding.
Dominique Bruns: I appreciate the 3D perspective and interactivity enabled by the HoloLens, a characteristic that might be especially useful for people that find it difficult to see in three-dimensions.
In my opinion, many further examples could be established and used for didactic purposes, e.g. chemical reaction mechanisms or biological folding events of proteins.
To find out more about how ETH teaching staff can start their own HoloLens project, visit theETH website.
Die EduApp ist eine der wichtigsten Lehrapplikationen der ETH. Ziel der EduApp ist es einerseits, die Interaktion zwischen Studierenden und Dozierenden im Hörsaal zu verbessern. Anderseits möchte diese Lehre-App Studierenden der ETH Zürich einen Mehrwert im Studienalltag bieten.
Im letzten Frühlingssemester haben 100 Dozierende Clickerfragen in ihrem Unterricht eingesetzt und damit 8’694 Studierende erreicht. Auch aus Sicht der Dozenten ist die EduApp eine wertvolle Ergänzung.
Dr. Ghislain Fourny (D-INFK): «Ich benutze seit 2016 die EduApp in allen meinen Vorlesungen und bin davon sehr begeistert. Es ermöglicht eine reiche Interaktion mit den Studierenden und gibt mir ein konstantes Feedback»
Prof. Dr. Christoph Heinrich (D-ERDW): «Ich habe im HS2017 zum ersten Mal regelmässig Clicker-Fragen in meiner grossen Geologievorlesung für die Erstsemestrigen am D-BAUG eingesetzt. Es war ein grosser Erfolg, nicht zuletzt wegen der Auflockerung, und ich bekam spontan viele positive Feedbacks».
Dr. Markus Kalisch (D-MATH): «Mit der EduApp bekomme ich sofortiges Feedback von den Studenten, selbst wenn die Vorlesung mehrere hundert Teilnehmer hat».
Dr. Meike Akveld (D-MATH): «Die EduApp gibt mir direktes Feedback darüber, ob verstanden wurde, was ich unterrichtet habe. Ich bitte immer einen der Studierenden die richtige Antwort zu erklären, was oft hilfreich ist. Ausserdem ist es für sie eine angenehme Abwechslung».
Neue Funktionen für den Clicker
Pünktlich auf das aktuelle Semester wurden in der der EduApp neue Funktionen im Bereich Clicker hinzugefügt. Mit der Funktion «Clicker» können Dozierende via EduApp Fragen stellen, die meist sofort im Unterricht beantwortet werden.
1. Zwischenresultate: Neu können Dozierende die Abstimmung der Clickerfragen in zwei Runden machen und die Zwischenresultate anzeigen.
2. Erweiterter LaTeX-Editor: Der Funktionsumfang des LaTeX-Editors zur Anzeige von mathematischen Formeln in Clickerfragen wurde erweitert. Nicht nur können Dozierende jetzt Formeln und Gleichungen im Text eingebunden darstellen, es gibt auch mehr Textformatierungsmöglichkeiten.
3. Flashcards: Studierende können neu mit der Funktion «Flashcards» bestehende Clickerfragen durcharbeiten (z.B. zur Prüfungsvorbereitung). Die neue EduApp-Funktion “Flashcards” wurde durch den «the Rectors Impulse Fund» ermöglicht.
Flipped learning helps evaluation of health delivery systems
When it comes to medical device and health services development, it is essential to conduct a rigorous analysis and propose fitting solutions. These should cover the specific situations that impact the user and the country’s system within which they are delivered. This not only requires knowledge in medical device evaluation and regulation, but also “soft” skills such as empathy. In order to maximise opportunities for his students to develop these competences, Prof. Dr. Walter Karlen flipped his class so that his students could practice the skills he intends for them to learn. The course in question is called Appropriate Health System Design, an elective course open to all master students of ETH.
What triggered this approach?
Comprehensive analysis and human centred design are very important competences in Dr. Karlen’s field. Encouraged and informed by what he learned in the didactic course Teaching at ETH: Committed and skilled, he adjusted his course design to become almost entirely flipped.
What exactly did he do?
Students work in groups to work their way through the challenges he sets for them every week. Readings assigned as homework prepare students to solve problems together during class time, while he is there to advise and support them. Using the principles learned during the didactic course, he ensures that he consolidates the results of their group activities and are shared in an online book so all students can benefit. This also means he can provide feedback on their progress towards the learning objectives.
Students attempt to improve existing devices (such as a standard asthma inhaler or an x-ray machine) or systems within which these devices are delivered. They do this according to the unique needs of fictional personas they collectively developed in the first weeks of the course. Then, they develop prototypes and evaluate them for appropriateness. They present their results in a final graded poster presentation to their peers and a panel of experts.
What were the results?
The response of the students has been very positive. They complimented the interactive course design and rated the course highly in the most recent evaluation. “I believe the students are inspired more. They will certainly remember this course next time they see a medical device. Even though adjusting the course plan has taken time, the effort was well worth it.”
Virtual field trips enhance Production Management classes
Using factory-visit apps and affordable cardboard viewers, Professor Torbjørn Netland and his team innovate how production management can be taught. By integrating Virtual Reality (VR) technology in the spring term course Global Operations Strategy at ETH Zurich, the teaching team provided students rare access to multiple factory sites and their inner workings. The students used VR apps in order to help them complete graded course assignments.
What triggered this experiment?
A student using the cardboard viewer
“Production and operations management is an applied field, but it is difficult to teach all the inner workings of a factory in a classroom. Because I cannot always bring students to the field, I wanted to bring the factory to the students. VR now offers unprecedented opportunities for doing so, and ABB had the app I needed. My impression is that students generally enjoyed this teaching innovation” – Prof. Dr. Netland
In optimal class conditions, students may be able to visit one factory in order to see how the concepts they are learning in class relate to the real world. However, field trips are resource intensive to organise and they have limitations. Not all students may be able to attend and they usually rely on their memory (or written notes) to recall relevant information later.
The use of the VR app enables students not only to revisit the factory as often as they would like, it also gives them access to spaces they may not have previously been able to see, such as a close-up view of a milling machine in action, or entry to a dust-free zone. Since the apps enable students to visit multiple company sites with a minuscule time and resource investment, the value of the virtual tours becomes very clear.
What exactly did they do?
With the help of two research associates, Oliver Flaeschner and Omid Maghazei, Prof. Netland compared the learning objectives of his course to the information already contained in the virtual tour apps produced by the ABB group. By adjusting their assignment questions for their teaching case, they created alignment between the content provided in the apps and the students’ assignment questions. With the help of the apps, their own smartphones and cardboard VR viewers, students navigated their way around five different factories in three countries, observed machinery and people in action and gathered information relevant to their assignments.
What were the results?
The teaching team conducted an evaluation survey, two focus groups and collected additional written feedback. The Faculty Development staff from the Educational Development and Technology unit (LET) provided support to ensure that student anonymity was preserved. The results showed that students initially found the opportunity to use VR fun and motivating. While there were some technical difficulties and temporary negative physical side-effects such as dizziness and headaches, in general students stated they enjoyed the opportunity to explore the virtual factory as often as they would like, and at their own pace and availability.
They also reported feeling a high degree of autonomy as they actively engaged with the information. The teaching team did not attempt to measure a learning gain this time, but are encouraged by this pilot project and are planning further development and evaluation of this teaching innovation. In the meantime, the results of the research have been written up and will be presented at the EurOMA Conference in Hungary in June this year.
Should readers wish to find out more, they are welcome to contact Prof. Dr. Netland directly for more information.
Oliver Flaeschner, Torbjorn Netland and Omid Maghazei
