# Author: Marinka Valkering-Sijsling

## How meaningful are clicker data?

Contributors: Meike Akveld (D-MATH), Menny Aka (D-MATH), Alexander Caspar (D-MATH), Marinka Valkering-Sijsling (LET), Gerd Kortemeyer (LET)

Among other things, ETH Zurich’s EduApp allows instructors to pose clicker questions during lectures. Instructors can interrupt lectures to ask questions from the students and get and give feedback on learning progress. Lecturers can also trigger phases of peer-instruction, where students discuss their initial answers to a question with one another and then reanswer the question – in effect, the students are teaching each other during those phases, thus “peer instruction”. By asking students to answer a question twice, lecturers gather data on student understanding. But how meaningful is this feedback data, in particular, when answering is voluntary and ungraded?

A group of mathematics instructors at ETH’s D-MATH worked with LET to analyze EduApp data using Item Response Theory (IRT), Classical Test Theory (CTT) and clustering methods. Over the course of the semester, 44 clicker problems were posed – 12 of them twice, as the instructor decided to insert a phase of peer-instruction. The following figure shows an example of the kind of problem being analyzed:

The problem shown was used in conjunction with peer-instruction; the gray bars indicate the initial student responses, the black bars those after the discussion. A simple, unsurprising observation is that after peer-instruction, more students arrived at the correct answer. What can we learn from these responses? CTT and IRT can provide psychometrics that help understand this instructional scenario.

When it comes to being “meaningful,” the “discrimination” parameter of a problem is of particular interest: how well does correctly or incorrectly answering a problem distinguish (“discriminate”) between students who have or have not understood the underlying concepts?

CTT simply uses the total score as a measure of “ability”, but also has a measure of discrimination (“biserial coefficient”). IRT estimates the probability of a student arriving at the correct answer for a particular problem (“item”) based on a hidden (“latent”) trait of the student called “ability” – typically, higher-ability students would have a higher chance of getting a problem correct. How exactly this probability increases depends on problem characteristics (“item parameters”).

In IRT, the ability-trait is determined in a multistep, multidimensional optimization process, where the difficulty and discrimination parameters of particular problems (“items”) feed back on how much correctly answering that problem says about the “ability” of the student; “high-ability” students are likely to get correct answers even on high-difficulty, high-discrimination problems.

The results of their study were extremely encouraging: using both CTT and IRT, almost all 44 problems under investigation exhibited strong positive discrimination in the initial vote. This means that the better the student understood the underlying concepts, the much more likely they were to give the right answers – and vice versa. A low discrimination, on the other hand, means a problem provides less meaningful feedback. For the handful of problems which had lower (yet still meaningful!) discrimination, this could be explained by other problem characteristics, for example, that at the time they were posed, they were still too hard or already too easy – but even that feedback is meaningful to the instructor for future semesters.

The truly surprising result of the study was that in all cases of peer-instruction, the problem had even stronger discrimination afterwards! Yes, unsurprisingly more students answer correctly after discussion with their neighbors (the problem becomes “easier”), but: peer-instruction does not simply allow weaker students to enter the correct answer, it apparently helps them to perform at their true potential.

For the purposes of the study, the clicker data had to be exported manually, but the next version of EduApp, slated to be released in December 2020, will allow export of data for learning analytics purposes directly from the interface – the following figure shows a sneak preview of that new functionality.

The exported data format is compatible with input for the statistics software R, and there are variety of guides available for how to analyze this data (https://aapt.scitation.org/doi/abs/10.1119/1.5135788 (accessible through the ETH Library) provides a “quick-and-dirty” guide).

The full study, including results from Classical Test Theory and clustering methods, as well an outlook for new EduApp-functionality is available open-access in Issue 13 of e-learning and education (eleed) under https://eleed.campussource.de/archive/13/5122.

Posted on ,

## Learning Autonomy with Self-Driving Cars: Duckietown goes MOOC.

Jacopo Tani and Andrea Censi are senior assistants in the research group headed by Emilio Frazzoli (D-MAVT), an internationally renowned specialist in autonomous systems. Together with Prof. Liam Paull of the University of Montreal, they lead the Duckietown project, which was conceived at the Massachusetts Institute of Technology (MIT) in 2015. The goal was to build a platform that was small-scale and cute yet still preserved the real scientific challenges inherent in a full-scale real autonomous robot platform.  Duckietown is now a worldwide initiative to realize a new vision for AI/robotics education. It teaches participants to programme autonomous vehicles to navigate a structured environment using rubber ducks as the passengers of the vehicles, and has now been used by over 80 universities in 23 countries worldwide. Their next endeavor is to create a series of massive open online courses (MOOCs) focused on the science and technology of autonomy through the lens of self-driving cars. In this multi-institution project, ETH will take leadership and develop the first course of the MOOC-series.

What will the MOOC be about and what do you seek to achieve for participants?

The Duckietown MOOC series will be about autonomy, or how to make machines take their own decisions to accomplish broadly defined tasks. This topic is both intellectually fascinating and very timely given the rapid progress of robotics and AI technologies in our daily lives. Autonomy will be studied through self-driving cars, an application with disruptive social potential.

Participants will engage in a sequence of software and hardware hands-on learning experiences whose particular focus is on overcoming the challenges of deploying robots in the real world. Our hope is that participants will gain useful skills and come to appreciate and understand the challenges of this technology, while at the same time having lots of fun!

What motivated you personally to make a MOOC?

The Duckietown project was developed to make the science and technology of autonomy accessible to the broadest possible audience, not only to those learners lucky enough to have access to premiere educational institutions where these topics are addressed. Building a Duckietown MOOC experience was a logical step towards achieving the mission of the project. We are grateful for ETH-Innovedum supporting our efforts and extremely excited to bring our vision for learning autonomy to the world.

What are the unique didactic challenges?

Teaching autonomy requires a fundamentally different approach compared to many computer science and engineering disciplines. There is extensive and diverse preliminary knowledge needed to really comprehend autonomy from the “pure” mathematics and physics to “modern” machine learning based approaches. Moreover, robots are real world machines, and theory and practice do not always play well together. To see the theory work in the real world it is necessary to translate the knowledge in software architectures, and deploy them on hardware platforms. Finally, there is a proliferation of hardware platforms and software tools out there, each with its own peculiarities, strengths and shortcomings. It is not always clear what tools are worth investing time in mastering, and how this competence will translate to different platforms.

How will you overcome these challenges?

To address these barriers of entry to learning autonomy, the MOOC “Self-Driving Cars with Duckietown” will have several distinguishing features, namely:

• Competency-based topic progression
The sequence of topics in the courses is determined by asking the question: “what is the most we can make our robot do, with the least amount of prior knowledge?” instead of “what is the best order to explain things?”. As learners progress through behaviors of increasing complexity to reach the final objective, it becomes naturally necessary to introduce new concepts and tools to address limitations to previous behaviors. This approach allows students to jump right in “the middle of things” (getting Duckiebots to do things!) and gradually re-iterate concepts through the various technical frameworks and implementation solutions that are so very important to align the theory with the practice, leading to a stronger comprehension of the how and why things happen.
• Hardware-based hands-on learning on a standardized platform (the Duckiebot) with open-source industry-widespread software tools
This is a robotics and AI MOOC where every participant will have the opportunity to follow along by doing real world experiments with their own robot at home. The Duckietown framework was designed, from the software stack (i.e., Python, ROS, Docker) to the Duckiebot and Duckietown city, to make the course accessible for all learners, both pedagogically and economically.
• Remote evaluations of hardware assignments
The last, but not least, distinguishing factor of this MOOC is the use of remote facilities (the Duckietown Autolabs) where reproducible performance assessment of hardware assignments is conducted in controlled environments. This feature enables remote grading of hardware assignment, which, to the best of our knowledge, is a first ever for a robotics MOOC.

Like to know more about Autonomy with Self-Driving Cars? Course starts at March 22, 2021 and will be published on the edX-platform.

Inspired to start your own MOOC project? Please have a look at our website and contact Marinka Valkering to discuss possibilities!

Posted on , 1 Comment

## Neue Funktionen für die EduApp

Die EduApp ist eine der wichtigsten Lehrapplikationen der ETH. Ziel der EduApp ist es einerseits, die Interaktion zwischen Studierenden und Dozierenden im Hörsaal zu verbessern. Anderseits möchte diese Lehre-App Studierenden der ETH Zürich einen Mehrwert im Studienalltag bieten.

Im letzten Frühlingssemester haben 100 Dozierende Clickerfragen in ihrem Unterricht eingesetzt und damit 8’694 Studierende erreicht. Auch aus Sicht der Dozenten ist die EduApp eine wertvolle Ergänzung.

Dr. Ghislain Fourny (D-INFK): «Ich benutze seit 2016 die EduApp in allen meinen Vorlesungen und bin davon sehr begeistert. Es ermöglicht eine reiche Interaktion mit den Studierenden und gibt mir ein konstantes Feedback»

Prof. Dr. Christoph Heinrich (D-ERDW): «Ich habe im HS2017 zum ersten Mal regelmässig Clicker-Fragen in meiner grossen Geologievorlesung für die Erstsemestrigen am D-BAUG eingesetzt. Es war ein grosser Erfolg, nicht zuletzt wegen der Auflockerung, und ich bekam spontan viele positive Feedbacks».

Dr. Markus Kalisch (D-MATH): «Mit der EduApp bekomme ich sofortiges Feedback von den Studenten, selbst wenn die Vorlesung mehrere hundert Teilnehmer hat».

Dr. Meike Akveld (D-MATH): «Die EduApp gibt mir direktes Feedback darüber, ob verstanden wurde, was ich unterrichtet habe. Ich bitte immer einen der Studierenden die richtige Antwort zu erklären, was oft hilfreich ist. Ausserdem ist es für sie eine angenehme Abwechslung».

### Neue Funktionen für den Clicker

Pünktlich auf das aktuelle Semester wurden in der der EduApp neue Funktionen im Bereich Clicker hinzugefügt. Mit der Funktion «Clicker» können Dozierende via EduApp Fragen stellen, die meist sofort im Unterricht beantwortet werden.

1. Zwischenresultate: Neu können Dozierende die Abstimmung der Clickerfragen in zwei Runden machen und die Zwischenresultate anzeigen.

2. Erweiterter LaTeX-Editor: Der Funktionsumfang des LaTeX-Editors zur Anzeige von mathematischen Formeln in Clickerfragen wurde erweitert. Nicht nur können Dozierende jetzt Formeln und Gleichungen im Text eingebunden darstellen, es gibt auch mehr Textformatierungsmöglichkeiten.

3. Flashcards: Studierende können neu mit der Funktion «Flashcards» bestehende Clickerfragen durcharbeiten (z.B. zur Prüfungsvorbereitung). Die neue EduApp-Funktion “Flashcards” wurde durch den «the Rectors Impulse Fund» ermöglicht.

Mehr zu den neuen Funktionen finden Sie auf der EduApp Service-Seite und in der aktualisierte EduApp-Anleitung.

Posted on ,

## ETHZ & WSL announce first MOOC on Landscape Ecology

What is a landscape? How has it evolved? How do we perceive landscapes?

In this MOOC participants learn theory, methods and tools to understand the landscapes we live in and to solve landscape-related environmental problems.  Leading landscape ecologists present case studies from around the world, where research in landscape ecology is needed, both to improve our understanding of land-use systems and to guide land managers in their decisions.

Course start Sep 10, 2018, by Prof. Felix Kienast and Dr. Gregor Martius

Posted on ,

## Responsive Cities or Webcartography?

In May two new ETH-MOOCs will start. Please feel free to join these courses!

 Responsive cities define the future of urbanization. Learn how responsive citizens use smart technology to contribute to planning, design and management of their cities! “Responsive Cities” is a self-paced course and will be open from the first of May 2018 till January 2019. Want to produce cartographic visualizations to better communicate the results of your project or research? Or you are just interested in using modern web technologies for cartographic purposes? Then “Webcartography II” is the course for you! Starts May 15.

Take the opportunity to communicate with learners from all over the world about webcartography or responsive cities! For more details about these two courses and other ETH-MOOCs: https://www.edx.org/school/ethx.

Posted on ,