{"id":1391,"date":"2020-11-27T15:28:31","date_gmt":"2020-11-27T14:28:31","guid":{"rendered":"https:\/\/wpethzprd.ethz.ch\/letblog\/?p=1391"},"modified":"2020-12-02T11:02:54","modified_gmt":"2020-12-02T10:02:54","slug":"how-meaningful-are-clicker-data","status":"publish","type":"post","link":"https:\/\/blogs.ethz.ch\/letblog\/2020\/11\/27\/how-meaningful-are-clicker-data\/","title":{"rendered":"How meaningful are clicker data?"},"content":{"rendered":"\n

Contributors<\/strong>: Meike Akveld (D-MATH), Menny Aka (D-MATH), Alexander Caspar (D-MATH), Marinka Valkering-Sijsling (LET), Gerd Kortemeyer (LET)
<\/p>\n\n\n\n

Among other things, ETH Zurich\u2019s EduApp<\/em><\/a><\/em> allows instructors to pose clicker questions<\/a> during lectures. Instructors can interrupt lectures to ask questions from the students and get and give feedback on learning progress<\/a>. Lecturers can also trigger phases of peer-instruction<\/a>, where students discuss their initial answers to a question with one another and then reanswer the question – in effect, the students are teaching each other during those phases, thus \u201epeer instruction\u201c. By asking students to answer a question twice, lecturers gather data on student understanding. But how meaningful is this feedback data, in particular, when answering is voluntary and ungraded?<\/p>\n\n\n\n

A group of mathematics instructors at ETH\u2019s D-MATH worked with LET to analyze EduApp<\/em> data using Item Response Theory (IRT), Classical Test Theory (CTT) and clustering methods. Over the course of the semester, 44 clicker problems were posed \u2013 12 of them twice, as the instructor decided to insert a phase of peer-instruction. The following figure shows an example of the kind of problem being analyzed: <\/p>\n\n\n\n

\"\"
Fig.1 Example of a clicker problem<\/figcaption><\/figure>\n\n\n\n

The problem shown was used in conjunction with peer-instruction; the gray bars indicate the initial student responses, the black bars those after the discussion. A simple, unsurprising observation is that after peer-instruction, more students arrived at the correct answer. What can we learn from these responses? CTT and IRT can provide psychometrics that help understand this instructional scenario.<\/p>\n\n\n\n

When it comes to being \u201emeaningful,\u201c the \u201ediscrimination\u201c parameter of a problem is of particular interest: how well does correctly or incorrectly answering a problem distinguish (\u201ediscriminate\u201c) between students who have or have not understood the underlying concepts?<\/p>\n\n\n\n

CTT simply uses the total score as a measure of \u201eability\u201c, but also has a measure of discrimination (\u201ebiserial coefficient\u201c). IRT estimates the probability of a student arriving at the correct answer for a particular problem (\u201eitem\u201c) based on a hidden (\u201elatent\u201c) trait of the student called \u201eability\u201c \u2013 typically, higher-ability students would have a higher chance of getting a problem correct. How exactly this probability increases depends on problem characteristics (\u201eitem parameters\u201c).<\/p>\n\n\n\n

In IRT, the ability-trait is determined in a multistep, multidimensional optimization process, where the difficulty and discrimination parameters of particular problems (\u201eitems\u201c) feed back on how much correctly answering that problem says about the \u201eability\u201c of the student; \u201ehigh-ability\u201c students are likely to get correct answers even on high-difficulty, high-discrimination problems.<\/em><\/p>\n\n\n\n

The results of their study were extremely\nencouraging: using both CTT and IRT, almost all 44 problems under investigation\nexhibited strong positive discrimination in the initial vote. This means that\nthe better the student understood the underlying concepts, the much more likely\nthey were to give the right answers \u2013 and vice versa. A low\ndiscrimination, on the other hand, means a problem provides less meaningful\nfeedback. For the handful of problems which had lower (yet still meaningful!)\ndiscrimination, this could be explained by other problem characteristics, for\nexample, that at the time they were posed, they were still too hard or already\ntoo easy \u2013 but even that feedback is meaningful to the instructor for future\nsemesters.<\/p>\n\n\n\n

The truly surprising result of the study was that in all cases of peer-instruction, the problem had even stronger discrimination afterwards! Yes, unsurprisingly more students answer correctly after discussion with their neighbors (the problem becomes \u201eeasier\u201c), but: peer-instruction does not simply allow weaker students to enter the correct answer, it apparently helps them to perform at their true potential.<\/p>\n\n\n\n

For the purposes of the study, the clicker data had to be exported manually, but the next version of EduApp<\/em>, slated to be released in December 2020, will allow export of data for learning analytics purposes directly from the interface \u2013 the following figure shows a sneak preview of that new functionality.<\/p>\n\n\n\n

\"\"
Fig. 2 The new “Learning Analytics” function in EduApp<\/figcaption><\/figure>\n\n\n\n

The exported data format is compatible with\ninput for the statistics software R<\/strong>, and there are variety of guides\navailable for how to analyze this data (https:\/\/aapt.scitation.org\/doi\/abs\/10.1119\/1.5135788<\/a>\n(accessible through the ETH Library) provides a \u201equick-and-dirty\u201c guide).<\/p>\n\n\n\n

The full study, including results from\nClassical Test Theory and clustering methods, as well an outlook for new EduApp<\/em>-functionality\nis available open-access in Issue 13 of e-learning and education (eleed)<\/em>\nunder https:\/\/eleed.campussource.de\/archive\/13\/5122<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Contributors: Meike Akveld (D-MATH), Menny Aka (D-MATH), Alexander Caspar (D-MATH), Marinka Valkering-Sijsling (LET), Gerd Kortemeyer (LET) Among other things, ETH Zurich\u2019s EduApp allows instructors to pose clicker questions during lectures. […]<\/p>\n","protected":false},"author":1537,"featured_media":1392,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"yes","footnotes":""},"categories":[249626],"tags":[293469,175064,175073],"class_list":["post-1391","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-featuring-teaching","tag-clicker-questions","tag-eduapp","tag-learning-analytics"],"_links":{"self":[{"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/posts\/1391","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/users\/1537"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/comments?post=1391"}],"version-history":[{"count":0,"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/posts\/1391\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/media\/1392"}],"wp:attachment":[{"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/media?parent=1391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/categories?post=1391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.ethz.ch\/letblog\/wp-json\/wp\/v2\/tags?post=1391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}