Musings on Assessing Interpretive Listening

listeningA couple of weeks ago I shared my thoughts about assessing interpretive reading.  In that post gave my opinion  that ACTFL’s IPA Template was a generally effective way to design an assessment of reading comprehension and that, with a couple of modifications, their rubric was well-aligned with the tasks on the template. I have reservations, however, about the use of the ACTFL IPA template to assess listening.  Here are a couple of my thoughts about assessing listening, please share yours!

Assessing Interpretive Listening is Important

By defining both listening and reading comprehension as Interpretive Communication, ACTFL has given us an out when writing IPA’s.  We can choose to include either one, but are not required to include both.   My guess is that when given the choice, most of us are choosing authentic written rather than recorded texts for the interpretive portion of our IPA’s.  There are several good reasons why this may be the case.

  1. Authentic written texts are usually relatively easy to find. A quick Google search of our topic + Infographie will often produce a content-filled, culturally-rich text with the visual support that Novice learners need. Picture books, ads, social media posts, etc. provide additional resources.  For our Intermediate students, our options are even greater as their proficiency allows them to read a wider variety of short texts, an unlimited supply of which are quickly located online.
  2. Written texts can be easily evaluated regarding their appropriateness for our interpretive assessment tasks. A quick skim will reveal whether a text being considered contains the targeted language and structures, culturally-relevant content, and appropriate visual support that we are looking for.
  3. Assessments of interpretive reading are easy to administer. We need only a Xerox machine to provide a copy of the text to each student, who can then complete the interpretive task at her own pace. When a student is absent, we can simply hand him a copy of the text and interpretive activity and he can complete the task in a corner of the room or any other area of the building where make-ups are administered.

Curating oral texts and assessing their interpretation, however, is considerably more time-consuming.  While we have millions of videos available to us on YouTube (my person go-to for authentic listening texts), videos cannot be skimmed like written texts.  We actually have to listen to the videos that our searches produce in order to evaluate whether they are appropriate to the proficiency of the students for whom they are intended.  In some cases, we have to listen to dozens of videos before finding that gem that contains the appropriate vocabulary, cultural content and visual support that our learners need.  When it comes to administering these assessments, we often face additional challenges.  In my school, YouTube is blocked on student accounts.  Therefore, I have to log into 30 computers in a lab (which is seldom available) or my department’s class set of IPads (sometimes available) for all of my students to individually complete a listening assessment at the same time. While many of us play and project our videos to the class as a whole, I think this places an undue burden on our Novice students who “require repetition, rephrasing, and/or a slowed rate of speech for comprehension” (ACTFL Proficiency Guidelines, 2012). A student who has her own device can pause and rewind when needed, as well as slow the rate of speech when appropriate technology is available.

In spite of these challenges to evaluating listening comprehension, I think we have a responsibility to assess our students’ ability to interpret oral texts. As Dave Burgess said at a conference I recently attended, “It’s not supposed to be easy, it’s supposed to be worth it.”  Assessing interpretive listening skills IS worth it. As the adage says, “we teach what we test.”  If we are not evaluating listening, we are not teaching our students what they need to comprehend and participate in verbal exchanges with members of the target culture.  While technology may allow us to translate a written text in nanoseconds, no app can allow us to understand an unexpected public announcement or participate fully in a natural conversation with a native speaker. In my opinion, our assessment practices are not complete if we are not assessing listening comprehension to the same extent as reading comprehension. As a matter of fact, I include separate categories for each of these skills in my electronic gradebook.  While others may separate grades according to modes of communication, I’m not sure this system provides as much information regarding student progress toward proficiency. Although both reading and listening may require interpretation of a text, they are clearly vastly different skills.  Students who are good readers are not necessarily good listeners, and vice versa. In their Proficiency Guidelines, ACTFL clearly differentiates these two skills, don’t we need to do the same when evaluating our students using an IPA?

Designing Valid Interpretive Listening Assessments is Difficult

 In my opinion, ACTFL has provided us very little direction in assessing interpretive listening.  While we are advised to use the same IPA Interpretive template, I find that many of these tasks do not effectively assess listening comprehension. Consider the following:

Key Words. While students can quickly skim a written text to find key words, the same is not true of recorded texts.  Finding isolated key words requires listening to the video multiple times and attempting to isolate a single word in a sentence.  I find this task needlessly time-consuming, as I will be assessing literal comprehension in other tasks.  Furthermore, this task puts some students, especially those with certain learning disabilities at a significant disadvantage.  Many of these students have excellent listening comprehension, but are not able to accurately transfer what they understand aurally into written form.

Main Idea. Although this task seems fairly straightforward, I question its validity in assessing comprehension for Novice and Intermediate learners. According to the ACTFL Proficiency Guidelines, Novice-level listeners are “largely dependent on factors other than the message itself” and Intermediate listeners “require a controlled listening environment where they hear what they may expect to hear.”  This means that all of my students will be highly dependent on the visual content of the videos I select to ascertain meaning.  Therefore, any main idea they provide will most likely be derived from what the students see rather than what they hear.  A possible solution might be for the teacher to provide a series of possible main ideas (all of which could be extrapolated from the visual information) and have the students choose the best one.  However, this task would certainly be unrealistic for our novice learners who are listening at word level.

Supporting Details.  I think this task on the IPA template is the most effective in providing us feedback regarding our students’ ability to understand a recorded text.  By providing a set of details which may be mentioned, we provide additional context to help our students understand the text and by requiring them to fill in information we are assessing their literal comprehension of what they hear.  In addition, this type of task can easily be adjusted to correspond to the proficiency level of the students. Providing information to support the detail, “Caillou’s age” for example, is a realistic expectation for a novice listener who is watching a cartoon.

Organizational Features While I see little value in this task for interpretive reading, I see even less for listening.  As previously mentioned, even intermediate listeners need to be assessed using straightforward texts so that they can anticipate the information they will hear.  Having my students describe the organization of a recorded text would not provide additional information about their comprehension.

Guessing meaning from context. As much as I value this task on reading assessments, I do not find it to be a valid means of assessing aural comprehension.  The task requires the teacher to provide a sentence from the text and then guess at the meaning of an underlined word.  As soon as I provide my students with a written sentence, the task becomes an assessment of their ability to read this sentence, rather than understand it aurally.      

Inferences As with the main idea, I think Novice and Intermediate listeners will be overly dependent on visual cues to provide inferences.  While I believe students should be taught to use context to help provide meaning, I prefer to assess what they are actually able to interpret verbally. ACTFL does suggest providing multiple choice inferences in the IPA template, but again the teacher would have to provide choices whose plausibility could not be derived from visual information in order to isolate listening comprehension.

Author’s Perspective. While I regularly include author’s perspective items on my assessments for my AP students, I feel this is an unrealistic task for Novice and Intermediate Low listeners.  Students who are able to understand words, phrases, and sentence-length utterances will most likely be unable to identify an author’s perspective using only the verbal content of a video.

Cultural Connections. Authentic videos are one of the best tools we have for providing cultural content to our students.  The content provided by the images is more meaningful, memorable, and complete than any verbal information could be.  However, once again it is difficult to isolate the verbal content from the visual, creating a lack of validity for assessment purposes.

 Conclusion

For now, I’m planning on using supporting detail or simple comprehension questions when formally assessing my students’ interpretive listening skills in order to ensure that I am testing what I intend to test. When practicing these interpretive skills, however, I plan on including some of the other tasks from the IPA template in order to fully exploit the wealth of information that is included in authentic videos.  I’m looking forward to hearing from you about how you assess your students on interpretive listening!

15 thoughts on “Musings on Assessing Interpretive Listening

  1. Tory

    There are ways to get around the You Tube blockage issue. There are websites that allow you to copy and download a video to your computer in QuickTime format and then you can share it with students on Google Drive. Keepvid.com is a good one.

    Reply
  2. Rebecca B.

    Really glad that you wrote this before I drafted my final exam. I’ve been feeling some hesitation about including interpretive listening as part of the exam, because I worry that I’ll end up testing their observation/visual skills more than anything to do with the language. In the Dark Ages, I just used the unit exams on the CDs from the textbook publisher, but at least I’m past that now!

    Reply
    1. madameshepard Post author

      I think we all did –at least they were native speakers– and my old days were only a couple of years ago!

      Reply
  3. Denise

    Thanks for your comments on using the ACTFL template for listening. I started experimenting with the reading one & found the same good/bad points you mentioned. I know I have some reading to do on the topic of proficiency, but do you know offhand at what level interpretive questions should be in the TL rather than English, according to ACTFL guidelines? In my practice, I have gone from English to all TL and am now questioning when & where using English would be appropriate.

    Reply
    1. madameshepard Post author

      I’m so glad you’ve found something you can use! Let me know if you have any questions.

      Reply
    2. madameshepard Post author

      Because I’m traveling I can’t give you a very definitive answer. However, the IPA interpretive template is in English and some of the sections require going from one language to the other. I start using French questioning on assessments in French 3, and all French in level 4. I try to include some of the same types of questions such as main idea, supporting detail and guessing meaning in context (but with a multiple choice French synonym).

      Reply
  4. Laura

    You are so right about the lack of transfer between reading and listening skills! I’m thinking of measuring along the 4 skills the proficiency guidelines are divided into anyway. What do you think?

    However I do think organizing structures and author perspectives can be useful in familiar formats like cooking videos or reality TV talent shows. Maybe those are the kinds of videos we should focus on for novices?

    Reply
    1. madameshepard Post author

      Yes, my grade book is divided into reading/writing/listening/speaking/misc. and each is weighted equally. The transfer between speaking and writing is, in my opinion, at least as tenuous as listening/reading. I definitely think those are great videos for novices. They provide the visual/contextual support that our students need. I do think the students use their knowledge of organizational structure/perspective to derive meaning from that type of video, I guess I’m just not sure that asking them to answer those questions provides me with additional information about their comprehension or allows me to give better feedback. What are your thoughts?

      Reply
  5. Cecile

    I agree 100% with your analysis. I spoke with one of the authors of “Implementing Integrated Performance Assessment” and he acknowledged that assessing the listening skill was still a work in progress. As you mentioned, I started implementing IPAs exclusively with reading as the interpretive task, but this past year I made it a goal to integrate listening also for a better rounded performance assessment. I use mainly main ideas and supporting details as drivers for my questions but I do occasionally use key word also, when a key word is repeated several times. This allows me to partially use the interpretive rubric, which I have modified to meet my needs… Thank you for this great post!

    Reply
    1. madameshepard Post author

      Thank you so much for your comment Cecile! I’ll be curious to see how ACTFL might change their template and rubrics for interpretive listening in future editions. Thanks for sharing your own ideas, too. I’d love to see your rubric, if you’d be interested in sharing.

      Reply
  6. Dawn

    I’m a little late getting to this conversation, but I have been struggling with assessing & teaching listening skills, too. I appreciate your thoughts–I had been finding the same things, and it is good to see I am not alone. I have not yet been using true IPAs, but I have been working with thematic units that assess in all modes and with all skills. My listening assessments so far have been a hybrid of textbook-produced audio with questions that I tailor to get more information and authentic audio (or produced-for-learners audio from the internet) with tasks that ask for discrete pieces of information.

    Reading your post, however, I wonder if a good strategy might be to use a written text followed by an audio or video source on the same topic as is done in the AP exam. The written text can provide schema & vocabulary that will assist the learners in understanding the audio, and the students can make comparisons regarding the difficult-to-assess features. For example, they might compare the authors’ perspectives or purpose; or you may have texts on the same topic from two regions of the world where students could make cultural comparisons. The difficulty, of course, will be in locating two sources that are closely enough related.

    Reply
  7. Pingback: Using Cartoons to Assess Interpretive Listening with Novice Learners - Madame's Musings

Leave a Reply

Your email address will not be published. Required fields are marked *