Skip to content

Shared Outcomes, Shared Practice: Evaluating an Instruction Program with One Assessment Technique (Guest Post)

By Meghan Wanucha Smith, Coordinator of Instructional Assessment, East Carolina University, wanucham16@ecu.edu

This post is based on a poster presented at 2018 Library Assessment Conference.

Background

Photo of Meghan Wanucha
Meghan Wanucha

At East Carolina University’s Joyner Library, librarians and library staff in Research & Instructional Services teach information literacy instruction for classes ranging from introductory composition to graduate-level research methods and use a variety of assessment techniques to gauge student learning. In previous program assessment efforts, we focused on lower-level composition classes with quizzes to test students’ abilities to use specific library resources. This time around, we wanted to know what students were learning in all of our classes to get a better sense of what the process of learning looked like in the entire instruction program. Could we design an easy-to-implement, shared assessment that would capture this information?

We started by tracking the learning outcomes that were taught and assessed the most across all of our sessions for one academic year. Then we designed a “vanilla” in-class worksheet around those shared outcomes as well as background information about students’ previous experience and knowledge. We tested this worksheet in a few classes in spring 2018 then launched in fall 2018, asking library instructors to use the worksheet in at least one instruction session.

Completed worksheets were collected and entered into Qualtrics for analysis. We created a rubric based on each shared learning outcome informed by the knowledge practices and dispositions from the ACRL Framework for Information Literacy. Each open-ended response was coded based on a rating of 1, 2, or 3 (highest). Responses that were left blank or with irrelevant information were removed from analysis.

Results

We analyzed a total of 349 worksheets used by 7 library instructors in 15 library instruction sessions, or 9% of our fall instruction sessions.Each open-ended question on the worksheet mirrored the research process, asking students to define their topic, find material, and evaluate the results:

  • Describe the research topic or question you practiced with during today’s session.
  • What kind of information sources are you required to use for your assignment?
  • Write down the title of the most useful source you found during today’s workshop.
  • How did you find this source? What tools, search terms, and/or filters did you use?
  • Why was this source the most useful one?

As to be expected, students’ research topics reflected the broad range of classes in which the worksheet was used. Many students expressed their topics in a few key words or a short phrase, most of which were relatively researchable with some narrowing in scope to search efficiently and effectively. Most students needed scholarly resources, which were mentioned in three out of five responses. Other common characteristics included reliable, credible, or peer-reviewed information sources, each mentioned about one in five responses.

We also asked students to identify the most useful source they found during the session, which proved particularly problematic for students: 30% either misunderstood the question or left it blank. We were attempting to capture a specific item title students found relevant during the session to match against source requirements and their research topics. In reality, many students offered statements about library tools or databases—while technically information sources, we actually wanted titles of articles, books, or websites.

About a quarter of responses mentioned using the library in some way to find that useful source, and about one in five used the library discovery layer, OneSearch, specifically. This makes sense in the context of an instruction session which tends to focus on library-specific resources (although not always).

The final session-specific question asked students to explain why their selected source was the most useful option. This was another problematic question for students with one-third either leaving the question blank or providing information that didn’t match the question. As this was one of the last research-related questions on the worksheet, this may be related to running out of time in the instruction session.

For general background information, we also asked students to tell us about what concepts were already familiar to them prior to the session. About a quarter (23%) of student responses mentioned knowing about library or librarian services. Databases were also mentioned as a familiar tool or concept (16%), as well as OneSearch, which was mentioned in 14% of responses. About one in 10 (11%) had heard about plagiarism or citing sources.

Lessons learned

Feedback from library instructors was mixed: The assessment was easy in that it did not take effort to create, however it was also too long and problematic for some to integrate into their teaching flow. We gained insight into what content is familiar to students beyond the typical “Have you had library instruction before?” feedback we solicit at the beginning of every session. We also captured a baseline of what students are comfortable doing—“translating” an assignment into the information types needed—and what needs work.

Most importantly, we assessed the assessment: It was far too complicated! We were asking students about FIVE separate learning outcomes in open-ended questions, resulting in quite a heavy cognitive load. Our spring 2018 testing of the worksheet had completely missed that problem: Only a couple of classes and instructors were involved in the testing, and those sessions were already designed around worksheets and well-planned content. We needed to collaborate and devote more time to the session planning process with library instructors so that all felt confident and comfortable with the new worksheet in any teaching scenario..

For spring 2019, we shortened the worksheet considerably to focus on the two key learning outcomes we were most interested in exploring and re-worded some of those problematic questions. Our hope is this results in better responses from students and an easier lift on library instructors to implement. We’re collecting these worksheets now and will be analyzing results as the semester wraps up.