Moodle is the primary course content platform at UMass Amherst, hosting online course pages for over 30,000 instructors and students.
At a glance
Timeline
14-week research project at UMass Amherst
My Role
Research, design, delivery, and analysis of usability testing
UI design samples
Objective
Conduct user research for Moodle to understand its strengths and weaknesses.
Team
Advisor
Janine Solberg
IT Stakeholders
Iris Chelaru, Rachel Shipman
Research and Design
Kylie Bane, Sohini Banerjee,
Emily Chen (me), Christiana Lee
Impact
A platform that checks all the boxes
1 year after this project, UMass announced a campus-wide transition to Canvas, beginning Fall 2023.
Reasons for the switch include Canvas’s mobile-first approach, and reduced complexity by having a single, university-wide LMS, both key priorities highlighted in our research.
Our team was credited in the Annual Report (p.10) from the UMass Information & Communication Technology Council (ICTC) for our contributions to improving the experience of IT services!
Process
Scoping the project
Exploring the problem space
Key insights
Design: Exploring solutions
Reflection
Background
What exactly is Moodle?
Moodle is a course content platform. Its competitors include Canvas, Google Classroom, and Blackboard.
In Fall 2021, the UMass Amherst IT department began exploring the possibility of switching to a new course content platform.
This discussion followed nearly two years of online instruction due to the COVID-19 pandemic. The whole educational landscape had changed, and UMass IT wanted to understand how the university should adapt.
Stakeholder discussion
We spoke to two members from IT to align on our project objectives:
Dan Cannity - Manager, Instructional Design and Technology
Anthony Laffan - Senior Manager, IT Service Desk
Together, we outlined the following research goals:
Identify short-term actions that could improve Moodle functionality with relative ease
Identify high-level problems or “wish-list” features that could be taken up for a long-term overhaul of Moodle, or as information to guide the selection of a new platform
Distinguishing our users
Moodle serves two groups of users: instructors and students.
A platform is only as good as its ability to serve its users. In order to assess Moodle, we had to first look at who we were trying to serve.
🧑🏫
Instructors
Structure, create, and deliver course content
Shape learning experience for students
🧑🎓
Students
Interact with course content
Determine effectiveness of the university
In a university setting, the role of an instructor is to help students learn.
Students, on the other hand, are ultimately who the university aims to serve—their primary users. Their outcomes determine the extent to which a university has met its goals.
So, we proceeded to assess Moodle through the student experience.
Determining our methods
What’s the best way to understand students’ relationship with their online course content?
Before we even looked at Moodle, we needed to get some context. What problems do students have, and when are they going to Moodle to solve them?
The IT department didn’t have data on students’ day-to-day experiences. They provided us with a 2011 survey and help desk data which only gave us insights into glaring issues with Moodle.
In an ideal scenario, we would be able to shadow students during a typical weekday, but this would be way too time-consuming as students ourselves.
Usability testing seemed like the best way to observe how students interact with Moodle in a realistic setting, within a reasonable amount of time.
Follow-up interviews would help us contextualize their behavior in their everyday experiences.
Designing our usability test
An interview and 7 tasks covering basic Moodle functionality
Instructors have full reign over how their course pages are designed, so no two Moodle pages are the same. For our usability test, we decided to create a mock Moodle course to include (and then test) the basic features that Moodle offers, as well as features used by our own instructors.
Our team compiled screenshots of Moodle pages from our own classes, Sohini from a class that she taught, and then we got building—thus, English 112 - College Writing was born.
Our mock Moodle course
We recruited 9 participants, each coming from a different major.
Instructors from different departments might use different Moodle functions and external sites depending on their needs. This can create a large disparity between how students experience Moodle.
To ensure that our participants would cover a large breadth of needs and use cases of Moodle, we selected for a wide variety in major.
Analysis overview
The amount of data we came out with was overwhelming.
We tackled this by allocating different tasks and interview questions to different team member, which we would each analyze across all 9 participants using affinity diagramming and thematic analysis.
Part I. Insights on the student experience
1. Students rely on Moodle for nearly all of their course information.
As the primary system used by UMass instructors, Moodle is almost always the first place that students go to find important information.
Some examples of when students use Moodle:
In the morning, to check what they have due for each class
During class, to open a file or follow along with lecture slides
Between classes, using their phone to check a recent exam grade
After class, to find a reading for their assignment
Submitting their assignment online
Taking an exam
Locating the Zoom link for an online class
Part II. Evaluating Moodle
1. Customizable…to a fault
There is a lot of information hosted on Moodle. Instructors have full control over their course page structure, so important information can be organized differently from course to course. Instructors also frequently use external sites for additional functionality.
The lack of consistency places the responsibility on students to find and keep track of information, which is not only time-consuming, but also causes unnecessary stress and uncertainty.
“I rely on Moodle to tell me when something is due, but I’m always anxious that I’m missing something.”
Noah*, Math Major
*Names have been changed
“If Moodle courses were structured the same, it would be a lot easier to find things.”
Elizabeth, Informatics Major
2. Mobile experience could use more attention
Google analytics revealed that upwards of 10% of Moodle’s traffic came from mobile devices. Our participants shed light on this, saying that they use Moodle on their smartphones and tablets when they can’t reach their laptops easily, or when they want to check something quickly, like a grade.
“[Moodle] is not formatted well for a vertical display.
John, Computer Science Major
“Sometimes the left navigation bar blocks the entire screen and won’t close.”
Jacob, Biology Major
3. The homepage is often overlooked
The Moodle homepage currently hosts site announcements and navigation to other pages. Most participants seemed to view the homepage as a step they had to go through to get to their courses.
“I look straight to the sidebar to get to my classes. The site announcements and maintenance windows have no importance to me, so I never really look at that.”
Nathan, Biochemistry Major
“I’m noticing the gray boxes above the orange—like the dashboard, calendar, all that stuff—but I’ve never clicked on them before.”
Elizabeth, Informatics Major
Moodle homepage
We have all these recommendations…now what?
As we were writing our final report, I started thinking about what our stakeholders would do with this information.
As the people directly troubleshooting with instructors and students, they were acutely aware of common issues with Moodle. But knowing about the issue doesn’t necessarily make it easy to identify a solution.
With the remaining time we had, Christiana, Kylie, and I created some wireframes to start thinking about how some of the issues we found could be addressed.
We weren’t able to do this for every single suggestion, but our goal was to lighten the mental load on our stakeholders by demonstrating how our suggestions could translate into tangible change.
I focused on redesigning the homepage.
Despite being the one page that all Moodle users cross, hardly anyone ever interacted with it. It was a huge opportunity to provide value, and address one of the largest issues we found in students’ experiences.
The problem
The current layout of Moodle places the responsibility of searching for, and keeping track of important information on its users.
How might we…connect students directly with high-priority information?
Pain-points, and their opportunity
Pain points
The homepage content has little relevance to students, who regularly look past it to get to their course pages.
Students often have to track important information across disparate platforms.
Course page structure, and thus experience of Moodle, varies drastically depending on instructor preferences.
Opportunity
The homepage can be used as a destination for curated, high-priority information, similar to a news feed.
Categorizing information across courses, not just within them, can help students decide what to prioritize (without needing to navigate inconsistent course pages).
Prototyping
Final Prototype
-
The most common pages that students access, migrated center-stage from the sidebar.
-
Anxiety-inducing events like deadlines are hidden beneath assignment details in current Moodle. This is a clear-cut way to show students what’s coming up.
-
Moodle site announcements are important, but not always relevant. Grouping time-sensitive information from IT and instructors can encourage students to check back for updates.
-
With the help of UMass branding guidelines and online inspiration, I learned how to create visual distinction between sections, emphasize information, and create visual consistency.
Before
The design process is a guide, not a rule.
As my first formal UX project, I had high hopes walking in. It seemed like this would be the opportunity where we’d be able to walk through the design process in its entirety! Spoiler alert: we didn’t. And I wouldn’t for the next project I worked on, either.
This was real life, which had a lot of constraints. It was a 14-week project, and for a lot of it, we were learning as we went—understanding what we had to do and how to do it, in parallel. This meant we had to…
Prioritize, prioritize, prioritize.
If it wasn’t for Dr. Solberg’s help, I would have been lost at sea with the amount of data that came out of our usability test. Working with her and our team showed me that the most important part of our work is making sure that our stakeholders hear what our users need most.
If we tried to do everything perfectly, we would have spent forever doing it. And those insights would have never seen the light of day.
