Helping NY Philharmonic with redesigning their archives website by conducting usability testing
Abstract: The New York Philharmonic, was interested in conducting usability testing of their current digital archives site with the help of Pratt Institutes Center for Digital Experiences. The primary focus of this project was to provide the NY Philharmonic archives, important findings and feedback from the user tests, to help them make informed decisions for the redesigning of the website. In this usability study, the “think-aloud” method was used and the tests were screen recorded as a form of raw data. Though the users were happy about the aesthetics and breadth of information available on the website, they experienced some usability issues with search results, while navigating the website and other minor findings.
About Client
The New York Philharmonic is one of New York City’s most prestigious cultural institution. Their digital archives website consists of 175 years of philharmonic history and more than 16,000 performances around the world. In 2011, they began their digital archiving project, cataloging and digitizing over 4 million pages or material, making it the “oldest and most comprehensive collection of any symphony orchestra”. The archive contains scores, music parts, printed programs, business documents, planning folders and photographs. The digital archives received an endowment from the National Endowment for the arts to overhaul their website and bring it into the 22nd century.
Our Team
Our Goals
-
Improve the exploratory experience for new users landing on the site, and
-
Evaluate and improve the experience for experts/researchers seeking out specific archival information.
My Role
- Formulating Screening Questionnaire
- Moderating 2 eye-tracking sessions
- Being a note-taker for 2 eye-tracking sessions
- Analysing Findings
- Providing Recommendarions
- Creating and designing Mockups on Figma
- Delivering Presentations to Clients
Client Kick-Off Meeting
Our first client meeting took place on a zoom call. All team members attentively listened to the client and asked the questions about the project to understand the project and client’s needs. Talking to the client cleared up many doubts and helped us to understand the type of users the site has. We asked the client if they can help us with expert users and participants for our study, which the client was happy to help with. In the client meeting, the client also mentioned that they desire to make the site more accessible to users with disabilities. The client also provided us with data analytics files for their website.
Preparation for recruiting Participants
Followed by the meeting, our group started to work on the materials which we can use to recruit participant. Drafting screening questionnaire and social media drafts was my responsibility. Other materials such as user profiles, tasks, post-test questions, consent form were drafted and reviewed by each member.
User Profiles
Participants
Our team reviewed the data we gathered from the google forms which we used for recruitment to decide which participants to select for the user testing interview. We preferred users who are Amateur’s (first time users) and Professors (expert users) with Explorers as a secondary group to gain insights and perspectives from different types of users.
We recruited 10 participants, with 3 “Expert” users and the remaining falls into the category of “Amateur’s” and “Explorers”. Users were working or a student in fields like research, music, education, genealogy, archives.
Participant’s Field or Study area
Tasks
Tasks were drafted according to the major key areas identified by our client an our team. Users were asked to “think-aloud” while navigating through the tasks. Open ended post task questions were asked to understand user’s thought process.
Below are the five tasks that were presented to users for the user test :
Task 0: Once you land on the home page, tell us any first impressions, thoughts, or anything else that comes to mind.
Task 1:
(a) Find a program that includes William Lincer, a member of the New York Philharmonic orchestra in the mid 1900s.
(b) Once you have found a program, find an image of William Lincer that is not included in this program.
User Goal:
I want to find a resource connected to my family member who I know was a member of the orchestra.
Task 2:
(a) Find the music for Tchaikovsky’s Nutcracker Ballet.
(b) Find the sheet music for a clarinet.
User Goal:
I want to find sheet music of a piece I want to play.
Task 3:
(a) Find the New York Philharmonic Archives’ digital exhibition about its first female musician.
(b) Once you have found the exhibition, find out where she was born.
User Goal:
I want to understand what types of resources are available through the NY Phil Archives website and find something that’s interesting to me.
Task 4:
(a) Find the New York Philharmonic Archives’ newly digitized set of press clipping scrapbooks, which are featured on their homepage.
(b) Once you have found the set, locate a press clipping scrapbook that documents one of the New York Philharmonic’s park concert series.
User Goal:
I want to explore newly digitized resources that are relevant to my research interests.
Task 5:
(a) Now, imagine you’re researching how orchestras like the New York Philharmonic reached audiences during the COVID-19 pandemic, which began around March of 2020. Find a recording of one of the virtual performances offered by the Philharmonic in the first few months of the pandemic.
(b) Going back to your search results, which soloist performed the most times during the first few months of the pandemic?
User Goal:
I want to find contemporary musical recordings and records for research.
Post–task questions:
1. How difficult was this task to complete?
2. How successful do you feel you were in completing this task?
Consolidated Analysis
After all the user tests were conducted, each team member addressed the problems their users faced on an excel sheet. All of these problems were consolidated and analyzed by each team member using the rainbow sheet method. Based on the the number of issues which were repetitive to different users , we came up with 5 recommendations according to the priority and severity of the issue.
Rainbow Sheet
Findings and Recommendations
Overall, Users liked the aesthetics of the website and found the breadth of information to be very useful. They used adjectives like attractive, well-designed, detailed and interesting while describing the website. They also commented positively about the type of resources available on the site.
Digital archives received an SUS score of 60.3. Our team identified 48 problems in total in our consolidated analysis. These problems occurred in locations such as search results, Home, Performance History, Resource, Features and All pages.
Finding 1 : Navigation, Usability and Accessibility Challenges
Main navigation and search bar on the homepage
During task 0, when users were asked to visit the homepage , several users faced issues with main navigation bar as it was styled in the same way and there was no indication of active page which lead to users ending up clicking on 'Digital Archives' thinking it will lead to another page. There was no proper signifier when the mouse was hovered over the text to indicate if the text is interactive or not, affecting the usability and accessibility of the website.Some users also noted issues with the size of the navigation bar.
Recommendation 1 – Improve navigation usability and accessibility by updating styling, spacing, and text.
Mockup 1
1.1 ) Add link styling to the navigation labels to make it clear that they are interactive links (and that they signify the active page)
1.2 ) Change the “Features” label to “ Exhibits”
1.3 ) Add “…from the New York Philharmonic” to the explanatory text by the search bar
1.4 ) Shrink the top navigation slightly to reduce white space
1.5 ) Additional adjustments to improve accessibility of main navigation and search
Finding 2 – Challenges navigating the carousel and with button contrast
Carousel on the homepage
The above screenshot shows a carousel slide with barely visible navigation buttons which confused the user on how to move to the next slide of the carousel.
Recommendation 2 – Update style and navigation elements on the carousel to make it more intuitive and accessible.
Mockup 2
2.1 ) Change the color of navigation buttons
2.2 ) Unhide the navigation arrows
2.3 ) Change the color of the ‘Explore’ button
Finding 3 – Results page layout prevented users easily navigating and filtering search results
Search results page layout
Two key issues were noticed in the search results page layout:
1 ) Search results showed 'No results' if there was no result available under 'Programs' section which lead to users thinking that there are no results overall as they missed seeing under other available sections like 'Scores', 'Parts', 'Images' and 'Business Documents'.
2 ) Most of the users did not notice the 'Sort by' option available at the right corner.
Recommendation 3 – Streamline search results page with consolidated sort and filter options.
Mockup 3
3.1 ) Consolidate sorting and filtering options.
3.2 ) Display search results content intuitively for users by adding “All Results” and displaying results consistently.
Finding 4
Search filter pop up modal
1 ) More than 200 pages are available in the search filter which ends up being overwhelming for the user
2 ) Dropdown options for soloists, conductors, and composers/works
3) Overlay of pop-up modal
Recommendation 4 – Move the pop-up modal underneath each filter, introduce a search bar for the filter options, and remove the additional layer of filtration options
Mockup 4
4.1) Move the pop-up modal underneath the filter options
4.2) Introduce a search bar under the filter options
4.3) Remove the additional layer of filtration options
Client’s reaction and
Next Steps
After delivering our presentation, a discussion followed in which the client mentioned that they want to improve and get more ideas about the carousel placement and functionality. Overall, the client was impressed with the amount of data and insights we’ve gathered.
If I could continue working on the project, I would like to see what changes they’re implementing and would love to conduct usability tests on the new changes to see how usable and accessible these changes are.
Takeways
The lesson I learnt from this project is “teamwork makes the dream work.” When everyone is involved in the project , the overall results would be superior . Communication and equal participation are the key assets for the team to work out well.
Click on the buttons below to see the report and presentation slide deck :
Conclusion
The NY Philharmonic Digital Archives website have plethora of archives, information and content available at convenience. They are working on their archives website to improve the discoverability and accessibility of the archival material. The 5 year grant project is funded by the National Endowment For the Humanities, and to update the work which is being conducted by them they partnered with Pratt Institute’s Centre for Digital Experience. Usability Tests were conducted with the help of 10 participants who completed 5 tasks over a zoom call while in the presence of one moderator and one note taker from the team.
Though , the website is rich with information and users had positive first impression of the website, the moderated user testing conducted on the archives website revealed some usability issues on locations such as search results page, carousel navigation, filter options and other minor issues.
The recommendations presented in this case study presents some quick fixes for the identified problems. Our team is confident that implementing these recommendations will improve the overall user experience and enrich the experience of the users navigating the digital archives website.