The More Complex the Less Success in Online Library Services: Evaluating the User Experience for International Students

Fatih Demir
Northern Illinois University

Wilson Hernandez Parraci
Northern Illinois University

The growing importance of web usability testing has led to improvements on millions of websites around the world. User Experience (UX) research involves use methods and procedures that inspect User Interfaces (UI) to identify the usability problems, pros, and cons of the design, and the level of user satisfaction (Demir, 2011). The increasing attention to the online applications has posed unlimited challenges, not just for the novice users, but also for the designers and creators of technology.

As the demand for more efficient and powerful communication systems and technological equipment grows, the need for manufacturing companies to improve their products and services increases. Consequently, engineers and multimedia designers are continuously conducting usability tests to make their products more appealing and competitive in the global technology market. Usability research is a way to be responsive to the needs of consumers by systematically evaluating users’ behavior in a controlled environment (Clark, De Leon, Edgar, & Perrin, 2012). In regard to the library services, usability studies play a vital role in making sure that users can find the desired information on library websites swiftly and precisely (King, 2003).

Literature Review

To enhance the user experience, international organizations created sets of guides for user interaction, such as standards, guidelines, and style guides to improve the usability conditions of websites and online services (Stone, Jarrett, Woodroffe, & Minocha, 2005). The purpose of the standards is to be aware of the functionality of technological advances and to make sure that they meet the needs and expectations of targeted audiences around the world. For example, a user interface standard consists of a set of internationally agreed principles focused on the human-centered design of multimedia user interfaces. Among the most trusted and well-known standards for usability evaluation is the International Standard Organization (ISO) which has created a number of human-centered standards for Multimedia UIs design such as human-centered design processes for interactive systems (ISO 13047). ISO standards refer to a product that can be used by a specified group of users “to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (Silva & Wijayaratne, 2015, p.138). These standards are fundamental to the improvement and development of the World Wide Web.  They are not just important guidelines for website development and design but also a standardized service to users around the world.

Usability testing is a very common method of evaluating websites. Even a small-scale usability test is essential and can capture design problems before the release of any website or service. Incorporating what is learned through usability testing into website modifications or redesigns leads to user-centered websites, and thus, more successful services (VandeCreek, 2005). In the last decade, there have been usability studies on the library websites that have focused on the students’ interaction with the system. Research done by the Helsinki Library Website (HLW) on the perspective of students found significant information gaps between the presented information by the library and the information desired by the students (Shumova, 2017). Moreover, the research results highlighted the overwhelming complexity of the library website as well as the issues with the ‘search box” used on the library website.

Proficiency in English when it is not a user’s native language is another obstacle for library web users, along with unclear information links and icons (Barisic, 2017). It is inevitable that the language barrier when using library websites poses difficulties, not only for English as a second language (ESL) students but also for some English native speakers. An important study on library websites was conducted with the Hong Kong University Library Mobile (HKUL) website. The researchers conducted a heuristic evaluation and found that HKUL had some usability issues, such as the inability to inform users of the waiting time for resources, the illogical organization of some information, the inconsistency in design, the lack of advance searching mechanisms, and the inadequacy of help resources (Reese, Dickson, Eddie, Kewin, & Patrick, 2016). Furthermore, Porat (2016) provided a selective review of current research and practice on user feedback in academic libraries. He divided the user feedback studies into four categories: Library as Place, Navigation, Satisfaction, and User Experience. His study addressed issues such as the use and navigation of the physical space and website, the degree of satisfaction, and overall experience. Although the researcher used certain informal methods to collect data, such as website feedback forms, suggestion boxes, and social media, his study mainly focused on a selection of systematic assessment processes undertaken to provide insight and to determine trends. His research mainly suggested improvements implemented as a result of survey findings.

When addressing usability studies on library websites, it is inevitable to deny the role of Information Communication Technologies (ICT). The recent developments in ICT have changed some people’s lives but have also posed a significant challenge to many others. The invention of new apps has contributed to the market, mobile websites, and virtual services and influenced the design of modern library web services, which has raised the level of complexity for the users in some cases (Liu & Briggs, 2015).   

To cater to users’ needs in the ever-expanding mobile technological world, higher education institutions have begun to mobilize their teaching and learning services. The implementation of mobile technology has created a new approach to promoting library use and value. It is expected that this will impact how libraries deliver services to their users in the future (Hashim, Tan, & Rashid, 2015). Many libraries have also been engaged in innovating high-tech and user-oriented services in non-traditional ways to provide students with a satisfactory learning environment (Mikkelsen & Davidson, 2011). To respond to the changes in the information society and user behavior, libraries should redefine their future functions and services to attract users of various physical and virtual resources (Jeng, 2005). On the other hand, adoption of new technology should be designed in a meaningful way to lead to ease-of-use and to reduce cognitive workload (Demir, Ahmad, Calyam, Jiang, Huang, & Jahnke, 2017). Research on the Library Mobile website of a university library in Taipei, Taiwan, investigated users’ reactions to the new mobile platform for the university library use. The study found that library websites using a mobile platform were more effective and accessible to fulfill students’ and users’ academic and leisure needs (Chanling & Hung, 2016).  

There are various case studies that evaluate the library web services in terms of usability and ease of use. Jang (2005) found that there is a strong relationship between effectiveness, efficiency, and satisfaction and users' preference to use the services. The research results show that the vast majority of library users prefer open-access library sources, followed by Google Books, through user-friendly interfaces because they provide ease of to access desired information and full-text documents (Matthew, Gilok & Lindsay, 2012). Much research conducted on the usability of the library websites suggests that digital library services must be designed in a way that is clear in navigation, simple, useful, and supportive of multiple information tasks (Warwick, Rimmer, Blandford, & Buchanan, 2005). A usability study on web services research results at King Khalid University in Saudi Arabia also indicates that complexity in design, lack of error recovery and inappropriate labeling causes failure in efficiency and effectiveness as well as decreases user satisfaction with the digital services (Jabli, Alghamdi, & Demir, 2018). Most library users tend to search for information on library websites. However, the research demonstrates that inexperienced library users having difficulties with the search features and the search engines that do not support users' activities well enough. Because lack of user guidance such as lack of suggestions, lack of system error recovery support, and inappropriate marking of the mandatory form fields may cause failures in accessing desired information (Zhang, Liu, Li, & Zhang, 2009).  

Although there is a wide variety of usability studies of library websites, only a few studies target international students and ESL students who use online library services. For this reason, this study focuses on the fundamental difficulties that international students face when interacting with such complex library websites. The targeted library website was evaluated by VandeCreek (2005), who found that “users reported problems when using Inter-Library Loans (ILL) services due to the unfamiliar terminology and poor design" (p.181). However, no other usability evaluation has been conducted since the website was redesigned based on the previous evaluation results. This research was the first to evaluate the new look of the library website. The study mainly focuses on the issues with user needs, user experience, navigation, and use of tools.

A central argument is that the basic underlying concept of usability is to make things compatible for people, which is to say that the things that people use to complete a task should not hinder the completion of that task by the nature of their design (Peterson, 2007). Additionally, the research team needs to consider that “all product design should begin with a thorough review of potential users” (Nielsen, 1993, p. 46) to determine the usability of the system. This evaluation process commenced with the analysis and perceptions of the user because "usability is both a philosophy to be followed during the design phase of the product development and a property of the final product to be evaluated, typically referred to as usability testing" (Peterson, 2007, p. 338). The purpose of this research was to take a concise picture of the current practices of a Midwestern college’s library website and evaluate to what extend useful to the international students.

Research Questions

The research questions are aimed at the three dimensions of usability (effectiveness, efficiency, and user satisfaction).

1. How much time do users spend to accomplish pre-determined tasks at the library web site?
2. How many pre-defined tasks do users complete successfully?
3. What is the satisfaction level of the users with the library web services?

Methods

The study was launched in the fall of 2017, and it was completed between November and December 2017. Accordingly, a limited number of participants (N=6) were invited to participate in the research to test the pre-defined tasks and to identify the design issues for the better development of the interfaces. Only international students were recruited for the study. A usability study was designed with pre-defined tasks. The System Usability Scale (SUS) survey was offered to measure participants’ subjective satisfaction with the library services. The SUS survey is a UX industry standard Likert type 10-items survey instrument that is widely used in UX research; its validity and reliability were proved with thousands of research to determine subjective satisfaction level of the users (Demir, Karakaya, & Tosun, 2012).

The study consists of phases, including (a) determining the demographics; (b) task test to analyze task completion success (effectiveness) and task completion time (efficiency); (c) subjective satisfaction with the web services by System Usability Scale (SUS) survey, and; (d) semi-structured post-task interview to get feedback about the participant’s experience.

Each participant attended an individual 1:1 session. Prior to each session, the researcher informed the participant about the details of the usability study and asked the participant to fill out the demographics survey. Then, the pre-defined tasks were offered one by one to allow the participant to complete one task at a time. The task session was followed by the satisfaction survey, and a semi-structured interview was conducted to gain information about their experience with the service and design as well as the obstacles they faced and their design recommendations.

The demographics and SUS survey were collected through printed survey instruments where effectiveness and efficiency rates were observed by the researchers. A stopwatch was used to monitor task completion rates, and an excel sheet was filled out for task completion success rates.  Additionally, participants’ interaction was recorded with a screen recorder software for further analysis. Post-interview feedback was transcribed to gain a better understanding of the users' perception.

Pre-defined task instruments

The participants were asked to complete the tasks one at a time on the college’s library website as follows:

• Task 1. Log in to the college’s library website with the credentials given by the researchers and check out the book by Steve Krug (2014). Don’t make me think, revisited: A common sense of approach to web usability; Third edition, Berkeley, CA: New Riders.
• Task 2. Look up the article “The revised learning process questionnaire: A validation of a Western model of students' study approaches to the South Pacific context using confirmatory factor analysis.” by Phan, H. P. & Deo, B.  
• Task 3. Search with the following keywords “Evaluation, education, research” and indicate how many full text online articles are available at the College’s Library Website?
• Task 4. Identify whether the College’s library provides Smart Classroom Reservation Service?
• Task 5. List the cultural events and academic activities advertised at the College's Library website?
• Task 6. Look for the number of e-books available using the keyword: "SPSS".

Procedure

Participants were scheduled for one-on-one sessions individually. First, the participants were given a consent form to sign. Second, they answered a demographic questionnaire. Third, the participants were assigned six tasks, one at a time, with no time limitation. All of the participants were given the same task list. The time was kept by the researches using a stopwatch from starting the task until they say they are done with the task. The success or failure with the tasks was also recorded in an Excel table by the researchers.  Upon completion of the pre-defined tasks, the participants were asked to fill out the SUS Survey. Finally, they participated in a semi-structured interview. The task test sessions were recorded using a screen recorder software called Flashback Pro in order to gain a deeper understanding of participants’ interaction and behaviors, such as facial expressions and mimics; the task test was recorded by an external camera as well.

Results

Demographics

All the participants were international students, three from China, two from India, and one from Saudi Arabia. The participants were selected with various educational backgrounds and fields of study; one person was from the Anthropology program, two were from the Engineering program, one was from the Health Department, and two were from the Computer Engineering program. Five of the six participants were between 20 – 30 years old, and one was above 40 years old. There were four male and two females. Four of the six participants indicated that they visited the library in-person once a week, one between two to four days a week, and one person visits the library regular daily basis. Regarding the level of expertise in technology, two of them identified themselves as beginners, two were intermediate, and two were advanced. All participants reported that they had some experience with the technology as the users of smartphones, tablets, laptops, educational websites, LMSs, library websites, and other kinds of communication applications available in the market.

Effectiveness Scores

Effectiveness is commonly measured through participants’ use of a product to accomplish tasks successfully (Demir, Karakaya, & Tosun, 2012). Ghinea, & Chen, . (2006) say that users with different characteristics have different perceptions on the effective use of a multimedia application. The participants of this study were from different nationalities, ages, academic backgrounds, and majors.

Effectiveness scores showed that the mean of the task completion scores was 47.22. These results indicate that, on average, less than half of the tasks were successfully completed. Table 1 shows that only one participant was able to successfully complete more than half of the tasks. None of the participants were able to successfully complete all of the tasks in one session.

Efficiency

Efficiency represents the time it takes for the user to complete or quit the task (Demir, et al., 2017). Shorter times in task completion denote more efficient products, and longer times denote an increase in complexity and difficulty to complete. The efficiency scores revealed that, on average, participants spent the most time on task three. As indicated in Table 1, some of the participants quit and moved to the next task. As shown in Table 2, task completion time on a task varied from 0:38 minutes to 12:10 minutes.

Satisfaction

Although the SUS score itself does not indicate whether users are satisfied with a system, it is an industry-accepted survey and is a widely used tool to measure users’ satisfaction level with a system. The average SUS score of 68 with a standard deviation of 12.5 was accepted as the average score by the most UX researchers because the average score of 68 is in the 50th percentile and represents the score that is higher than 50 percent of all tested systems and applications (Lah & Lewis, 2016). Adopting Sauro and Lewis’ (2016) curved grading scale (Table 3), satisfaction results indicated that the library website's SUS score averaged 46.5. To this end, the curved grading scale of library website indicated that participants’ satisfaction was at an F level, which is the worst level possible.

Demir et al., 2017; Nielsen, 1993; Stone, et al., 2005 claim that users with a lot of experience are more likely to rate the use of the systems as favorable, and users with less or no experience will rate the study as less usable. Nevertheless, the discomfort and negative perceptions of the use of the system were unanimous. 

Interview Results

After doing the tasks, all the participants complained about the way the library website is designed. Some of them complained about the required number of steps to complete a single task. They reported that this kind of difficulty made them feel lost in the system and eventually cause them to quit, as stated in one participant’s feedback below:

Sophia: “I assumed that it is an easy task to find an article by putting the keywords into the search box and hit the search icon. However, this easy task required me multiple steps including (1) go to a search article section, (2) click on the menu (3) look for the particular journal database -which I have no idea on which database would help me to land what I am looking for, and (4) enter the keywords. I recommend designers to create one single search box to run a search across the multiple databases.”  

Interview results showed that all of the participants complained about the complexity of design and lack of guidance for academic search. Moreover, all of the participants mentioned that a single search box would be the best option for them to access desired information, regardless of selecting a database beforehand. It is also obvious that the design issues led to trouble navigating and locating the appropriate link quickly. Particularly, the users who defined themselves as novice users experienced obstacles distinguishing terms such as article database, journals, e-books, reference sources, google scholar, and College’s digital library.

One of the main features that affect efficiency is complexity in design. The time to complete the pre-defined tasks was measured to obtain a better understanding of the efficiency as well as the obstacles that users were faced with. The complexity in design, such as unnecessary back and forth navigation, made the users feel lost in the website and thus delayed their task completion.

Beth: “After completing six tasks…some of the tasks were quite difficult in the sense it took a lot of time to find the particular article or the catalog number which I was looking for. First, I thought that finding a book or a catalog number is a simple task but finding the articles by the authors or by the subjects…some of the functionalities are not there on the website…I am exhausted in locating the desired information…because I have lost and spent a lot of time on the website trying to find and got confused with so many links. I am not a native speaker and many icons look the same to me such as a journal, article database and college's digital library.”

Discussion

Library websites are dynamic source of information that needs to be updated on a continuous basis to improve their interactions with thousands of international and local users, to enhance their efficiency and effectiveness, and also to maintain the contentment of the members of their academic communities. Regarding the accomplishment of the tasks, just one of the participants could do the assigned tasks with minimal difficulties (33.3 %), while another participant was able to finish just one assigned task (16.67%). Three participants had trouble completing tasks two, three, and six. Almost all of the applied instruments showed the complexity of the system as one of the most salient factors in the unsuccessful accomplishment of the assigned tasks.

The literature clearly states that the navigation and information architecture should be designed in a clear, attractive manner that also meets the users’ needs and expectations (Chanling & Hung, 2016; Hashim, et al., 2015; Nielsen, 1993; Peterson, 2007). As suggested by Stone et al. (2005), international standards, guidelines, and style guides including but not limited to web accessibility and universal design principles should be followed to ensure better online service outcomes. 

The lack of appropriate labeling and use of jargon that students may not know was one of the biggest challenges for the participants. Although the targeted library web site evaluated by VandeCreek (2005) suggested the appropriate labeling for a better experience, this was not applied to the design in 2017. 

The interview results also showed that the main purpose of visiting the library website is to run a search to access the desired information. However, the unitive design of the search function of the library website was problematic for most of the participants. Four of the six participants indicated that they assumed they would enter the search time and hit enter to access the full-text documents. Unfortunately, the library website requires them to select a database beforehand where there is no systematic support to represent how to achieve a successful search on the system. The results are also in parallel to literature which addressed the difficulty in searching as one of the arduous challenges for library users (Zhang at al., 2009).

Another salient result was the difficulty in finding events and services offered by the library website. The majority of the participants (five out of six) mentioned that the list of events and services were not listed clearly, and the rogue design was the main reason for the delay in task completion time. Additionally, the participants suggested creating a page for Events or Services for efficiency in browsing the events. 

On the other hand, the results clearly show that the subjective satisfaction level with the library website is at the F level, which is the worst possible result for a website. From these findings, it can be said that the library web services should be redesigned in a more user-centered way to meet the needs and expectations of the students, to increase the satisfaction level, and to provide expected service to the research community at the university.

Limitations

Regardless of the statistical arguments behind the number of the participants, only five users recommended testing a protocol design and predicting design quality; however, recruiting more participants in a study may yield to discover more usabilityproblems such as design issues, and system complexity (Barnum et al. 2003). In some usability research, only three participants would be enough to identify the most problems whereas 100 participants would be too few in some usability tests; therefore, the complexity of the tested system, target audience, and research questions are key to determine the number of participants (Demir, Karakaya, & Tosun, 2012).

This study was completed with only six participants. Although most usability research suggests that there is no certain number of participants to identify usability problems (Alroobaea & Mayhew, 2014), some usability literature argues that five users are enough to find 95% of the major usability problems (Nielsen, 2012; Demir, Karakaya, & Tosun, 2012) and only three participants is enough to find 85% of usability problems in domain specific-to-context inspection studies (Alroobaea & Mayhew, 2014).

Conclusion

Digital libraries are the main assets for the academic community and often require user-friendly interfaces for information searchers. It is therefore critical to create digital library services that enhance the user experience with digital collections. A college’s library services should take into consideration the creation of a user-friendly and easy to use library website, especially for the non-native students.

The results show that the complexity in design increases the time to complete the task, causes users to quit, and eventually decreases the effectiveness rates on successful task completion. Moreover, library users demanded the use of a single search box like a Google search. However, testing library web sites requires users to select a database prior to launching a search. Therefore, the search box design was one of the main reasons for users to fail in order to access desired search results.

There is no doubt the library website is an important asset to the students. However, complexity in design, mislabeling, and jargon may create obstacles, particularly for those who are not native speakers. The language barrier creates deeper hesitation and dissatisfaction in use of library services. The time allotted to the completion of tasks is crucial for the engagement and efficacy of the College’s Library website users. Thus, the library website should offer the users easy and quick answers for their academic search activities.

As a suggestion to the policymakers and practitioners, it is of great importance to take into consideration the possibility of making the initial screen of the college’s library website more user-friendly, useful, and less overwhelming for international students with basic English proficiency. The library web service designers should also consider changing the labeling of icons and links to make them clearer, especially for the non-native users. Prior to redesigning the website, it would be a great investment to conduct research to identify the students’ needs and expectations from the online library services. The new design should also represent the students’ mental model, not the designers’ mental model. Future researchers may conduct research for better navigation templates and the most intuitive search operations of the library web services. The satisfaction evaluating various library web services would also be helpful to determine the design outlines and features of the top rated library websites.