The Impact of Web Accessibility Overlays on the Usability and User Experience for People with Permanent Visual Impairments

1 Introduction

The internet is an essential element of today’s information society, connecting billions of people worldwide. As of January 2024, 5.35 billion individuals are using it, representing 66.2% of the world’s population (Statista 2024). The inability to easily access digital information can result in significant disadvantages (Schmutz, Sonderegger, and Sauer 2016; Adam and Kreps 2006, 1). According to the CRPD (2008), having access to information and communication technologies, including the web, is a basic human right.

In Europe, one out of every four adults, or 87 million people, have some form of impairment (European Council 2022). The majority of those affected are women. According to the WHO (2023c), globally, one in every six individuals¹ experiences a significant impairment.

Impairment is a wide-ranging term that can be broken down in various ways. As per the ICF , mentioned by the WHO (2019) in their report, an impairment is a broad term used to describe a health condition that causes a problem in the function or structure of a person’s body. McLeod (2023) states that impairments can be differentiated into physical impairments, such as sensory, orthopaedic, and other non-cognitive impairments, or mental impairments, like learning impairments, other cognitive impairments, and mental health conditions. Impairments can also be classified as visible or invisible (or non-visible). Additionally, impairments can be differentiated by how obtrusive they are and to what extent they interrupt the flow of social interaction (McLeod 2023). Zahra and Brewer (2017), writing for the WAI, categorise impairments into:

1. Auditory

   These impairments can be mild to uncorrectable hearing loss in one or both ears. Some people use hearing aids, and others can hear sounds but eventually not sufficiently enough to understand speech.

2. Cognitive, Learning and Neurological

   These impairments can be behavioural and mental health disorders, neurodiversity and neurological disorders. These impairments impact how well someone can hear, move, see, speak, and understand information. Examples are attention deficits, dyslexia, autism or learning impairments.

3. Physical/Motor

   These impairments include missing limbs, muscular control or sensation limitations, lack of coordination, pain restricting movement, and joint disorders.

4. Speech

   These impairments can cause difficulty in producing recognisable speech, such as issues with clarity or loudness.

5. Visual

   These impairments can be low vision due to vision loss in one or both eyes, blindness (substantial and uncorrectable vision loss in both eyes), colour blindness or increased sensitivity to bright colours. The primary causes of vision impairment and blindness worldwide are refractive errors² and cataracts³ (WHO 2023b). Globally, an estimated 43.3 million individuals are blind, with over 90% having a preventable or treatable cause and an additional 295 million experiencing moderate to severe visual impairment (Forrest et al. 2023). At least 2.2 billion individuals suffer from near or distance vision impairments (WHO 2023b). Although vision loss can affect people of all ages, the majority of individuals with vision impairment or blindness are aged over 50 years. Projections suggest that by 2050, 61 million individuals will be visually impaired and 474 million will have moderate to severe visual impairment (Forrest et al. 2023).

When discussing impairments, severity and duration must also be considered, as they can have diverse effects on an individual’s daily life. According to the Persona Spectrum defined by Microsoft (2016), each type of impairment can occur as a permanent, temporary or situational one (Figure 1). A person with a permanent visual impairment might be blind, while someone with a temporary impairment could be suffering from a current eye disease. In contrast, a situational impairment may include situations such as being a distracted driver.

Although some sources employ the Medical Model Definition of Disability, which utilises the term disability to depict the aforementioned impairments, this thesis adopts the Social Model Definition of Disability. The Social Model Definition differentiates between the terms impairment and disability (Retief and Letšosa 2018). Disability is understood as a socially constructed disadvantage or restriction of activity caused by a contemporary social organisation that takes no or little account of people with impairments and thus excludes them from participating in divstream social activities (UPIAS 1976). Hence, disability is a social construct that can be changed and eliminated (Retief and Letšosa 2018). The term impairment in this thesis refers to an individual’s physical, sensory or cognitive difference. Therefore, when not referring to the social implications, the thesis uses the person-first expression of people with impairments.

Although a survey conducted by Sharif, McCall, and Bolante (2022) aimed to provide insights into the language preferences of and for people with impairments found that 49% of people preferred identity-first language compared to 33% who preferred person-first language, this paper uses person-first language to emphasise the Social Model Definition of disability. In addition, Sharif, McCall, and Bolante (2022) recommend that authors and researchers respectfully ask individual users for their language preferences, as the results of their survey show that the detected preferences varied across impairment categories. For instance, individuals with mobility impairments, neurological disorders, or chronic illnesses preferred person-first language over identity-first language. Additionally, language preferences varied among gender identities, age groups, and countries.

In this thesis, the identity-first term disabled people, however, is used when writing about individuals with impairments who are disabled by society. Advocates of the Social Model Definition argue that the phrase disabilities is rooted in the perspective of the Medical Model and, therefore,disabled people more accurately conveys the oppression and discrimination faced by those with impairments daily. Therefore, in the context of this thesis, these expressions are used accordingly.

1.1 Accessibility of Digital Services

Accessibility refers to the extent to which products, systems, services, environments, and facilities can be used by individuals with a wide range of user needs, characteristics, and capabilities to achieve identified goals in specific contexts of use, including both direct use and access technology (Technical Committee ISO/TC 159 and Subcommittee SC 4 2019). According to Sauer, Sonderegger, and Schmutz (2020), accessibility is not only concerned with the design of websites and consumer products but also with the design of buildings (e.g. wheelchair access) and transportation. However, accessibility research places a strong focus on the design of websites due to the crucial role of barrier-free access to the web for social inclusion for users with impairments.

The number of individuals living with an impairment is on the rise, primarily due to a rise in chronic diseases resulting from non-communicable diseases⁴ and an ageing population (WHO 2023c; European Council 2022). Impairments are, therefore, part of being human and accessibility of digital services is vital to the social integration of people with impairments (Alonso et al. 2010; WHO 2023c, 1).

There are about 1.11 billion websites on the internet, with approximately 18%⁵ of them presently active and used in some form. Approximately 252,000 new websites are created every day (Huss 2023). An accessibility evaluation of the top 1 million websites’ home pages was conducted by WebAim (2023), revealing that 96.3% contained accessibility errors⁶. The total number of distinct accessibility errors detected on the one million home pages was 49,991,225, resulting in an average of 50 errors per page. Users with impairments should assume that they will encounter errors on one in every 21 elements on a homepage with which they interact. Digital services should be designed and technically implemented to facilitate inclusion and eliminate barriers.

"The power of the Web is in its universality. Access by everyone regardless of disability⁷ is an essential aspect."

WebAim (2023) reports that over the last five years, 96.1% of all accessibility errors identified in their evaluations can be attributed to six different areas: (1) low contrast text, (2) missing alternative text for images, (3) empty links, (4) missing form input labels, (5) empty buttons and (6) missing document language. By addressing these issues, accessibility across the web can be significantly improved. Accessible digital services ensure equal opportunities and access for all, regardless of people’s abilities or environmental constraints (S. L. Henry and McGee 2018). Building accessibility into the design and development of digital services is essential. As S. L. Henry (2022a) outlined on the WAI website, people with impairments must be able to perceive, understand, navigate, interact with and contribute to the web.

1.2 Accessibility Laws

Digital services are subject to various legal regulations worldwide, which may differ from country to country. In Sweden, the Discrimination Act is the applicable legislation (Swedish Government 2008), while theE-Government Act (E-GovG)applies in Austria (National Council Austria 2004). In the USA, the ADA of 1990, as amended by the ADA Amendments Act in 2008, is applicable (U.S. Congress 2008).

In 2019, the Accessibility Act (Directive 2019/882) was established by the EU as the legislation mandating accessibility for selected everyday products and services for individuals with impairments (European Union 2019). It covers the products and services that the EU has identified as the most important for individuals with disabilities and which are likely to have differing accessibility requirements across EU countries (European Commission n.d.). EU Member States must carry out regular compliance checks, review and follow up on complaints, whereupon companies take necessary corrective actions. Individuals must know the appropriate authority to approach with their complaint (European Commission n.d.). All EU member states are obliged to integrate the European Accessibility Act into national law by June 2025. Effective from June 28, 2025, customers will have the right to file complaints with national courts or authorities if products or services do not adhere to the new regulations.

In May 2023, the Federal and National Council of Austria unanimously approved the new EU regulations (National Council Austria 2023). The Federal Act enacted by the Council outlines the accessibility requirements for the products and services covered by the Accessibility Act. It also mandates that companies only introduce products and services compliant with the Accessibility Act while enforcing market surveillance (National Council Austria 2023).

Global legal regulations and the previously noted statistics highlighting the number of people with impairments emphasise the need for websites to be accessible. In general, the number of digital accessibility lawsuits continues to rise. A report by UsableNet (2023) reveals that as of the end of 2022, 4035 cases have been filed in the USA. These include cases filed in federal court and those filed in state court in California and New York. 95% of these cases allege accessibility issues with desktop websites. eCommerce websites are the most common targets, accounting for 84% of the lawsuits examined, with food service coming in second at 7%. Education is now the third most impacted sector in 2023.

The study of Loiacono and Djamasbi (2013) demonstrates that legislation affects the accessibility of websites in two ways: directly, by mandating adherence to accessibility standards, and indirectly, by influencing the number of sites that undergo accessibility testing. These findings indicate that when a company is legally required to comply with website accessibility standards, it is more likely that the website will be accessible.

Although regulations are in place and intend to assist those with impairments, they pose new business challenges. As per Egger, Zimmermann, and Strobbe (2022), several potential justifications exist for businesses not making websites accessible, although they may be required to. A shortage of personnel with the required skills, the cost factor and a lack of time for development can be named as possible examples. According to Huq et al. (2023), previous research has identified that time, finances, and resource costs hinder the prioritisation of web accessibility. These factors are the most significant barriers to implementing web accessibility (Huq et al. 2023). According to Sauer, Sonderegger, and Schmutz (2020), practitioners often argue that implementing accessibility requires significant effort and incurs high costs. However, they believe that considering accessibility early in the development process would be an excellent investment rather than only towards the end. To serve these challenges for businesses, providers of what are known as Accessibility Overlays have emerged.

1.3 Accessibility Overlays

In recent years, the market for accessibility overlays - also known as "accessibility widgets" or "accessibility tools" - has experienced rapid growth. Accessibility overlays refer to a range of technologies aimed at enhancing the accessibility of a website (Groves 2021).

An accessibility overlay generally appears on a website as a toolbar, plugin, app or widget (Figure 2). They are usually activated by clicking on a small circular button that appears on the edge of a website, floating above the content. After clicking this floating action button, the accessibility overlay opens. Consequently, the users can utilise the accessibility overlay to customise the website to their needs, such as altering the font size, adjusting the colour contrast, or deploying text-to-speech. Users can activate specific features with a single button click or select an "accessibility profile" to implement multiple adaptations simultaneously. Examples include profiles for colour blindness, dyslexia, or epilepsy.

Some variations of accessibility overlays aim to fix any errors in website code that prevent the easy use of access technology, also called assistive technology, by scanning the code and automatically attempting to repair identified errors (Groves 2021). A blog post by B. Henry (2020), writing for The Paciello Group Interactive (TPGi), indicates that these accessibility overlays incorporate AI automation and machine learning to identify and resolve accessibility issues on websites. These fixes can include predicting high-level tags and attributes and generating alternative text for images. When referring to an accessibility overlay in this thesis, the term encompasses the widget, toolbar, or plugin, including variants with and without automated remediation.

To add an accessibility overlay to a website, a website owner can insert a short snippet of JS into the page’s source code. According to a marketing claim by AudioEye (2022b, 14), an accessibility overlay company, the inserted JS code snippet improves website functionality. In addition, many companies providing accessibility overlays claim to comply with legal regulations, which further appeals to many businesses, as stated by Egger, Zimmermann, and Strobbe (2022).

Within the accessibility community, however, accessibility overlays are highly criticised, according to Egger, Zimmermann, and Strobbe (2022), arguing that over 600 accessibility experts, lawyers, and contributors to accessible web content guidelines signed a statement that accessibility overlays should be removed and that other research already demonstrated the limitations of these technologies. As of March 11, 2024, 856 individuals have signed the mentioned statement (Groves 2021). The extent to which individuals with impairments, for whom these accessibility overlays are designed, also appreciate them is an ongoing topic of discussion.

To date, little scientific research has investigated accessibility overlays. Although there is extensive research on (web) accessibility (see e.g. Cane (2023), Thoo et al. (2023), Huq et al. (2023), Campoverde-Molina, Luján-Mora, and Valverde (2021), Potluri et al. (2021), Mack et al. (2021), Min Htike et al. (2021), Ming et al. (2021), Brulé et al. (2020), Nuñez, Moquillaza, and Paz (2019), Piccinno et al. (2019), Vollenwyder et al. (2018), Schmutz, Sonderegger, and Sauer (2016), Trewin, Marques, and Guerreiro (2015), Loiacono and Djamasbi (2013), Power et al. (2012), Kane et al. (2011), Murphy et al. (2008), Lazar, Feng, and Allen (2006), Adam and Kreps (2006), Theofanos and Redish (2003)), little attention has been paid to accessibility overlays. Research by Egger, Zimmermann, and Strobbe (2022) suggests that accessibility overlays only provide limited support for the accessibility of websites. In addition, the accessibility overlays they examined in their 2020 study showed too many limitations to achieve compliance with legal regulations. The study of Kohler (2023) on mobile users with visual impairments concluded that accessibility overlays did not significantly improve accessibility for users navigating with a screen reader access technology and were largely ignored by study participants. Besides the two just-named studies, the academic literature on the topic of accessibility overlays redivs scarce. Although they have existed for many years, discussions and lawsuits about them have only begun in recent years. This may account for the insufficient exploration in that particular dodiv. Thus, this thesis will be an essential contribution to the knowledge base on accessibility overlays, making it a fundamental piece of research.

1.4 Research Question

The research of this thesis aims to contribute empirical knowledge⁸ to the ongoing debate surrounding the potential benefits and challenges of accessibility overlays by answering the following div research question: What impact do accessibility overlays have on the usability and user experience of individuals with permanent visual impairments browsing a desktop website?

Vision is the most dominant of our senses and plays a critical role in every facet and stage of our lives (WHO 2023b). Moreover, the probability of vision impairment affecting more individuals is set to rise with the increasing population growth and ageing (WHO 2023b; Forrest et al. 2023). As a result, it makes sense to choose individuals with permanent visual impairments as the focus group for this thesis.

Usability is a clearly defined concept (Aizpurua, Harper, and Vigo 2016). According to the ISO 9241-11:2018 standard, usability refers to the extent to which specified users can use a system, product, or service to effectively, efficiently, and satisfactorily achieve their goals in a particular context of use (Technical Committee ISO/TC 159 and Subcommittee SC 4 2018). However, websites serve a greater purpose than simply fulfilling specific tasks. According to Aizpurua, Harper, and Vigo (2016), people view the internet as a means to achieve informational goals and use it for communicating, leisure and social activities. Therefore, the success of a website may depend not only on its actual features but also on how users experience them.

UX refers to the perception and responses of individuals resulting from the use or anticipated use of a product, system or service, as per ISO 9241-210:2010 (Technical Committee ISO/TC 159 and Subcommittee SC 4 2010). Whilst UX aims to enhance the general experience of individuals, usability focuses on the functional aspects of technical systems (Richter and Flückiger 2013). Usability and UX are crucial to successfully adopting any information system (Nikou and Economides 2019), making them a strong focus of this research.

This thesis assesses accessibility in two ways: (1) through the perceived accessibility reported by participants and (2) accessibility regarding compliance with guidelines such as the WCAG standard. This results in the following two sub-questions for the research:

1. How do individuals with permanent visual impairments perceive accessibility overlays?

2. How effectively does incorporating an accessibility overlay on a website ensure compliance with the success criteria of the WCAG 2.1 standard?

Evaluating accessibility concerns about accessibility overlays directly with the people affected is essential. However, the topic has not been sufficiently explored in a scientific setting (Egger, Zimmermann, and Strobbe 2022; Kohler 2023). Section 2 goes into further detail on related work in this field of research. A study by Aizpurua, Harper, and Vigo (2016) found that perceived web accessibility significantly correlates with UX. At the same time, the relationship between UX and conformance to accessibility guidelines is more elusive, suggesting that it is more correlated with usability (Aizpurua, Harper, and Vigo 2016). These findings illuminate the interrelationship between the sub-questions and the div research question.

The author answers the research questions of this master’s thesis by performing an evaluation study. The study consists of two parts. First, a technical evaluation of how three selected accessibility overlays perform on a website, based on the success criteria of WCAG 2.1 standard, is executed. Then, a user study is conducted to assess the usability and user experience of 21 individuals with visual impairments through interviews, observations and questionnaires. The research results of this thesis aim to contribute empirical knowledge to the field of research on accessibility overlays.

2 Related Work

The results of a literature review performed by the author are displayed in the following sections. The review explores studies on accessibility overlays, analysing their technical capabilities and their effect on the usability and experiences of individuals with impairments. Additionally, an analysis of research regarding web accessibility and how people interact with digital services is conducted to gain a deeper comprehension of the subject matter. Furthermore, the literature review concentrates specifically on individuals with visual impairments as they are the focus of the study. In addition to these topics, this section includes information on accessibility standards, the process of testing the accessibility of a website and the marketed features of accessibility overlay companies.

The literature review consists of previous research and media publications, including articles from newspapers and blogs. The issue of accessibility overlays is heavily debated and often discussed in blog posts, so it is reasonable to look at these sources. Gregor and Hevner (2013, 349) state that it is essential to carry out the literature review carefully and include prior knowledge related to the addressed problem, as well as work that has been published under a different label but with similar aims. Prior work can be empirical research studies, theories, reports from practice and developed artefacts that solve a similar problem (Gregor and Hevner 2013). Nevertheless, caution is applied when citing unscientific sources.

The literature study was conducted using the following keywords: Web Accessibility, A11y⁹, Accessibility Overlays, Visual Impairments, Visual Disabilities, Web Accessibility and Low-Vision, Web Accessibility and Blind, Website and Disabilities, Digital Services and Accessibility, Accessibility Overlay and Experience, Accessibility and Usability, Accessibility and User Experience.

Thoo et al. (2023) argue that many scientific papers lack disclosed details on the visual impairments of study participants. Some papers target both blind and low-vision individuals and use non-specific terms such as "visually impaired users" and "blind and visually impaired people". Thus, the keywords blind and low-vision were added to the search to cover such cases.

2.1 Usability and UX

Sauer, Sonderegger, and Schmutz (2020) argue that there is a significant overlap between the concepts of Usability and UX. This overlap has sparked a lively debate in the scientific community regarding the meaning and utility of these two concepts.

2.1.1 Defining Usability

According to Sauer, Sonderegger, and Schmutz (2020), Sarodnick and Brau (2006) stated that the concept of Usability was first introduced in 1981 by Shackel (1981) as a replacement for terms like "user-friendliness" and "ease of use". Since then, it has been defined through various perspectives, emphasising the ease and effectiveness of use by humans. Subsequent definitions have broadened to include perspectives on usability that are product-oriented, user-oriented, performance-oriented, and context-oriented (Sauer, Sonderegger, and Schmutz 2020; Bevan, Kirakowski, and Maissel 1991). Each perspective highlights different aspects of usability, ranging from inherent product characteristics to the subjective experience of users.

The ISO 9241-11:2018 standard defines usability as the degree to which specified users can effectively, efficiently, and satisfactorily achieve their goals using a system, product, or service in a particular context of use (Technical Committee ISO/TC 159 and Subcommittee SC 4 2018). However, according to Sauer, Sonderegger, and Schmutz (2020), the researchers Tractinsky (2017; Hirsch and Levin 1999) have criticised the concept of usability as being too broad and covering diverse phenomena. Furthermore, the differentiation between formative and summative testing introduced complexity to the concept. Formative testing aims to improve user-product interaction iteratively, while summative testing evaluates system quality against standards or other products. This differentiation suggests two distinct conceptions of usability that challenge integration into a single definition (Gediga, Hamborg, and Duntsch 1999; Lewis 2014). Aizpurua, Harper, and Vigo (2016), however, argue that usability is a clearly defined concept.

2.1.2 Defining UX

The concept of UX is often considered complex and not well-defined, encountering similar challenges to usability. According to Sauer, Sonderegger, and Schmutz (2020), researchers have proposed various definitions that fall into three main categories.

1. Holistic View

The holistic view encompasses a broad range of outcomes from user-device interaction, including cognitive and emotional responses (Sauer, Sonderegger, and Schmutz 2020; Blythe et al. 2003). It refers to UX as all a person's actions, sensations, considerations, feelings, and sense-making when interacting with a technical device or service.

2. Extension of Usability

UX is described as an extension of usability. The scope is expanded to include affect and emotions, in addition to efficiency, effectiveness, and satisfaction (Sauer, Sonderegger, and Schmutz 2020).

3. Primary Focus on Emotion

This perspective focuses on the emotional outcomes of interactions. Sauer, Sonderegger, and Schmutz (2020) state that Desmet and Hekkert (2007) consider UX a set of affective outcomes influenced by aesthetics, emotion, and meaning and that Vermeeren et al. (2008) define UX as specific emotions, such as anger, joy, excitement, unease, and satisfaction, which are influenced by factors such as aesthetics, usability, and the personal significance of the product to the user.

Like usability, the concept of UX is an ISO norm (ISO 9241-210:2010). It refers to the perception and responses of individuals resulting from the use or anticipated use of a product, system or service (Technical Committee ISO/TC 159 and Subcommittee SC 4 2010).

Some authors emphasise the positive impact of user experience on usability (Lewis 2014). However, there is less agreement among scientists regarding the broader and more diverse understanding of user experience compared to the concept of usability.

Sauer, Sonderegger, and Schmutz (2020) presented the concept of IX as a higher-level term, including UX and usability. The new term IX is proposed also to include important ideas related to accessibility. According to ISO standard 9241-210:2019, accessibility refers to the extent to which products, systems, services, environments, and facilities can be used by individuals with a wide range of user needs, characteristics, and capabilities to achieve identified goals in specific contexts of use, including both direct use and access technology (Technical Committee ISO/TC 159 and Subcommittee SC 4 2019).

The researchers assert that a new concept is necessary to reconcile opposing views on the definitions of usability and UX. Specifically, the debate between "Usability is a part of UX!" versus "UX is a modest extension of usability!". However, IX is considered an umbrella term and cannot be measured. Nevertheless, the elements constituting it, such as accessibility, user satisfaction, emotion, and efficiency, can be measured (Sauer, Sonderegger, and Schmutz 2020).

As this concept is new, this thesis examines usability, user experience, and accessibility separately using the methods already established in these research fields.

2.2 The Complexity of Web Accessibility

Adam and Kreps (2006) argue that the complexities of web accessibility are best understood and analysed against relevant discourses, such as the legal discourse, the web accessibility discourse, the digital divide discourse and the social construction of disability discourse.

Legal discourse refers to the legal regulations established worldwide to prevent discrimination against individuals with impairments. The web accessibility discourse is centred around movements such as the W3C, which establish accessibility standards. The digital divide concept refers to the division between those who have access to information and communication technologies and those who do not. This term was first coined in 1995 and has since become a central concern in policy discussions surrounding social inclusion and exclusion. The discourse on the social construction of disability is based on the Social Model Definition of Disability. While Adam and Kreps (2006) explain that other models to describe impairments, such as the Medical¹⁰ and Charity¹¹ Model Definitions of Disability exist, they highlight the Social Model. The Social Model views disability as a socially constructed disadvantage or restriction of activity caused by a contemporary social organisation that does not take people with impairments into account, thus excluding them from participating in mainstream social activities (UPIAS 1976). According to Adam and Kreps (2006), the Social Model is a more radical approach which emphasises that locating disability in the individual as opposed to society is a political decision. Additionally, they state that other researchers argue that research has primarily focused on analysing specific impairments and developing technical solutions to address them, reflecting the dominant medical paradigm of disability. Technology can create disability by being designed in a way that excludes specific individuals from using it (Adam and Kreps 2006). In essence, it is the responsibility of society as a whole to ensure that the internet and web are accessible, rather than placing the burden on individuals with disabilities to obtain specialised technical aids or to navigate through poorly designed and frequently inaccessible websites. Therefore, according to Adam and Kreps (2006), it could be argued that a poorly designed website may be considered as creating a disability.

Adam and Kreps (2006) suggest that part of the reason for the persistence of inaccessible websites lies in crucial gaps in the engagement between these discourses. They further state that the discourse on web accessibility has had minimal overlap with the discourses on digital divide, social construction, and legislation. This means it has not been in the best position to take advantage of the more radical arguments of the social construction and legal discourses on barriers and rights, using their arguments positively to fuel its campaign. Without a united voice from the complete spectrum of relevant discourses pressing for the active rights of people with impairments against constructed barriers to web accessibility, a relatively passive liberal approach toward disabled access may, according to Adam and Kreps (2006), prevail. The issue with liberal discourse is that while there may be a general desire for a more equal and fair society, the deeper social and cultural structures that contribute to inequality are often overlooked.

2.3 Accessibility Standards

There are many publicly available web accessibility guidelines to help build a shared understanding of web accessibility, as well as to guide design choices and provide a benchmark for evaluation (Vollenwyder et al. 2018, 5). The W3C, for instance, strives to develop open standards that ensure the long-term growth of the web. In particular, the W3C Web Accessibility Initiative (WAI), founded in 1996, produces standards and support materials specifically for accessibility. These standards can assist multiple stakeholders, including developers and designers, in comprehending and implementing accessibility to make their websites and applications more approachable for all users (Wille, Dumke, and Wille 2016; Cao and Loiacono 2019). Cao and Loiacono (2019) refer to earlier studies demonstrating the importance of web accessibility guidelines as a critical resource for incorporating accessibility into websites and apps.

2.3.1 WCAG

One of these standards refers to the WCAG. The WCAG documents explain how to make web content more accessible to people with impairments and are an international standard. Web content refers to the information found on websites, including text, images, sounds, and the code or markup that defines how a page is structured and presented (S. L. Henry 2022b). According to Vollenwyder et al. (2018, 5) and Schmutz, Sonderegger, and Sauer (2016, 2), the WCAG are the most widely known and commonly used set of recommendations. WCAG 1.0 was initially published in 1999 and has since been updated and replaced by subsequent versions. WCAG 2.0 was released in December 2008, followed by minor updates as WCAG 2.1 in June 2018, and the latest update, WCAG 2.2, was recently released in November 2023. While new updates to the current standard are still actively being developed, work on WCAG 3.0 is also underway. WCAG 3.0 is set to become a W3C Standard in the coming years. It is expected to maintain the fundamental and specific accessibility requirements outlined in WCAG 2.2 but will introduce a new structure, conformance model, and broader scope (S. L. Henry 2023).

The WCAG comprise four principles: Perceivable, Operable, Understandable and Robust. These principles are further broken down into guidelines. WCAG 2.1 offers 13 guidelines with clear objectives for achieving web accessibility. Each guideline contains testable success criteria and is associated with one of three levels: A (low accessibility), AA (high accessibility), and AAA (highest accessibility). Level A is the minimum level. Level AA includes all Level A and AA requirements. It is common for organisations to aim for Level AA compliance (WAI 2020). Level AAA compliance is achieved when all Level A, AA, and AAA requirements are met. In total, WCAG 2.1 has 78 success criteria distributed across these three levels.

Alonso et al. (2010) assert that one of the primary objectives of the WCAG is testability. Although the WCAG are widely accepted, Vollenwyder et al. (2018) criticise them for not considering cognitive impairments sufficiently and for formulating guidelines that are too difficult to evaluate. According to a study by Alonso et al. (2010), beginners often have difficulty interpreting the success criteria.

2.3.2 ARIA

ARIA, also known as WAI-ARIA, is a specification developed by the W3C aimed at enhancing the accessibility of applications by supplying additional information to access technologies. It is useful for dynamic content and advanced user interface controls developed using HTML, JS and related technologies. According to Nurthen, Cooper, and Henry (2006), ARIA is essential for ensuring that users with impairments have access to all website functionalities, especially for people who rely on screen readers or cannot use a mouse.

ARIA presents a framework for adding attributes to recognise features for user interaction, their interrelations, and their current state. WAI-ARIA 1.2 is the most recent version released as a finalised W3C Recommendation on June 6, 2023. However, the first rule of ARIA use is to employ a native HTML element or attribute possessing the required semantics and behaviour built-in (Faulkner and MacDonald 2018). This method is more effective than re-purposing an element and adding an ARIA role, state, or property to make it accessible, according to the standard.

2.3.3 BIK BITV-Test

The BIK BITV-Test, according to DIAS (2018), is the established test procedure for evaluating web accessibility in Germany, complying with the BITV law enacted by the Federal Ministry of Justice Germany (2011). BIK stands for "Barrierefrei Informieren und Kommunizieren" (accessible information and communication), whilst BITV stands for "Barrierefreie Informationstechnik Verordnung" (accessible information technology ordinance). The BITV-Test comprises 92 comprehensive test steps for evaluating whether information-oriented websites are accessible to users with impairments (DIAS 2018). The test steps include all success criteria outlined in WCAG 2.1 level A and AA, along with 38 additional requirements.

2.4 Testing the Accessibility of a Website

Accessibility testing aims to assess whether a given solution is accessible to a range of people, including people with different impairments (Piccinno et al. 2019). In a systematic literature review by Nuñez, Moquillaza, and Paz (2019) on web accessibility evaluation methods, it was determined that (1) automatic tools, (2) evaluation by experts (manual testing), and (3) user tests are the most widely used techniques.

Assessing the accessibility of a website typically consists of analysing a website’s conformance with the requirements of official guidelines, such as the WCAG. This analysis can be undertaken with automated and manual accessibility tests. During an analysis, a website is checked against the WCAG success criteria. Depending on the use case, different criteria apply. Usually, a website is checked against Level A and AA of the WCAG. If needed, some criteria of Level AAA can be evaluated as well.

Additionally, user tests with individuals with impairments can be conducted to provide deeper insight and discover accessibility errors that might be overlooked. Sauer, Sonderegger, and Schmutz (2020) note that user testing is not commonly conducted in accessibility evaluations for specific reasons. Usability engineers argue that testing with people with impairments requires too much time and financial resources. Additionally, gaining access to users with functional limitations may be difficult. Accessibility practitioners may also be hesitant to conduct user testing due to uncertainty about accommodating the needs of users with impairments during the testing procedure. However, Sauer, Sonderegger, and Schmutz (2020) do not find these arguments convincing. They believe that the field of accessibility research would benefit from user testing. Understanding the users and how they work with their tools is essential, as accessibility guidelines based solely on technology are insufficient to meet the needs of all individuals (Theofanos and Redish 2003).

2.4.1 Automated Accessibility Testing

There are many automated accessibility testing tools on the market. When assessing accessibility, the selection of techniques and tools depends on many factors, including preference, stage of development, role and context (Piccinno et al. 2019). A survey conducted by WebAim (2021b) found that the majority of accessibility practitioners use a variety of tools to assess web accessibility. The top five most frequently used tools include Axe¹² (64%), followed by a browser's DevTools/Inspector (63.9%), WAVE¹³ (53.0%), accessibility browser bookmarklets or user scripts (40.1%) and Lighthouse¹⁴ (34.6%). Other examples are the W3C Markup Validation Service¹⁵, Siteimprove¹⁶ and ANDI¹⁷.

Some testing tools provide solely automated results, while others also assist in the manual testing process. One example is ANDI. ANDI stands for Accessible Name and Description Inspector and is a free accessibility testing tool. It automatically detects accessibility issues. In addition, it displays the accessible names of interactive elements, revealing what a screen reader would say to the end user.

2.4.2 Manual Accessibility Testing

Automated tools offer a first impression of the accessibility of a website but the results have to be analysed manually to identify false negatives and put the findings into context. Although automated tools are widely used in research, they cannot always identify all web accessibility issues (Nuñez, Moquillaza, and Paz 2019). Adam and Kreps (2006) state that the WCAG guidelines, for example, cannot be adequately tested through automation and require human verification due to their complexity. Additionally, they argue that many organisations, including those focused on disability, mistakenly believe that merely passing such automated tests fulfils their obligations to ensure web accessibility. According to Nuñez, Moquillaza, and Paz (2019), an evaluation by an accessibility expert is often the most efficient way to ensure compliance with specific standards.

Besides reviewing automated accessibility testing results, manual tests include reviewing the HTML document structure, reviewing the website on smaller devices and with different color preferences, navigating with a keyboard only and using a screen reader for navigation.

A variety of tools, browser extensions and browser bookmarklets can be used to assist manual accessibility testing. Browser bookmarklets work by injecting JS into the website to highlight specific information about the website like alternative texts for images, headings or landmarks. They can also be used to change font sizes and spacing and view the website in different colors.

More extensive tools are the Web Developer¹⁸ and Axe browser plugins. The browser plugin Web Developer adds various tools to a browser, assisting with manual accessibility testing. Some of the added tools are turning off JS, CSS and cookies, visualising HTML form details, outlining (background) images, headings and external links or displaying hidden elements. The Axe browser plugin offers functionality for manual accessibility testing in addition to its automated accessibility analysis. It can visualise landmarks, headings and accessible names of elements.

2.4.2.1 Screen Reader Navigation

Navigating websites using access technology, also called assistive technology, is an essential component of accessibility testing. Many individuals with impairments require access technology to access websites. Access technology empowers individuals to maintain a healthy, productive, autonomous, and dignified lifestyle (WHO 2023a). It is estimated that over 2.5 billion people worldwide require one or more access products to assist them (WHO 2023a). With an ageing global population and an increase in non-communicable diseases, it is estimated that over 3.5 billion individuals will require at least one access product by 2050, with a significant percentage of older individuals requiring two or more. Access technology includes hearing aids, wheelchairs, communication aids, spectacles, prostheses, pill organisers, and memory aids. In the digital sphere, access technology additionally incorporates tools like refreshable braille displays¹⁹, browser and system settings, or screen readers.

Especially individuals with visual impairments can benefit from various access technologies. 68% of users with visual impairments utilise two or more access technologies, whilst 23% rely on four or more distinct types (WebAim 2018). They can use screen magnification software or adjust system settings to change the display size. Browser zoom controls and text sizing settings can also assist in making page content easier to see. High contrast mode or settings and browser settings to alter colours can also help. Additionally, custom styles, such as those achievable through user style sheets, can enhance readability. Highlighting tools can further assist with reading. It is critical to ensure that websites are accessible to all users via such access technologies.

The most frequently used access technology is a screen magnification (48.4%), closely followed by a screen reader (45.2%) (WebAim 2018). The most prevalent group to use a screen reader are those with blindness, followed by individuals with low vision (WebAim 2021a). However, other impairments such as motor, deafness, and cognitive impairments can also result in individuals using a screen reader. Screen readers convert visual semantics, content and hierarchical information into a linear output stream, usually in the form of audio or braille (Potluri et al. 2021). When searching for information on a lengthy website, most screen reader users (67.7%) navigate through the headings on the page (WebAim 2021a). Therefore, it is vital to ensure that a website has an adequate number of headings during the manual stage of an accessibility analysis. A smaller percentage of screen reader users (13.9%) use the "Find" feature, while 8.1% read through the page and 7.1% navigate through the links. Only 3.2% navigate through the landmarks/regions of the page. There are very different ways one can navigate a website. Ideally, each of these approaches should be easily accessible, as screen readers offer various mechanisms for users to locate the element they seek (Cane 2023).

WebAim (2021a) performed a survey, analysing screen reader preferences with approximately 1500 respondents. Most respondents were between 21 and 40 years old (38.3%), with the second-largest age group being people aged between 41 and 60 (31.4%). 79.5% of the participants reported they were blind, 21.9% had reduced vision, and 6.4% identified as deaf-blind. The subsequent paragraphs offer further insight into the utilisation of screen readers by individuals with impairments, as discovered by WebAim (2021a).

JAWS²⁰ was the primary screen reader for 53.7% of users in 2021, while NVDA²¹ was used by 30.7%. In contrast, within Europe, NVDA had higher usage than JAWS (41.6% vs. 40.2%). The third most utilised screen reader is VoiceOver, accounting for 6.5%. Primary usage of both NVDA and VoiceOver has decreased significantly in the past two years (Figure 3). The most frequent reason for selecting a primary screen reader is pre-existing familiarity or expertise, as reported by 46.3% of respondents. The second and third most cited reasons are features and availability. 71.3% of participants reported using multiple screen readers. Nevertheless, the most commonly used screen readers are JAWS, NVDA and VoiceOver (Figure 4).

The majority of screen reader users express high levels of satisfaction with their primary screen reader, with 97.3% of NVDA users, 95.3% of JAWS users, 93.7% of VoiceOver users, 91.5% of ZoomText/Fusion users, and 87.5% of Narrator users indicating that they are either "very" or "somewhat satisfied". More than 50% of the respondents reported being advanced screen reader users, while 67.5% regarded themselves as advanced internet users.

Among users of screen readers, Google²² is the most frequently employed browser, accounting for 53.6% of usage. Microsoft Edge²³ was second at 18.4%, followed by Firefox²⁴ at 16.5% and Safari²⁵ at 5.1%. 91.1% of screen reader users operate on Windows, compared to 6.5% on Mac. 90% of respondents also use screen readers on mobile devices. Within the mobile screen reader market, iOS devices remain dominant, with 71.9% of respondents adopting their use. Consequently, VoiceOver²⁶ ranks as the most popular screen reader, with Safari emerging as the leading browser employed on mobile devices.

The survey conducted by WebAim (2021a) found that 42.3% of screen reader users perceive the accessibility of web content to have remained the same over the past year, while a similar number (39.3%) believe that it has improved. These figures suggest that there is still scope for enhancement.

2.4.2.2 Keyboard Navigation

Another part of manual accessibility testing is navigating the website with a keyboard. This test determines if each element can receive focus, if the focus remains visible during navigation, if interactive elements behave as intended, and if the user accidentally gets trapped somewhere on the website. Approximately 60.4% of individuals with visual impairments always or often rely on a keyboard for browsing websites (WebAim 2018, sect. "Keyboard Usage"). Furthermore, those with more severe vision loss exhibit an even greater tendency to use the keyboard to navigate.

2.4.3 User Testing

User testing can provide valuable insights into how people with impairments perform specific tasks. According to Campoverde-Molina, Luján-Mora, and Valverde (2021), one can assess a user’s ability to navigate a website with ease and observe their consequent actions. Nuñez, Moquillaza, and Paz (2019) state that the key benefit of user testing lies in directly identifying specific accessibility issues and violations, as the test can be designed to suit the purpose. One may adopt international standards or establish their own set when identifying criteria. In general, accessibility issues can be identified through various methods such as observations, interviews, questionnaires, and user comments (Campoverde-Molina, Luján-Mora, and Valverde 2021). Sauer, Sonderegger, and Schmutz (2020) argue that accessibility should also encompass designing for positive experiences and emotions beyond functional design, focusing solely on efficiency and effectiveness. This means that accessible design should offer users fun, enjoyment and positive emotions when interacting with the device. However, according to the researchers, it is essential to note that measuring such affective experiences cannot be achieved through methods not involving a user. Therefore, objective user-based evaluation methods should become prominent in accessibility evaluation procedures.

2.5 Visual Impairments in Accessibility Research

Mack et al. (2021) discovered that research on users with visual impairments had been the primary focus of accessibility publications over the past three decades (1994-2019). However, the terminology used to describe visual impairments varies across regions, cultures, and research disciplines. For instance, Thoo et al. (2023) found that the World Blind Union’s list of national blind and low-vision federations uses the terms blind or visually impaired to refer to different levels of vision loss, while in other cases, they are mentioned separately. In the research field of HCI, papers focusing on blindness and low vision often use WHO statistics to support their work. According to the WHO (2019), a "vision impairment occurs when an eye condition affects the visual system and one or more of its vision functions", as grounded in the ICF. Thoo et al. (2023) state that researchers in eye health and vision loss provide a more detailed classification of visual impairments by referring to moderate and severe visual impairment categories as low-vision. In the computer science community, the terminology varies. For example, Thoo et al. (2023) note that review papers often distinguish between blind and low-vision individuals, indicating that the term visually impaired does not include those who are blind. Brulé et al. (2020) used the term people with visual impairments to refer to various subsets of visual impairments, with blindness considered a smaller subset. In contrast, Mack et al. (2021) used the term blind and low-vision to encompass the two major categories and other commonly used terms such as visually impaired, differently sighted, or having vision loss. However, it is important to note that the understanding of vision may be subject to change due to recent advances in vision-related disciplines. This thesis employs the term people with visual impairments to encompass all forms of visual impairment, including low vision and blindness.

2.5.1 Experiences of Users with Visual Impairments Navigating the Web

A number of researchers have studied the experience of users with visual impairments on the web. Theofanos and Redish (2003), Murphy et al. (2008) and Power et al. (2012) specifically aimed at identifying issues experienced by that user group.

The study of Theofanos and Redish (2003) aimed to understand how people with visual impairments interact with websites and what implications this has for designers and developers. The researchers conducted an observational study with sixteen users with visual impairments. They found that screen reader users need to know the browser, the screen reader, and the website itself, resulting in a significant cognitive load. It was also observed that several users did not utilise or have knowledge of various software features. Given the cognitive demands of web browsing and the use of access technology, it is not unexpected that several of the study’s participants were not familiar with the full array of functions provided by the screen reader software they employed (Theofanos and Redish 2003). Moreover, the participants accessed websites that offered a "text version" or a "screen reader version". However, only a small proportion (two of the 16 participants) preferred using a text version. The others argued that having two versions is unnecessary and that one accessible website version is better. Theofanos and Redish (2003) conclude that it is not enough to have guidelines based on technology to meet all users’ needs truly. Understanding the users and how they work with their tools is also necessary.

The investigation of Murphy et al. (2008) engaged 30 computer users who were blind or partially sighted. It aimed to comprehend the difficulties experienced by the community of individuals with visual impairments better when accessing the internet. According to Murphy et al. (2008), despite being familiar with screen readers, participants showed a strong desire to explore new web browsing solutions. A qualitative research analysis led Murphy et al. (2008) to create a multimodal web browser in the form of a browser plugin with audio and haptic feedback. This prototype system aimed to provide spatial awareness to users with visual impairments through non-visual channels using haptic feedback and non-speech sounds. The researchers hoped their system, which was in the prototyping stage at the time of their research, would provide an alternative to screen reading technologies and offer a new interaction experience for internet users with visual impairments.

Power et al. (2012) conducted an empirical study analysing the issues that 32 blind users encountered whilst browsing the web. Task-based user evaluations were administered on 16 websites. The findings showed that only 50.4% of the encountered problems were addressed by the success criteria outlined in the WCAG 2.0. Furthermore, the study revealed that a few developers are implementing the current version of WCAG, and even when the guidelines are put into practice on websites, it is unclear whether people with impairments will experience fewer difficulties.

However, there is a lack of studies investigating subjective aspects of users with visual impairments' interaction with the web (Aizpurua, Harper, and Vigo 2016). Lazar, Feng, and Allen (2006) collected data on challenges faced by 100 web users with visual impairments and how these challenges affected their mood. The analysis produced a number of findings. Contrary to previous research on users without impairments, the moods of individuals with visual impairments were not significantly affected by time lost during tasks. Instead, their mood was notably influenced by the impact on work, indicating that the frustration's effect on their ability to complete tasks mattered more than the time consumed. Users with visual impairments responded to frustration by seeking workarounds rather than giving up. This approach seemed tied to their determination to accomplish their work. The study revealed that users' strategies for managing frustration by prioritising solutions over immediate frustration could benefit the wider population. Additionally, the research found a quadratic relationship between mood changes and anxiety, indicating that users are more affected by frustration when either excessively anxious or overly relaxed. Encouraging individuals to maintain a composed and receptive attitude can potentially alleviate the impact of frustration and foster a positive emotional state. In general, the findings from Lazar, Feng, and Allen (2006) indicate that frustration significantly contributes to the decline of participants' moods.

2.6 Research on Accessibility Overlays

The academic literature on the topic of accessibility overlays is scarce. Although they have existed for many years, discussions and lawsuits about them have only begun in recent years. This may account for the insufficient exploration in that particular domain. Thus, this thesis will be an essential contribution to the knowledge base on accessibility overlays, making it a fundamental piece of research. The identified related work in the research field of web accessibility overlays is detailed in the subsequent paragraphs.

A survey conducted by WebAim (2021b, sect. "Overlay, Plugin, Widget Effectiveness") found that the majority of web accessibility practitioners (67%) and respondents with impairments (72%) consider web accessibility overlays that automate accessibility changes in web pages as "not at all" or "not very effective". However, 2.4% of respondents perceive them as very effective.

According to Huq et al. (2023), the primary obstacles to implementing web accessibility are time, financial constraints, and resource costs. These challenges often result in practices that conflict with the values held by the accessibility community. As an example, Huq et al. (2023) name the use of accessibility overlays. The researchers continue by stating that this practice has two significant implications that require attention. The first implies that if accessibility is reduced to just being an issue of regulatory conformance, companies prefer to choose the less expensive solution, even if it does not ensure an accessible product. The second foresees a market filled with comparable products that aim to fix inaccessible software by technical means. While Huq et al. (2023) cannot anticipate the technical effectiveness of accessibility overlays, they state that reliance on them hinders accessibility integration in the core development process and diminishes the need for accessibility awareness among practitioners.

A study by Kane et al. (2011) presents three software-based "access overlays" that aim to improve the accessibility of large touchscreen interfaces for blind users. Through a comparative user study with 14 blind participants, the three access overlays "edge projection", "neighbourhood browsing", and "touch-and-speak", were evaluated against the screen reader VoiceOver. The results showed that the proposed techniques eased the interaction process and gave users a more accurate understanding of the screen layout. Significantly, the study reported a strong preference among participants for the newly introduced techniques over the existing VoiceOver solution, highlighting their potential to improve the usability of touchscreen devices for users with visual impairments. However, the access overlays described within the study are more similar to access devices than the definition of accessibility overlays of this thesis. Besides, the access overlays of Kane et al. (2011) were designed for a large two-handed touch screen, not a desktop computer or laptop. Participants were also not browsing the web but interacted with maps and selected targets. Furthermore, the participants did not possess the large touch screen, which meant they were unfamiliar with it and did not explore or know of other ways of interacting with it. Therefore, the study differs significantly from the research on accessibility overlays on the web in this thesis. The outcomes of the study from Kane et al. (2011) cannot be directly applied to the discussion of findings from the research in this thesis but rather showcase different investigations in the area of overlays.

A recent study by Egger, Zimmermann, and Strobbe (2022) looked at the possibilities and limitations of accessibility overlays as a support for accessible websites. The research examined the effectiveness of accessibility overlays in improving website accessibility to meet legal obligations such as the ADA and the European Union Directive 2016/2102. It investigated whether these overlays genuinely enhance accessibility and identified their limitations and potential for improvement. Nine accessibility overlays were initially considered, and three (accessiBe, EqualWeb²⁷ and UserWay²⁸) were analysed in detail in a comparative study. Egger, Zimmermann, and Strobbe (2022) defined seven metrics for their comparative study based on the WCAG 2.1 and the information provided by the accessibility overlay companies. The BIK BITV-Test and four additional metrics were used to validate the changes made by accessibility overlays. Although user tests were not part of the study, the perspectives of website owners were taken into account. Significant differences between the examined accessibility overlays and their ability to adjust the website were observed. accessiBe achieved the best result in the study by repairing more accessibility errors for users with visual impairments and those employing keyboard navigation than its competitors. However, none of the overlays achieved full compliance with legal regulations. Egger, Zimmermann, and Strobbe (2022) proposed potential areas for these technologies to improve, emphasising the requirement for ongoing development and enhanced transparency regarding their limitations. The results also underscore the necessity of educating website owners about manual adjustments and the need for future research into the effectiveness of accessibility overlays.

Kohler (2023) summarised in-person research sessions conducted by the Nielsen Norman Group (NN/g)²⁹ in 2023 with mobile device users who are blind or partially blind and have to rely on screen readers. The blog post explores screen reader users’ challenges on mobile devices, specifically highlighting their difficulties in accessing information sequentially and their inability to scan page content efficiently. As part of the research, the efficacy of accessibility overlays was investigated. Kohler (2023) and his team studied whether accessibility overlays are practical and recognisable for users with visual impairments. To answer this question, they had participants perform tasks on mobile websites that utilised accessibility overlays. Although participants encountered the floating button to access the accessibility overlay functionality during task completion, not a single user opened them voluntarily. Both fully and partially blind study participants almost entirely ignored them. When participants did not voluntarily access the accessibility menus, the researchers encouraged exploration. After a brief investigation of the accessibility overlay by the participants, the researchers concluded that the overlays were redundant and restrictive. Screen reader users typically become highly familiar with their screen reader. Therefore, there is no reason for them to abandon a system they already use efficiently to learn a new one that may be less effective and will only be available on a subset of sites (Kohler 2023). Despite the aim of accessibility overlays to aid users and adhere to ADA requirements and the WCAG standard, the study concludes that although accessibility overlays may be advantageous to sighted individuals and may guard companies legally, they do not significantly improve accessibility for screen reader users. True accessibility necessitates comprehensive testing with real users with impairments, as most problems are structural and cannot be solved by an accessibility overlay (Kohler 2023). As per Kohler (2023), enhancing accessibility for screen reader users demands concerted effort and an understanding of the unique browsing experience individuals with visual impairments have compared to sighted users.

3 Methods

The research questions were addressed through an evaluation study. Initially, a preliminary study was conducted to obtain further insights into accessibility overlays before commencing the evaluation study. Hereafter, three accessibility overlays and one website for the study were chosen, following a similar method to that of Egger, Zimmermann, and Strobbe (2022). Subsequently, a quantitative technical evaluation was conducted on the selected overlays and website to determine their current accessibility status based on the success criteria of the WCAG. It is worth noting that this thesis only employed the WCAG, whereas the study of Egger, Zimmermann, and Strobbe (2022) also used the BIK BITV-Test. This choice was made because the BIK BITV-Test is not very common worldwide, written in German, and created in Germany.

Afterwards, the usability and user experience were assessed in a user study via interviews, observations, and questionnaires. This thesis sets itself apart from Egger, Zimmermann, and Strobbe (2022) through the implementation of a user study with individuals who have impairments. Their study concentrates exclusively on the technological facets of accessibility overlays without exploring their effects on individuals with impairments. While accessibility guidelines are a helpful starting point, prior empirical research conducted by Power et al. (2012) indicates that the WCAG only address approximately half of the issues users with visual impairments face when navigating the web. As a result, a website may satisfy the guidelines and still not be accessible enough for users. The findings of Aizpurua, Harper, and Vigo (2016) support that view by stating that a website with a significant number of violations of the WCAG success criteria may still be perceived as accessible. On the contrary, a website that complies with the guidelines may not always be perceived as accessible (Aizpurua, Harper, and Vigo 2016).

The last phase of the research for this thesis concerned the evaluation of the findings. The following sections describe each stage of the evaluation study in detail.

3.1 Preliminary Study

The results of the literature review described in Chapter 2 revealed a lack of research on accessibility overlays. As a result, the author performed a preliminary study to gain practical insight into how practitioners discuss the topic. A web search was conducted to find comments and opinions that stem from online surveys, blog posts, and public statements. In addition, expert interviews were conducted with accessibility consultants Susanne Buchner-Sabathy and Sina Bahram. Both experts were chosen for interviews due to their extensive experience in web accessibility. Buchner-Sabathy is a consultant and expert in digital accessibility. She grew up with a visual impairment and has been blind for a few years. The interview with Buchner-Sabathy was conducted online in German on June 28, 2023. The translated interview questions can be found in the Appendix F.1. Bahram is a blind computer scientist, consultant, business owner/entrepreneur, invited expert to the W3C's ARIA Working Group, and former researcher who has worked in and around digital accessibility for the past 23 years. The interview was conducted online on October 6, 2023, and the questions can be found in the Appendix F.4.

Further, representatives from UserWay and EyeAble, companies that sell accessibility overlays, were interviewed. Both representatives were selected for interviews because of their familiarity with and knowledge of accessibility overlays. Although both companies sell an accessibility overlay, their approaches differ. Specifically, UserWay employs automated remediation in conjunction with its accessibility overlay, while EyeAble opposes automation. Lionel Wolberger represented UserWay, and Chris Schmidt represented EyeAble. Schmidt is the Co-Founder and Chief Marketing Officer at Web Inclusion GmbH, the company behind the accessibility overlay EyeAble. The interview was conducted online in German on August 8, 2023. The translated interview questions can be found in the Appendix F.2.

Wolberger is a serial entrepreneur who has authored patents, holds a Ph.D. in anthropological approaches, serves as Chief Operating Officer at UserWay, and is a member of the W3C Accessible Platforms Architectures Working Group. The interview was conducted online on October 17, 2023, and the questions asked can be found in the Appendix F.3.

All interviews were transcribed and, where necessary, translated from German into English. The author analysed the transcripts to summarise expressed opinions into topics. The identified opportunities and challenges of accessibility overlays are presented along with direct quotes and comments found online in Chapter 4.

3.2 Technical Preparation

As a prerequisite to the evaluation study, a technical preparation phase was required. During this phase, the website and accessibility overlays for the evaluation study were chosen. Furthermore, technical arrangements were made in advance of both the technical evaluation and the user study. Detailed descriptions of each step of the technical preparation phase are provided in the following paragraphs.

3.2.1 Accessibility Overlay Selection

The initial stage of the technical preparatory phase involved selecting suitable accessibility overlays. A Google search was performed to identify accessibility overlays for the study, as most users rely on Google when conducting searches (Urman and Makhortykh 2023). Because search results on Google are customised to the user's location, it should be pointed out that the search was performed in Southern Europe. Different search terms, such as "accessibility overlay", "accessibility overlay company", or "web accessibility overlay", were entered into the Google search bar. The search terms were adjusted depending on the search results obtained. Companies that appear on the first page of search results were selected for further analysis. According to a study by Urman and Makhortykh (2023) in the DACH region (Germany, Austria, Switzerland), over 97% of all clicks are made on the first page of search outputs, with over 50% of users clicking on the first search result. The objective was to identify five to ten companies that provide accessibility overlay services to obtain a suitable sample size for the analysis. Then, their marketed product features were compared to provide a basis for selecting three accessibility overlays for the evaluation study.

3.2.2 Website Selection

The second step of the technical preparatory phase was the selection of a website for the user study. The author coded no own inaccessible website to avoid potential personal bias. Instead, an inaccessible public website was selected. The selection criteria for the website were as follows.

1. Website Size

   The website must have subpages, apart from the homepage, to provide multiple tasks for user study participants while avoiding monotony.

2. Content-Language

   The website content must be available in both English and German, as these are the languages spoken by the target group.

3. Usage of Different Media Types

   The website must contain a range of media, including images, text, video and sound, to ensure diversity in the tasks carried out during the user study. Media diversity also necessitates extra steps for accessibility, which is of interest when investigating the capabilities of accessibility overlays.

4. Accessibility Errors

   The website must feature accessibility issues for examination of their impact on users with impairments, as well as to explore the effectiveness of accessibility overlays in addressing them.

3.2.3 Technical Set-Up

The inaccessible public website was cloned into a GitHub³⁰ repository. This step was essential to ensure the same website with the exact content was available for every participant joining the user testing. Once the website was cloned, some minor changes were made. All existing tracking links, for example, a Google Tag Manager³¹ snippet, were removed from the code. This was done because the author had no access to the tracking results of the actual live website, and it was also not needed for the study. Additionally, the meta tag <meta name="robots" content="noindex, nofollow" /> was added to the HTML <head> section to hide the website from crawlers and not list it in search engine search results.

Next, four different branches were created and deployed on a server. The first branch displayed the website as-is. The three other branches were for each of the selected accessibility overlays. Each branch had its own domain.

• fusilli.dnikub.dev - no accessibility overlay

• farfalle.dnikub.dev - accessibility overlay 1

• rigatoni.dnikub.dev - accessibility overlay 2

• spaghetti.dnikub.dev - accessibility overlay 3

The selected names for the custom domains were pasta types, prepended to the existing domain of the author’s website. This naming scheme allowed for quick randomisation of site order during the user study without providing clues about the chosen accessibility overlays.

3.3 Technical Evaluation

As mentioned earlier, the evaluation study consisted of a quantitative technical evaluation and a mixed-method usability and user experience evaluation. The technical evaluation examined technical metrics regarding accessibility and consisted of automated and manual accessibility testing. These tests were performed to acquire the current accessibility status of the selected website before adding the accessibility overlays. WCAG 2.1 by WAI (2022) served as the accessibility compliance reference since it was the latest valid version at the time of the study (July 2023). This study examined Level A and AA of the WCAG. WCAG 2.1 success criteria that do not apply to the website were excluded. Additionally, only the desktop version of the website was evaluated. This reduction was made due to the scope and time restraints of the thesis.

The results of this first technical evaluation served as the basis for the study and subsequent comparison with and between the accessibility overlays. After the evaluation of the selected website was completed, each copy with its respective accessibility overlay underwent testing. This second evaluation aimed to assess the resolution of identified accessibility issues through the use of accessibility overlays, and to identify any newly arisen errors. The result was four technical automated and manual accessibility evaluations.

All accessibility tests were performed on a 15-inch computer with MacOS 13 (Ventura³²) using the Web Browsers Brave³³ and Safari. Safari was explicitly used for testing websites with the Mac built-in screen reader VoiceOver, as it ensured the highest level of compliance with this access software. Additionally, it is the second most utilised web browser globally (Fleck 2023). Although Google Chrome is the most widely used browser worldwide, with 63.6% of internet users utilising it according to Fleck (2023), Brave was chosen for its advanced privacy features. Both Brave and Google Chrome are built on the same Chromium³⁴ code base, indicating that the majority of their code is shared. Additionally, the author installed an AdBlocker in the form of a browser extension within Brave. AdBlockers suppress the use of tracking pixels on websites (Kamps and Schetter 2020). In 2022, 35.7% of global internet users utilise an AdBlocker to safeguard their privacy, optimise their browsing experience by blocking online ads, and reduce load times and data consumption (Statista 2023; Kamps and Schetter 2020).

The next sections detail the process of automated and manual accessibility testing.

3.3.1 Automated Accessibility Testing

The technical evaluation started with automated accessibility tests. The tools utilised were the previously mentioned automated testing tools WAVE, Lighthouse and the Axe Accessibility Insights for Web. Only the homepage of each website was used in this testing step. Other pages on the websites were analysed during manual testing. This reduction in automated testing aimed to simplify the testing procedure. Since manual tests were to be carried out regardless, it was unnecessary to check all subpages automatically. Moreover, the homepage consisted of a navigation, a footer, and several content elements, indicating that most interactive content was already present on the homepage. Furthermore, it should be noted that the WebAim (2023) survey also only assessed the homepage of each website.

These automated tools give a first impression of how accessible a website is and where specific issues are. However, the results cannot be taken as-is. They have to be analysed manually to identify false negatives.

3.3.2 Manual Accessibility Testing

The first step of manual testing is analysing the automated testing results. As mentioned in section 2.4, doing this will put the findings into context, identify falsely found errors and give information on how to remediate the issues. A variety of Browser bookmarklets^35,36,37 that inject JS into the website were used to assist with manual accessibility testing:

• Images Bookmarklet

  It displays alt attributes in black-on-yellow text on top of all images and adds a green dashed outline around images with alt values and a solid red outline around images with no alt attribute.

• Grayscale Bookmarklet

  It switches all colours on the website into black, white and grey.

• Lists Bookmarklet

  It highlights the location and accessibility properties of <ul>, <ol>, <li>, <dl>, <dd>, and <dt> HTML elements.

• Iframes Bookmarklet

  It visually highlights the properties and location of <iframe> elements.

• Title Bookmarklet

  It displays all titles set to HTML elements in black-on-yellow text.

• Tabindex Bookmarklet

  It shows if HTML elements have a custom tabindex.

• Landmarks Bookmarklet

  It highlights all landmark roles on the website.

• Headings Bookmarklet

  It displays opening and closing tags around each <h1> to <h6> HTML element and WAI-ARIA role=heading aria-level=1 to aria-level=6.

• Forms Bookmarklet

  It inserts different coloured outlines to HTML forms, depending on the presence of for and id attributes. If they exist, they are visually displayed.

• ARIA Bookmarklet

  It visualises the used ARIA attributes over each element in black-on-yellow text on top of the images.

• Text Spacing Bookmarklet

  It changes the line height (line spacing) to 1.5 times the font size, spacing following paragraphs to 2 times the font size, letter spacing (tracking) to 0.12 times the font size and word spacing to 0.16 times the font size on the website.

• List Links and Buttons Bookmarklet

  It detects all links and buttons (actual HTML <a> and <button> elements and elements with role=link or role=button) and lists the visible text, the accessible name and the link target.

Additionally, the Web Developer browser plugin was used for manual accessibility testing. As described in section 2.4, it offers a toolbar with different functionality, such as highlighting headings, alternative texts, link texts, landmarks, labels and colour contrast between elements. The plugin can also turn off JS, CSS and cookies, visualise HTML form details, outline (background) images and external links or display hidden elements.

3.3.2.1 Keyboard Navigation

As mentioned in section 2.4, navigating the website with a keyboard only was part of the manual testing process. In this evaluation, this step checked if every element could receive focus, if the focus was always visible while navigating, if interactive elements behaved as they should and if the user accidentally became trapped somewhere on the website. Besides navigating the website with a keyboard only, the Axe Browser plugin was used in this evaluation to visualise the tab stops on the website. Not every internet user navigates with a mouse. Some might use a keyboard only. However, elements should still be available to them. Therefore, it is essential to evaluate keyboard accessibility.

3.3.2.2 Screen Reader Navigation

Another step of the manual accessibility testing process was navigating the website with a screen reader. This action was essential as the thesis focuses on people with visual impairments, and many within that group use access devices like screen readers to navigate and access web content. As previously mentioned, VoiceOver was used in this study. Although it is only the third most popular screen reader according to WebAim (2021a), it was the one available to the author and, therefore, used for the study.

A website's accessibility to a screen reader can be assessed by using one to navigate the website in question. Screen reader users have a distinct approach to website navigation compared to mouse or keyboard users. They usually scan the website aurally to determine whether to proceed further (Theofanos and Redish 2003). Hence, they do not listen to every word on the page, similar to sighted users who do not read every word. Many users set the voice of their screen reader to speak fast. Some users navigate by jumping from heading to heading or link to link (WebAim 2021a). Others use shortcuts and a screen reader built-in search function. It may be difficult for a blind user to find a particular keyword if it is not located at the beginning of a link or a line of text, according to Theofanos and Redish (2003).

3.4 Usability and User Experience Evaluation

According to Theofanos and Redish (2003), accessibility guidelines based solely on technology are insufficient to meet the needs of all users. Understanding the users and how they work with their tools is essential. Therefore, this thesis conducted an additional usability and user experience evaluation. The usability and user experience evaluation, further called user study, was a contextual inquiry. According to Salazar (2020) and Richter and Flückiger (2013), a contextual inquiry consists of in-depth observation and interviews of a small user sample to better understand work practices and behaviours, user needs and backgrounds. Furthermore, they state that this type of field study is well-suited for understanding the user’s interaction with a complex system and in-depth process, as well as the point of view of expert users. One of the greatest strengths of this methodology is that it uncovers aspects that one would not expect, as well as low-level details such as superstitious behaviour and illogical processes that have become habitual and invisible (Salazar 2020). Therefore, the contextual inquiry was suitable for this user study as it resulted in more detailed findings on the research topic of accessibility overlays. The user study of this thesis consisted of interviews, tasks, observations and questionnaires, which are detailed in the following sections. However, first, the selection criteria and recruitment process of the study’s participants are described.

3.4.1 Requirements for Participants and Recruitment

Various organisations that work with people with impairments, mainly within the DACH region (Germany, Austria, Switzerland), were contacted and asked to take part in the user study. The DACH region was chosen due to the author's origin in this area. The author's native language, German, facilitated access and connections with institutions and organisations in this region, making the recruitment process more effortless. The appeal was also posted to accessibility groups on social networks like Slack³⁸, LinkedIn³⁹, X⁴⁰ (former Twitter) and Mastodon⁴¹ and used hashtags to reach an even broader, also English-speaking, audience.

The spectrum of impairments is broad and individual. Nonetheless, for this thesis, a limitation had to be established. The research aimed to concentrate on one particular impairment and one of the three severity types of impairments previously outlined in section 1. The author chose to explore the category of permanent visual impairments. This decision was made because the author had easier access to individuals within that community, not because that particular group is more significant than others. In 2015, it was estimated that around 1.34 million individuals, representing 18.4% of the resident population over the age of 15 in Austria, had a permanent impairment (Baldaszti and Statistik Austria 2016). To specify, 3% of this group (40,200 individuals) had visual impairments. Regarding Switzerland, 1.1% of the Swiss population (77,000 individuals) experienced severe or complete visual impairments in 2017 (Federal Statistical Office of Switzerland 2017), while in Germany, the figure reached 558,725 in 2021 (Federal Statistical Office of Germany 2021).

3.4.1.1 Group Characteristics

Inclusion and exclusion criteria were defined to find suitable user study participants. Every interested individual filled out a study pre-selection questionnaire (Appendix E.1). This pre-selection questionnaire, also called a screening questionnaire or screener, had the goal of gathering participants' information to quickly identify and prioritise the suitable candidates who are representative of the target audience and exclude people that are not an appropriate fit (Fessenden 2021).

The following list further elaborates on this study's inclusion and exclusion criteria.

1. Regularly uses the internet

   The participants need to be used to browsing the internet and, therefore, to know interaction patterns subconsciously. They must also be aware of what a good UX for them feels like. This way, the rating of their experience during the study is more realistic than if they rarely use the internet.

2. No knowledge about accessibility overlays is required

   Participants need not have heard of or used accessibility overlays before. This thesis also investigates whether the target group interacts with overlays, regardless of whether they have encountered them before.

3. 18 years or older

   Due to legal reasons, all participants have to be adults. In the EU, where this thesis is written, people are legally considered adults at 18.

4. Fluent in English or German

   The study can be conducted in either English or German, as the author is proficient in both languages. Depending on the participants selected, the website, overlays, and interview questions can be provided in the chosen language.

5. Has a visual impairment with no vision corrected to normal

   If participants would have their vision corrected to normal with, for example, glasses, they would be excluded from the study. However, they are included if their impairment forces them to make adaptations to the device they use for browsing the web. These adaptations could, for example, be made by adapting font sizes, colours or contrast, zooming in or using an access technology like a screen reader.

   If participants have another impairment besides a visual one, this is no exclusion criterion. When analysing the study findings, it will be noted and considered as the UX may be further impacted by additional impairments. The main focus, however, remains visual impairments.

6. Has a permanent visual impairment

   All three types of impairments mentioned in section 1 are targeted by accessibility overlay companies. However, users with a permanent impairment are most impacted by an inaccessible website. Users with situational or temporary impairments are excluded from the study as their impairment is only short-term.

7. Has their own device to access the internet

   Study participants must use their own devices during the user study. This setting creates a more realistic user experience as participants may have changed settings on their devices to accommodate their needs. In addition, if they use access technology, they can use the tool they are used to and know how to operate.

8. Able to participate online

   Participants need to be able to join a video call and share their screen when they participate online. Otherwise, the author could not make detailed observations during the user study. Participants also need to be able to participate in the set timeframe for the study.

The objective was to recruit approximately 20 participants. According to Nielsen (2012), a small number of participants can provide valuable insights into the strengths and weaknesses of design and usability. They recommend using five participants for a usability study as this amount is sufficient to identify most usability issues, even when compared to larger study sizes. Richter and Flückiger (2013) agree with them, stating that five to seven people are sufficient to evaluate the main application scenarios and to initiate targeted improvement actions. They continue by expressing that the required number of test participants predominantly relies on the study's objectives. For quantitative studies that aim to produce statistically significant results, at least 20 participants are necessary, which is four times the required amount for qualitative studies (Nielsen 2012). As this user study gathered qualitative and quantitative data, it appeared reasonable to recruit around 20 participants. Nevertheless, if a smaller number was obtained, it would not be a concern.

3.4.2 Study Setting

The user study began with the author introducing herself and what to expect. According to Montoya (2023), participants must be relatively unaware of the hypotheses and aims of the study to not jeopardise the study's validity by intentionally altering their behaviour or responses to match the hypotheses. Hence, each participant was provided with as little information about the study as possible. The purpose of the research was to investigate the impact of accessibility overlays on the user experience and usability of people with permanent visual impairments, which was made clear to all individuals. However, they were not apprised of the debates and controversies surrounding accessibility overlays. Additionally, extensive details of the functionality of accessibility overlays and their marketing claims were also not provided to the participants. Participants were also not equipped with information on how to activate or interact with accessibility overlays. Each participant entered the study with their existing understanding of accessibility overlays in order to decrease the likelihood of any modified behaviour to match a hypothesis. If the participants did not know what an accessibility overlay was, they were told the following:

"An accessibility overlay is special software that is placed over a website by the website operator to make it more accessible to people with impairments. It adds additional features such as reading text aloud, enlarging content, adjusting colour contrasts and keyboard controls to make it easier to use. By using an accessibility overlay, people with visual, hearing or motor impairments can better understand and interact with the content."

The participants received information about the study's duration and procedure to ensure they were aware of what to expect. A within-subjects approach was taken. With a within-subjects approach, all participants experience and are observed on all experimental conditions (Montoya 2023). This approach results in the need for fewer study participants as they are not compared with others but with themselves (Figure 5). In this study, the same participants tested all four versions of the selected website in randomised order. Randomisation affects and minimises transfer and learning across conditions in within-subjects designs (Budiu 2023).

To compute an exclusive sequence of websites for each participant, the formula for permutation without repetition was applied. A nonrepeating permutation is a sequence of n distinct objects. After n placement possibilities for the first object, only n − 1 possibilities come into consideration for the second object, followed by n − 2 for the third object, and so forth until the last object, which has only one free place remaining. The following factorial indicates the number of possible permutations of n objects:

n!=n*(n-1)*(n-2)*...*1

The four websites utilised in this study yielded 24 distinct permutations of the order in which the websites were presented:

4!=4*3*2*1=24

Moreover, this conforms to the aim of recruiting 20 study participants, mentioned in section 3.4.1.

The participants were required to complete four tasks on four websites, equating to a total of 16 tasks. The tasks and the observational methodology are outlined in section 3.4.3. Additionally, interviews and questionnaires were carried out to gather supplementary data on participants' encounters and experience with accessibility overlays, as described in section 3.4.4.

The user study was conducted online with Zoom⁴², and the participants were asked to share their screens. The author chose Zoom as the video conferencing tool for the user study because it provided the most accessible experience out of all available options at that time. Zoom had the ability to schedule and record meetings and could be used directly in the browser. The participants did not need to install software, making it potentially more compatible with their browser and operating system. Other platforms, such as Microsoft Teams⁴³ or Google Meet⁴⁴, are often too complicated or not keyboard accessible, hindering users from unmuting their audio or sharing their screen. An accessible tool is vital for a user study with people with impairments.

Performing the study in-person would potentially have brought more profound insights into the UX, but adaptions were made for the well-being of the study participants with impairments. By performing the study online, every participant could join from the comfort of their home and participate on their device. Using the participants' devices was crucial for this study as it created a more realistic approach. The participants navigated the websites with their access technologies, tools and browser or system settings. It is important that participants use devices and technologies which they know how to operate and are comfortable with. Forcing participants to bring their devices to in-person user testing is unnecessarily complex and, in some cases, impossible. Performing the study in the users' natural environment adds context to the contextual inquiry. In this case, an online study was more convenient and less time-consuming for participants.

The participants were directed to work at their own pace, avoid speaking out loud whilst completing a task but to raise any questions or frustration that they may have had. Following this, the participant's comprehension of the consent form was checked. The consent form was dispatched one week before the scheduled testing. Informed consent is an indispensable component of ethical user research that honours the participants and protects them from harm (Fessenden 2022). The ethical facets of this study are further enumerated in section 3.4.5. Once screen sharing had started, the recording and user study began. It was estimated to last one to one and a half hours long.

A pilot study to assess the feasibility was conducted online with two participants. Both had permanent visual impairments but did not meet all the group characteristics set for the actual user study. One had their vision corrected to normal and neither adapted their browser or system settings nor used access technology. Therefore, their recorded time spent on a task was multiplied by two to gain an approximate sense of the study duration for people with access technologies. The interview questions and tasks were adapted according to the feedback received.

3.4.3 Tasks and Observations

The participants were given four tasks to complete. Each task was repeated on each of the four websites. The four given tasks were virtually the same for each website. A slight variation was added to help participants avoid becoming bored and to motivate them to interact with all the websites. According to Moran (2018), tasks must be as realistic as possible, emotionally neutral and sufficiently descriptive without unnecessary verbosity. They must not provide any priming clues to participants and allow them to solve the task independently. If a task fails to produce the intended insights, it can be modified. According to Richter and Flückiger (2013), the quality of the results of a usability study depends largely on how tasks are designed. In a qualitative study, it is acceptable to alter the task phrasing, eliminate existing tasks or even introduce new ones mid-study (Moran 2018). The task instructions were read out loud to the participants during the user study.

1. Task 1 involved making the website accessible and usable, if possible and necessary. The aim was to assess if participants interacted with the accessibility overlay and made any additional adaptations to their devices.

2. In Task 2, participants had to explore an artist profile on the website and listen to a short excerpt of their performance. Task 2 aimed to investigate the user's interaction with various content types on the website, such as the audio player, search function, and menu.

3. For Task 3, users were asked to select a press photo from the "Press" page and describe its content to assess whether users could identify, comprehend, and utilise the image download button. Additionally, it was examined if alternative texts for images were available and conveyed the appropriate meaning.

4. Task 4 asked users to display only specific posts on the "News" page. The aim of task 4 was, again, to investigate user interaction with the website and its varied content types, for instance, the filter option and menu.

The exact wording of the tasks given can be found in the Appendix D.1.

Another important aspect of a contextual inquiry is observing the user and enquiring information about their needs to comprehend their actions and motives (Salazar 2020; Richter and Flückiger 2013). In this qualitative method, the author took note of the completion of tasks and the task duration, as well as the reasons for items being (in-)accessible or (un-)usable. The participants did not think aloud while completing the tasks but worked independently. This decision was made because if a user applied a screen reader, their verbalisation could have interfered with what the screen reader was saying, making it more challenging to observe. Clarification and follow-up questions were asked after a participant had completed their task.

3.4.4 Interviews and Questionnaires

In addition to metrics such as task completion, duration, and noted observations, interviews and questionnaires were conducted to gather additional data about the participants' experiences with accessibility overlays.

3.4.4.1 Interviews

Several open-ended interview questions (Appendix E.3) were asked during the study. Depending on the answers given, some questions were omitted, and additional spontaneous questions were added. The semi-structured interviews offered a more in-depth understanding of the observed interactions, identified problems and general perception of the target group.

At the beginning of the user study, before looking at the first website, questions about the participants' navigation style with (access) devices, their internet usage and their system settings were asked. This was done to contextualise observations during the study and to understand technical errors that may have occurred. The participants were also questioned about their knowledge and perception of accessibility overlays. These questions were designed to provide information about how each individual approached the study and to help analyse the observed interaction.

After each task, the participants were asked to briefly summarise how they completed the task. This information helped in the analysis of observations, as participants did not comment on their interactions while performing the task. Especially when using access technology such as a screen reader, the author may not have captured every detail of the interaction with the website. In addition, technical difficulties or unexpected behaviour, if any, were discussed. After each task on the first website, the participants were requested to rate the task's difficulty. On each subsequent site, the participant was asked if the task differed from the previous site.

After each website, questions were asked about the experience of navigating the website. In addition, if the website had an accessibility overlay, the participants were asked if they had noticed it was there, why they had or had not interact with it, and if it had been helpful to them. At the end of the study, after the participants had interacted with all four websites, they were asked final questions about accessibility overlays.

Additionally, expert interviews with accessibility advocates and accessibility overlay company representatives were conducted to supplement the limited scientific research available on accessibility overlays at the time of this thesis. The aim of conducting these interviews was to integrate viewpoints from various perspectives alongside identified scientific research and online blog publications. Appendices F.1 to F.4 list all questions asked by the author during each interview. The full transcripts, however, are not published since the thesis scope does not permit verification of the mentioned (marketing) claims by some of the interviewees. The thesis solely presents the individual's viewpoints on accessibility overlays in the form of selected quotes without the intention of promoting any software solution.

3.4.4.2 Questionnaires

The questionnaire used four times during the user study was the AUS. It is a free tool for measuring the usability of a digital product, explicitly designed for access technology users, and therefore well suited to this user study. It was created by the company Fable⁴⁵ and is licensed under a Creative Commons Attribution 4.0 International License⁴⁶. The AUS was modelled after the SUS, a 10-question survey to measure users' subjective perceptions of a system's usability. According to its creator Brooke (2013), the SUS is considered reliable and measures both learnability and usability in alternating questions to avoid response biases. They continue stating that the SUS is technology-neutral, meaning it can also be used for websites, despite its creation in the 1980s before websites existed. The AUS, like the SUS, also consists of ten questions alternating from positive to negative notions, forcing participants to read each statement carefully and respond accordingly. This alternating model minimises extreme response and reduces agreement bias (Fable 2023). The author translated all ten questions into German for the study (Appendix E.2).

In 2022, the average AUS score, according to Fable (2023), after compiling data from 2,100 usability sessions, was 65. This is just three points off the average SUS score of 68, which is based on decades of research. Fable (2023) further states that the reliability of the AUS in the consistency metric Cronbach’s alpha is 0.93, which is very similar to the reliability of the SUS at 0.92. Cronbach's alpha was developed by Lee Cronbach in 1951 and provides a measure of the internal consistency of a test or scale, expressed as a number between 0 and 1 (Tavakol and Dennick 2011). If alpha is too high, this may indicate that some items are redundant, as they test the same question differently. A maximum alpha value of 0.90 is recommended. However, Fable (2023) states that its Cronbach's alpha value shows that "the AUS has excellent reliability and that all ten questions effectively measure usability". As the score is nearly equivalent to the SUS, the AUS appeared to be a suitable choice for incorporation into the user study.

Aaron Bangor and Miller (2008) produced a figure comparing the mean scores of the SUS by quartile, adjective ratings, and the acceptability of the overall SUS score (Figure 6). The figure demonstrates that products rated at least passable have SUS scores above 70, with better products scoring in the high 70s to upper 80s. Very superior products score better than 90. Products with scores below 70 should be subject to increased scrutiny and continued improvement and should be considered marginal at best. Products with scores below 50 should be a cause for significant concern and are deemed unacceptable (Aaron Bangor and Miller 2008). Fable (2023) does not provide an interpretation scale for the AUS but refers to the just described SUS interpretation.

Each of the ten AUS questions can be rated on a 5-point Likert Scale. A Likert Scale is a range of responses from positive to negative, with a neutral score in the middle (Likert 1932). It usually has an odd number of categories (3, 5, 7 or 9), with the middle category indicating indecision, ambivalence or lack of opinion on the part of the respondent (Jiří et al. 2021). For the AUS, a scale position is assigned to each option of the Likert scale to quantify the perceived usability (Table 1). To calculate the AUS score, positively worded questions (1, 3, 5, 7, and 9) get subtracted by one and then multiplied by 2.5. Negatively worded questions (2, 4, 6, 8, and 10) get subtracted from five and multiplied by 2.5. In the end, all individual scores are summed up.

According to Fable (2023), the best time to use the AUS is at the end of task-oriented research. The author decided to let participants fill out the AUS after each website, resulting in four completed questionnaires per party. However, there was a problem with the use of the AUS in this user study. The participants interacted with the same website four times. The only difference was the added accessibility overlay. Although the order in which the websites were presented was randomised, the participants were still able to learn how to navigate the website faster and more efficiently with each use. However, the AUS question remained the same. This resulted in potentially falsified AUS scores for the second, third and fourth sites viewed. This limitation meant the scores could not be taken out of context or viewed separately. They were all related and had to be viewed calculated as an average or as a timeline evaluation. Despite this limitation, the author still chose to use the AUS because it could help participants review and more accurately evaluate their experience with the website. At the same time, the four calculated AUS scores visualised the learning that occurred on the user's end.

In addition to the ten questions of the AUS, each participant was asked to rate their UX on a 5-point Likert Scale from "very bad" to "very good". According to Trewin, Marques, and Guerreiro (2015), accessibility researchers often use Likert scales to gather feedback on proposed accessibility technologies and to compare technologies. The 5-point scale, instead of a 7 or 9-point scale, was chosen to save time and at the same time make it easier for all participants. As the AUS is already employing a 5-point Likert scale, participants are used to it. In addition, when reading a larger rating scale with access devices like a screen reader, it can become overwhelming as an increased amount of time is needed to answer the question. The gathered information added additional context to the usability findings of the AUS result, the task performance and observations.

3.4.5 Ethical Considerations

Myers and Venable (2014) suggest that ethical issues should be considered early in the research process before any activities affecting people are conducted. Moreover, ethical issues must be constantly kept in mind throughout the research project. Thoo et al. (2023) encourages researchers to collectively reflect and discuss potential guidelines regarding collecting and reporting information about participants with impairments, as well as the methods for gathering such information.

Some precautions were taken to comply with data protection laws. Ensuring adequate safeguards for privacy is crucial not only for those directly involved in the project but also for those who may use or be impacted by any developed software or information system in the future, as Myers and Venable (2014) have advised. After completing the screening questionnaire, each participant was randomly assigned a name. This name was used in all notes, interviews, questionnaires and communications to refer to that participant. In addition, all recorded data from the user study was stored offline on the author's hard drive only. The material was not synced to the cloud or any other online storage. No one other than the author interacted with the unsorted data. Participants were able to withdraw their consent to participate in the study at any time. Informed consent was essential for this study.

Informed consent involves the exchange of information between the researcher and the participant (Fessenden 2022). Myers and Venable (2014) argue that informed consent must be obtained from any individual who is in any way involved with the research project. Typically, the researcher informs the participants about the nature of their involvement in the study, as well as the potential consequences of participating. The participants must fully comprehend the terms of the study to make an informed decision about whether they would like to participate voluntarily (Fessenden 2022). It is crucial to ensure that participants are provided with comprehensible and accurate information about the study and its potential risks and benefits Myers and Venable (2014). Additionally, they should be able to consent voluntarily and without compulsion or pressure. Failure to obtain informed consent can result in serious ethical violations, which can impact both the participants and the credibility of the research.

All the user study participants signed a consent form before starting the study. The form included details on the study's aim, the study's conductor, data collection and storage procedures, the purpose and duration of data retention, methods for anonymising data, and individual rights (including the right to access, authorise deletions, restrict processing, and portability of data). The understanding of the consent document was assessed at the beginning of the study. Furthermore, it was brought to everyone's attention that certain tasks could be challenging, frustrating, and mentally and physically exhausting due to the website's inaccessibility. The term "inaccessible" not only implies unattainability of specific data but may also refer to certain elements that induce discomfort or unease, such as excessive brightness in colour schemes or dynamic content. The well-being of the participants held utmost significance. The participants were allowed to pause or terminate the study at any point. The author routinely inquired about the participants' well-being and whether they wished to continue. The experts interviewed were briefed on the academic paper's contents, the purpose of the interview, and its transcription, with quotations that will be integrated into the thesis.

3.5 Evaluation of Results

After collecting data during both the technical evaluation and the user study, the last step of the evaluation study involved analysing the obtained results.

Before the technical evaluation, a large table of all relevant WCAG 2.1 success criteria was established. This table was updated throughout the evaluation and included data such as code examples, screenshots, and evidence from both automated and manual accessibility tests conducted on each website. During the evaluation phase, the table was analysed to determine the number of WCAG success criteria that had not been met across all four websites. Furthermore, it was utilised to classify detected accessibility issues according to their severity and exhibit them in Chapter 5 of this thesis.

The user study resulted in both quantitative and qualitative data. Initially, individual AUS and UX scores for each participant and website were calculated. Next, the interview responses were transcribed. Finally, the user study notes and transcripts underwent a thematic analysis. This commonly used qualitative method allows to translate observations and apply statistical analysis to determine the validity of the themes (Boyatzis 1998, 7). The author followed the six-phase approach of Braun and Clarke (2012), which includes familiarising themselves with the data, generating codes, identifying themes, reviewing themes, defining and naming themes, and reporting. To organise, identify themes and make sense of all the data gathered in the user study, an affinity diagram was constructed. When analysing qualitative data, an affinity diagram provides a practical method for exchanging ideas (Richter and Flückiger 2013). During this method, relevant observations and findings are noted on cards and subsequently collectively analysed, arranged and interpreted in an evaluation workshop. Because of the groups of thematically related cards that emerge, this technique is called an affinity diagram (Richter and Flückiger 2013).

4 Preliminary Study

The potential benefits and disadvantages of accessibility overlays to address inaccessible websites are currently the subject of ongoing debate among accessibility consultants, accessibility advocates, accessibility overlay companies and people with impairments.

In general, discussions around accessibility overlays tend to focus on three different topics:

1. Accessibility overlays with automated remediation

2. Accessibility overlays without automated remediation

3. Improvements or alternatives for accessibility overlays

For each topic, there are advocates and critics. Furthermore, some arguments are grounded in regulatory viewpoints, while others are rooted in financial or ethical considerations. The subsequent sections elaborate upon the diverse opinions, showcasing comments expressed through online blog posts, surveys, public statements, related lawsuits or interviews conducted by the author.

4.1 Lawsuits Related to Accessibility Overlays

Discussions surrounding accessibility overlays have already resulted in legal action. Numerous lawsuits against companies using accessibility overlays concerning their compliance with accessibility guidelines exist and have been covered by the media, examples are given below.

• Murphy vs. Eyebobs⁴⁷ (accessiBe)

  Eyebobs' is an online glasses company that faced a lawsuit in January 2021 for failing to comply with accessibility requirements from a blind plaintiff. As per the settlement, Eyebobs' will have to make several changes to its policies, procedures, and personnel to ensure greater accessibility to its digital platforms (Byrne-Haber 2021).

• LightHouse vs. ADP⁴⁸ (AudioEye)

  ADP, a large HR software company, had implemented the AudioEye accessibility overlay. However, blind employees still filed a lawsuit against the company. In December 2021, a settlement was reached to enhance the accessibility of ADP's products for San Francisco LightHouse for the Blind employees (Feingold 2022).

• Quezada vs. U.S. Wings⁴⁹ (accessiBe)

  U.S. Wings has purchased a one-year partnership with accessiBe to ensure their website remains compliant, resulting in three audit reports confirming the website is WCAG 2.1 compliant. The court finds that U.S. Wings has not demonstrated that they have undoubtedly fixed the accessibility concerns on their website as multiple barriers still allegedly exist.

• Paguada vs. YieldStreet⁵⁰ (UserWay)

  YieldStreet, an alternative investment platform, sought to dismiss an accessibility lawsuit against them by incorporating UserWay's accessibility overlay into their website. Nevertheless, an expert witness found accessibility errors that persisted despite the overlay, leading to Judge Lorna G. Schofield denying the motion for dismissal.

Legal action explicitly taken against companies using accessibility overlays is on the rise. In 2020, more than 250 lawsuits were initiated against companies utilising accessibility overlays in the USA (Taylor 2020). The registered complaints persisted in 2021, with 400 filed lawsuits (Taylor 2021; Jason 2022). From January to June 2023, plaintiffs filed 414 lawsuits against websites featuring active accessibility overlays, compared to the 315 suits documented in 2022 over the same period (UsableNet 2023).

While some accessibility overlay companies face lawsuits, there are also legal disputes with accessibility advocates who publicly express their concerns and advise against using particular accessibility overlays. Specifically, the French accessibility overlay provider FACIL'iti has taken legal action against Julie Moynat, a web designer and web accessibility consultant, in her personal capacity and against the French digital accessibility firm Koena⁵¹ (Feingold 2021). Similarly, AudioEye, an accessibility overlay company, has taken legal action against consultant, writer and speaker Adrian Roselli⁵², who has been involved in the development of accessible user interfaces for 30 years and is a member of several W3C working groups (Roselli 2023).

According to disability rights lawyer Feingold (2023), these lawsuits harm people with impairments as such litigation can affect those who advocate for the inclusion of disabled people online. Feingold (2023) and the attorneys of Roselli (2024) believe that AudioEye has filed a Strategic Lawsuit Against Public Participation (SLAPP), a type of lawsuit that targets activists worldwide.

"Instead of embracing Adrian Roselli's expertise to better its product, Audio Eye, in my opinion, has strategically singled out a top leader in this space for his participation in this urgent conversation."

The outlined legal situation underlines the importance of scientific research in this area. While all parties aim for a universally accessible web, they disagree on how to achieve it.

4.2 Arguments Against Accessibility Overlays

The following sections present a breakdown of online comments and expert interviews arguing against accessibility overlays.

4.2.1 Viewpoints Opposed to Accessibility Overlays Expressed Online

Several individuals and organisations have publicly expressed their opposition to accessibility overlays and advised against their usage. In this context and thesis, the term accessibility overlay is used to refer to the widget, toolbar, or plugin with automated remediation unless stated otherwise.

1. The EDF (2023) and the IAAP have issued a joint statement to clarify the limitations of accessibility overlays and to warn that this technology does not make websites accessible or compliant with European accessibility legislation. In addition, the EDF (2023) states that certain accessibility overlays may hinder the use of access technologies that individuals with impairments utilise to access online content. The EDF (2023) suggests that technology buyers should engage with digital accessibility specialists, people with impairments and their representative organisations to gain insight into user requirements and how they can be met.

2. The NFB in the USA has accused the accessibility overlay company accessiBe of engaging in conduct detrimental to the progress of blind individuals in society. Specifically, the board of the NFB (2021) believes that accessiBe dismisses the concerns of blind persons about its products and approach to accessibility in a peremptory and scornful manner. Additionally, the board reports that accessiBe disrespects blind technology experts in the press and other communications. Moreover, the company fails to recognise that blind experts and individuals who regularly use screen readers have a deep understanding of accessible technology. Furthermore, accessiBe overemphasises the effectiveness of automated testing, neglecting to recognise the significance of involving real humans in the process of user testing. In addition, promoting false claims of achieving compliance with Web Content Accessibility Guidelines through integrating a single line of code is misleading, according to NFB (2021). As a result, the National Federation of the Blind withdrew accessiBe’s sponsorship of its national convention (NFB 2021).

3. According to the Directorate-General for Communication (DG COMM) of the European Commission (n.d.), accessibility overlays may decrease website accessibility for some users. They also state that claims that a website can be made fully compliant without manual intervention are unrealistic, as no automated tool can cover all criteria of the WCAG. DG COMM does not endorse any accessibility overlays.

4. The WACA initiative recommends against using accessibility overlays as "inaccessible websites remain inaccessible despite overlays". Considering accessibility from the beginning of website creation is much easier, according to WACA (2023). They continue stating that it is optimal for users to use their own access technology, such as screen readers, magnification software, and special mouse devices, as it best suits their individual needs. Additionally, implementing a website according to international WCAG criteria ensures optimal accessibility for all users, regardless of their impairment or access technology.

5. The Federal Monitoring Centre for Accessibility of Information Technology (BFIT-Bund) in Germany states that accessibility overlays cannot currently make a website with barriers completely barrier-free (BFIT-Bund 2023). In addition, using overlays may create further obstacles on a website. However, they can enhance existing accessibility by meeting additional criteria for the highest conformance level of the WCAG, according to the BFIT-Bund (2023). Nonetheless, no negative interactions are allowed to occur for users after an accessibility overlay is embedded.

6. 856 individuals (as of March 11, 2024) have signed the "Overlay Factsheet". According to its creator, Karl Groves, the Overlay Factsheet is a non-biased, community-driven, fact-based information sheet to educate customers on web accessibility overlays. All signatories agree never to advocate, recommend, or integrate an overlay which deceptively markets itself as providing automated compliance with laws or standards. In addition, they will always advocate for the remediation of accessibility issues at the source of the original error. Further, they will refuse to stay silent when overlay vendors use deception to market their products. More specifically, they advocate for the removal of web accessibility overlays and encourage the site owners who have implemented these products to use more robust, independent, and permanent strategies to make their sites more accessible (Groves 2021).

4.2.2 Viewpoints Opposed to Accessibility Overlays from Expert Interviews

The author interviewed Susanne Buchner-Sabathy, a digital accessibility consultant; Sina Bahram, a computer scientist, consultant, business owner and former researcher; and Chris Schmid, the Co-Founder and Chief Marketing Officer at Web Inclusion GmbH, the company behind the accessibility overlay EyeAble, to gain further insights.

In the experience of Buchner-Sabathy, what accessibility overlays promise often does not work reliably or entirely. Bahram claims that accessibility overlays can even introduce spurious speech events into a website, redirect keyboard input, and change expected behaviour. In addition, according to Bahram, in his experience, the websites of accessibility overlay companies are also frequently inaccessible, as well as their tools.

"I mean, I think their metric is 'we labelled some buttons'. That's not accessibility. That's baseline best practices for coding and programming."

Schmidt, speaking on behalf of EyeAble, agrees with Bahram as he considers it essential for an accessibility overlay to be accessible itself. He adds that their product, EyeAble Assist, is both accessible and certified as WCAG-compliant by an external testing agency. The available test report displays that EyeAble version 1.11.140 was last assessed by the company BITV-Consult, with Detlef Girke as the consultant, between October 21, 2022, and December 22, 2022, and was found to be compliant with BITV regulations (Girke 2022). Details of the BITV standard are outlined in section 2.3.3.

When asked why their website is inaccessible, Schmidt from EyeAble states that they did not consider accessibility from the beginning of their website creation. In addition, according to Schmidt, their website builder software is pushing them to "massive limits in terms of accessibility". EyeAble plans to relaunch a new website at the beginning of 2024 that is BITV certified.

"Otherwise, we don't need to sell what we sell, and in the end we are also untrustworthy."

Schmidt further claims to incorporate as many people with impairments as possible into the process of creating the new website, such as blind people, individuals with attention deficit disorders and people with learning difficulties.

Generally speaking, Bahram does not believe that accessibility overlays can reduce barriers or facilitate access to digital content. Instead, he argues that accessibility overlays introduce patterns that are either hostile toward disabled users or entirely contradict the expected interaction patterns that have been adopted and learned by both access and non-access technology users over the past two to three decades.

"I want to be very clear about my assertion. [Accessibility] overlays do not make a website better, they make it worse a majority of the time. And they also perpetuate a lack of the need and impetus for then fixing the accessibility issues in the first place. So, they not only not work, but they also make the world worse for disabled people."

The main criticism of Buchner-Sabathy on accessibility overlays is that they create false expectations about their effect. In her impression, the expectation that an accessibility overlay will make a website accessible is not only created for website owners who install an accessibility overlay on their site but also for site users.

"That [accessibility overlays make a website accessible] is simply not the case. Certain features can be compensated for with AI, but all the WCAG criteria will never be taken into account; it is not possible at all. [...] You need human understanding to decide what the accessible solution would be."

Schmidt from EyeAble shares the opinion of Buchner-Sabathy. He argues that employing an accessibility overlay, with automation or without, does not make a website more accessible according to the WCAG standard. Schmidt further states that WCAG compliance can not be achieved by an accessibility overlay that modifies HTML, automates processes, and produces texts that "you don't understand at all". He also clarifies that human intervention is still required.

"You can tell me what you want, but there, you don't even address the WCAG; there, you use the WCAG as a sales tool, and that's all. Period. [...] Accessibility should be of, for and with people with disabilities. Period. Anything else won't work."

Schmidt asserts that EyeAble does not refer to WCAG or BITV when advertising their accessibility overlay. According to their website, their accessibility overlay complies with some regulations but not all WCAG requirements. Schmidt further explains that the company changed its phrasing from accessibility to approachability when marketing its accessibility overlay to clarify that they do not promise compliance. The author checked their website five months after the interview to verify this claim. On the German website, specifically on the accessibility overlay page, the company uses the term "accessibility" ("Barrierefreiheit" in German) six times and "approachability" ("Zugänglichkeit" in German) seven times (January 2024).

After being asked why EyeAbles' website references client websites that are still inaccessible but all use the accessibility overlay, Schmidt stated that accessibility is a process and making a website WCAG compliant takes time.

"I can imagine that in six months' time, there will be a lot more press releases saying 'Hey, we've finally managed to make our website accessible'. But it's going to take a while for that to happen, and in the meantime [our clients are] saying OK, I'm on my way' and you're just seeing EyeAble Assist."

Buchner-Sabathy goes on to argue that institutions and large companies that implement accessibility overlays on their websites are wasting money without achieving the desired effect and neglecting to invest in the skills of their development team.

"So instead of helping developers to code accessible websites, companies are being paid for a promise that will not be kept. And I don't think that's OK."

Companies utilising accessibility overlays opt for simplistic solutions to complex problems, according to Bahram. He further contends that instead of tackling the underlying issues and coming up with more systemic solutions that would help everyone, companies are trying to use a "Band-Aid" technique to fix their problems.

Bahram points out that "many companies that use these access overlays think they are doing the right thing due to false claims and promises, in my opinion".

"[Accessibility overlay companies] are trying to sell a kind of a Band-Aid, an off-the-shelf solution to make your accessibility problems go away."

Another critique, voiced by Schmidt, is that some of their competitors are pigeonholing individuals with impairments through the use of impairment profiles within their accessibility overlays. He argues that impairments exist on a spectrum and cannot be generalised.

4.3 Arguments for Accessibility Overlays

The subsequent sections present a breakdown of online comments, expert interviews, and a white paper advocating for accessibility overlays.

4.3.1 Viewpoints In Favour of Accessibility Overlays Expressed Online

Market dynamics, usage trends, and financial metrics, alongside other supporting voices, encourage the use of accessibility overlays.

1. The list of available accessibility overlay companies is long and continues to grow. The Web Almanac (2022, chap. 11) report shows that 1.6% of desktop websites used one of 22 specific "accessibility apps", compared to only 1% in 2021. While not all of these identified accessibility apps are accessibility overlays, it is evident that the demand for software aimed at improving accessibility is steadily growing. Out of the detected accessibility overlay companies, three are the most popular: accessiBe⁵³, AudioEye⁵⁴ and UserWay. Of all the 8,360,179 websites analysed by the Web Almanac (2022, chap. 11), accessiBe is utilised on 0.37%, AudioEye on 0.35%, and UserWay on 0.49%, making them the most commonly used overlays in the dataset. However, UserWay is not used on high-traffic websites, i.e., websites ranked in the top 1000 by visits. Generally, the use of accessibility overlays and accessibility apps is less common for high-traffic websites compared to low-traffic websites (Figure 7). Merely 0.3%, or three websites, amongst the top 1000 websites used an overlay.

   

2. Many companies selling accessibility overlays, particularly those identified as the most common, receive notable funding and generate millions of dollars in revenue (accessiBe 2022; AudioEye 2022a; UserWay 2022). This fact further highlights the increasing popularity of accessibility overlays among business owners and investors.

3. A website called "Overlay Facts" was created as a counter to the Overlay Factsheet by Karl Groves. The site uses the term "edge technologies" to refer to accessibility overlays (Lemon Pulse 2022). It aims to provide a detailed, objective description of accessibility overlays, including their purpose and the issues they address. It also addresses controversies surrounding these technologies. The website does not indicate the entity responsible for it. However, it mentions that the founders have expertise in digital marketing and are directors with the aim of promoting the digital domain. Further examination reveals that Lemon Pulse, a digital marketing and Search Engine Optimisation enterprise, seemingly developed the website, yet their precise role involvement remains uncertain.

4.3.2 White Paper on Accessibility Overlays

The accessibility overlay company AudioEye published a white paper in April 2022 that illustrates their research and approach to accessibility overlays. The subsequent paragraphs outline some statements.

AudioEye (2022b, 18) asserts that automated solutions for identifying and addressing accessibility issues, commonly known as accessibility overlays, are a controversial topic, frequently presented in polarising perspectives. AudioEye (2022b, 14) states that some competitors falsely advertise overlays as "reliable and complete accessibility solutions", which can be misleading. accessiBe is the specific competitor referred to by AudioEye (2022b, 14) in this instance.

"The misleading discourse and false promises made by some vendors cause a reductive attitude towards technology in general, falsely generalizing all solutions as simply 'overlays'."

Furthermore, AudioEye (2022b, 13) states that businesses of any size are vulnerable to accessibility vendors who sell inadequate solutions and create self-serving, false narratives around compliance. Evaluating different accessibility providers is further complicated by the lack of transparency, as claimed by AudioEye (2022b, 15).

AudioEye (2022b, 18) asserts that a "sustainable and scalable solution, which businesses of all sizes can afford to implement on an ongoing basis", is required. At the same time, AudioEye (2022b, 13) employs the expressions "cheap automation-only solutions" and "expensive manual audits" when describing its competitors. The inclusion of the subjective terms "cheap" and "expensive" introduces emotional bias and implies that neither option is suitable. This subjective view is a marketing claim and cannot be taken as fact.

As per AudioEye (2022b, 13), both options fail to provide maintained accessibility and compliance and instead have "a damaging effect". According to AudioEye (2022b, 18), neither approach caters to the existing challenges in digital accessibility. They continue stating that automation is needed to keep up with the dynamic nature of websites in today's fast-paced digital transformation. However, while affordable, these solutions alone cannot address "the many complex accessibility issues" according to AudioEye (2022b, 24). Additionally, AudioEye (2022b, 26) claims that relying solely on manual testing and remediation poses a risk of leaving accessibility issues uncovered and unaddressed between audits. Moreover, they maintain that it is impossible to keep pace with the constant flow of new content and website modifications through a manual process alone. Further, they argue that for most website owners, employing manual accessibility consultants is not financially viable.

What distinguishes (AudioEye 2022b, 25) from their competitors, according to their statement, is that they have a hybrid approach to accessibility and depend on human expertise to guarantee that their solution always provides the "highest level of accessibility" to individuals with diverse impairments. They claim this combination is the most effective method of "breaking down barriers to an accessible, usable and inclusive internet for people with impairments".

"Unlike automation-only providers, we recognize that there's still a large number of issues that technology cannot solve today. We also understand that without human contribution, automation will not improve on its own and will not serve its purpose."

This means that the AudioEye accessibility overlay is an automated solution, but the company, according to its own statement, does not claim to be "reliable or complete" like its competitors. However, it is unclear from their description whether their proposed solution refers solely to their accessibility overlay or includes other components. Furthermore, it is questionable how a combination of their negatively described "cheap automated solutions" and "expensive manual audits" creates a hybrid solution.

4.3.3 Viewpoints in Favour of Accessibility Overlays from Expert Interviews

The author interviewed Lionel Wolberger who is the Chief Operating Officer at UserWay. He is also a serial entrepreneur, holds a Ph.D. in anthropological approaches, and is a member of the W3C Accessible Platforms Architectures Working Group.

Wolberger affirms that UserWay does not distribute an accessibility overlay, but rather an "AI-powered widget" that enables functionality using JS. Nevertheless, according to the definition used in this thesis, it is an accessibility overlay.

Wolberger argues that adding accessibility through a JS layer is a classic and effective approach.

"It is a good and proper step forward that accessibility providers should be able to add value through JavaScript."

Wolberger says that UserWay communicates to its customers that it wants them to do an audit and remediate at the core. In addition, they advise to always keep the accessibility overlay on a website because "it keeps telling you things are OK".

According to Wolberger, the best way to get on a path to accessibility is to add UserWays technology. He claims that there is consensus in the field across the board that automation is necessary to achieve accessibility on a widespread scale.

"I think we all agree on that. There is consensus across the board that automation has a good home in the continuous integration pipeline. When the developers are working, you'll find almost everybody using automation there."

The author was unable to verify this consensus within the scope of this thesis.

Wolberger is optimistic about the advancements AI can make in improving accessibility. He stresses the importance of keeping humans involved in the process, which he refers to as the "human in the loop" approach - a user-centred solution to AI.

"We don't hand over to AI. 'Go make decisions for us'. We have AI accelerate ourselves and always have checkpoints where the human gets involved at the right moment."

Wolberger continues by saying that some people complain about AI regarding alternative texts for images.

"Image alts require context, so anyone complaining about AI will right away say 'ohh AI labeling of images is junk'. But not in UserWay because UserWay is 'human in the loop'."

Human in the loop in this case refers to UserWay suggesting automated alternative texts to its clients, who can then decide whether any revisions are required to suit the context. Wolberger addresses critics who state that some customers buy the accessibility overlay but are not the humans in the loop by stating that for UserWay, it is about progress, not perfection.

"There is a human in the loop, but many, many of our customers, they're just too busy making their pizza. They're too busy selling their hats. And so, then the question becomes, is the entire community better if those people put some automation or not? And then the question becomes perfection or progress? [...] You know at scale you're talking about a lot of experiences."

Wolberger thinks more access to features is "overall a good thing". He believes that everyone wants people to be able to find what they need. Wolberger claims that research indicates specific individuals cannot configure their operating system settings and that when a system undergoes an update, "these particularly vulnerable populations struggle even more to keep up with the changes". The author was unable to verify this claim within the scope of this thesis.

"Same with the browsers. Things move around and things change. So, we want, you want people to know you can make all your fonts bigger in the operating system, you can do it in your browser but adding one more place to do it is not a bad thing. It's a good thing. It ends up increasing awareness. And some people will use it here and some people will use it there."

Wolberger asserts that the intended users for UserWay are individuals who are blind, motor-impaired, or deaf. He also claims that UserWay has access to "highly knowledgeable" users who are blind, motor-impaired, or use screen readers, in order to evaluate the effectiveness of their services for people with impairments.

Wolberger believes that the increasing popularity of accessibility overlays among website owners is due to the perception of actively aiding individuals with impairments by installing an accessibility overlay. He continues stating that some accessibility advocates said one can not feel good about adding an accessibility overlay because errors should be eliminated at the source.

Wolberger disagrees with this view. He maintains that accessibility overlays can help everyone.

"To keep it simple, number one: Putting something in the front window that says I am involved with accessibility, and I care is a very powerful signal. I do not agree that it is a fake signal, or that it is a way out. I think it's a well-meaning signal to say I'm in on this and it's a way of committing your organization."

Wolberger further claims that when a website uses an accessibility overlay, users also attribute "very positive sentiments" to the site.

"You're helping inclusion and diversity and they react positively to that. You don't get that if you paid for an audit. The visitor that comes to the site wants to see: Are you doing something or not?"

According to Wolberger, a group of individuals have reported negative experiences and trauma associated with accessibility overlays due to a particular competitor's misconduct. He continues to criticise competitors that detect if one keeps pressing the tab key to activate their accessibility overlay automatically.

"And some blind advocates are furious about this. And I'm a privacy guy. I agree with them. It's like you're tracking them. It's not nice."

Wolberger states that UserWay does not incorporate the concept of "turning on" since "it just works". He proceeds to argue that individuals disfavour the act of turning things on.

Wolberger states that it is necessary to conduct research to verify the extent of individuals' benefit from accessibility overlays. Additionally, he contends that advocates for accessibility demand evidence that such overlays are helping and not causing harm. Wolberger predicts that if a screen reader user were to compare their experience on a website with and without the accessibility overlay of UserWay, it would be worse without UserWay. However, Wolberger explains that the sole reason why UserWay has not carried out any research so far is the high cost and time-consuming nature of such an endeavour.

"The bottom line is, people don't understand what these technologies do. Some people are upset and frightened even. And they're just, like, prove it. And the only reason we haven't proved it yet is because it's just expensive."

Wolberger concludes that "new technologies are always bumpy".

"I think we're better served not by people saying shut it down, get rid of it, get rid of it. But really engaging and saying this is the part I don't like. This is the part you need to improve. Because we listen and we improve very quickly."

4.4 Arguments Surrounding Accessibility Overlays Without Automated Remediation

The term accessibility overlay, as used in this thesis, refers to the widget, toolbar, or plugin, including variants with and without automated remediation. However, some individuals and companies advocate for accessibility overlays without automation, whilst others argue against both.

Schmidt acknowledges that EyeAble Assist is an accessibility overlay but does not consider it on the same level as UserWay, DigiAccess⁵⁵, and "all the others" as they do not "promise accessibility at the touch of a button". He believes that their accessibility overlay EyeAble Assist is at the end of the accessibility process.

According to Schmidt, only their two other products, a WCAG checking software and a WCAG dashboard, can be used to improve accessibility. Their accessibility overlay, as Schmidt mentioned, is only for approachability after the underlying website is accessible. Schmidt claims that the company offers its accessibility overlay for help beyond the implemented WCAG requirements. Schmidt continues by stating that some people have no access technology and, for them, an accessibility overlay without automation is a good option. According to Schmidt, the target audience of EyeAble does not consist of screen reader users due to the presence of system-wide read-aloud functions. Instead, their target audience comprises, amongst others, older individuals, those who do not receive state aid, and people with a migration background who prefer to have texts read to them for multitasking activities.

Buchner-Sabathy asserts that it can be helpful to offer additional accessibility features at the touch of a button, but not in the form of an accessibility overlay. Instead, such features could be implemented in the menu bar. According to Buchner-Sabathy, although it may seem redundant, it can be helpful for people who are not tech-savvy and unfamiliar with the browser's means or tools. Bahram, on the other hand, argues that such buttons, as well as accessibility overlays, are redundant since the issues they address can be resolved within the browser or system settings.

"[These settings] can either be enabled as a one-time configuration change for somebody who is not as technically fluent. For a more technically fluent audience, they can be toggled on and off at will and even customized so as to not necessitate the need for accessibility overlays."

Bahram believes that if a person does not know of such operating system settings or browser settings, one should teach them, instead of teaching them how to use an accessibility overlay. On the contrary, Schmidt contends that removing accessibility overlays altogether is not inclusive.

"When you get a call several times a day from a nice grandmother thanking you for your work, and she has no idea that such [browser or system] settings exist, has no idea about WCAG, and does not benefit from them at all; when testers keep giving us feedback like 'hey, ZoomText⁵⁶ is already good, but ZoomText and EyeAble is even better', then we are not willing to stop making our tool available to these people just because there is a community that is understandably angry at overlay technologies."

In opposition, Bahram argues that claims such as the one of Schmidt should be taken "with a huge grain of salt". He is not asserting that those claims are untrue, but he argues that it is a hyperbolic emotional example designed to tug at the heartstrings instead of illustrating an actual event. Bahram believes that a grandmother could have the same emotional reaction when her grandchildren show her how to turn on high contrast mode on her computer or if somebody hands her an iPad and teaches her how to use it. He further states that there are multiple ways to create this experience that do not involve an accessibility overlay. Bahram claims that accessibility overlay companies are capitalising on the grandmother's reaction to having access or perceived access to something.

"I think it's deeply concerning that overlay companies appear to be really using what is sometimes referred to as inspiration porn⁵⁷ or the capitalization of the difficulties of disabled audiences in order to motivate and fulfill a for-profit objective. "

Bahram continues to state that accessibility overlays employ an excellent marketing strategy, but he questions the data supporting the efficacy of their solution.

"I think that marketing should not be used as a substitute for evidence. [...] I live in an evidence-based world in which I care about reality, not performance."

4.5 Arguments Surrounding Alternatives for Accessibility Overlays

Bahram does not believe that there is any way to improve accessibility overlays. In general, Bahram maintains that the solution to the accessibility overlay debate is not to use them. Instead, the underlying website should be made accessible. In addition, Bahram suggests adjustments should be made at the browser or system level, under the user's control, and not within an accessibility overlay. Furthermore, Bahram states that a regulatory perspective should be considered. He asserts that, within legal and compliance frameworks, it is crucial to establish that an accessibility overlay does not meet the obligations under any EU, American, or Canadian regulation to meet the accessibility and inclusive design requirements set out in the standards. Buchner-Sabathy also questions legal security promises, as well as data protection (how data gets collected, processed and by whom).

"[An accessibility overlay] doesn't create legal certainty, and it goes against universal design⁵⁸. [...] It's not about putting ramps over steps, it's about providing step-free paths."

However, Bahram thinks that automated solutions like AI and machine learning can undoubtedly be used for accessibility in the future, but under the premise that the effort comes from within the community.

"It needs to be disabled-led. It needs to be supported by evidence. [...] It needs to be based on sound principles, not on 'how much money can we grab out of weaponizing and capitalizing on the fears of companies around litigation' [...]."

Bahram states that within the disability community, there is an expression of "nothing about us, without us", meaning you need to involve the community in decisions.

"There is a long history within the disability space of those who either have good or bad intentions that are not part of the communities they claim to be serving, that do things. And this is a deeply problematic practice."

To describe this practice of not involving the disability community, the term disability dongle was established by Liz Jackson in 2019. It defines "an outcome in which designs or technologies 'for' disabled people garner mainstream attention and accolades despite valid concerns disabled people have about them" (Jackson, Haagaard, and Williams 2022).

Bahram suggests following an inclusive design methodology that promotes accessible document practices and prioritises creating an accessible website to help companies avoid releasing inaccessible content. In addition, Bahram argues that accessibility overlay companies are nowhere near as serious as proper accessibility companies with subject matter experts and employed disabled people. Furthermore, Bahram senses that the debate surrounding accessibility overlays should not solely focus on functionality or exist only in a vacuum. He reasons that it is crucial to consider implications and that arguments should be holistic and consider the positive and negative impacts.

5 Results and Evaluation

An evaluation study was used to address the research questions, whereby three accessibility overlays and a website were initially selected for the study. Subsequently, a quantitative technical evaluation was carried out to determine their current accessibility status based on the WCAG 2.1 success criteria. Following this, the usability and user experience were assessed through interviews, observations and questionnaires. The results of each evaluation stage are detailed in the following sections, beginning with the findings of the preliminary study.

5.1 Preliminary Study Summary

This section provides a summary of the data gathered from the preliminary research illustrated in the previous chapter.

5.1.1 Legal Situation Concerning Accessibility Overlays

Chapter 4.1 outlined the increasing legal challenges related to the use of accessibility overlays on websites. As overlays gain popularity among companies seeking to enhance website accessibility, individuals with impairments have filed lawsuits, citing failures to comply with established accessibility guidelines. Additionally, some accessibility overlay companies have pursued legal action against critics, including accessibility consultants and advocates, raising concerns about the potential suppression of critical discourse in the accessibility community.

5.1.2 Viewpoints Opposed to Accessibility Overlays

Individuals, organisations, and experts have widely criticised accessibility overlays. Entities such as the EDF (2023), the IAAP, the NFB (2021), the WACA (2023), the DG COMM of the European Commission (n.d.), the BFIT-Bund (2023) and 856 signatories of the "Overlay Factsheet" (Groves 2021) have issued statements and actions reflecting deep concerns over the effectiveness of accessibility overlays. They argue that overlays do not fully comply with (European) accessibility legislation and may even hinder users with impairments from navigating a website. Additionally, critics voiced that some companies make misleading claims about achieving WCAG compliance through automated solutions alone. Moreover, it is stated that overlays cannot replace addressing accessibility issues at their source and may even create new barriers.

The expert interviews presented insights from accessibility advocates and professionals who critique the efficacy and approach of accessibility overlays. Bahram, Buchner-Sabathy and Schmidt expressed concerns about the reliability, unintended consequences, and false expectations of accessibility overlays. Buchner-Sabathy suggests that accessibility overlays may not always fulfil their promises and, according to Bahram, may even introduce new issues such as unexpected behaviour. Schmidt and Buchner-Sabathy argue that accessibility overlays may not necessarily make websites more accessible according to WCAG standards. In addition, Bahram raised ethical concerns about offering simplistic solutions to complex accessibility problems and suggested addressing the root issues instead of using accessibility overlays. Schmidt also emphasised the importance of making accessibility overlays accessible.

The three experts collectively emphasised the necessity of human intervention for proper website accessibility and criticised companies using accessibility overlays as a substitute for thorough, inclusive web development practices. These perspectives highlight a consensus for more genuine and manual interventions in web accessibility efforts rather than relying on automated overlay solutions.

5.1.3 Viewpoints In Favour of Accessibility Overlays

There is a mixed yet growing interest in accessibility overlays, as reflected in online commentary, expert interviews and a white paper.

During the expert interview, Wolberger advocated for the effectiveness of JavaScript-based accessibility solutions and emphasised a human-centred approach to AI in enhancing web accessibility. Wolberger discussed criticisms of AI, specifically in generating alternative texts for images, by highlighting UserWay’s "human in the loop" methodology, which involves user oversight on automated suggestions. Despite challenges and criticisms, including negative feedback from some users and the high costs of conducting empirical research to validate the efficacy of overlays, Wolberger remains optimistic about the role of AI in accessibility, emphasising the importance of progress over perfection and the positive signals sent by their use, suggesting that they contribute to inclusion and diversity. Wolberger acknowledges the potential benefits of accessibility overlays and the controversies surrounding their use.

AudioEye's white paper from April 2022 examined the debate surrounding accessibility overlays. It criticised the industry for polarised views and misleading claims, mainly targeting competitors for advertising overlays as complete solutions. The white paper highlighted the challenges businesses face in navigating the accessibility market, worsened by a lack of transparency and the spread of false narratives by some companies. AudioEye (2022b) criticised both low-cost automated solutions and high-priced manual audits offered by competitors, arguing that neither approach adequately maintains website accessibility or compliance. Instead, AudioEye (2022b) advocated for a hybrid model combining automation and human expertise. The white paper presented AudioEye’s position as striving for a balance between technological solutions and necessary human oversight to achieve a more accessible, usable, and inclusive web. However, questions remain regarding the specifics of their hybrid solution and its effectiveness compared to other market offerings.

Market trends showed an increase in the use of accessibility apps, with a 1.6% usage rate on desktop websites, according to a report of the Web Almanac (2022, chap. 11). This report indicates a preference for solutions such as accessibility overlays, particularly on low-traffic websites. The financial success of accessibility overlay companies, demonstrated by their significant funding and revenues (accessiBe 2022; AudioEye 2022a; UserWay 2022), highlights the commercial viability and growing popularity among businesses and investors. This data suggests a growing market despite the controversies surrounding their effectiveness and implementation.

5.1.4 Viewpoints Surrounding Accessibility Overlays Without Automated Remediation

While discussing arguments surrounding accessibility overlays without automated remediation, the interviewed experts expressed divergent views on their utility and ethical implications. Schmidt, representing EyeAble, distinguished their product from others by emphasising that it is an additional approachability tool once a website meets accessibility standards. In contrast, Buchner-Sabathy and Bahram argued against the need for accessibility overlays, proposing that browser or system settings could replace many of their functions. They advocated for educating users on these features as a more inclusive solution. Bahram, in particular, raised ethical concerns and pointed out that overlay companies exploit emotional narratives for profit without substantiating the effectiveness of their products. This debate emphasised the complex dynamics between improving web accessibility, user education and the potential for commercial use of accessibility technology.

5.1.5 Viewpoints Surrounding Alternatives for Accessibility Overlays

Discussing alternatives to accessibility overlays, the interviewed experts Bahram and Buchner-Sabathy emphasised scepticism towards the efficacy and ethical standing of accessibility overlays in meeting legal and universal design standards. They advocated for direct improvements to website accessibility and adjustments at the browser or system level, underlining that overlays do not fulfil regulatory obligations in the EU, America, or Canada. Buchner-Sabathy criticised overlays for lacking legal certainty and violating universal design principles, suggesting a shift towards creating environments accessible to everyone without additional modifications. Bahram highlighted the potential of AI and machine learning in enhancing accessibility, stressing the importance of community-led initiatives and the involvement of the disability community in decision-making processes. Bahram called for an inclusive design approach, emphasising the necessity of consulting with subject matter experts and incorporating feedback from the disabled community to ensure genuinely accessible solutions.

5.2 Technical Preparation Results

The technical preparation phase was necessary before conducting the evaluation study. It involved selecting the website and accessibility overlays for the evaluation. Additionally, technical arrangements were made in preparation for both the technical evaluation and the user study. The following paragraphs describe the results obtained from the technical preparation phase.

5.2.1 Accessibility Overlays Selection

Various accessibility overlay companies exist on the market. As described in section 3.2.1, a Google search was conducted to find them. Typing the search string "accessibility overlay" on Google did not list any companies but rather blog posts speaking against overlays. "Accessibility overlay company" made UserWay show up within the search results. When trying "web accessibility solution", again UserWay, but also accessiBe, accessiWay⁵⁹, and EqualWeb were found. "Digital accessibility solution" showed AudioEye in the search results. "Accessibility assistance software" brought up EyeAble⁶⁰. The author stopped the search after discovering six accessibility overlay companies, as this number is a reasonable basis for analysis. All of these businesses were listed on the first page of search results at the time of searching.

Table 2 compares each overlay's pricing, accessibility standard or law compliance and installation details. In addition, the following paragraphs give some more information on the capabilities of each company (in randomised order).

• EqualWeb

  Founded in 2014 in Israel, EqualWeb claims on its website to be the world's number one web accessibility solution. Its "Auto AI Accessibility Widget", an accessibility overlay, assures compliance with multiple standards including ADA and WCAG 2.1 AA (PPM of 39$). By pairing the accessibility overlay with their in-house expert remediation solution one can achieve "full compliance", adding compliance with AODA and EN 301549.

• EyeAble

  Founded in 2020 in Germany, EyeAble provides three software services to enhance web accessibility: "Audit", "Report", and "Assist". The EyeAble Assist accessibility overlay cannot be purchased as a standalone product. As per their website, their accessibility overlay meets some regulations but not all WCAG requirements. One can use their auditing software to identify and correct errors with the help of instructions.

• AudioEye

  Founded in 2005 in the USA, AudioEye claims to use a hybrid approach combining automated and manual remediations to eliminate up to 95% of accessibility errors on their customers' websites (PPM of 49$). Nonetheless, it is important to note that their human expertise is not to be equated with manual accessibility website testing. AudioEye utilises human expertise to improve the automation of their accessibility overlay. In addition, AudioEye does not guarantee compliance with legal requirements but asserts that it gives its customers a four times higher level of protection against legal claims than its competitors. Thus, it was categorised as "Partly" in Table 2.

• UserWay

  Founded in 2016 in the USA, UserWay claims to be the web accessibility market leader (Figure 8), with over 1 million websites worldwide trusting them. They sell an accessibility overlay called "accessibility Widget", with its newest version (V4.0) available since April 2022. UserWay offers a free "Lite" version of their overlay and a paid "Pro" package (PPM of 49$) with "over 100 AI-powered functions that ensure WCAG & ADA compliance". Besides their accessibility overlay, UserWay offers accessibility-related services like an accessibility scanner, accessibility audits, an Microsoft Office integration and a managed accessibility package that includes "manual reviews, testing and updates by experts".

  

• accessiBe

  Founded in 2018 in Israel, accessiBe also claims to be the web accessibility market leader (Figure 8), with 247,778 websites trusting them (as of March 2024). Their accessibility overlay "accessWidget" promises compliance with ADA, AODA, EAA, WCAG and more within the "Standard" package (PPM of 49$). The package also includes AI-powered features, such as daily compliance monitoring, screen reader and keyboard navigation and monthly accessibility compliance audits. More expensive packages offer premium add-ons like a monthly remediation report with full documentation and an automatic Google Analytics⁶¹ integration. Besides their accessibility overlay, accessiBe also offers "accessFlow", an "all-in-one platform to test, monitor, and remediate the accessibility of websites and web applications", according to their website.

• AccessiWay

  Founded in 2021 in Italy, AccessiWay aims to enhance a website's accessibility through its "accessWidget". The accessibility overlay they offer has the same name as accessiBe's because they market and distribute the accessiBe software (AccessiWay 2023). The primary distinguishing factor of AccessiWay from accessiBe is that they are slightly more affordable and do not promise conformance to legal regulations with their accessibility overlay alone. The company states that accessWidget cannot ensure full compliance with WCAG 2.1 for every site element since incompatibility with some elements is always possible. They suggest scheduling a consultancy session after installing accessWidget to achieve compliance.

The accessibility overlays selected for the study were UserWay, accessiBe and AudioEye. These three companies claim to have the most users and being market leaders (Figure 8). In addition, the Web Almanac (2022, chap. 11) identified their popularity in their survey. The rising number of websites utilising accessibility overlays implies that a more thorough examination of them and their impact on individuals with impairments is needed. Furthermore, the substantial funding and significant revenue of the underlying companies mentioned in section 4.3 underscore the need to test their software and assess their marketing assertions.

The previously mentioned custom domain for the copy of each website in combination with each selected accessibility overlay was as follows:

• fusilli.dnikub.dev - no accessibility overlay

• farfalle.dnikub.dev - UserWay

• rigatoni.dnikub.dev - accessiBe

• spaghetti.dnikub.dev - AudioEye

AccessiWay was not selected as they utilise the accessiBe software. Their terms and conditions state that

"AccessiWay markets and distributes licenses for 'Saas' (Software as a Service) under the specific authorization of AccessiBe Ltd, an Israeli company registered under no. 51-585530-2, holding intellectual property rights to the software, AccessiWay Systems, and owner of the servers through which the licensed 'Saas' services are provided."

As accessiBe was already selected for testing, it was redundant to test AccessiWay. EyeAble was excluded because they only offer a year minimum contract, and they do not sell their accessibility overlay alone. EqualWeb was excluded from being tested due to the scope restrictions of this thesis. It was only possible to test three overlays within the given time frame and with the available financial capabilities.

All three selected overlay companies were contacted to ensure their terms of service allow the usage within a research project. In addition, a request for an extended trial period for the duration of the user study was made. This request was made due to the limited financial capabilities of the thesis. UserWay agreed and enabled the author to use their service for free for a month. accessiBe declined. As AudioEye was chosen last, there was not enough time to agree on a more precise plan for an extended testing period before starting the user study. Therefore, the overlays of AudioEye and accessiBe were purchased for a month.

For this study, only the basic version of each accessibility overlay was selected. No additional functionalities were purchased due to two reasons, the first being the limited financial capabilities. Secondly, the author intended to examine how far one can get with the overlay's functionalities and automated fixes by only purchasing the most inexpensive option. Furthermore, the author adopted the viewpoint of the website owner from an economic standpoint. As a result, any modifications made in the accessibility overlay admin dashboard were restricted to evident changes. This restriction meant that only settings present during the set-up process were changed or when the author was later notified directly by the accessibility overlay via email. This stance was adopted to narrow the scope of the thesis further and to evaluate the efficacy of automated fixes provided by each overlay, given their marketing claims.

5.2.2 Website Selection

The author selected the website of an Austrian music festival, shown in Figure 9. The festival's logo and name are blacked out throughout all images within this thesis as they are irrelevant to the study's outcome. This website was selected because it fulfils all of the selection criteria. It is available in both English and German and has a wide range of interactive content types such as audio, video, search, sound, images and filters. It does have accessibility problems at present as well.

5.2.3 Adding the Accessibility Overlay

The whole process of implementing an overlay on each copy of the website consists of the following:

1. Creating an account on the website of the accessibility overlay company.

2. Starting the trial period.

3. Selecting the position and styling of the accessibility overlay on the website.

4. Adding one line of JS code to the website.

5. Deploying the website.

6. Verifying that the JS code is added, meaning the accessibility overlay is implemented.

The implementation of the three selected accessibility overlays required a two-hour set-up process. Additionally, depending on the company, the author had to wait up to 48 hours for the automated scans and fixes to be completed. The set-up process for each accessibility overlay is detailed in the following paragraphs. This description does not directly answer a research question. However, it contributes to the general understanding of how accessibility overlays function, as it provides insight into the effort required by a website owner to implement them. Therefore, the author decided it is essential to include it in this thesis.

5.2.3.1 General Design Decisions

As mentioned earlier, an accessibility overlay appears on the website as a floating action button. All three companies allow customisations regarding the design. It is possible to customise where the button should be positioned on the website, its colour, and what icon within the button should be used.

The same design changes are made to each accessibility overlay for the study. The colour and icon are left as-is. As Figure 10 illustrates, all buttons are blue, round and contain a white icon of a stick-figure person.

The positioning of accessibility overlay buttons on the website varies by default. The author repositioned it to the bottom left corner due to lack of space. Figure 9 illustrates that a cookie banner takes the bottom right corner, the top right corner is occupied by the navigation and the top left by the logo. The bottom left corner is the sole available spot where the accessibility overlay button does not sit above another element. As all three accessibility overlay companies positioned their button in the bottom right corner of their own website at the time of the study, it seemed logical to choose the bottom part of the page as well.

5.2.3.2 AudioEye

After adding the JS code and deploying the website, the next step is to verify the installation in the AudioEye admin dashboard. The verification, however, did not work. Even after clicking the verification button and visually checking the presence of the overlay button on the live website, the AudioEye admin dashboard still reported: "AudioEye is not installed". The issue was eventually resolved after contacting support. Beforehand, the author's interactions with the admin dashboard were limited due to several locked features.

Throughout the paid month, the support had to be contacted three times. The first situation was the one just mentioned immediately after signing up. The second was directly after the end of the trial period. Although the overlay was successfully purchased, it did not appear on the website. The third time support was contacted was towards the end of the trial when the overlay suddenly disappeared, and the verification was reversed. This reversal happened even though the paid subscription and the JS code implemented were still valid. AudioEye explained that this was a technical problem they were currently experiencing due to a recent update but were actively working to resolve.

AudioEye provides a "Homepage Accessibility Score" upon registration before any JS is injected into the website. The initial score without any accessibility overlay was 76. However, after the overlay was installed, the score increased to 94. The score was fluctuating. This alteration was anticipated since AudioEye informs the user that it requires approximately a week to analyse the website, and therefore, the rating may still vary. At the end of the study, the score with the overlay was 87, while the score without the overlay was suddenly 79 (Figure 11). AudioEye declared it had resolved 67% of the errors it detected. All of these were classified as low severity. 4% of the issues "need to be fixed at source" and 29% "need custom fixes".

5.2.3.3 accessiBe

Setting up accessiBe turned out to be more complicated. Firstly, the sign-up process was unsuccessful due to the inability to create a user account using an Austrian phone number. The author required assistance from support to resolve the issue and complete the sign-up process. Secondly, implementing the accessibility overlay on the website required greater effort. Figure 12 demonstrates that modifying the overlay's position, appearance, or language necessitates replacing the entire JS code. Consequently, the website must be redeployed, extending the set-up duration significantly.

The admin interface itself provides a list of audits with the note "verdict: compliant". The audit examines compliance with WCAG 2.1 level AA success criteria and is downloadable. It lists a requirement along with a code example and states whether it succeeded or failed.

5.2.3.4 UserWay

After creating a user account successfully, UserWay presents an "Accessibility Score" on its admin dashboard (Figure 13).

(a) The Accessibility Score before adding the JavaScript code.

(b) The Accessibility Score and detected issues after adding the JavaScript code

(c) The Accessibility Score after remediating the issues via button click.

Prior to incorporating the JS code into the website, the score was 54% (Subfigure 13a). Following its implementation, the score increased to 96%. Furthermore, identified accessibility errors are listed, four of which are at the WCAG conformance level AAA, two at AA, and 37 at A (Subfigure 13b). The problems can be resolved with the click of a button, and subsequently, the notification "Congratulations! UserWay's AI-Powered solution has successfully remediated all accessibility issues." appears. The Accessibility Score now stands at 100%, as shown in Subfigure 13c. During the paid month, the score kept changing. On some days, it was at 96%, on others at 98%. UserWay's AI seems to audit the website constantly. Later during the month of testing, it detected four inaccessible PDFs on the website. These might be related to the decreased Accessibility Score. Remediation of the PDFs is possible with extra payment but was not done for the study.

After a while, it became possible in the admin dashboard to edit the auto-generated alternative texts for all images or mark them as decorative. Figure 14 illustrates that certain alternative texts are more precise than others. One image, for instance, portrays a drawing of an ear encompassed by red circles, while its textual description states "a close-up of a cat's face". Furthermore, all generated alternative texts were in English, even though the website is in German. The author did not modify any of the generated texts as this functionality was not available during the set-up process, and the displayed Accessibility Score was already at 96%. Despite being aware of the potential duration of up to 48 hours for the website scan, the author proceeded with the approach of a business owner, who possesses limited knowledge of accessibility concerns and would only make adjustments following explicit directions.

The paid month ended with an Accessibility Score of 99% and, according to UserWay, a "low lawsuit risk". On their admin dashboard, UserWay displays "Websites with a score of 75% or below are at risk of lawsuits.". Although the PDFs that were detected remained inaccessible and the generated alternative texts were never modified, the automatic fixes executed by UserWay resulted in them promoting an accessibility score of almost 100%. In section 5.3, the author lists the results of the manual accessibility assessment for this website, conducted by the defined accessibility testing methodology from section 3.3.

5.3 Technical Evaluation Results

All tests were conducted to assess the accessibility of the specified website and the website combined with each accessibility overlay. During testing, the provided button for activating the accessibility overlay was not used. The technical evaluation only analysed automatically applied changes by each overlay, requiring no action from the user. This decision is founded on the overlays' multiple features to accommodate individual impairments, making it impractical to enable them all, as some features are incompatible with each other.

WCAG 2.1 was used as the reference for accessibility compliance. This study focussed on levels A and AA of the WCAG. The total number of success criteria for both levels is 50, but 9 success criteria were excluded as they were inapplicable to the website (success criteria 1.2.4, 1.4.2, 2.1.4, 2.2.1, 2.3.1, 2.5.1, 2.5.2, 2.5.4 and 3.3.4). As a result, there are 41 remaining criteria.

Table 3 lists the number of failed success criteria for each website and how they are divided into levels A and AA. accessiBe and UserWay have the fewest failures, while AudioEye has the same number as the website without an accessibility overlay. However, whilst UserWay removes two errors, accessiBe eradicates four but introduces two new ones.

While two overlays remediated some accessibility errors, neither achieved 100% compliance with WCAG 2.1. The subsequent sections describe the automated and manual testing process and detail some of the accessibility errors found.

5.3.1 Automated Accessibility Testing Results

Various tools were employed to conduct automated accessibility tests on each website's homepage. The following sections present the results, ordered by the tools used.

5.3.1.1 Detected WAVE Errors and Alerts

Without any accessibility overlay present, the tool WAVE detects 13 errors and 56 alerts, with 11 of the alerts being false negatives identified during the manual accessibility test. Of the 30 detected features, about 29 are false positives. According to Table 4, the website using accessiBe had fewer WAVE errors (1), while the website using AudioEye had more (14). UserWay's website had an identical error count.

However, UserWay's and accessiBe's accessibility overlays automatically activated specific features whilst browsing with access devices or a keyboard. Subsequently, after the activation of UserWay, all errors were eliminated. For accessiBe, the activation of its overlay was triggered by running the WAVE check. Therefore, the error count presented in Table Table 4 is with the active accessibility overlay. The website utilising accessiBe displays a lower WAVE alert count than the website lacking an accessibility overlay.

5.3.1.2 Detected Lighthouse Score and Errors

The Lighthouse score for the website without any accessibility overlay is 98 out of 100 (Table 4). Two mistakes, one of which occurred three times, were detected and reduced points: links do not have discernible text, and <iframe> elements do not have a title. Only UserWay achieved a perfect score of 100. accessiBe had the lowest score of all, at 88. This low score is caused by links lacking a clear name, [aria-*] attributes not matching their roles, and list items not being placed within their parent.

5.3.1.3 Detected Axe Errors

Axe identifies the same error for link text as Lighthouse, but seven times (Table 4). The number of identified errors remains the same for UserWay and AudioEye. However, on the website using accessiBe, the score almost doubles to 12 detected errors.

5.3.2 Manual Accessibility Testing Results

During the manual accessibility testing, additional technical difficulties arose. AudioEye and accessiBe were both blocked by the installed AdBlocker in the author's browser. With the active AdBlocker, neither the overlays nor their corresponding activation buttons are loaded. Furthermore, there were additional difficulties when browsing the website with accessiBe implemented. Occasionally, the entire overlay failed to load or previously made settings reset upon navigating to a different page on the same website. When accessing the website where UserWay is enabled using only a keyboard or a screen reader, the accessibility overlay automatically activates additional features. These features are available to sighted users only when manually enabling them by clicking the overlay button.

As previously stated, accessiBe is the sole accessibility overlay that failed two success criteria that would have passed without the overlay being present. This issue can be attributed to the extensive usage of ARIA by accessiBe. To elaborate, success criteria 1.3.1 and 2.4.1 do not pass on the website because the <main> element's assigned aria-role of "presentation" is not allowed, and aria-role="navigation" has been set on the <ul> list element within the <nav> element. Moreover, each <li> element within the <ul> has been designated as a root menu item with the assistance of ARIA.

In general, manual accessibility testing has identified numerous WCAG errors, resulting in the failure of success criteria. If not specified otherwise, the errors occur across all four websites.

1. Some links lack an accessible name as they are only represented by an image without any alternative text description.

   1. Fails: Success criterion 1.1.1

2. Some alternative texts for images only provide copyright information or the name of the person pictured without any additional details.

   1. Fails: Success criterion 1.1.1

   2. UserWay: All images, except those in the slider, now have alt text. However, they frequently contain inaccuracies or lack sufficient description, particularly for logos. Therefore, this success criterion still fails.

3. Sound files do not have transcripts.

   1. Fails: Success criterion 1.2.1

4. Some embedded YouTube videos lack captions.

   1. Fails: Success criterion 1.2.2

5. An audio description for videos is not available (only captions of spoken content).

   1. Fails: Success criteria 1.2.3 and 1.2.5

6. Form labels are missing in multiple locations, including the newsletter subscription and search field.

   1. Fails: Success criteria 1.3.1, 1.3.5, 2.4.6 and 3.3.2

   2. UserWay: The existing form label is still empty, but an accessible name is provided through an additional <span> element, resulting in the newsletter subscription and search input fields now having accessible names. Success criterion 3.3.2 now passes.

   3. accessiBe: The existing form label for the newsletter subscription and search is still empty, but the <input> element now has an ARIA label, ensuring compliance with success criterion 3.3.2.

7. Multiple content elements should be coded using HTML heading elements from <h1> to <h6>.

   1. Fails: Success criteria 1.3.1 and 2.4.6

   2. accessiBe: The accessibility overlay alters some potential headings to actual headings using ARIA. However, some elements that are now headings should not be (Figure 15), which is why success criterion 2.4.6 still fails.

      

8. Emphasis has been incorrectly utilised in some cases, whereby elements have only been styled using CSS to be bold or italic instead of being coded using HTML (<strong>, <b> or <i>).

   1. Fails: Success criterion 1.3.1

   2. accessiBe: Some text within paragraphs, previously marked as bold with CSS, has been turned into a heading. As can be seen in Figure 16, this heading interrupts the sentence flow within a paragraph.

      

9. After clicking the main menu button, access technology users cannot view the menu contents directly. Instead, they must navigate to the bottom of the page as the content appears after all website content.

   1. Fails: Success criteria 1.3.2 and 2.4.3

   2. AudioEye: The accessibility overlay button is the final element on the page. While screen readers receive an announcement of its shortcut, keyboard users must manually navigate through previous elements to access it for adaptations.

   3. accessiBe: When navigating with the keyboard only, the focus becomes trapped when the menu is opened or the search function is used. This behaviour makes success criterion 1.3.2 pass. However, navigating with a screen reader sometimes results in unexpected focus shifts.

10. The buttons used to display sub-menu options lack an accessible name. Screen reader users only hear the word "button".

    1. Fails: Success criterion 1.3.3

    2. UserWay: The buttons that open the sub-menu now have the attribute aria-label with the string "button" added. Although this may seem like an improvement, it fails to provide more context of what the button is doing.

    3. accessiBe: Screen reader and keyboard users can now not access the sub-menu button. This inability leads to an additional failure of success criterion 2.1.1.

11. No autocomplete functionality has been implemented for the newsletter sign-up form.

    1. Fails: Success criterion 1.3.5

12. Most of the text has sufficient contrast. However, the language switcher lacks sufficient contrast, as do some of the linked textual logos.

    1. Fails: Success criterion 1.4.3

13. Some images with text on top of a gradient overlay have a fallback background colour with adequate contrast. However, specific images lack sufficient contrast when text exceeds three lines or text placed underneath headings is lengthy, causing the heading to move up to the top.

    1. Fails: Success criterion 1.4.3

14. Some content on sub-pages overlaps with other content on the page due to an unresponsive container. In particular, the text is cut off by overlapping elements, such as the logo above the menu button, making it difficult to read.

    1. Fails: Success criteria 1.4.4 and 1.4.10

15. A picture of text is present on the website within the slider on the home page. Additionally, it has no alternative textual description.

    1. Fails: Success criteria 1.4.5 and 2.5.3

    2. UserWay: The accessibility overlay deemed this image purely decorative. Thus, no alternative text was provided.

16. Some elements implement a title attribute that displays content only upon hover but not upon focus.

    1. Fails: Success criterion 1.4.13

    2. accessiBe: No content is displayed solely for mouse users as images no longer have a title attribute. Additionally, input fields now display a tooltip upon focus and hover, making this success criterion pass.

17. The website showcases an automatic image slider that endlessly rotates without the possibility of manual control.

    1. Fails: Success criterion 2.2.2

18. The slider header images are linked, but they lack alternative text. Additionally, certain links lack an accessible name due to being represented solely by an image without alternative text.

    1. Fails: Success criteria 2.4.4 and 4.1.2

19. Some links on the website have identical names ("View details" or "Download").

    1. Fails: Success criteria 2.4.4 and 2.5.3

20. The keyboard focus is not always visible, for example, on buttons such as "Menu" or "Subscribe".

    1. Fails: Success criterion 2.4.7

    2. UserWay: Custom focus styles are now in place, enabling this success criterion to pass. However, visibility is poor in grayscale mode.

    3. accessiBe: Custom focus styles are in place, and the keyboard focus is always visible, resulting in a pass for this success criterion.

21. Some words and phrases in the text are in other languages and have not been translated.

    1. Fails: Success criterion 3.1.2

22. A screen reader user does not receive notifications of errors that occur during interaction with a form.

    1. Fails: Success criteria 3.3.1 and 4.1.3

23. The provided error messages are displayed below the input field but lack helpful information. The message "Enter a valid email address" is not sufficiently informative.

    1. Fails: Success criterion 3.3.3

24. An ARIA label is present for the social media menu but not for the other menus on the website.

    1. Fails: Success criterion 4.1.2

5.3.3 Best Practices for Accessible Websites

Some best practices were identified during testing. Best practices are a category for accessibility errors that do not directly fail a WCAG success criterion but do impact users' experience and should be removed whenever possible. The following list includes ten suggestions to fix detected errors that do not fit into another category:

1. Essential elements such as buttons and links should be adapted to high contrast, grayscale and other colour settings.

2. It is important to avoid skipping heading levels (<h1> to <h6>) in order to prevent confusion for readers.

3. Only one <h1> heading should be used on each page of a website.

4. Providing additional context can be helpful. The website being examined offers a language switcher that could benefit from more descriptive surrounding text. That way, users will not be surprised with links named "EN" and "DE".

5. Announcing links that open new pages helps users with visual impairments navigate.

6. Avoiding all-cap text in HTML prevents screen readers from dictating words that should not be.

7. Applying hover styles to links assures better distinguishability.

8. A skip link on the website would allow users to skip entire content blocks. On the website within the study, it would be helpful if the logos within the <footer> could be skipped.

9. The alternative text should not include the term "image" as it is redundant since the screen reader already announces it.

10. The website sometimes uses <spans> for image descriptions, but it would be more appropriate to use the alt attribute that is already provided by the <image> element.

5.4 Usability and User Experience Evaluation Results

The user study, or usability and user experience evaluation, began by selecting study participants who fit the pre-defined group characteristics. A total of 36 individuals completed the screening questionnaire, whereby three individuals were excluded as they did not meet all the selection criteria. Out of the remaining 33 candidates, only five were English-speaking. The author decided to exclude these participants and conduct the study solely in German to simplify and shorten the preparation process of the study. Twenty-eight study participants remained. Three of them never replied to select a participation date. One participant was unable to partake due to technical difficulties. Another three participants actively revoked their will to participate due to various reasons. Twenty-one user study participants remained and took part in the study. In the following sections, they are encoded as participants T1 to T21.

The user study has resulted in 30.5 hours of interview recordings to transcribe. The median session duration was 90 minutes, with the shortest recording lasting 40 minutes and the longest lasting 130 minutes.

As stated in section 3.4.2, the website order underwent randomisation to minimise transfer and learning across tasks and websites. Each website combination was distinct. Six participants viewed the website without an accessibility overlay as the first website. Websites utilising UserWay, accessiBe, and AudioEye were presented as the first website five times. Table 5 lists all combinations for each participant.

Twenty of the 21 research participants navigated on each of the four websites. Participant T14 only engaged with three websites, leaving out accessiBe, due to fatigue and exhaustion as a result of their impairment. However, this does not constitute an exclusion criterion. The fatigue resulting from engagement with these specific websites emphasises the diversity of impairments and the need for accessible experiences. Thus, the integration of participant T14's results and feedback are crucial.

The data collected from the user study was categorised and evaluated using affinity diagramming. The affinity diagram produced for this research is not a conventional affinity diagram but a large table. The information contained within it was sorted, grouped, colour-coded and evaluated like it would be done in a traditional affinity diagram. The decision to produce an affinity table is based on the amount of quantitative data and its correlation with qualitative data. The quantitative data in this study (e.g. task duration and completion or AUS and UX scores) is closely linked to the qualitative data (e.g. observation notes and user comments). Furthermore, a table facilitates the detection of changes in user behaviour and measured values with every additional website viewed, as well as any learning effect that may occur. As a result, aggregating all collected data into a single table is a suitable decision.

The creation of the affinity table, in alignment with the six-phase approach of Braun and Clarke (2012), uncovered the following main themes and subthemes:

1. Participant Background and Technology Use

2. Impact on UX and Website Usability

   1. Challenges with Accessibility Overlay Recognition, Interaction and Activation

   2. Website Interaction: Task Duration and Completion

   3. Accessibility Overlay Effectiveness

3. Awareness and Perception of Accessibility Overlays

   1. Pre-Study Awareness and Initial Judgement

   2. Post-Study Perception Changes

      1. Willingness of Engagement with Accessibility Overlays

4. Desired Improvements and Expectations for Accessibility Overlays

These themes are used to discuss all the data collected in Chapter 6 of this thesis. The following sections present the quantitative and qualitative data collected during the user study grouped by the chronological order in which it was collected.

5.4.1 Demographics

The author aimed to collect as little personal and sensitive data as possible. Therefore, the demographic analysis concentrates on technical aspects such as access technology and browser usage, as well as an individual's impairment information, rather than their age, name or gender.

5.4.1.1 Duration of Visual Impairment

All 21 participating individuals have had their visual impairment for different lengths of time (Figure 17), with the majority experiencing it since birth (11 participants). Three participants reported living with their visual impairment for 20 to 30 years, while another three reported 30 to 50 years. One individual is living with their visual impairment for 10 to 20 years, one for 5 to 10 years, and two individuals have had it for 2 to 5 years.

5.4.1.2 Type of Visual Impairment

Eight of the 21 research participants with visual impairments stated they were blind. In addition, two participants were deaf-blind and required assistance from a sign-language translator and speech-to-text interpreter to participate in the study. The remaining eleven participants provided no further details regarding their visual impairment.

5.4.1.3 Internet Usage

All but one of the study participants use the internet daily for browsing, whereas the remaining individual uses it weekly. Seventeen participants used a computer with Windows, whereas four used one with MacOS. Two of the MacOS users chose Safari as their browser, whereas Firefox was the preferred choice among most of the other participants (9), followed by Chrome (7) and Edge (3).

5.4.1.4 Access Devices

Nineteen participants reported using access technology, with 16 of them utilising a screen reader. Figure 18 illustrates that eight opted for JAWS, four for NVDA, two for VoiceOver and two for Fusion (a combination of JAWS and a zooming function). Notably, two JAWS users stated employing NVDA as a backup in case JAWS failed.

5.4.1.5 Browser and Operating System Adjustments

The two participants who reported no use of access technology made adjustments on their computers. These modifications included alterations to the contrast and font size, usage of a larger monitor, and a screen magnifier. Overall, 13 out of 21 participants stated that they made adjustments to their browser or operating system. Among these modifications were changes such as inverting colours, utilising a dark mode, enlarging the mouse pointer, deactivating animations, installing extensions for their screen reader and utilising a refreshable braille display.

5.4.1.6 Accessibility Overlay Perception (Pre-Study)

When questioned about their familiarity with accessibility overlays, only eight respondents were aware of their existence. Three were uncertain, and ten had never heard of or encountered them. After providing a short description of what an accessibility overlay is (see section 3.4.2), participants were asked to express their first impression. The participants who were previously unaware of the existence of accessibility overlays responded positively to some extent. Eight found them potentially helpful. However, some concerns were raised:

"It could be useful in principle, but first, JAWS itself is often not fully compatible with websites and has a lot of legacy code, and second, website operators might think that an overlay is enough and just put it on top of a website."

Participant T7

"Possibly useful, e.g. colour contrast enhancement or reading aloud, but my own tools already do that anyway."

Participant T16

"I have already adjusted a lot of things, I do not think adding an overlay would help."

Participant T11

Participants who had been exposed to accessibility overlays prior to the study were mostly negative and sceptical about them.

"A scarecrow that you put a suit on. You save the web designers from having to create an accessible website, but they should just deal with it!"

Participant T1

"I have used AccessiBe before but did not have a very good experience. I have the impression that an accessibility overlay is not yet an alternative to an accessible website."

Participant T17

"Whenever I come across it, it's a feeling of being trapped, because I can only navigate in a certain area and then I can't get back to the website."

Participant T19

"I wonder how such an overlay would really work, how can software do that? [...] A website is such an individual product that I think it's almost impossible to put something clever on top of it."

Participant T10

5.4.2 Accessibility Overlay Recognition

The study measured participants' recognition and interaction with accessibility overlays, using four possible outcomes:

1. Detected

   Refers to a user who noticed the presence of the accessibility overlay but did not necessarily interact with it further.

2. Interacted

   Refers to a user who has opened the accessibility overlay after detecting it. This user may also engage further by reading its options and capabilities. It is important to note that activating the detected functionality is not necessarily implied.

3. Activated

   Refers to the act of a user proactively activating functionality of the accessibility overlay with a keyboard shortcut or by using the available button.

4. Attempted Activation

   Refers to a user's attempt to access the accessibility overlay or activate its functionality (through a shortcut), but the endeavour was unsuccessful.

Just over half of the study participants voiced that they could detect the accessibility overlays on the website (Table 6). The majority of users (71%) detected UserWay on the website. Figure 19 visualises how participants were distributed across the different scenarios: detected, interacted with, activated, or attempted to activate.

Table 6: Number of study participants who detected, interacted with, activated, or attempted to activate the accessibility overlay and its features while navigating.

Detection and Interaction with Accessibility Overlays
	Detected	Interacted	Activated	Attempted Activation
UserWay	71% (15 users)	9 of 15	5 of 9	1
AccessiBe	60% (12 users)	9 of 12	8 of 9	0
AudioEye	60% (12 users)	6 of 12	5 of 6	1

On the website utilising UserWay, the second item to be announced, after a "skip to main content" button, was a button that could enable accessibility features of the accessibility overlay. In English, it said, "Enable accessibility for visually impaired". The study participants heard or read the German version ("Aktivieren Sie die Zugänglichkeit für Sehbehinderte"). Directly following this button was another button labelled "Open the accessibility menu". This button offered a shortcut to displaying the accessibility overlay options, allowing users to toggle various functions on and off.

accessiBe and AudioEye also made announcements to their visitors. However, they presented some challenges to participants. The first button presented to screen reader users on the website that uses accessiBe said, "Use this website in a screen reader mode". Although the accessibility overlay and website were in German, the text on this button was in English. Fluency in English, in addition to German, was not a prerequisite for study participants. Therefore, there is a possibility that some individuals may have had difficulties understanding the button text.

Additionally, participant T6 encountered an issue while using their screen reader to navigate the website. The just-named buttons were not announced to the participant as they were only available when navigating with a keyboard or screen reader shortcuts, not with a mouse. This participant used their screen reader in a way that only read-aloud the parts of the website the user was hovering over with their computer mouse. The participant detected the accessibility overlay button but did not recognise it as such. Upon hovering over it, no announcement was made to the user. Consequently, the user opted not to click on the button. Participant T6 never interacted with the accessiBe overlay during the study. However, the same issue arose on the website that used UserWay. The participant did not interact with that accessibility overlay either.

When users accessed the website with AudioEye installed with a screen reader, they were automatically presented with a lengthy message.

"This website is AudioEye-capable and is optimised for accessibility. To open the AudioEye toolbar, press 'shift + ='. Some assistive technologies may require the use of a passthrough feature prior to this keystroke. For more information, enable the 'Explore your accessibility options.' button." [Translated from German]

Participant T13 acoustically understood AudioEye as "Audio AI". They further stated, "I've seen this somewhere before but have no idea what it is". Another participant (T3) also expressed uncertainty about the company name in the presented message and enquired, "What is AudioEye?"

A participant with reduced vision, who did not use a screen reader and was not blind, recognised a button with a stick figure in the lower left corner of the screen. Nonetheless, they did not comprehend its function.

"Is this the symbol for the accessibility overlay? Such a stick figure does not automatically mean that it is an overlay, it is just a stick figure."

Participant T14

Participant T17 was also unsure about how to identify an accessibility overlay.

"I don't have the experience to know how to recognise whether an accessibility overlay is on the page. I don't know how I would identify it."

Participant T17

In addition to the just-named issues, certain participants had an AdBlocker active on their devices. Subsequently, it was discovered during the observations that the AdBlocker prevented AudioEye from being loaded onto the website for three participants, while accessiBe was blocked for one participant. The author already identified this issue during the technical manual accessibility evaluation, outlined in section 5.3.2. Consequently, participants were requested to disable the AdBlocker for the duration of the user study. This yields four participants who were aware that the specific website they were navigating had an accessibility overlay, compared to other participants who only speculated that it would have one due to the study's context.

5.4.3 Accessibility Overlay Interaction

As listed in Table 6, not all participants who detected an accessibility overlay interacted with it. Of those who detected UserWay, 60% interacted with the overlay, while for accessiBe, the percentage was 75%, and for AudioEye, it was 50%. Even fewer participants chose to activate the accessibility overlays after the interaction.

The reasons for individuals not interacting with or activating an accessibility overlay are diverse. When questioned, participant T9 explained that their experience on the website with accessiBe during this study resulted in them not trying to activate the AudioEye accessibility overlay.

"Sorry, I'm not activating anything now, that annoyed me too much earlier."

Participant T9

Participant T5 stated that they did not activate the detected accessibility overlay because they did not trust it. They continue by saying that they prefer to look at the original version of a website as accessibility overlays "create additional elements that look like accessibility but don't actually help".

Another participant stated,

"Experience has shown that nothing changes for me when I activate the accessibility overlay. That's why I'm going to try it without."

Participant T19

A similar statement was made by participant T20.

"I didn't activate it because my previous experience with this type of accessibility overlay is very poor and it wouldn't have helped me to solve the tasks anyway."

Participant T20

Participant T15 reported that while navigating the second website within the user study, they did not perceive any need to modify the website using an accessibility overlay because the website was functional and satisfactory. However, they said that it could be different on another website.

In general, accessiBe had the greatest number of individuals activating the accessibility overlay in total and concerning the number of participants who detected it. However, the user study observations identified challenges when attempting to activate the accessibility overlays provided by each of the three companies.

For UserWay, one participant expressed confusion regarding the terminology used. In the German version of the accessibility overlay, the terms "Barrierefreiheit" and "Zugänglichkeit" were employed. These terms can be translated to "accessibility" and "approachability". The participant became even more confused by the differentiation between "blind" and "visually impaired" within the impairment profiles, as a person who is blind is also visually impaired.

"I would have to be very interested in this website to work my way through this."

Participant T1

Participant T10 was uncertain as well as to which individuals the "visually impaired" profile was targeting and whether it included blind individuals. Participant T17 additionally questioned whether there is a specific profile for screen reader users as they could not identify one.

Two study participants, T7 and T18, were unsure if activating the accessibility overlay was successful.

"I can't see if the button [for activating the accessibility overlay] is doing anything."

Participant T7

Participant T12 however reported no issues with activating the accessibility overlay.

"I thought it was very clear that you could see whether the mode was activated because the text then changed to 'deactivate'."

Participant T7

Regarding accessiBe, participants voiced similar issues and questions. One Participant was confused as to why impairment profiles exist.

"Do I have to choose a profile now? What if multiple apply to me?"

Participant T8

Three other participants were unsure if activating the accessibility overlay was successful. Participant T18 stated that they think they activated it. Participant T17 said that they pressed the button "but it still says 'not pressed'".

"The accessibility overlay says to press the switch, but it is a toggle switch and not a switch. For a switch, I would press [the shortcut] 'B' and find nothing because it is not a switch. Also, it's not the first link on the website. Additionally, it still reads 'press to activate' even though I've already activated it."

Participant T7

With the accessibility overlay from AudioEye, some study participants found it challenging to activate it with the presented shortcut. The key combination to press was not understood by participant T1 at all. Participant T7 had to use their refreshable braille display to understand the key combination as their screen reader only announced "Press Shift". However, shift had to be pressed in combination with the equal sign. Participant T10 pointed out that shift has to be pressed any way to press the equal sign on a German keyboard layout. However, they did not manage to activate the accessibility overlay, even after knowing the combination. Participant T8 also struggled with identifying the correct combination but eventually managed to activate the overlay. However, they reported that they received no feedback on whether the activation was successful.

In general, most participants managed to activate the accessibility overlays in the end. However, one person on the website with UserWay and one on the website with AudioEye could detect it but encountered difficulties when attempting to interact with the overlay and gave up trying to activate it.

5.4.4 Deactivating the Accessibility Overlay

It is worth noting that not all participants employed the accessibility overlay they activated for all tasks.

Starting with tasks 3 and 4, participant T3 activated AudioEye, as they had not previously identified the accessibility overlay on the website. During task 2, study participant T9 deactivated the suggested screen reader mode of the accessibility overlay accessiBe due to navigation difficulties. Otherwise, they would have been unable to complete the tasks.

For participant T12, the accessibility overlay of accessiBe deactivated itself during navigation. It was noticed during task three.

"I seem to have deactivated the screen reader mode. Well, I didn't deactivate it, but it was deactivated, presumably by navigating back on the page."

Participant T12

After realising the situation, the participant did not reactivate the accessibility overlay and instead stated, "It seems to work better this way".

5.4.5 Identified Accessibility Errors

Despite the implementation of accessibility overlays, numerous accessibility errors were found on all three websites during the technical evaluation and summarised in section 5.3.2 of the results. The user testing identified additional errors that are elaborated below. Not all of these identified errors failed the WCAG 2.1 success criteria, but all of them affected the usability or UX of the participants, highlighting the importance of user testing.

1. Unclear Label for Audio Samples

   The audio sample provided for performing artists was not clearly labelled as such. Several participants found the embedded SoundCloud⁶² component but had problems identifying it as an audio sample. This struggle was due to them not knowing what SoundCloud was, buttons labelled in English when they were not fluent, or being afraid to click the play button because it was unclear what it was for and what it would play.

2. Text not Read-Aloud

   The subpages containing information about the festival's performing artists were not read aloud to users hovering over it with their mouse. This error was likely caused by the order of HTML elements on these pages. A container holding a header image was placed above the container holding the body text.

   1. None of the three accessibility overlays fixed this error. Additionally, a participant navigating the website with UserWay reported that they had to highlight text for it to be read aloud, which "does not work if I cannot see the text".

3. Insufficient Colour Contrast

   Participant T21 reported that the floating action button of the UserWay accessibility overlay was inaccessible due to insufficient contrast. The participant had made several browser and system settings adjustments, including changing the contrast.

4. Confusing Filtering Options

   The news page had a filter navigation for filtering blog posts. Visually, they were presented in a horizontal line with red underlining and space between them (Figure 20). Directly after each filter option was a red circle with a number in it. This number indicated the amount of blog posts associated with each filter option. Visually, this relationship is arguably quite clear. It was not for people navigating with a screen reader, as no additional information was provided for access devices. A screen reader would read "Festival 202219" instead of, e.g. "Festival 2022 - 19 entries". Participants were confused about the meaning of the numbers and took longer to complete tasks where they were asked to filter for a specific topic.

   

   1. This problem persisted on all websites, even those with active accessibility overlays. The number of blog entries on the website using accessiBe was separated, resulting in the text: "Festival 2022 19". However, there was still no additional context. Participants commented that they noticed this change but were still unsure about the meaning of the number.

5. Confusing Search Button Text

   The automatic addition of the words "Color Baltic Sea" to the search button on the website using accessiBe caused confusion and resulted in participants hesitating to click the button.

6. Screen Reader Text not Matching Refreshable Braille Display Text

   Participants reported the problem that their refreshable braille display displayed an image description while their screen reader's link list displayed a different text. This error occurred when navigating over a component that previewed a blog post. This component consisted of an image, a headline, a preview text and a link to the post. However, the image and headline were also linked to the blog post. When the focus was set on this component, the alt text of the image was used as the link text in the refreshable braille display, as the image was the first element within that component.

7. Inaccessible Input Field

   Several participants using a screen reader to navigate could not reach the search input field when the accessibility overlay from accessiBe was active. This issue resulted in anger, frustration, disappointment and the overlay being turned off.

   "I have to admit if things as serious as the search function don't work, I'll probably close the website and never go back."

   Participant T9

In addition to revealing new accessibility errors, the user study provided context as to how some already identified errors explained in section 5.3.2 impact users with visual impairments.

1. Error 7: Heading Levels and Order

   Some participants found the hierarchy of headings confusing, particularly on websites with accessiBe and AudioEye.

2. Error 9: Open Menu Position

   Several participants reported confusion because the menu content was located at the end of the page instead of directly after the button to open the menu. Some would have never found the menu content without the author's help.

3. Error 13: Images over Text

   Multiple participants reported issues with text placed over images. Some found it exhausting to read, and others could not read it at all.

4. Error 17: Automatic Image Slider

   A participant using a digital magnifying glass experienced issues with the moving image slider on the home page. They quickly became lost and confused about their location on the website and the events taking place.

5. Errors 6 and 22: Form Labels and Notifications

   Some participants using a screen reader reported that the website did not provide feedback on the success or failure of their search conducted via the input field. Additionally, the participants suggested including labels for input fields would be helpful. They reported confusion between the email and search fields while completing study tasks.

5.4.6 Average AUS Score

There was no big difference in each site's rounded average AUS score. Table 7 displays the calculated averages. It can be seen that the website with AudioEye had the worst score, followed by the site without an accessibility overlay and the site with accessiBe. The website without an accessibility overlay and the one with accessiBe scored 57, while UserWay scored 58 and AudioEye scored 55. However, the minor difference in the quantitative score is negligible, as it only differs after the decimal point. Nevertheless, the average AUS score was lower than the calculated standard average AUS score of 65. This lower score indicated that the participants had difficulty navigating the website, which negatively impacted their usability.

It is important to note that the AUS and UX scores of participant T12 for the website with AudioEye were excluded due to technical difficulties on the side of AudioEye. The accessibility overlay did not load and could not be tested. Therefore, the documented scores were excluded from all calculations, which impacted the overall average for that website.

Table 7: The average AUS score for each website. It is further differentiated between users who recognised the accessibility overlay, those who did not, those who interacted with it and those who activated it.

Average Accessible Usability Scale Score
	Average AUS	Not Detected	Detected	Interacted	Activated
No Overlay	57 (57.05)	/	/	/	/
UserWay	58 (57.90)	67.33 ∧	54.13 ∨	53.00 ∨	57.60 ∨
AccessiBe	57 (57.45)	64.75 ∧	52.58 ∨	47.56 ∨	45.00 ∨
AudioEye	55 (55.45)	63.38 ∧	50.17 ∨	39.33 ∨	39.20 ∨
The score is between 0 and 100. The average of the AUS is typically 65.

Interestingly, the AUS score was higher for people who did not recognise the accessibility overlay. In two out of three cases, the AUS score was slightly above the average of 65. At the same time, the AUS score was lower than the overall average for people who recognised the accessibility overlay. It continued to decrease for people who chose to interact with the overlay. The individuals who then chose to activate features of the accessibility overlay resulted in even lower AUS scores in two cases. accessiBe now had an average AUS of 45 and AudioEye 39.2. Only UserWay was able to improve its score to 57.6. However, these scores are all still below the overall average within the study.

5.4.7 Average UX Score

The rounded average UX score, as can be seen in Table 8, was identical for all websites with a score of three ("neither good nor bad"). Upon examining the unrounded values, the website with accessiBe received the lowest average user experience score. The website without accessibility overlay was ranked second-last, followed by AudioEye and Userway. However, the differences are only in a few decimal points.

Table 8: The average UX score for each website. It is further differentiated between users who recognised the accessibility overlay, those who did not, those who interacted with it and those who activated it.

Average User Experience Score
	Average UX	Not Detected	Detected	Interacted	Activated
No Overlay	3 (3.30)	/	/	/	/
UserWay	3 (3.45)	4.00 ∧	3.21 ∨	3.25 ∨	3.60 ∧
AccessiBe	3 (3.18)	3.88 ∧	2.68 ∨	2.56 ∨	2.57 ∨
AudioEye	3 (3.32) ∧	3.88 ∨	2.91 ∨	2.80 ∨	2.80 ∨
The score is between 1 and 5, five being "very good" and one "very bad".

It should be noted that two participants' comments during the user study have impacted the overall average. Participant T9 was uncertain about their evaluation of their user experience on the website implementing accessiBe. Section 5.4.3 already mentioned their negative experience with that particular accessibility overlay. In order to complete tasks, the participant was forced to deactivate the overlay and subsequently declined to activate any other accessibility overlay from a different company for the remainder of the user study. When evaluating their experience on the website with accessiBe, the participant reported that their experience was positive after turning off the overlay and gave it a rating of four. However, when the accessibility overlay was active, they rated their experience as very bad (1). The participant was not willing to choose only one rating, so the author calculated the average of both values, resulting in a rating of 2.5 for the website.

Participant T3 first looked at the website without any integrated accessibility overlay. They rated their UX as three. The second website viewed had AudioEye integrated and received a rating of 3 regarding the participants' UX. Next was UserWay, rated with a 5, followed by accessiBe with a rating of 4. After viewing all four websites, the participant stated that they preferred the website without an accessibility overlay the most.

"I find my way around websites without an accessibility overlay and that's why I actually found the first website the best, because I'm used to it."

Participant T3

This preference was not reflected in the UX score given. If the participants' rating for the website without an accessibility overlay were to improve from three to five, the highest possible rating, the average UX score would increase as well. The author chose not to modify any values but instead excluded the UX scores provided by participant T3 for all websites. This decision was made due to multiple reasons. The comment provided by the participant was made at the end of the study. Their memory of their experience with the first viewed website may be imprecise. Furthermore, if the value for the best-perceived website were to be increased, the values for the other websites would have to be decreased to be lower than the rating of the best-perceived website. As the participant's comments did not align with their actions, the score was excluded from the calculations of the quantitative results.

When analysing the data in Table 8, an equivalent decrease in the UX score, as observed for the AUS score, can be detected. Individuals who detected the accessibility overlay rated their UX lower than average. The rating further decreased for those who engaged with and activated the overlay, except for the website employing UserWay. On this site, activating the accessibility overlay resulted in slightly improved UX ratings among participants, but still lower than average. Additionally, similar to the findings of the AUS score, participants who did not notice the accessibility overlay rated their UX higher than average.

The comparable score alteration for the AUS and the UX indicates a correlation between usability and user experience.

5.4.8 Learning Effect Across Websites

Participants interacted with the same website four times. The only difference was the added accessibility overlays. Although the order of presentation was random, there was still a learning effect that impacted the task duration and usability.

5.4.8.1 Task Duration

The average duration of a task decreased progressively with each website viewed (Table 9). This decrease could be attributed to the learning effect. The highest reduction in duration occurred between the first and second websites viewed, where the duration was almost halved. Subsequently, the average duration only decreased by approximately 10 seconds. When calculating the median to exclude anomalies, the average duration initially decreased but stabilised afterwards, similar to the calculated average. The longest task duration was ten and a half minutes, whereas the shortest average duration was 10 seconds. The average task duration over all four websites is almost two minutes (115 seconds). The median is 70 seconds.

Table 9: Average, median, lowest and highest task duration for the initial, second, third, and last websites visited by the participants during the user study.

Task Duration for Websites in Sequential Order
	1st Website	2nd Website	3rd Website	4th Website
Average	185 sec	100 sec	93 sec	81 sec
Median	160 sec	60 sec	60 sec	50 sec
Shortest	15 sec	10 sec	15 sec	15 sec
Longest	10 min 25 sec	7 min 40 sec	6 min 25 sec	8 min

As Table 10 illustrates, participants experienced the longest average task duration while using the website that had accessiBe installed. On average, they devoted slightly over two minutes to each task on that website. Additionally, the longest task duration, over ten minutes, was by a user with the accessiBe accessibility overlay activated. On the other hand, the website that employed UserWay had the shortest average task duration, roughly one and a half minutes. Nevertheless, when computing the median, the site without an overlay had the shortest task duration.

Table 10: Average, median, lowest and highest task duration for each of the four websites of the user study.

Task Duration
	Average	Median	Shortest	Longest
No Overlay	115 sec	63 sec	15 sec	8min 30 sec
UserWay	101 sec	70 sec	10 sec	6 min 10 sec
AccessiBe	132 sec	70 sec	15 sec	10 min 25 sec
AudioEye	114 sec	68 sec	15 sec	7 min 40 sec

5.4.8.2 Usability and User Experience

An increase in perceived usability could be observed when calculating the average AUS score for each of the websites navigated. As listed in Table 11, for the initial website visited by users, the AUS score was lower compared to the second website. Subsequently, a score increase could be observed for the third and fourth websites. The continuously improving usability of each website suggested a learning effect. This conclusion was reinforced by observations made by the author during the user study and the documented decrease in task duration listed above. Moreover, multiple participants addressed their learning effects directly during an interview.

Table 11: Average AUS and UX scores for the initial, second, third, and last websites visited by the participants during the user study.

Average AUS and UX Scores for Visited Websites in Sequential Order
	1st Website	2nd Website	3rd Website	4th Website
AUS	49.90	58.90 ∧	59.90 ∧	60.70 ∧
UX	3.10	3.45 ∧	3.33 ∨	3.42 ∧

The average user experience score, however, did not continuously grow. The average UX score increased from the first to the second website. On the third website, the score decreased again. For the fourth website, the score rose once more. Nonetheless, the second website viewed by study participants obtained the highest overall UX score. Concluding, the average UX score was not significantly influenced by the learning effect.

5.4.9 Task Completion

Participants' task completion was measured using four possible outcomes:

1. Completed

   The participant successfully completed the task.

2. Partly completed

   The participant needed the help of the author to complete the task.

3. Not completed

   The participant thought they completed the task, but they did not.

4. Abandoned

   The participant gave up trying to complete the task.

Table 12 illustrates that the website lacking an accessibility overlay had the greatest number of abandoned tasks. It was also the first website viewed in four out of six abandoned instances. Three tasks each were abandoned on the websites with accessiBe and AudioEye, but only one task each was abandoned after viewing the websites as the first of four. The task completion varied among individuals who had the accessibility overlay enabled. On both the websites with UserWay and AudioEye, 5 out of 21 people had it enabled, while on accessiBe, there were 8 participants.

Table 12: Task completion for each website's total number of tasks, categorised by active and inactive accessibility overlays.

Task Completion
	Completed	Not Completed	Partly Completed	Abandoned
No Overlay	48	6	2	6
UserWay	56	2	4	1
UserWay On	14	1	0	0
UserWay Off	42	1	4	1
accessiBe	47	6	4	3
accessiBe On	15	3	3	3
accessiBe Off	32	3	1	0
AudioEye	47	8	4	3
AudioEye On	10	2	1	1
AudioEye Off	37	6	3	2

With the UserWay accessibility overlay activated, there were no instances of tasks being abandoned or partly completed, whereas with deactivation, one task was abandoned, and four were partly completed. One task was not completed in both circumstances. Similarly, AudioEye saw a reduction in the number of abandoned, partly completed and not completed tasks with the accessibility overlay activated. AccessiBe was the sole accessibility overlay among the three where the amount of abandoned, partly completed, and not completed tasks deteriorated instead of improving.

5.4.10 Accessibility Overlay Perception (Post-Study)

At the end of the user study, participants were asked several questions about their experience and perception of the accessibility overlays. If participants had not engaged with the overlays during the allotted tasks, they were requested to do so swiftly before responding to the ensuing questions.

5.4.10.1 Preference: Accessibility Overlay or System Settings

Participants were asked about their preference for adapting websites to their needs, either via an accessibility overlay or browser and system settings. The results indicated that the majority (52.4%) preferred to only use their access technology, browser, or system settings (Figure 21).

"After a brief investigation, I haven't found anything suitable for me. When I'm surfing a website, I don't want to spend half an hour going through a list of what I can activate. It also seems that the accessibility profiles are only for the visually impaired and not for people who are blind."

Participant T10

"Personally, I prefer to use system and browser options because then it is simply a global setting and does not have to be readjusted from website to website."

Participant T11

Participants T16 and T6 concurred that it is more sensible to adjust the settings once, thereby eliminating the need to readjust them frequently.

In contrast, only one person indicated a preference for an accessibility overlay, but under the premise that it was available on every website.

"I would like it to be on the each homepage. That would be a help. I could imagine that. But then every website must have such settings. Then I can also make settings quickly on other devices. You're not necessarily always on the same device."

Participant T2

However, six participants expressed a liking for a combination of the two.

"I need the browser and the system settings for general navigation on the computer. That's why it's always activated. I don't know exactly what the overlay can do yet. But I would use a combination if I did, because I always need the magnifying glass, for example."

Participant T4

"I have set my accessibility settings so that it works for me. But when navigating websites, I sometimes change these settings or use accessibility overlays because then I don't have to change my system settings again when I go to another website."

Participant T21

For an accessibility overlay to benefit participant T13, it required an announcement at the beginning of a website.

"If it's somewhere else on the page, I wouldn't turn it on, I'd just have a look at the website. If I already know a website and know how to navigate it, I don't need the accessibility overlay either.

However, if I use a page often and the accessibility overlay saves my settings, this can make sense. But if I know that I'm just going to Google and only use the page once, then I think it's quicker to just look at the page myself."

Participant T13

Participant T14 stated that they would not use an accessibility overlay on their personal computer as they had already set up their computer so that they could use it properly. However, they could imagine using an accessibility overlay when they were forced to use a different device due to various circumstances.

"For example, if I'm in a hotel on someone else's computer, I can imagine using the accessibility overlay [...] but I think I'll fail because not everything is so intuitive. However, it's probably better than nothing. But I don't see any point in using it on my computer now."

Participant T4

Participant T15 expressed a preference for the inclusion of accessibility overlay capabilities in a browser plugin. They did, however, acknowledge the need to conduct further investigations on accessibility overlays, as they did not find most of the introduced features beneficial. However, the potential for an overlay to provide a partial improvement for them was not ruled out.

5.4.10.2 Features of Accessibility Overlays

Participants were asked whether they considered specific aspects and features of the accessibility overlays to be helpful or disruptive.

Participant T4, among others, stated that one has to take the time to investigate accessibility overlays, their abilities, and how they are beneficial to oneself.

Participant T16 reported liking the "high contrast and page structuring features". Participant T21 expressed enjoyment regarding the larger text, contrast, colour inversion, zoom, and read-aloud functionality. Participant T11 liked the available impairment profiles and believed that the epilepsy profile could assist light-sensitive individuals, such as themselves, in not losing focus as quickly. Furthermore, they appreciated the numerous functionalities provided by the accessibility overlays. On the contrary, participant T21 disliked the complexity and wished that accessibility overlay companies would "keep it simple" in order for them to navigate more easily.

Some participants appreciated the concept of the improvement but criticised its implementation. T5 stated they enjoyed the accessibility overlay's addition of more headings but did not like too many being created during user testing. The same issue was voiced by participant T7.

T6 appreciated the text magnification option but noted that Microsoft already incorporates such a feature. Similarly, participant T18 liked the alternative texts of images but pointed out that "many pages already have alternative texts without me having to activate anything extra beforehand".

Some participants found particular aspects of accessibility overlays to be unnecessary or disruptive. Participant T19 found none of the features beneficial except for the read-aloud button. However, they mentioned that a screen reader would be required for them to access and activate the button, thus making it useless.

Participant T1 expressed confusion about the description for opening the accessibility overlay and reported that the skip links were not functional. Participant T7 communicated multiple things they disliked.

"I would like to know whether a switch has been switched on or not. I don't like it when I don't know that. The correct labelling of text input fields would also be good. And if [the focus] just jumps away, like on the last page, then that's confusing."

Participant T7

Participant T20 questioned the tendency of accessibility overlays to being "very data-hungry".

"[This] is very unpleasant, I don't like that at all."

Participant T20

5.4.10.3 Preference: Website With or Without Accessibility Overlay

Participants were asked if they would prefer a website with an accessibility overlay integrated over a website without one.

Several participants expressed a preference for a website featuring an accessibility overlay. T4 and T18 stated they prefer the overlay, but only if it functions correctly. However, during the user study, participant T4 did not detect any accessibility overlay and completed all tasks without encountering one. In addition, they expressed uncertainty about the specific alterations made by an accessibility overlay on a website and that they need to spend more time investigating it. Participant T18 tested the accessibility overlays during the study, but faced issues with all of them. Their user experience was average throughout the study period and did not show any improvement.

Participant T6 prefered the overlay as it provided more information to them, such as alternative image texts. However, it was not realised by T6 that an accessibility overlay was implemented on a website while conducting the study. The only instance where they encountered the alternative text option inside the overlay was when prompted to interact with it towards the end of the study.

Participant T21 favoured a website with an accessibility overlay on the condition that the button to activate it was located in the website's header. The participant interacted with the overlays during the study and stated that some functionality was useful to them. However, they emphasise the importance of a website being accessible in the first place.

"What is very important to mention again: The website must be accessible according to WCAG. This must be taken into account. An accessibility overlay can never replace an accessible page. Because, for example, a braille display cannot be controlled via the accessibility overlay."

Participant T21

Participant T12 was uncertain. They noted that the effectiveness of accessibility overlays depended on the website being accessed.

"If I open a website and I have the feeling I can't navigate it, then I would definitely prefer a version that uses such an accessibility overlay."

Participant T12

In the study, participant T12 tested two overlays and reported positive results for one, while the other had a negative impact on their user experience.

For participant T8, it also depended on the complexity of a website.

"If I'm only looking for a single piece of information then it would be too complicated for me to take an extra look at the accessibility overlay. But if there's an accessibility overlay on a site that I use often, then I'd be up for it straight away if the website itself isn't quite optimal yet."

Participant T8

During the user study, the participants' user experience and usability scores decreased when using the tested accessibility overlays, as opposed to the website without such an overlay.

Participant T14 would prefer an accessibility overlay solely on a foreign computer. They declined to use an accessibility overlay on their personal device based on the adaptations they have already made, negating any need for it. Furthermore, they mentioned that their energy level also influences their decision.

"It always depends on how much energy I have and what I have planned for the rest of the day. Sometimes it's just too hard for me to click through and search, especially if every accessibility overlay is different. Then I'm also disappointed when I stumble across one that doesn't work so well."

Participant T14

In contrast, participants T5, T10, T19 and T20 preferred a website without an accessibility overlay.

"I prefer a website that doesn't use an accessibility overlay. Because then I'm dealing with something that a person has built and not something that a system has generated."

Participant T5

"At the moment it probably wouldn't make any difference to me as I can work with everything my computer has to offer, such as the magnifying glass/zoom functionality."

Participant T11

Participant T15 contended that a website should be programmed with accessibility as a default feature. However, they thought that an accessibility overlay could be added on top.

"If a website doesn't have good usability, then an accessibility overlay can't work wonders either. Therefore, a website should be well programmed by default, and if an accessibility overlay is offered, why not? Just like with the read-aloud functions on some websites. I think that's a good thing, because why not if it helps some people? But to judge the quality of a website solely on that is a bit too narrow."

Participant T15

Participant T2 stated that if a website was accessible by default, they personally would not need an accessibility overlay. During the study, participant T2 did not identify any accessibility overlay. Still, they reported an outstanding user experience and above-average usability on all websites, even though they do not comply with the WCAG standard for accessibility.

Participant T1 said that with an accessible website without accessibility overlay incorporated, they were sure that the website works 100% with their screen reader.

"With an overlay, I only maybe have the certainty that it will work with my screen reader."

Participant T1

If a website was inaccessible, participant T1 preferred an accessibility overlay. However, this statement conflicted with the author's observations. T1 achieved the best AUS and UX scores on the website without any overlay. Furthermore, T1 repeatedly expressed the belief that accessibility overlays do not work and do not reduce barriers.

5.4.10.4 Website Ownership Impact on Acceptance

The participants were asked whether their acceptance of accessibility overlays integrated into websites changes depending on the website's owner. Acceptance in this context did not automatically imply that participants endorse the implementation of an accessibility overlay on a website. The purpose of the question was to appraise whether their response varies based on the website owner. 12 participants answered with "No", four answered with "Yes", three were unsure and one did not comment.

Participants T12 and T17 asserted that the website owner does not impact their acceptance of accessibility overlays. Participant T10 believed there was no general justification for using an accessibility overlay. Participant T11 contended that company size was irrelevant regarding innovations. Participant T21 expressed indifference, stating that their personal preference was the most important factor.

"Data is collected anyway, so the whole world knows that I'm blind anyway. And I'm also selfish, I take what's most convenient for me."

Participant T21

Participant T7 also stated that they, as an end-user, did not care how big a company was, but they cared about whether or not the company's website was accessible.

"I know the small companies like to talk their way out of it. That bothers me a bit socially. People always criticise the big bad companies, but most of the time they are actually better. As a user, I don't care whether I go to an online shop or a 5-person website, as a user I have to be able to use it. If small companies think [accessibility] through from the beginning, they don't have so much hassle. It is much more difficult to make a website accessible afterwards."

Participant T7

Participants T5 and T18 believed that larger companies had a greater responsibility to create accessible websites and should not use accessibility overlays.

"For a large company, I would say accessibility is something that absolutely has to be included. [...] Of course, it is also good and important that smaller companies deal with accessibility, but it may well be that it is more demanding for them. I expect large companies to adhere to the existing guidelines. Period."

Participant T18

Participants T13 and T3 concurred and added that they would not understand why a large company would use an accessibility overlay. Nonetheless, both asserted they would understand it if a smaller enterprise employed it, as they believe "it is the more economical alternative". Participant T2 asserted that the financial feasibility of accessibility should be ensured for small businesses, and public funding would be beneficial.

"Accessibility should be a given, regardless of the website. But unfortunately, this doesn't really work in reality."

Participant T6

On the contrary, according to participant T20, there were few reasons why a company, regardless of its size, could not create an accessible website without resorting to an accessibility overlay. The participant found it irrelevant whether the company was small or large. In addition, participant T20 suggested that organisations advocating for individuals with impairments should carefully consider whether employing an accessibility overlay is beneficial.

"In my view, you lose a lot of credibility that way."

Participant T20

Participants T3 and T18 argued that identifying an accessibility overlay on a website indicated to them that the website owner had deliberately considered web accessibility and had previously encountered the term. Nonetheless, a critical view was also expressed by participant T3.

"But you can always quickly add something and then say we are barrier-free and diverse and so on, but whether it really is that way is another matter."

Participant T3

5.4.10.5 Improvements for Accessibility Overlays

Participants were asked if they could think of any improvements for accessibility overlays.

Participant T3 wished for compatibility with screen readers. Participant T21 desired accessibility overlays not to become extensive lists of features to maintain simple navigation. Participants T8 and T6 wished to be able to toggle specific functions swiftly using keyboard shortcuts. This feature already exists, but the participants were not aware of it.

Participant T9 requested feedback on button activation due to difficulties with activating the accessibility overlay. Additionally, they, as well as participant T12, asserted that the focus should not randomly shift to other elements. Furthermore, participant T12 requested that accessibility overlays display a particular HTML version of a website that is simpler, more coherent to navigate and loads faster.

Participants T11 and T16 requested a modification to the placement of the accessibility overlay button. Furthermore, participant T11 expressed dissatisfaction with the stick-figure icon, as they do not interpret it as a button for website editing.

Participant T15 expressed a desire for a tutorial detailing the capabilities of the accessibility overlay. They noted that without it, it can be quite laborious to sift through each option and comprehend them independently. Additionally, participant T15 stated that there is room for improvement in the interaction between the browser and accessibility overlay.

"If I have already zoomed in in my browser, the accessibility overlay does not work properly. You would have to reset everything here to set up the accessibility overlay correctly."

Participant T15

Participant T13 and T14 stated that standardising the structure of accessibility overlays available in the market would expedite the process of resetting features on a website that utilises a different overlay company.

"It's a disaster that every website is different. If everything is always different everywhere, then it's not a relief and I think that's a shame. It has to be standardised. When it comes to accessibility, I can't say that everyone has a bit of artistic freedom because it's not about art. You have to ask people with impairments and set up EU-wide or nationwide committees and then the disabled people say what we agree on, what works best for us and that should then apply. There simply have to be a few basic structures that everyone should adhere to. People with impairments should simply be included."

Participant T14

Participant T18 and T5 also expressed that people with impairments should be integrated into the development process of accessibility overlays. Furthermore, participant T5 stated that the automation of accessibility overlays should be improved.

"Just because it looks like many elements have now been made accessible doesn't mean it's optimal. If I have one thousand headings, it doesn't really help me to get an overview. One thousand jump labels don't help me either. I'm also careful with image descriptions because it has the potential to produce a lot of incorrect things."

Participant T5

Participant T20 believed that adaptions should happen on the user-agent side. Nevertheless, T20 acknowledged that numerous users are unaware of these possibilities.

"I think there should be more advertising for existing features from browser manufacturers. It would be more sustainable to learn how to change something in the system settings, and not in the accessibility overlay, because then I have it on all pages and don't have to do it all the time."

Participant T20

In the opinion of participant T15, accessibility overlays should not claim to remove barriers.

"I think they should change their marketing strategy by saying you can improve user experience and usability with an accessibility overlay. This is an everlasting point of debate. Accessibility overlays shouldn't claim to encounter barriers and want to remove them, because the barriers shouldn't actually be there. The accessibility overlay companies should finally take a different approach and say 'our technologies help people to optimise information and content to their own needs' and not say that they are removing barriers. In this sense, an accessibility overlay is a tool for optimising content to suit individual needs."

Participant T15

Participant T20 also recommended a differentiated marketing approach to accessibility overlay companies. T20 suggested these companies should partner with browser vendors to promote native browser features that accomplish the same results as accessibility overlays.

"For example, when installing or using [the browser] for the first time, you are shown what it can do."

Participant T20

Additionally, T20 believed that the sense of accessibility overlay was "relatively exaggerated" as most access technologies are easily accessible, and browsers and systems have built-in features such as screen magnification, text size, contrast reduction, or enhancement.

"If you say, 'Oh this is the panacea, now we no longer need screen readers and have the golden path to the kingdom of heaven', as is often done by companies like accessiWay, for example, or even with 'We are the pioneers and with us on your website you won't have a problem with the accessibility laws', that is, of course, a lie, and lying is bad. Lying hurts and lying destroys the world."

Participant T20

However, participant T20 suggested that individuals who feel at ease turning on an accessibility overlay for specific intentions should do so since it was a matter of personal preference. Moreover, they believed that by using an accessibility overlay, people would eventually become aware of their browser or system settings.

5.4.10.6 Effectiveness in Reducing Barriers

Participants were asked if they think that accessibility overlays help to reduce barriers and facilitate access to digital content.

Some participants were unsure and stated that they needed more time to test accessibility overlays before they could state their opinions. Participants T2, T3 and T21, however, said that accessibility overlays help in reducing barriers.

"Overlays enable people with visual impairments to actually use a website. This ensures equal usability. An accessibility overlay is preferable to just programming an accessible website because (a) not every operating system has as many accessibility options as Apple and (b) it is easier for people who are not so computer-savvy to change the settings directly."

Participant T21

However, participant T3 asserted that the overlay only works if no other browser or system settings are already active.

Participant T6 stated that an overlay could facilitate access if it functions more reliably than it currently does. Participant T18 stated that they believe accessibility overlays can be effective, but only under the condition that they function correctly.

"Only if they work properly. And only then! Because the ones I've tried here exist, but they're not very useful. Maybe I'm just not very experienced, but then they have to be so good that I can use them intuitively and don't have to take an extra course to be able to use them."

Participant T18

On the contrary, participants T10, T1 and T20 said that overlays do not facilitate access to digital content.

"[Accessibility overlays] draw attention to the fact that there are assistive technologies and that there is a need for adjustments to be made. But the shortcut via a JavaScript snippet and then you think everything is fixed is misleading."

Participant T20

Participant T9 stated that it is difficult to answer this question.

"I'm just asking myself why the accessibility overlay would actually be needed if programmers could take accessibility into account when programming. My amateurish thought is it wouldn't be any more work to simply do this when programming."

Participant T9

Participant T11 voiced that they are "rather neutral, but tending to be in favour" of the accessibility overlay.

"Because people who don't want to or can't deal with the system settings certainly have the opportunity to customise the website without having to worry about changing anything in the system."

Participant T11

The data presented in this section was collected from an evaluation study that included technical and user evaluations. These evaluations comprised automated and manual accessibility testing and user experiences collected through interviews, questionnaires, and observations. The following chapter provides a detailed discussion of this data.

---

Skip footnotes

59. https://accessiway.com↩︎

60. https://eye-able.com↩︎

61. https://marketingplatform.google.com/about/analytics/↩︎

62. https://soundcloud.com↩︎

6 Discussion

The research results of this thesis aim to contribute empirical knowledge on accessibility overlays. The following sections summarise and interpret the key findings and their contribution to the field of research. To structure the findings, the themes identified during affinity diagramming outlined in section 5.4 are used. Additionally, the limitations of the present work are discussed by outlining the difficulties faced during the research.

6.1 Participant Background and Technology Use

To answer the research questions, a user study with 21 participants was conducted. The 21 user study participants, who had varying visual impairments and ages as well as differing durations of impairment, utilised their personal devices, access technology, and corresponding settings. Out of all individuals in the study, most participants had lived with their visual impairment since birth. Furthermore, 19 out of 21 participants reported using access technology. Thirteen also adjusted their browser or system settings, indicating they already had the necessary tools in place. The popularity of screen readers is identical to the survey by WebAim (2021a), with JAWS being the primary screen reader for study participants, followed by NVDA and VoiceOver. However, the preferred browser's statistics differ, as most employed Firefox instead of Google Chrome.

It is not apparent that the aforementioned factors influenced participants' interactions with and perceptions of accessibility overlays. According to Thoo et al. (2023), a study by Min Htike et al. (2021) found that people with similar levels of visual acuity perceived different benefits due to their distinct preferences and techniques when using their residual vision. The study's findings support this claim, as participants reported varying accessibility needs and preferences. These varying approaches and needs also highlight the importance of user testing as reported by Sauer, Sonderegger, and Schmutz (2020; Campoverde-Molina, Luján-Mora, and Valverde 2021; Nuñez, Moquillaza, and Paz 2019).

6.2 Impact on UX and Website Usability

The main research question of what impact accessibility overlays have on the usability and user experience of individuals with permanent visual impairments browsing a desktop website can be answered by stating that website usability and UX are both, on average, not impacted by accessibility overlays but positively and negatively impacted depending on whether the overlays are recognised or not. The following sections elaborate upon and discuss this conclusion.

The participants evaluated their UX using a five-point Likert scale. The websites all received an average UX score of 3. Upon examination of the unrounded values, it is clear that they differ only by a few decimal points. As this difference is so small, it can be considered negligible. It, therefore, cannot be determined which website provided the better UX as their values are almost identical. This finding shows that, on average, accessibility overlays did not impact the UX for participants with visual impairments.

The AUS score was used to measure website usability. On average, each website scored between 55 and 58 across all participants, which is lower than the standard average AUS score of 65 on a scale from 0 to 100. According to the interpretation of Aaron Bangor and Miller (2008) of the SUS score, scores below 65 are considered marginal at best, and scores below 50 are unacceptable. Fable (2023) suggests that this interpretation can also be applied to AUS scores. Thus, the study results indicate that the usability of all four tested websites was poor. Although there was a slight variation in the recorded values, they all fell within the same range of 50 to 65. The website without an accessibility overlay had accessibility errors, which explains the poor usability and below-average score. However, adding an accessibility overlay did not significantly improve the website's usability.

The research also showed that usability and UX are closely linked, as demonstrated by the similar changes in their measured scores. However, a learning effect only affected the average usability but not the average UX. The AUS score, measuring usability, increases with each of the four websites viewed, whereas the UX score remains relatively constant.

Interestingly, the participants who did not recognise the accessibility overlay achieved the highest AUS and UX scores, with an AUS score between 63 and 67 and a UX score of 4. However, individuals who detected, interacted with, or activated any accessibility overlay scored lower than the website average for usability and UX, with both scores decreasing further the more participants interacted. This finding is unexpected, as accessibility overlays aim to improve usability, accessibility, and UX for individuals with impairments, according to resources such as Egger, Zimmermann, and Strobbe (2022; AudioEye 2022b; Groves 2021). After analysing the user study results, it can be concluded that the decreasing scores are due to challenges in recognising, interacting with, and activating the accessibility overlays, further elaborated in the following section.

6.2.1 Challenges with Accessibility Overlay Recognition, Interaction and Activation

Just over half of the study participants could identify the website's accessibility overlays. Similar to the findings of Kohler (2023), participants largely disregarded them. The lack of recognition may be due to its placement. Participants who used a digital magnifying glass, in particular, suggested that the button to access the accessibility overlay options should be located at the top of the website, near the menu. It is also possible that individuals may struggle to recognise the accessibility overlay due to unfamiliarity and a lack of knowledge on how to identify them. Most participants in this study stated that they had never heard of the term and were unsure of its meaning.

Another identified issue affecting the recognition, but also the interaction and activation of the accessibility overlay is the choice of words. Some participants found the company names and English language confusing, as well as unclear instructions and the inconsistent use of terms for accessibility and impairments across all overlays. However, accessibility overlay companies are not the only ones who use inconsistent terminology. Researchers like Thoo et al. (2023), Mack et al. (2021) and Brulé et al. (2020) also use varying terminology to describe visual impairments. In addition, Sharif, McCall, and Bolante (2022) pointed out that language preferences and terminology vary among gender identities, age groups, and countries. Adam and Kreps (2006) also noted the challenges of establishing a neutral terminology that is universally accepted. Unsurprising this is also the case for study participants, as each individual may have a unique interpretation of visual impairment.

UserWay was the most frequently detected of the three accessibility overlays. This may be due to their superior communication with screen reader users. Participants were presented with UserWay's concise announcement in German, prompting them to skip to the main content or open the "accessibility menu". Compared to accessiBe and AudioEye, UserWay's announcement was significantly shorter and did not include their company name. However, UserWay had the fewest activations out of all the observed accessibility overlays despite being the most detected.

Several reasons as to why participants did not interact with or activate accessibility overlays could be identified. One reason was a lack of understanding of what they were interacting with and what to do next. Theofanos and Redish (2003) found that screen reader users require knowledge of the browser, screen reader, and website, resulting in a significant cognitive load. If the overlay has unclear instructions, it could add to the cognitive load, making it difficult for the user to understand. Some participants reported that they did not notice any changes when they tried the accessibility overlay or had a negative experience with a previously tested one, leading to reduced confidence in the effectiveness of this technology. Other reasons given include participants stating that they do not require the functionality of the accessibility overlay to interact with the website because they are comfortable with their own tools. This finding is consistent with the research of Kohler (2023), which found that there is no reason for users with access technology to abandon a system they already use efficiently in order to learn a new one that may be less effective and only available on a subset of sites.

These findings emphasise the significance of word choice and user testing, especially as the study results show that the users could not always comprehend the information and instructions presented, regardless of which accessibility overlay was tested. The methods and text presented by accessibility overlays do not sufficiently explain to users without prior knowledge why they would benefit from them, nor do they convincingly encourage activation. The author, once again, agrees with the researchers Sauer, Sonderegger, and Schmutz (2020; Campoverde-Molina, Luján-Mora, and Valverde 2021; Nuñez, Moquillaza, and Paz 2019) that user testing is crucial to understanding user needs and creating usable products.

6.2.2 Website Interaction: Task Duration and Completion

When examining task completion, it was found that the website used for the study had accessibility errors, as tasks were frequently abandoned. However, the website, without an accessibility overlay, had the highest number of abandoned tasks. This observation shows that participants were likely more frustrated when navigating that website. Lazar, Feng, and Allen (2006) discovered that individuals with visual impairments tend to seek workarounds rather than give up when faced with frustration. It is important to note that participants abandoning tasks is a significant issue, as research indicates that this is not their initial response. The task's duration also indicates that participants took their time to solve each task when encountering it for the first time. The average task duration decreased progressively with each website viewed, which can be attributed to the learning effect. Although the presentation order of the websites was random, a learning effect still impacted the task duration. Spalding and Ross (1994) state that individuals frequently compare new with similar old instances and observe shared characteristics. Therefore, if a participant abandoned a task during the study, it is likely that they were frustrated and had attempted all possible solutions.

The impact of accessibility overlays on task completion, however, varies depending on the overlay used. Not all overlays are equally effective in addressing the needs of users with visual impairments. There is a need for a more thorough examination of the distinct features and functionalities of various overlays and their interaction with diverse web content and user requirements.

When looking at task completion alone, the UserWay and AudioEye overlays have improved task completion rates, indicating they can enhance website usability and accessibility for people with visual impairments. This improvement may be due to various reasons, such as providing more intuitive navigation aids or better technical addressing of common accessibility issues. This advancement aligns with the intended purpose of accessibility overlays to enhance the accessibility and navigability of web content for users with impairments. However, when combining this finding with the change in measured AUS and UX scores presented above, it is apparent that while task completion appears promising in favour of accessibility overlays, usability and UX suggest otherwise. If participants interact with accessibility overlays, their usability and UX are worse than if they did not. This result contradicts the arguments just mentioned to explain the improvement in task completion.

6.2.3 Accessibility Overlay Effectiveness

The accessibility overlay was not always activated for participants who used it. It either turned off accidentally or was intentionally turned off by participants due to issues with website interaction. Additionally, sometimes an activated AdBlocker in the browser made the overlay disappear. This suggests that the technology may not always be reliable, which is consistent with the statements of Buchner-Sabathy and Bahram, who claim that accessibility overlays often do not work reliably or entirely.

Despite the implementation of accessibility overlays, numerous accessibility errors were found on all three websites during the technical evaluation and the user testing. As Wille, Dumke, and Wille (2016) and Cao and Loiacono (2019) have noted, accessibility standards and guidelines are crucial for ensuring that websites and apps are accessible to all users. The findings from automated and manual accessibility testing demonstrate that incorporating accessibility overlays onto a website could marginally decrease identified WCAG violations in two of the three instances tested. It was found that UserWay eliminated two errors. AccessiBe eliminated four errors but introduced two new ones. The error count on the AudioEye-enabled website remained unchanged. The error count detected varied across the accessibility software used for automated testing, with it being higher for some accessibility overlays and lower for others. However, the manual accessibility test that was conducted identified further errors in addition to those identified by the automated test. This result aligns with Nuñez, Moquillaza, and Paz (2019), who stated that automated accessibility tests alone cannot always identify all web accessibility issues. Adam and Kreps (2006) argue that relying solely on automated web accessibility software results in designing problematic information systems that reflect a liberal view of equality and access, meaning that while there may be a general desire for a more equal and fair society, the deeper social and cultural structures that contribute to inequality are often overlooked. This view arises due to the lack of engagement between relevant discourses, the lack of involvement of people with impairments in the testing of websites, and the prevalence of a passive view of access, accepting what a tool gives without considering what access truly means (Adam and Kreps 2006). Furthermore, in accordance with the discourse on the social construction of disability, it is important to acknowledge that the existence of numerous inaccessible websites contributes to the creation of disability. Therefore, according to Adam and Kreps (2006), it can be argued that the design of technologies that rely on automated testing not only reflects liberal politics but also perpetuates disability itself.

Despite overlays remediating some accessibility errors, neither achieved full WCAG 2.1 AA compliance. These findings align with those of Egger, Zimmermann, and Strobbe (2022), who concluded that none of the overlays they examined achieved full compliance with legal regulations. The results are also consistent with the statements of various entities, including the EDF (2023), the IAAP, the NFB (2021), the WACA (2023), the DG COMM of the European Commission (n.d.), the BFIT-Bund (2023), as well as the opinions of the interviewed experts Bahram, Schmidt and Buchner-Sabathy. They argue that overlays cannot replace addressing accessibility issues at their source. They also note that overlays may not necessarily make websites more accessible according to WCAG standards and may even create new barriers.

Thus, this research has answered the sub-research question of how effectively the incorporation of an accessibility overlay on a website ensures compliance with the success criteria of the WCAG 2.1 standard by finding that accessibility overlays do not ensure conformity with the success criteria of the WCAG 2.1 AA standard.

6.3 Awareness and Perception of Accessibility Overlays

Participants were surveyed about their perceptions of and personal preference for accessibility overlays before and after the user study.

6.3.1 Pre-Study Awareness and Initial Judgement

Prior to the study, only 8 out of 21 participants were familiar with accessibility overlays. This number is surprising as the topic is heavily debated within the community, as outlined in section 2 and section 4. It raises the question of whether end-users for whom accessibility overlays are created are truly included in discussions about their usefulness. However, the study revealed that an AdBlocker can remove an accessibility overlay from a website without the user noticing, which could make users unaware of their existence. Additionally, according to statistics from the Web Almanac (2022, chap. 11) report, less than 2% of desktop websites use accessibility overlays. This is a small percentage compared to the entire web and could explain the unfamiliarity with the topic.

Participants who were already familiar with accessibility overlays before the study expressed mainly negative and sceptical attitudes towards them due to bad experiences with them or scepticism about the technical possibilities available today. Individuals previously unaware of accessibility overlays responded positively to a certain extent. Nonetheless, apprehensions were expressed concerning the necessity of accessibility overlays if browser and system settings are already in place, and doubts about their effectiveness were voiced. The research of Kohler (2023) on accessibility overlays for mobile devices provides a possible explanation. Their study concluded that users of access technology become highly familiar with their tools, making it unnecessary to abandon an efficient system to learn a less effective one that may only be available on a subset of sites. The participants of this thesis research may have been sceptical for the same reasons prior to the user testing.

6.3.2 Post-Study Perception Changes

After the study, the majority of participants were uncertain about the effectiveness of accessibility overlays in reducing barriers.

One participant questioned the need for accessibility overlays and whether it would be more convenient to program accessible website code directly. Other participants have expressed that they do not want to encounter accessibility overlays on any website due to a lack of trust in their abilities, their bad experience or because they are satisfied with the tools they currently use. Three more participants clearly stated that in their perception, accessibility overlays do not facilitate access to digital content. Others appreciated the concept of improving accessibility through overlays but criticised their implementation and the changes they make on a website. For instance, one of the overlays attempted to add more headings but ended up adding too many. The lack of resolution of accessibility errors is consistent with the results of this study's technical evaluation and the findings of Kohler (2023). Additionally, it was noted by participants that several accessibility overlay functionalities already exist in browser or system settings or as access technology. This finding agrees with a statement made by the interviewed expert, Bahram, who questioned the necessity of accessibility overlays. The issues they address can be resolved within the browser or system settings, making accessibility overlays redundant, according to Bahram.

Some participants in the user study stated that there is no justification for a website to use an accessibility overlay. They emphasised the importance of website accessibility, regardless of the company size or available resources. A few participants even believe larger companies are more responsible for making their websites accessible and should not rely on accessibility overlays. One participant argued that a company loses credibility by employing an accessibility overlay. On the contrary, two other participants argued that identifying an accessibility overlay on a website indicated that the website owner had deliberately considered web accessibility and had previously encountered the term. These statements from participants support Wolberger's claim that when a website uses an accessibility overlay, users have a "very positive" perception of the site. However, this was not the case for all participants.

In contrast to the negative perceptions of accessibility overlays, three participants appreciated the idea and the aim of accessibility overlays, believing they could be helpful under the premise that they work correctly and reliably, which they currently do not. This positivity aligns with Murphy et al. (2008), who found that participants demonstrated a strong desire to explore new solutions despite being familiar with their access devices. Some participants suggested that an accessibility overlay could be used as access technology, such as for the read-aloud function, if the website was already accessible. However, this scenario was not addressed during the user study, and the participants speculated about this possible use case. Future studies should explore this idea.

In summary, most participants hold a sceptical view of the current state and abilities of accessibility overlays to reduce barriers and facilitate access to digital content. This rejection towards accessibility overlays is similar to the survey results of WebAim (2021b, sect. "Overlay, Plugin, Widget Effectiveness"), which found that the majority of individuals with impairments consider web accessibility overlays as "not at all" or "not very effective". However, participants of this thesis user study were additionally asked about their willingness to engage with accessibility overlays in the future to understand their post-study perceptions better.

6.3.2.1 Willingness of Engagement with Accessibility Overlays

After the study, only one out of 21 participants stated preferring the accessibility overlay as the only means to navigate a website, but only if it is available on every website they encounter. However, two participants pointed out that individuals with impairments are not necessarily always on the same device, sometimes possibly even on public ones. In such scenarios, an accessibility overlay could, according to the participants, be helpful as it is probably quicker than altering system settings and eventually better than having no accessibility options at all. Additionally, individuals may not always have access to sighted assistance and would benefit from increased independence. This voiced idea of a possible use-case for accessibility overlays is, again, reminiscent of the findings of Murphy et al. (2008), who reported that although their study participants were familiar with access technology, they expressed a strong interest in exploring new web browsing solutions. However, to date, this is a hypothetical scenario that requires further research. Perhaps the concept of the interviewed expert Buchner-Sabathy about accessibility features and including them directly in the menu bar rather than as an accessibility overlay could be equally beneficial. As Buchner-Sabathy noted, while it may appear redundant, it can assist those who are not tech-savvy and unfamiliar with the browser's tools. To address these ideas, future research on accessibility overlays could investigate their impact when no previous device settings have been made, and no access devices are set up and ready to use.

The majority (52.4%) of participants prefer to solely use their access technology, browser, or system settings. These participants expressed that they do not want to spend a lot of time adjusting overlay settings because they prefer to use their time differently. They argued that it is not worth their time to set up the accessibility overlay if they just want to quickly look up information. Additionally, they do not want to readjust accessibility overlays on new websites. Furthermore, some found the accessibility overlay too complex. Although not all participants use a screen reader, these findings are similar to the ones of Kohler (2023), who reported that there is no reason for users of access technology to abandon a system they already use efficiently to learn a new one that may be less effective and will only be available on a subset of sites. Additionally, they stated that accessibility overlays are redundant for screen reader users specifically as they do not significantly improve accessibility.

However, 28.6% of study participants prefer using a combination of an accessibility overlay and their access technology, browsers, and system settings. These participants argue that they may need to adjust their system settings depending on the website they are interacting with. They do not wish to alter their system settings for each website, but they can use the accessibility overlay to make adjustments solely for a particular website. Some additionally appreciated specific functionalities of the accessibility overlays for personal use. Examples included contrast and font size options, colour inversion, zoom, and read-aloud functionality. Further participants also found the "accessibility/impairment profiles" helpful, while others were unsure which category they fall into. This finding aligns with a statement made by Schmidt from EyeAble, who argued that impairments exist on a spectrum and cannot be generalised and critiqued that some of their competitors are pigeonholing individuals through the use of impairment profiles.

Surprisingly, this reported preference contradicts the recorded AUS and UX scores, which show that usability and UX are poorer for individuals who use an accessibility overlay. However, some participants expressed that they could imagine using an accessibility overlay, although they did not test it or were not fully satisfied with its execution during the study. It appears that the concept and idea behind accessibility overlays are valued, even if the execution of the accessibility overlay was not entirely satisfactory.

An explanation could be a potential response bias and an apparent discrepancy between voiced statements and observed actions. Ming et al. (2021) state that response bias concerns all researchers conducting studies with people, particularly those working with participants with impairments because the diverse needs of these participants require methodological adjustments. Additionally, differences in impairment identity between the researcher and participant can influence power dynamics. According to Trewin, Marques, and Guerreiro (2015), positive bias in accessibility research may arise from various sources. For instance, participants who volunteer for accessibility research studies, as they did for this research, may have a very positive attitude. Another explanation could be the verbal presentation of questions that may make it difficult to express negative opinions. Additionally, Trewin, Marques, and Guerreiro (2015) discovered that participants in non-anonymous accessibility research studies provided more positive average ratings than those in typical usability studies, especially when answering questions about a proposed innovation. Furthermore, they observed that participants with visual impairments were less sensitive to usability issues and that their subjective ratings did not correlate as strongly with objective performance measures. This phenomenon and a positive response bias may have also occurred in the study of this thesis, possibly explaining the observed discrepancy.

The appeal of accessibility overlays to individuals with impairments may also lie in the idea of accessibility. If websites are designed with, for example, reliable contrast options, users would not have to adjust their system settings because a website owner ensures accessibility. During the study, multiple participants highlighted the importance of their well-being, which is why websites must be accessible, according to them. Users seem to desire simple and easily accessible functionality.

The author speculates that the observed contradicting behaviour may be due to a desire for access rather than the actual effectiveness of the accessibility overlay itself. Although the web has existed for over three decades and legal regulations were established worldwide, over 96% of websites still contain accessibility errors (WebAim 2023), disabling individuals with impairments and resulting in digital divide, inequality and hope for improvement (Retief and Letšosa 2018; Adam and Kreps 2006).

The presented results answer the sub-research question of how individuals with permanent visual impairments perceive accessibility overlays by finding that they are primarily perceived negatively, with individuals questioning their effectiveness in reducing barriers and reporting upon personal experiences. However, some are still considering using accessibility overlays alongside their access technology, browser or system settings, provided the overlays function reliably in the future.

6.4 Desired Improvements and Expectations for Accessibility Overlays

This research has demonstrated that accessibility overlays, in their current state, do not eliminate all accessibility errors and do not improve usability or UX. While the interviewed expert Bahram does not believe that there is any way to improve accessibility overlays, participants themselves voiced multiple ideas.

Participants desire improved compatibility with access technology and browsers and the absence of technical malfunctions caused by the overlay. Additionally, they suggest positioning the floating action button near the top for easier identification and providing more feedback during interaction with the accessibility overlay. They also want a tutorial and standardisation to ensure that all accessibility overlays on the market have a similar, clear, and user-friendly interface, making interaction faster and easier. Individuals also wished for accessibility overlays to assist in optimising a website to their needs rather than promoting the reduction of barriers. According to participants, a website should have no barriers in the first place, but individuals may require further customisation, which could be done by accessibility overlays. The participants also want accessibility overlay companies to be truthful in their marketing strategy about the capabilities of their technology. This statement is in line with Bahram and Buchner-Sabathy's questioning of promised compliance with accessibility standards and legal security promises. The findings of the technical evaluation in this thesis also highlight the need for further development and enhanced transparency regarding the limitations of accessibility overlay companies, which aligns with Egger, Zimmermann, and Strobbe (2022) previous emphasis. The results, identical to Egger, Zimmermann, and Strobbe (2022), also highlight the importance of educating website owners about manual adjustments and the limitations of accessibility overlays.

Adam and Kreps (2006) pointed out that part of the reason for the persistence of inaccessible websites lies in a passive liberal approach due to crucial gaps in the engagement between the different discourses of web accessibility, digital divide, social construction, and legislation. The missing engagement may also be found in the development of accessibility overlays, which could explain why they do not deliver on their promises. Although there seems to be a widespread aspiration for a more equitable and fair society, the underlying social and cultural structures perpetuating inequality appear to be disregarded. Accessibility overlay companies should integrate more people with impairments into their development process, aligning with Bahram's expression of "nothing about us, without us", meaning you need to involve the community in decisions. This integration was also demanded by the study participants. As Kohler (2023) already pointed out, true accessibility necessitates comprehensive testing with real users with impairments, as most problems are structural.

While Wolberger of the company UserWay argues that it is about progress, not perfection, the author disagrees. The results point at accessibility overlays being a disability dongle, meaning "an outcome in which designs or technologies ‘for' disabled people garner mainstream attention and accolades despite valid concerns disabled people have about them" (Jackson, Haagaard, and Williams 2022). In addition, existing alternatives, including browser and system settings and access devices, that have been in use for years, render accessibility overlays in their current state as an inferior solution, after evaluating the results of this study. Currently, the author suggests that it is more sustainable to teach individuals with impairments how to operate access technology that has been available for a more extended period rather than teaching them to use accessibility overlays until they are further researched and improved. Moreover, prioritising the enhancement of some users' experience over the deterioration of others' regarding web accessibility is ethically questionable and also needs to be addressed in future research.

6.5 Limitations

Several study limitations could be identified during the research process. The subsequent paragraphs address these limitations and considerations, as they could have affected the study's outcome.

6.5.1 Conducting the Study Online

The decision to conduct the study entirely online introduces several limitations.

Firstly, using an online setting restricts the participation of individuals who prefer or rely exclusively on in-person interactions. Some individuals may encounter irregular internet connectivity or may favour or necessitate alternative communication methods due to accessibility issues. Online research inherently eliminates those who encounter obstacles to internet accessibility due to economic, geographical, or technological constraints. This exclusion may even disproportionately impact individuals with impairments. This exclusion could neglect valuable insights from specific segments of the community of individuals with visual impairments and implies that the representation of individuals could be limited.

Secondly, although the website and study materials were made accessible with great effort, people who use access technologies may still face difficulties. For instance, screen reader compatibility, keyboard navigation, or other accessibility features might not function optimally, adversely affecting participation. In this study, technical difficulties caused one participant to withdraw their participation.

Thirdly, the evaluation of accessibility overlays was conducted using the video conferencing tool Zoom. Participants shared their screens to allow the author to observe their interaction. However, the Zoom options (e.g. mute, stop screen share, turn on video) were sometimes present on the screen, which could have impacted the interaction. Furthermore, Zoom could not replicate all access technology functionality, such as specific colour profiles and magnifying software. However, participants could still use their access technology as they were used to and did not notice this limitation, but the author could not see them. Nonetheless, this restriction may have led to overlooked observations or connections regarding interaction issues.

6.5.2 Financial Constraints

Financial restrictions impacted several aspects of the research, including the scope of data collection, sample size, technologies and potential outreach strategies. Consequently, the study might have overlooked opportunities to examine a broader range of data sources or undertake more thorough and detailed analyses. Financial constraints may have limited the investigation of a broader range of usability scenarios concerning accessibility overlays for individuals with permanent visual impairments.

Due to the financial constraints, the study had limited availability of technologies, specialised tools or software that could enhance the research process. This limitation may have affected the thoroughness and efficacy of data analysis, potentially compromising the study's findings.

The financial constraints also coincide with limitations in participant recruitment. Inadequate funding has constrained the authors' capacity to engage with a broader and more diverse pool of participants, influencing the sample's representativeness. Additionally, offering incentives would eventually have increased the number of willing participants who could have been recruited. A small sample size could potentially reduce the study's statistical power and the findings' generalisability.

6.5.3 Sample Size

The study's sample size could impact the generalisability of the findings and may have overlooked nuances. This pertains to both the sample size of participants in the study and the quantity of chosen accessibility overlay companies. Despite striving to create a diverse and representative sample, it may not have sufficiently encompassed the full range of variability and diversity within the population of individuals with permanent visual impairments. This restriction may have hindered the exploration of different user experiences, preferences, perspectives and challenges users face when interacting with a website that utilises an accessibility overlay. Consequently, the results may have restricted applicability to a broader range of scenarios and user demographics.

6.5.4 Variation in Impairment Types

The study centred around individuals with permanent visual impairments, intending to understand how accessibility overlays specifically impact this population. However, the field of impairments is vast, encompassing different types of impairments beyond visual ones. The impact of accessibility overlays may differ significantly regarding other kinds of impairments. Focusing solely on permanent visual impairments restricts the generalisation of findings to cover a wider range of impairment types. The effectiveness, usability and user experience of accessibility overlays for people with other types of impairments remain uncertain and unexplored within the scope of this research.

6.5.5 European-Centric Perspective

Whilst the related work Chapters incorporate a US-centric view of the research topic, the user study focuses predominantly on European-centric perspectives. This focus may limit the universality and cross-cultural applicability of the study findings. The research may have overlooked regional nuances in accessibility needs and variations in the effectiveness of accessibility overlays across different geographical locations or cultural contexts by focusing on particular regions.

Different regions may have different cultural attitudes and approaches to accessibility. Neglecting to consider these differences may limit the study's ability to offer universally applicable insights into the effectiveness and reception of accessibility overlays.

Furthermore, discrepancies in technological infrastructure and availability can significantly affect the use and efficacy of accessibility tools such as overlays. Regions with diverse technological advancements or disparities may pose differing challenges and opportunities beyond this study's scope.

Additionally, user behaviour and preferences around technology may vary significantly based on cultural backgrounds and societal norms. These variances in user behaviour could impact the approval, usage trends and perceptions of accessibility overlays, which may not have been adequately investigated in the study's confined geographic scope.

6.5.6 Language

The research was conducted in German, which may limit the generalisability of findings from individuals with diverse linguistic backgrounds. The website content and research materials were presented exclusively in German, potentially excluding insights and experiences of individuals who primarily interact in other languages or modes of communication. This linguistic focus may inadvertently limit the inclusivity and representation of individuals who interact primarily in languages other than German. Findings from a study conducted solely in German may not be universally applicable across diverse linguistic groups. Therefore, the generalisability of the study's conclusions to global populations may be restricted.

6.5.7 Desktop-Only Devices

Although mobile device usage is higher than desktop usage, this study focused exclusively on desktop devices. According to Ceci (2024) writing for Statista, mobile devices account for approximately half of the web traffic worldwide, consistently hovering around the 50% mark since the beginning of 2017 before surpassing it in 2020. In the last quarter of 2023, mobile devices (excluding tablets) generated 58.67% of global website traffic (Ceci 2024). It is vital also to consider the impact of accessibility overlays on mobile devices. As mobile devices use touch screens, the interaction and accessibility requirements differ. Further research is recommended to explore the effects of accessibility overlays on mobile websites, especially in relation to the work of Kohler (2023), who has already begun investigating this area.

6.5.8 Economic Perspective

The author adopted a website owner's economic perspective for the evaluation study. This perspective meant that modifications to the accessibility overlay admin dashboard were limited to visible alterations present during the initial setup or when explicitly reported via email by the accessibility overlay company. The author did not intervene further. The study excluded human involvement, relying predominantly on automated solutions. This focus could have influenced the study's results, as frequent adaptations in the accessibility overlay may alter the functionality and accessibility of the website.

7 Conclusion

In conclusion, in their current state, accessibility overlays do not benefit every individual with a permanent visual impairment. A website's usability and a user's UX are impacted by them. While accessibility overlays are currently not increasing usability and UX to an acceptable level, both usability and UX are slightly better in the case of individuals who are unaware of the accessibility overlays' existence on a website. This contradicts the idea of accessibility overlays that aim at improving usability, accessibility, and UX for individuals with impairments (Egger, Zimmermann, and Strobbe 2022; AudioEye 2022b; Groves 2021). Additionally, the results of this thesis show that users generally experience a worse UX and reduced usability when they realise a website has an accessibility overlay. This negative impact is further compounded when users interact with or activate the overlay. This decrease can be attributed to issues with activating and interacting with the accessibility overlay and its user interface due to several reasons such as inconsistent structure, misleading wording (Sharif, McCall, and Bolante 2022; Thoo et al. 2023; Mack et al. 2021; Brulé et al. 2020) and cognitive load (Theofanos and Redish 2003), navigation issues on the website while using the overlay, previously negative experiences with overlays and the feeling that "nothing had changed" (Kohler 2023), which led participants to deactivate the accessibility overlay.

Accessibility overlays also do not ensure conformity with the success criteria of the WCAG 2.1 AA standard. This result aligns with findings of previous research (Egger, Zimmermann, and Strobbe 2022; Nuñez, Moquillaza, and Paz 2019) and statements made by interviewed experts and various entities (EDF 2023; NFB 2021; WACA 2023; BFIT-Bund 2023; European Commission n.d.) arguing that overlays currently cannot replace addressing accessibility issues at their source, do not make websites more accessible according to WCAG standards and do not align with legal regulations.

The present study further revealed that while many people are unaware of accessibility overlays, as Kohler (2023) also demonstrated, they tend to question their effectiveness in reducing barriers (WebAim 2021b) or view them negatively once they become aware of them. Most individuals are not willing to interact with accessibility overlays in the future and prefer to only use their existing access technology, browser and system settings. However, a significant number of individuals with permanent visual impairments may consider using accessibility overlays in certain situations alongside their access technology, browser or system settings, provided that the overlays function reliably at one point in the future, although their experience with overlays has mainly been negative thus far. This positivity can possibly be attributed to a combination of positive response bias (Ming et al. 2021; Trewin, Marques, and Guerreiro 2015), a strong desire to explore new solutions (Murphy et al. 2008) and the desire for access in a world where the majority of websites contains accessibility errors (WebAim 2023), disabling individuals with impairments and resulting in digital divide and inequality (Retief and Letšosa 2018; Adam and Kreps 2006).

The current form of accessibility overlays does not deliver on their promises and was identified as disability dongle, meaning "an outcome in which designs or technologies 'for' disabled people garner mainstream attention and accolades despite valid concerns disabled people have about them" (Jackson, Haagaard, and Williams 2022). In conclusion, the author suggests teaching individuals with impairments how to operate access technology instead of teaching them to use accessibility overlays in their current form. Further, it is proposed to design and program accessible websites following accessibility standards, such as the WCAG, as various legal regulations require. At the same time, the research showed and agrees with previous research (Sauer, Sonderegger, and Schmutz 2020; Campoverde-Molina, Luján-Mora, and Valverde 2021; Nuñez, Moquillaza, and Paz 2019; Kohler 2023), that individuals with diverse impairments must be incorporated into the development process to understand user needs, identify errors that go beyond the WCAG success criteria and create usable products. Additionally, as Adam and Kreps (2006) reported, engagement between the discourses of web accessibility, digital divide, social construction, and legislation are encouraged to reduce the persistence of inaccessible websites in the long run. Lastly, accessibility overlay companies are urged to invest in research and remediate the identified issues within their technologies while considering ethics and not prioritising the enhancement of some users' experience over the deterioration of others.

7.1 Future Work

The study on the impact of accessibility overlays on the usability and user experience of people with permanent visual impairments browsing desktop websites has provided valuable insights. Nevertheless, several areas remain for further exploration to enrich and extend this research. Hence, this study proposes some potential future research directions.

7.1.1 Examining other Impairments

Future research could adopt a more inclusive approach by considering a broader range of impairments to overcome the limitations of this study's target group selection. These could include other permanent impairments like motor, cognitive, or auditory, but also temporary and situational ones. Incorporating a range of user groups representing diverse impairments can enhance the understanding of how accessibility overlays affect usability and user experience across different impairment types. This inclusivity can enrich the research results and ensure that the findings are more robust and applicable to the needs of a broader population of people with impairments.

7.1.2 Ethics in Web Accessibility

The research identified differing viewpoints and significant differences in approaches among practitioners, researchers, and companies in the field of web accessibility, particularly regarding technological innovation and prioritisation of enhancements. Future research should address the ethical implications of promoting technology for a specific user group while compromising the experience and usability of others.

7.1.3 Examining Interest in Accessibility Overlays Despite Poor Experience

Some participants are still considering using accessibility overlays alongside their access technology, browser or system settings, provided they function reliably in the future, despite their poor experience. Future studies should explore this phenomenon and the identified positive participation bias in web accessibility towards non-functioning technology to understand its origin and how it can be addressed and interpreted in future work.

7.1.4 Assessing Accessibility Overlays on Accessible Websites

Several participants proposed the idea of implementing an accessibility overlay for websites that are already accessible. Further research is needed to determine whether this would have a different impact on UX and usability than the results of this study.

7.1.5 Assessing the Accessibility of the Accessibility Overlay

Further research could be conducted to examine the accessibility of the accessibility overlay widget/pop-up/tool itself. This examination could entail assessing accessibility overlays' compatibility with access technologies and adherence to accessibility guidelines. It is particularly intriguing because any mechanism used to make accessibility adaptations to a website should be accessible, functional, and navigable for individuals with impairments in the first place.

Investigating these highlighted areas in forthcoming research can improve our understanding of the impact, usability and user experience associated with accessibility overlays across different user groups and contexts. Tackling these areas can play a significant role in enhancing web accessibility and promoting inclusive digital environments for individuals with diverse abilities.