Feature request: Implement accessibility features?

[juanreina921]

Version/Branch of Dear ImGui:

not applicable to current situation

Back-ends:

not applicable to current situation

Compiler, OS:

not applicable to current situation

Full config/build information:

not applicable to current situation

Details:

My Issue/Question:
ImGui has no current implementation of accessibility for Asistive software, but there is a way for an implementation to have such support.
XXX (please provide as much context as possible)

hello there!

This issue, as the title states, is about the lack of accessibility support in imgui, especially since this will become increasingly important soon enough. This will also present a convenient solution to making imgui have accessibility support, for windows, linux and mac OS, all at once, with minimal effort on your part. But before we get to that, some background and terminology explanation is required, because last time this was discussed, #4122, people were confused, rightly, by most of what was talked about in there, so it's my turn to attempt to make it right

background

For a very long time now, I have been passionate about the chiptune scene. However, the only thing I could do is archive people's chiptune, but that I did a lot of, and I like to think that I did it well too. The problem is that I wanted to do more at some point, to give back to the community in the only way I know how, by making something. However, in this scene, a lot of UI is written with imgui, because imediate mode ui is generally better than regular retained mode UI, such as gtk for example. So then, I was trying to talk to people about making that software accessible to us too, and corroborating multiple such instances together, this is how such a conversation typically sounds like:

• me: hello there, I'm a blind person who uses a screenreader, insert details about how screenreaders work here. Could you perhaps help me make this accessible, for me and my people? currently, nothing at all reads, it looks like a blank window, a black canvass
• person at the other end: ahem, what's that more exactly/sure, let's see what's possible. A long discussion usually follows, about how accessibility is typically implemented, more on that below
• person goes and actually tryes to implement some accessibility in the app, in whatever way they think it should work. It generally doesn't work at all, or is just exploratory and it's abandoned, because
• person: there are no accessibility API's here, I can't subclass or annotate anything like in qt or gtk, and not even native controls are accessible. Nothing of what you told me applies, and not even self-voicing it with tts would work that well, because events are slightly unpredictable with keyboard navigation
• me, inwardly prepairing for an unfavourable answer and kinda knowing what's coming: what ui is this written in?
• person: imgui
• me: a long, long pause, followed by lots of disappointment

This has happened a couple times already, and not only with trackers either. An mml editor is written in imgui, so are some other products, such as some hex editors, which could be useful in some scenarios, music players, you name it, it's probably there.

What I'm trying to get at here is that more and more people are adopting imgui, because as I said above, imediate mode ui does allow one to make applications much quicker, however this does leave a lot of people behind. And not before long, there will be stronger accessibility related laws in the EU and beyond, at which point concerned people might open more issues about this, and there's very little you could actually have done in the past to fix this problem.

That's why I'm opening this now, I want what's best for everyone involved, our community and a lot of people using accessibility tools in general, the consumers of your library because they don't want to have such a risk of legal issues in the future, if they don't already, you because you won't have to deal with absolutely no accessibility on your against imgui list, and such a label would be damaging in the near future, if it's not already.

We don't know how many people didn't use imgui, even if it would have made their lives easier, just because it doesn't have accessibility. It's not just about having the issue yourself either, it's about human decency and empathy, and for example, there's a lot of that in the fediverse, a lot of people from there won't touch a product or library, on the basis of pure principle alone, if it has known hard showstoppers when it comes to accessibility, the same way a lot of people from there don't boost images if they don't have alt text, from principle.

alright, you heard the term screenreader a lot up to now, I'll try to properly explain that, as well as how it actually works. Hint, not like someone thought in that issue:

what's a screenreader anyway?

In the simplest terms, a screenreader is a piece of software which uses diverse output and input methods to convey to a visually impaired person what's happening on their computer. Output methods could be speakers, headphones, braille displays, vibration engines inside controllers perhaps, while inputs could be the keyboard, the touchscreen, etc, which then pass through the screenreader and get slightly modified, for example we have screenreader specific shortcuts which give us even more information about where we are on the desktop, or the touchscreen gestures, such as flicking a finger to the right to go to the next item, or double tapping anywhere on the screen to click a focused object

Did you notice that in that entire definition, I didn't mention the screen or reading it at all? well, that's because a screenreader...doesn't actually read the screen! What it reads, we call this semantic information, enough of that combined and you got what we call an accessibility tree.

An accessibility tree is essentially a hierarchy of components, which logically represents your application. For example, at the root, you may have your window, which has afew properties, such as it's role of window, its bounding box, the location of the top-left corner of the bounding box, the title if any, etc. Underneath that, you have your regular buttons, checkboxes, etc, those also have the same properties as a window, but their role is button or checkbox, they have a state, aka clicked or focused, they have a text or label associated to them, etc. Those we call the leaf nodes, because there's nothing under them, but for example, you could group some of those under a pannel, which then would be a node with however many children, etc. See? this isn't very hard, and it's also how web browsers work, your browser's DOM does that kind of transformation, because javascript has to act on the information you see on your screen in some way, and that kind of design does help with that. Firefox's or chrome's accessibility inspectors could help a lot in understanding this

Now that you know all that information, you may be asking, what would a screenreader see if imgui doesn't support sending those virtual node objects to a virtual channel? Well, nothing, and by that, I mean absolutely nothing, a blank spot on the desktop.

Alright, so how would this be fixed, how was it fixed in the past, surely other people faced this? The way it was done before was horrible and inefficient, and now there's a new way which is a lot better, both of those will be discussed in the following

OS native accessibility, the past and the future

Before recently, the only possible options for making stuff accessible were:

• write it in a toolkit which supports accessibility, gtk, qt, electron, etc are all good options
• implement the accessibility logic yourself, which is undesirable because it's platform-specific and tedious besides
• use text to speech solutions to essentially self-voice every control in your toolkit, making more or less a screenreader inside every app using it. That's also not recommended, because that requires a high level of maintenance, especially when it comes to lots of different ways to access information that the screenreader natively provides, which you would have to duplicate, and at some point, it becomes more complicated than an actual screenreader

So then, a new option appeared relatively recently, it's called accesskit, which I believe would be awesome for integrating with imgui, and since it was specifically made for imediate mode UI, it would be perfect for that. Here's what I know about access kit, all good reasons for which adding it should be considered

• it's cross-platform, working with all platforms it supports behind the adaptor abstraction. This only works with windows, mac OS and linux, but that's very important as well, even ifimgui supports android and such too, plus you'll get accessibility support for those the moment access kit adds them, with minimal disturbance to your existing code
• its architecture is based on imediate UI paradigms, which means you provide it with tree diffs of what changed between the current frame and the previous one, not unlike what you already do with rendering, and it figures out the rest in such a way that it just works with all supported screenreaders and operating systems. This is huge, this means that accessibility is coupled with the visual part of the screen more than ever before, which can fix so many of the existing weird little timing bugs and so on which were present in other UI libraries, because the coupling is much better now. Furthermore, you don't need to keep ui state, no more than what you already tell the user to write, aka in relation to what control is that one, you provide it also when a node should have focus, according to whatever heuristics you have, in that tree diff, same for other actions the screenreader asks to do, and then you react to those as if they were done by the mouse. That's that, for most of your native controls, you only have to stick an extra field in there with the accesskit node that represents, then manipulate your internal state according to what action the screenreader asks. I know it's not that simple, but it's definitely much better than even a year ago, when you had to do stuff like the above, definitely not fun
• it's written in rust, so less memory management issues over all, plus a relatively simpler dependency chain if you just take it vendored like that, though you can also actually integrate it with sources and all. Most important though, it has C bindings, which means it would integrate nicely with imgui. I know, you probably have to wrap it for better c++ usage, but in general it's pretty straight-forward for what it does
• because the screenreader sees that as native ui, the user can read it in the way they configured their screenreader for the rest of the system, which is incredibly useful for people with braille displays or other esoteric forms of output and input
• since the accessibility is OS native, other tools which make use of that would be able to support the user more, for example zoom which uses focus and such, input methods for languages like japanese and chinese, possibly right to left languages as well if there are input methods for that, etc
• some projects which embedded accesskit, as you'll find on its website are: egui, which is a rust alternative to imgui, slint which is another imediate mode UI for rust, as well as possibly others I forgot. So yes, there's prior art, and although the implementation is a bit wonky at the moment, it's better than nothing by miles
• because it manages only the accessibility layer, focus and such stuff is still on you, so you can still use the existing infrastructure to make it work. Furthermore, our screenreaders have something called object navigation, where we can inspect the accessibility tree without moving focus, so we could go, if absolutely needed, in places which aren't reachable by the keyboard. So, as I said, win win for everyone

Finally, the relevant links:

• accesskit website: https://accesskit.dev/
• C bindings, including an example accessible application, which uses sdl to manage focus and such, for a dummy application which is invisible to everything but the accessibility infrastructure: https://github.com/AccessKit/accesskit-c
• python bindings: https://pypi.org/project/accesskit/

So, what do y'all think? could this be revisited again, now that there's a universal framework to make this kind of UI accessible? I really hope so, and would be glad to do the testing to make sure it works, at least on windows, and I know people who can test on linux too.

Screenshots/Video:

Video of a demo using Furnace; a tracker software using ImGUI for it's UI.
https://drive.google.com/file/d/18PyGNQp0SeoIT5y96Xipm_9uX-IOhLOV/view?usp=sharing

Minimal, Complete and Verifiable Example code:

No response