Proposal for location based object relationships

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Proposal for location based object relationships

Leonard de Ruijter-4
Hey All,

It has been possible for years to navigate objects using next, previous, parent and child relation ships. Normally, this offers more than enough functionality. Sometimes however, one might be interested in getting more insight in how objects are ordered on screen. For example, when you navigate by next and previous object over the desktop, you won't be able to see how icons are arranged on screen visually. Another example is the bad accessibility implementation in most Delphi applications, making it almost impossible for blind users to find labels for controls.

I therefore propose adding additional relationships to the NVDAObjects, namely left, right, above and below. In the case of left, for example, this property should either return a tuple of adjacent objects to the left, or the object which is left to the point at x=left,y=bottom-top, or whatever algorithm we choose. Same applies for right, above and below.

I think this functionality will offer benefits for the following cases:

1. Screen review entirely depends on display model information. With location based object relations, it might be possible to offer screen review in many more situations
2. In creating add-ons for applications, finding labels for unlabeled objects which are on screen but not associated to a control
3. Improving experience for braille users who prefer a line based approach (à la line mode in JAWS, physical mode in SuperNova) over an object/structured based mode people will have to use in situations where screen review is unavailable.

The only thing that worries me, is how resource intensive this feature would be. ANy how, all thoughts about this proposal are more than welcome.

kind regards,
Leonard

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nvda-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nvda-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal for location based object relationships

derek riemer
In the case APIs exist for this I like it. However what does left mean exactly? Like should I look outsside the bounding box of this object, but left, so I'd give back everything on screen? left of {(x,y),(x,y2)} or just every item left of {(x,0),(x,BOTTON_OF_SCREEN)}?

On Wed, Apr 19, 2017 at 2:20 AM, Leonard de Ruijter <[hidden email]> wrote:
Hey All,

It has been possible for years to navigate objects using next, previous, parent and child relation ships. Normally, this offers more than enough functionality. Sometimes however, one might be interested in getting more insight in how objects are ordered on screen. For example, when you navigate by next and previous object over the desktop, you won't be able to see how icons are arranged on screen visually. Another example is the bad accessibility implementation in most Delphi applications, making it almost impossible for blind users to find labels for controls.

I therefore propose adding additional relationships to the NVDAObjects, namely left, right, above and below. In the case of left, for example, this property should either return a tuple of adjacent objects to the left, or the object which is left to the point at x=left,y=bottom-top, or whatever algorithm we choose. Same applies for right, above and below.

I think this functionality will offer benefits for the following cases:

1. Screen review entirely depends on display model information. With location based object relations, it might be possible to offer screen review in many more situations
2. In creating add-ons for applications, finding labels for unlabeled objects which are on screen but not associated to a control
3. Improving experience for braille users who prefer a line based approach (à la line mode in JAWS, physical mode in SuperNova) over an object/structured based mode people will have to use in situations where screen review is unavailable.

The only thing that worries me, is how resource intensive this feature would be. ANy how, all thoughts about this proposal are more than welcome.

kind regards,
Leonard

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nvda-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nvda-devel




--

Derek Riemer: Improving the world one byte at a time!

  • University of Colorado Boulder Department of computer science, 4th year undergraduate student.
  • Accessibility enthusiast.
  • Proud user of the NVDA screen reader.
  • Open source enthusiast.
  • Skier.

Personal website



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nvda-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nvda-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal for location based object relationships

James Teh
In reply to this post by Leonard de Ruijter-4
Hey Leonard,

While there are probably valid use cases for spatial navigation, it seems to me that your use cases actually call for correct logical relationships, not necessarily separate spatial relationships.

1. Screen review is inherently spatial because we can't do any better. However, if we're talking about being able to do flat style review with objects (AKA browse mode everywhere), we want to work with logical flow, not necessarily spatial flow. It's true that we do need to know where one line ends and another begins and that's a new concept we'd have to introduce. However, if we have three lists side by side, we don't want to render one item from each list on the same line, for the same reason we don't want to do this on the web: they aren't semantically related, just as columns in a newspaper aren't meant to be read "across". Also, spatial navigation is problematic for RTL languages: it's difficult to know which way separate objects are meant to be read logically. It's better to allow the content dictate that (with the ability to override this where it's clearly broken).

2. Finding labels might need to be done spatially, but they aren't necessarily to the left of the control; they might be above it or even a column header in a table style control. It'd be better to introduce a labelledBy relation (which might fall back to spatial navigation if necessary).

3. The requirements for a line based approach for braille are very similar to those for point 1.

What I'm suggesting is that the core should only ever use logical relationships for flat review, braille, labelling, etc. Spatial navigation might be used by specific implementations, but those implementations will have a clearer idea of when it's appropriate to use this.

To give a concrete example, you mention the labels are in the wrong position in the tree for Delphi apps. I'd argue next and previous should be overridden for Delphi apps to order things better and that labelledBy should be implemented to return the correct label.

All of this gives authors the opportunity to provide the correct logical information where possible while still allowing us to fall back to heuristics. In contrast, if we always use spatial navigation for certain things, we will potentially override correct implementations.

Jamie

On Wed, Apr 19, 2017 at 6:20 PM, Leonard de Ruijter <[hidden email]> wrote:
Hey All,

It has been possible for years to navigate objects using next, previous, parent and child relation ships. Normally, this offers more than enough functionality. Sometimes however, one might be interested in getting more insight in how objects are ordered on screen. For example, when you navigate by next and previous object over the desktop, you won't be able to see how icons are arranged on screen visually. Another example is the bad accessibility implementation in most Delphi applications, making it almost impossible for blind users to find labels for controls.

I therefore propose adding additional relationships to the NVDAObjects, namely left, right, above and below. In the case of left, for example, this property should either return a tuple of adjacent objects to the left, or the object which is left to the point at x=left,y=bottom-top, or whatever algorithm we choose. Same applies for right, above and below.

I think this functionality will offer benefits for the following cases:

1. Screen review entirely depends on display model information. With location based object relations, it might be possible to offer screen review in many more situations
2. In creating add-ons for applications, finding labels for unlabeled objects which are on screen but not associated to a control
3. Improving experience for braille users who prefer a line based approach (à la line mode in JAWS, physical mode in SuperNova) over an object/structured based mode people will have to use in situations where screen review is unavailable.

The only thing that worries me, is how resource intensive this feature would be. ANy how, all thoughts about this proposal are more than welcome.

kind regards,
Leonard

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nvda-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nvda-devel



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nvda-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nvda-devel
Loading...