Motivation / Problem

Right now, our QA team tests YaST with OpenQA very much based on keyboard shortcuts: Set the keyboard focus to the "User Name" input field with Alt-U, enter a user name, [Tab] to get to the next field, enter more text, finally Alt-N to activate the "Next" button.

This works quite well in general, but it happens that those keyboard shortcuts change when some other widget is added to the dialog that might compete with one of those keyboard shortcuts that is already taken: An input field "Name" might request the same Alt-N keyboard shortcut as the "Next" button. This might also happen only under certain circumstances, for example on a per-product basis where some product needs more user input and thus more widgets.

The YaST UI always had an automated keyboard shortcut conflict resolver to sort this out so within the same dialog, those shortcuts are always unique. But that might mean that the keyboard shortcut that an OpenQA test case relies on now does something different, thus making the test fail (sometimes even in very subtle ways that are hard to debug).

For one thing, that keyboard shortcut conflict resolver is a very useful thing to have for the user, so we wouldn't like to give it up for the sake of easier testing; for another, the behaviour would still change (albeit in slightly different ways) when we would deactivate that conflict resolver.

Approach for a Solution

So here is another approach: Use the widget IDs that YaST already uses internally and set the keyboard focus to a widget with that ID or activate a button with that ID: For example, the "Next" button internally uses an ID "next", "Name" typically uses "name" etc., independent of the target language the user chose for the user interface (in German, it would be a "Weiter" button, but it still uses "next" as its ID).

Basically, it would work like this:

  • Press a special key combination to prepare for sending a widget ID; say, Ctrl-Alt-Shift-I
  • Send the widget ID: "next", then [Return]
  • YaST activates a widget if it's a button or a similar widget, or just sets the keyboard focus to it otherwise

Feature Request

Fate #324098

Affected Packages

Further Reading

YaST UI

UI Examples

YaST UI Examples

(Ruby examples despite the misleading package name)

Check out that repo, go to the examples/ subdirectory and start an example like this:

/usr/lib/YaST2/bin/y2base ./HelloWorld.rb qt

(for the Qt (graphical) UI)

/usr/lib/YaST2/bin/y2base ./HelloWorld.rb ncurses

(for the NCurses (text based) UI)

Most comprehensive example: Events.rb

/usr/lib/YaST2/bin/y2base ./Events.rb qt

Watch the YaST log in another shell window with

tail -F ~/.y2log

C++ UI Examples

https://github.com/libyui/libyui/tree/master/examples

Looking for mad skills in:

yastui openqa libyui-ncurses libyui-qt

This project is part of:

Hack Week 16

Activity

  • almost 2 years ago: riafarov liked Make YaST Testing Independent of Keyboard Shortcuts
  • almost 2 years ago: ancorgs liked Make YaST Testing Independent of Keyboard Shortcuts
  • almost 2 years ago: okurz liked Make YaST Testing Independent of Keyboard Shortcuts
  • almost 2 years ago: teclator liked Make YaST Testing Independent of Keyboard Shortcuts
  • almost 2 years ago: shundhammer liked Make YaST Testing Independent of Keyboard Shortcuts
  • Show History

    Comments

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      Implementation Ideas

      The Simplistic Approach: Use a Pop-Up Dialog

      Extend the key event handler in the Qt UI so it catches the special key combination (Ctrl-Shift-Alt-I or whatever), then open a simple Qt pop-up dialog to let the user enter the widget ID and rely on the standard Qt behaviour that [Return] closes that dialog.

      Pro

      • Simple
      • Easy to test
      • It's obvious what's wrong when the process gets stuck during testing - there is still an open pop-up on the screen

      Con

      • The pop-up will easily (and very likely) get in the way of OpenQA's needles (screenshots); OpenQA would have to make sure the pop-up is closed and the screen redrawn to show the original dialog (the dialog under test) so the current screenshot can be compared against the expected "needle".

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      The Behind-the-Scenes Approach: Send Widget IDs via a Named Pipe (or Socket)

      This would create a named pipe in, say, /tmp/YaST/widget-ids where the test application (OpenQA) would write the widget ID, and the YaST UI would read from that named pipe in parallel to keyboard and mouse input (Qt does provide this with the QSocketNotifier class).

      Pro

      • No pop-up dialog that might get in the way; the screen (the dialog under test) remains unchanged.

      Con

      • Synchronisation with keyboard input will become a problem; if the test application fills the keyboard buffer very quickly with some widget IDs sent to that named pipe in between. When should the widget IDs be used? There will be race conditions for sure.
      • Security implications: We have to make sure that this named pipe does not open a security hole. It must be writable by root only, and YaST has to check for that and not use it if it is not. Also, it needs to be in a separate directory below /tmp/ because of the sticky bit, and that subdirectory also needs to be accessible only by root. Remember to double-check with the security team.

      TO DO

      • Figure out who creates the directory and the named pipe and when
      • Find a strategy what should happen if there are multiple instances of YaST running; maybe also put a YaST-pid file there and check if that process is still alive? (send a signal #0 to that process)
      • Clean up after YaST / the test ends

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      The Combined Approach: Key Combo and a Named Pipe

      Use the keyboard to tell YaST to read one widget ID from a named pipe: Ctrl-Shift-Alt-I, then write the widget ID to the named pipe followed by a newline.

      Pro

      • Also no pop-up dialog that might get in the way
      • No synchronization problems with keyboard buffer vs. named pipe

      Con

      • More complicated
      • Need clear logging (y2log) to make it possible to debug situations where a test gets stuck somewhere in between: log something like "received Ctrl-Shift-Alt-I; reading /tmp/YaST/ids", and when finished "Read widget ID 'next' from /tmp/YaST/ids".
      • Same security implications as with the named pipe approach

      TO DO

      Same as with the named pipe approach

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      Needed Tools in the Ecosystem

      • The YDialogSpy needs to be extended to always show a widget's ID, preferably as the first item under its properties (even though it's not strictly speaking a property).
      • Maybe add another special key combination that pops up a dialog showing only the widgets with IDs (i.e. not the complete widget tree, not all the layouts and spacings and alignment widgets) along with their "debug labels"

    • lslezak
      almost 2 years ago by lslezak | Reply

      Interesting idea, but it solves only half of the usual openQA issue. One problem is sending the input, the other problem is checking the output (the UI state). I have proposed another solution (REST API) here. It additionally solves the other problem.

      Moreover I already have a working proof of concept with HTTP + JSON (read-only so far, just dumps the current dialog in JSON response).

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      @lslezak: I discussed several approaches with people from our QA team in the past, and they seem to be very reluctant to use things that they consider too intrusive; very much like "don't fratzernize with the enemy" (the application under test). I can understand those reservations, yet I also see the benefits of checking widget content on the logical level rather than just comparing pixels in screenshots. Both have their pros and cons.

      The approach in this project might be considerably easier for them to accept since it's not that far away from what they already do: Send key events and then compare screenshots. Only this time it's no longer just plain and dumb key events, they come with some semantics: The widget ID, not just a keyboard shortcut that can (and does) change for lots of reasons.

      IMHO we can easily and we should do both: Giving QA the feature to send widget IDs and keep using their current way of testing with screenshots, and in parallel offer deeper introspection with checking actual internal widget state. Both have their merits, depending on what exactly a test wants to achieve.

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      Seel also the reladed project here.

    • lslezak
      almost 2 years ago by lslezak | Reply

      I see, this project is simpler so it might be easier to accept and integrate it into the current openQA. We can implement both and see how it evolves later...

    • okurz
      almost 2 years ago by okurz | Reply

      I think a special keyboard shortcut + widget id is a nice approach for a potentially problematic button so I am looking forward to the results of this hackweek project. I doubt we want to use it in all occasions though as it will be slower (more keypresses to send) and less obvious but it could be more robust of course.

      Regarding "don't fratzernize with the enemy" I would rather phrase it differently: Test as the user does. What we do with openQA is using exactly the interface that the user has available and will most likely use. That is for example shortcuts where it makes sense, e.g. I assume everybody can agree that "alt-n" for the next button is something that should be expected to work. In other cases we can click on buttons identified by "needle matching" where multiple needles can be created and it's easier to maintain then changing screenshots by versions/products/distributions. We can also do send_key_until_needlematch, e.g. "press tab until you reach <button>", where "<button>" is also identified by needles. Pressing a shortcut is just the fastest way because it is just one specific key press rather than comparing screen and then clicking or hitting keys multiple times until the right button is reached. I think all examples where we had this problem of changing hotkeys so far were mainly because of the following problems: Test writers do not know that YaST has this (awesome) hotkey conflict resolver. So we just need shundhammer to help us understand how it's working and he is doing a great job in doing so already :)

    • coolo
      almost 2 years ago by coolo | Reply

      I don't see how a named pipe could work at all - openQA does not run in the SUT. openQA accesses the system remotely - through VNC or doing a ssh installation.

      What can work is a special spy for special cases - <hotkey> to create the popup, enter something (I think QA engineers would prefer the label text instead of the widget ID) and then popup closes and action is triggered. This wouldn't interfere with screen matching at all.

    • coolo
      almost 2 years ago by coolo | Reply

      I don't see how a named pipe could work at all - openQA does not run in the SUT. openQA accesses the system remotely - through VNC or doing a ssh installation.

      What can work is a special spy for special cases - <hotkey> to create the popup, enter something (I think QA engineers would prefer the label text instead of the widget ID) and then popup closes and action is triggered. This wouldn't interfere with screen matching at all.

      • shundhammer
        almost 2 years ago by shundhammer | Reply

        Coolo, good point; I hadn't thought about that.

        That would leaves us with an alternative approach I had considered, but I wanted to keep it simple: Use a socket. For testing, nc (netcat) would do just nicely to feed it.

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      Since I had intended to try the popup solution first anyway as a very simple proof of concept, this would be one version, maybe even one to stay as an alternative to a socket connection. Both would feed the same internal queue anyway.

      As for widget labels: This is also very fragile; they might change due to proofreading or changed product manager preferences about wording, so widget IDs are definitely the prefered way. But as outlined here, we need some better tools to get the widget IDs; one is always displaying them in the YDialogSpy. A popup-based solution might use a combo box to provide some suggestions which should be obvious enough, maybe even with part of the widget label or whatever; we'll need to play with that to get an impression what it feels like.

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      Result

      Implemented the popup-based version for both the Qt and the NCurses UI. So far, this seems to work well.

    • riafarov
      almost 2 years ago by riafarov | Reply

      Conclusion and future steps

      Defined goal was reached, pop-up approached was implemented and integrated with openqa to prove the concept. Here is link to the branch with changes: os-autoinst-distri-opensuse During the project I've learned more about yast gui internals. This functionality can be used in the tests with constantly changing keyboard hotkeys in first place.

      After seeing this possibility to have more stable tests using yast gui internals, we can use advantage of it and stop relying on needles and keyboard shortcuts. After that I was introduced to YCP functionality which is there for a long time and already has a recorder and a player. Documentation for the Yast Programming language can be found here. This functionality can be used for testing and already contains the functionality to read widget properties and set them, as well as interact with them simulating user input.

      YCP macro can be executed when application is started by simply calling: /usr/lib/YaST2/bin/y2base ./HelloWorld.rb qt --macro macro.ycp Or, it can be also called during the run using ctrl-alt-shift-p key combination and then selecting ycp file.

      This part makes it possible to operate on lower level and hence make automated tests more stable and scalable.

      Pros

      • Usage of widget IDs allows to write tests which are resistant to fonts and layout changes
      • Usage of YCP allows to avoid syncronization issues (when some control is not yet shown, but we try to act on it)
      • YCP allows to avoid send keys problems we experience in openQA
      • Reduced execution time as we are able to set controls property to certain value instead of typing
      • Macro recorder allows to partly reduce test code development
      • Tests developed in this way can be easily executed by developers and significantly reduce feedback cycle
      • Can be combined with needle matching approach
      • Easy migration to the solution (e.g. only newly or unstable developed tests are developed using YCP)

      Cons

      • Learning curve for test developers
      • Missing functionality and requires further development (e.g. not possible to verify values in the table or select row with certain cell)
      • Unstable behavior if used with installer (unfortunately due to some bugs installer may crash during macro execution, some trivial actions like pressing button may not work if change installers page, etc.)
      • Missing openQA integration (even though it's easy to run such a test, and log errors in y2log file, the results won't be parsed and displayed in friendly way on the dashboard)
      • Works only with Yast GUI, so usage scope is limited

      All the disadvantages are mainly caused by the fact that YCP is not widely used, but that can be solved with reasonable time investment. In order to prove benefits of the solution we could implement one of unstable tests using YCP and run them in parallel to calculate time spent on maintenance of both solutions.

    • shundhammer
      almost 2 years ago by shundhammer | Reply

      Packages

      Popup-based solution implemented when pressing Shift-F6 in these packages:

      • libyui-3.4.0
      • libyui-qt-2.48.0
      • libyui-ncurses-2.49.0