Testing GUIs is hard: unit tests require a tremendous amount of mocking and often don't capture the exact user input anyway, integration tests on the other hand are difficult to setup as they often require a dedicated infrastructure with virtual machines.

libuitest should become something like the image recognition part of openQA: it receives a video or image stream from some source and searches for matches (e.g. button found, sound played). It should also be able to send clicks, key presses, strings, etc. back via an abstract API to a backend.

It should be compatible to openQA's needles, but the main way how to write tests should be done via a more generic API: `

help_menu = find_menu("Help", search_area_percent=((0, 0), (100, 0)))

help_menu.click()

about_my_program_button = find_menu("About Foo", search_area_percent=(help_menu.location, (100, 0)))

about_my_program_button.click()

window = find_window("About Foo")

about_foo_text = get_text_from_area(window.area)

assert(about_foo_text == f"This is Foo version {env.FOO_VERSION}")

The image recognition part should be handled by OpenCV and the ocr via tesseract. It would be also nice to use some simple machine learning so that libuitest could be trained to find buttons, menus, windows, etc. independently of the used GUI and theme (or at least robustly if you tell it which theme & GUI you are using).

Looking for hackers with the skills:

machinelearning testing image_recognition opencv tesseract

This project is part of:

Hack Week 19

Activity

  • 5 months ago: Pharaoh_Atem liked libuitest - a generic GUI testing library
  • 5 months ago: mwilck liked libuitest - a generic GUI testing library
  • 5 months ago: dancermak added keyword "machinelearning" to libuitest - a generic GUI testing library
  • 5 months ago: dancermak added keyword "testing" to libuitest - a generic GUI testing library
  • 5 months ago: dancermak added keyword "image_recognition" to libuitest - a generic GUI testing library
  • 5 months ago: dancermak added keyword "opencv" to libuitest - a generic GUI testing library
  • 5 months ago: dancermak added keyword "tesseract" to libuitest - a generic GUI testing library
  • 5 months ago: dancermak started libuitest - a generic GUI testing library
  • 5 months ago: dancermak originated libuitest - a generic GUI testing library

  • Comments

    Be the first to comment!

    Similar Projects

    Artificial Intelligence playground for Data Scientist by afesta

    Project here: https://confluence.suse.com/displ...


    Packaging libnvidia-containers and nvidia-container-runtime-hook by jordimassaguerpla

    This is a follow up to https://hackweek.suse.co...


    Finish Machine Learning introductory course by dmacvicar

    I started [Standford's machine learning course]...


    Colorizing old images on my NAS (or run machine learning deoldify algorithm on an edge TPU) by jordimassaguerpla

    For my 40th birthday I got from my friends a ve...


    Testing GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the rocket chat channel! [https://chat.su...


    ethtool ops for netdevsim by mkubecek

    This can be seen as a subproject of [ethtool ne...


    labgrid: add support for sispmctl and remote ykush access by mbrugger

    labgrid [0] is an embedded board control python...