openQA has a well earned reputation as a 'full system' testing tool, able to test a system end-to-end from the operating system to it's applications on a number of different platforms and architectures, including VM's & Bare Metal.

But one area of weakness is it's usefulness as a testing tool for developers or packagers. openQA can easily test a package once it's INSIDE a distribution, but how do you test that package BEFORE submitting it to the distribution?

openSUSE/SLE has the concept of 'staging', where we utilise OBS and openQA to basically create 'what-if' builds's for the whole OS where we can do basic validation tests to ensure a proposed check-in doesn't break the OS or it's basic functionality, but staging doesn't test the package itself

This hackweek project will experiment with solutions to this problem.

Concepts to be investigated include

  • Providing reference disk images for package-specific tests for each OS (SLE, Tumbleweed, Leap)
  • Injecting test code into openQA without needing it to be submitted via github. This might include openQA tests being stored as part of the OBS package.
  • Loading package-specific tests in openQA by feeding openQA variables over its REST API
  • Booting these reference images to either a console or X11 as required by the test
  • Testing and providing the results in a meaningful way to the Developer

I will try to find solutions to the above which will work in the following contexts

  • a Developer working 'packageless' with just a git repo and a local compiler (unlikely, but still want to think about it)
  • a Developer working on a package in their OBS Home/Devel project
  • Release Management/Devel Project reviewers of an OBS submit request wanting to see the results of Package Testing BEFORE deciding whether or not the package can be checked-in or forwarded to staging

Looking for hackers with the skills:

perl openqa obs

This project is part of:

Hack Week 15

Activity

  • about 1 year ago: tjyrinki_suse liked openQA Package Testing
  • about 1 year ago: cdywan liked openQA Package Testing
  • over 3 years ago: tbechtold liked openQA Package Testing
  • over 3 years ago: pgeorgiadis liked openQA Package Testing
  • over 3 years ago: szarate liked openQA Package Testing
  • over 3 years ago: bchou liked openQA Package Testing
  • over 3 years ago: okurz liked openQA Package Testing
  • over 3 years ago: pluskalm liked openQA Package Testing
  • over 3 years ago: osukup liked openQA Package Testing
  • over 3 years ago: diegoakechi liked openQA Package Testing
  • All Activity

    Comments

    • RBrownSUSE
      over 3 years ago by RBrownSUSE | Reply

      Part 2 - Autogenerate X11 tests from Desktop files in packages - including booting the image for the first time, running the app for the first time, and autoneedling for the first time

    • tbechtold
      over 3 years ago by tbechtold | Reply

      Maybe interessting is the system Debian/Ubuntu uses called autopkgtest. See also https://ci.debian.net/

      • pgeorgiadis
        over 3 years ago by pgeorgiadis | Reply

        I watched the relevant talk at FOSDEM and it seems really great the fact that they are running all the upstream testsuites plus the fact that they trigger these testsuites against the package's reverse dependencies... (!) But autopkgtest it's tighten only to Debian, which means we can't just take it and use it as it is. We have to come up with our own equivalent system. But the most important part here is that the developers/maintainer has to take care about these upstream tests, so IMHO there has to be an agreement that the src rpm should include the tests into them, and build service has to have the resources + the knowledge on how to run them; then either reject or forward the update up to openQA.

    Similar Projects

    Design patterns for os-autoinst-distri-opensuse by ybonatakis

    The idea is to hack os-autoinst-distri-opensuse...


    openQA log-viewer firefox plugin by asmorodskyi

    Idea is to write FF plugin which would process ...


    Modernize Mash deployment by seanmarlow

    Mash is a Python based CI/CD pipeline for aut...


    Migrate more OBS service scripts to pure systemd by enavarro_suse

    Following the work started in the last hackweek...


    OBS Project Monitor page redesign by vpereirabr

    Exactly what problem will this solve?

    Th...


    MicroOS Desktop by RBrownSUSE

    [Video Recording of openSUSE Conference sessio...


    distributed build for Ceph in containers by denisok

    Investigate possibilities for the distributed b...