Let's make reposync faster

Every day,

Multiple times a day,

Every SUSE Manager customer,

Every Red Hat Satellite customer,

Every Spacewalk user,

And every Uyuni user...

...spends a lot of CPU and wall clock time in reposyncing.

Intro

A lot of that time is wasted by an old, overcomplicated and most of all inefficient algorithm that contributes heavily on heat dissipation and user patience depletion!

HackWeek hackers, we can change that!

Past attempts only partially succeeded: https://trello.com/c/inl9Wu0p/40-reduce-global-warming, https://trello.com/c/dYAR0J8K/13-reduce-global-warming-take-2

But we have better tools now!

Tooling

py-spy to the rescue: introduction

Install with: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python get-pip.py pip install py-spy

Trace a running spacewalk-repo-sync with: py-spy --nonblocking --pid `ps aux | grep spacewalk-repo-sync | grep -v grep | awk '{print $2}'` --flame output.svg --duration 10

Look at the results with:

python -m SimpleHTTPServer 8666

And point your browser to http://<NAME_HERE>:8666/output.html. Here is one such example:

Flame Graph

Current remarks:

  • we currently spend a lot of time in lookup functions
  • lookup functions SELECT rows at every INSERT
  • this is especially bad for checksums, capabilities and some other cases
  • design comes from Oracle and can probably be changed!

Looking for mad skills in:

python performance databases postgresql

This project is part of:

Hack Week 18

Activity

  • 14 days ago: mkoutny liked Uyuni: improve spacewalk-repo-sync performance
  • 23 days ago: Pharaoh_Atem liked Uyuni: improve spacewalk-repo-sync performance
  • about 1 month ago: PSuarezHernandez liked Uyuni: improve spacewalk-repo-sync performance
  • about 1 month ago: ebischoff liked Uyuni: improve spacewalk-repo-sync performance
  • about 1 month ago: moio added keyword "postgresql" to Uyuni: improve spacewalk-repo-sync performance
  • Show History

    Comments

    Similar Projects

    Run and manage your Ansible cluster using Salt! by PSuarezHernandez

    At SUSE we've implemented a module on Salt call...


    Evaluate mirrormanager (or mirrormanager2) for download.opensuse.org mirror management by lrupp

    As there is no progress around [MIrrorPinky](ht...


    Make "salt-toaster" available to be used outside SUSE by PSuarezHernandez

    The salt-toaster (https://github.com/openSUSE...


    Porting Askbot to Python 3 by rbueker

    During the last year Askbot, a question and ans...