I would like to change the way "quilt setup" is implemented.

At the moment, we call rpmbuild and intercept the calls to tar and patch in order to record the location where archives are extracted and the order and options of the patches which apply to them. Then we replay that record to create our own quilt-compatible source tree.

While this works good enough in simple cases, there are two drawbacks:

  • We duplicate archive extraction and patch application, which incurs a performance penalty.
  • We miss extra commands from the spec file, so the source tree we produce is not exactly what is specified in the spec file.

What I would like to do is keep intercepting calls to patch, but instead of recording them for later use, I'd like to replace them on the fly by quilt import and quilt push. That way rpmbuild will create a quilt-compatible source tree, which we can just copy over to the final location.

There will certainly be a few issue to solve on the way but I can't foresee any blocker. I hope I'm not overseeing a big obvious and unsolvable problem. If we can get there then I expect "quilt setup" to be better and faster.

Looking for mad skills in:

bash rpmbuild

This project is part of:

Hack Week 11

Activity

  • almost 5 years ago: jdelvare removed keyword perl from Reimplementation of "quilt setup"
  • almost 5 years ago: rwill liked Reimplementation of "quilt setup"
  • almost 5 years ago: cxiong liked Reimplementation of "quilt setup"
  • almost 5 years ago: sleep_walker liked Reimplementation of "quilt setup"
  • almost 5 years ago: matejcik joined Reimplementation of "quilt setup"
  • Show History

    Comments

    • matejcik
      almost 5 years ago by matejcik | Reply

      I was thinking of implementing quilt-like functionality on top of git. This might help my goals as well, so for now I'm joining your project ;)

      • jdelvare
        almost 5 years ago by jdelvare | Reply

        Jan, at least two quilt-like git-based implementations exist, named stgit and guilt. You should probably give them a try if you are interested in this topic.

    • rwill
      almost 5 years ago by rwill | Reply

      If you need a "non-trivial" test object, you may want to consider 'grub2', which usually fails to 'setup' due to '%if ! 0%{?efi}' or some such... (c:

      • jdelvare
        almost 5 years ago by jdelvare | Reply

        Thanks for the pointer, Raymund. I'll make sure to test my new code on the grub2 spec file.

      • jdelvare
        almost 5 years ago by jdelvare | Reply

        The problem with the grub2 spec file is that it uses a syntax which older versions of rpmbuild do not recognize as valid. This has nothing to do with "quilt setup". But it's easy enough to fix the spec file itself as far as I can see, so let's just do that.

    • jdelvare
      almost 5 years ago by jdelvare | Reply

      The new backend for "quilt setup" is working, and the performance improvement is very nice. For example, the kernel-default package (which was the slowest one I was aware of) took 1 min 48 s to setup before, while with the new --fast option, it only takes 25 s. So I think the project can be called a success :-)

      There are a few technical implementation details which still need to be sorted out, I'll discuss that with upstream. There are some possible improvements left on my to-do list as well.

      It took a few tries to get things right. For example, I originally wanted to get rid of the md5sum step altogether, before I remembered that patches are typically passed to us through stdin, so we still need a checksum of all patch files in order to resolve the contents to a filename. I was still able to save time by skipping md5sum on archive files.

      Another problem I hit is that my original implementation used "quilt import" to add each patch to the series file. This created a copy of each patch, instead of linking back to the original files in the source directory. That made it impossible to refresh the patches directly as "quilt setup" currently allows. I first worked around that by deleting the copies and linking back afterward, but that was both inefficient and fragile. The proper fix was to manually add the file to the series file instead of calling "quilt import". That required some work so that all the paths were correct both when rpmbuild is applying the patches and later when the working tree is available to the user. The trickiest part was to make it all work also when options -d and/or --sourcedir are used.

      Finally, contrary to my original intention, "quilt push" isn't called on every patch. This turned out to be a bad idea from a performance point of view, because individual calls to "quilt push" are much slower than a single call to "quilt push -a". So, it is still up to the user to apply the patches with "quilt push", same as before.

      Two user-visible differences exist between the original implementation of "quilt setup" and the new, faster one:

      • Headers in the generated series file are incomplete. This prevents reusing the series file for a future call to "quilt setup". I think most users don't care about this feature though, so they won't notice. This could be fixed, but at the price of some of the performance gain, so I don't want to do it by default. Maybe this can be implemented as an option later, but that only really makes sense if we get rid of the original implementation of "quilt setup" (which my patches do not.)
      • Patch failures are no longer reported, the user will see them on "quilt push". A great side effect of this is that all patches are added to the series even if one fails to apply in the middle. So I consider this change a feature, as this avoids the "quilt setup" / "rm -rf" cycle the user had to go through before when some patches do not apply.

      Also a nice side effect for this project is that I came up with a few bug fixes, cleanups, and performance improvement patches for quilt along the way. 6 of them are already upstream and 6 more have been posted for review.

    • jdelvare
      almost 5 years ago by jdelvare | Reply

      The result of my work is packaged at: https://build.opensuse.org/package/show/homeadd-emojibranchesadd-emojitools:scm/quilt

      Just pass --fast to "quilt setup" to use the new code, and enjoy the performance boost. If anything doesn't work as expected, please report to me.

      Note: for complex packages such as the kernel (or any package including patches in archives), you want to use option -d as well, otherwise some of the performance gain is lost. This is a known bug and I have an idea how it can be fixed, but I did not have the time to implement it yet.

    • jdelvare
      almost 5 years ago by jdelvare | Reply

      Patch/RFC was finally posted to the quilt-dev list.

    • jdelvare
      over 4 years ago by jdelvare | Reply

      The quilt setup "fast mode" feature has been committed upstream.

      • jdelvare
        about 4 years ago by jdelvare | Reply

        I have just committed the final pieces of "quilt setup" optimization upstream. It will be faster than ever in v0.65.

    Similar Projects

    This project is one of its kind!