Description:

The network connection to ibs from Beijing is quite slow. And ibs doesn't supply the search function of tags in spec files, like versions, requirements, or patches, etc...

Tasks to complete the projects:

  1. Spider the specs in ibs.
    • Already implemented a spider of another web via Python, so plan to choose golang without any 3rd party framework to fetch. To familiar with goroutine for parallel work, though threads may more effective since network should be the bottleneck so far...
  2. Save the results into database with tags like projects, package name, etc...
    • Not decided yet which database to pick, or even sql/nosql model... Current idea is mysql/memcached or redis, depends on how to compose the tables/key of database.
  3. Analyse and polish the code to make it more fast.
    • Try to make the local engineer become high availability cluster, low latency and higher throughput. Using Galera or Pacemaker for availability? nginx or haproxy for load balance? In memory database for accelerating the read? No idea at the moment, should depends on the bottleneck and the needs. Though a simple engineer for internal won't require much, but may learn more while improving/thinking about the questions.
  4. Check and compare the performance like TPS...
  5. A nice web page to show the result?
    • It is not the 1st priority task of me... let's see in future:)

The target of the project are:

  1. Learning and familiar with golang. Currently i have nearly 0 experience on golang for a real project:)
  2. Learn more while constantly questioning the system.
  3. Have fun.

I don't think the project can be finished within 1 week. Hopefully i can at least get something after that and continue the task when have spare time. First to do it and have fun:)


Comments

  • wanghaisu
    about 2 years ago by wanghaisu | Reply

    Seems sub-list of mark down is not supported... at least i failed:(

    • wanghaisu
      about 2 years ago by wanghaisu | Reply

      works but not good as expect:(

      • hennevogel
        about 2 years ago by hennevogel | Reply

        Patches welcome add-emoji

        • wanghaisu
          about 2 years ago by wanghaisu | Reply

          I found it is a bug of redcarpet 3.4.0 which hackweek use, refer to issue redcarpet doesn't release a new version for more than 1 year:( Will see a better solution.

          • wanghaisu
            about 2 years ago by wanghaisu | Reply

            Solution for example like an extend markdown generater for CommonMark, which is GitHub flavored at least. Anyway, i need to setup a hackweek locally to do a real test before any submission:)

Similar Projects

This project is one of its kind!