Software Collections Daemons Made System-wide

This post is about an idea that has been here for some time already, but I'd like to get some more feedback especially from Software Collections users. So, if you're familiar with this term, read on.

 

Familiar with Software Collections concept?

Shortly, the Software Collections (SCL) concept is a really cool technology, that is especially usable on RPM-based Linux distributions that deliver one version of a component and support it for long time, like Red Hat Enterprise Linux or CentOS. SCL allow you to install a newer (or just different) version of any component without affecting the rest of the system (which is crucial).

That all is done by installing files into separate directories that don't collide with the base system. For Software Collection called rh-mariadb101 (that delivers MariaDB 10.1 Server), the files are located at:

• /opt/rh/rh-mariadb101/root/usr/bin (binaries)
• /etc/opt/rh/rh-mariadb101/my.cnf.d (config files)
• /var/opt/rh/rh-mariadb101/lib/mysql (data)
• /var/opt/rh/rh-mariadb101/log/mariadb (error log file)

Why this is not enough?

This all has worked very well for several years and people love SCL. But even the beloved ones have some issues. And what we hear from users, the issues with Software Collections concept currently are basically those:

• we need to use scl enable, which changes $PATH and other environment variables, so the binaries placed in different location are visible by the system
• scripts originally written for "normal" MySQL use full paths (like /usr/bin/mysql) which does not work when we only have the Software Collection installed
• Data directory, config files and log files are on different location than it is common

So, we tried to look at the issues above and tried to come up with a solution. An easy take would be not using Software Collections concept, but that would effectively mean to break something on the base system (either we would update some dependency, we would force users remove client library because there are some shared files with the server, and probably many more that I'm even not aware about). That means using Software Collections is still valid, we just need to remove the obstacles put by the alternative path used.

 

What to do with that?

The approach I prefer is based on two changes:

• have a symlink on usual places in /etc/ and /var to make it easier for users to find the stuff they're looking for
• have a wrapper for every binary, the wrapper is located in /usr/bin, so no scl enable is needed; the wrapper runs scl enable internally and then executes the binary located under /opt/rh/rh-mariadb101/root/usr/bin

The wrapper is quite easy and can be shared (symlinks then head to the wrapper); the wrapper can be then located in /opt/rh/rh-mariadb101/root/usr/libexec/scl_wrapper:

#!/usr/bin/bash
source scl_source enable @SCL@
exec `basename "$0"` "$@"

For example mysql tool is then available by having such a link:

/usr/bin/mysql -> /opt/rh/rh-mariadb101/root/usr/libexec/scl_wrapper

Both, the wrappers and the symlinks, are available in a separate sub-package, so just having them available does not change anything for currently working solutions, which I really like about this approach honestly.

It means only users who would like to use the Software Collection packages as normal components will be allowed to install a rh-mariadb101-mariadb-system-widesub-packge (cool name, huh?). Those users will need to accept the fact that they cannot install mariadb and mariadb-server packages from the base system any more, because those will obviously conflict with the rh-mariadb101-mariadb-system-wide package.

 

Does it really work?

A working solution is also available in my testing Copr repository:

https://copr.fedorainfracloud.org/coprs/hhorak/mariadb-wrapper/

For dependencies, you need either Red Hat Software Collection channel enabled on RHEL-7, or use the packages from the SCLo SIG group on CentOS 7. See more info at:
https://www.softwarecollections.org/en/scls/rhscl/rh-mariadb101/

That Copr will show you, that after installing the package rh-mariadb101-mariadb-system-wide, you can start doing things like the MariaDB 10.1 Software Collection would be normal package:

#> service mariadb start
#> service mariadb status
#> echo "select @@version;" | mysql
#> mysql_upgrade --force
#> less /etc/rh-mariadb101-my.cnf
#> less /etc/rh-mariadb101-my.cnf.d/mariadb-server.cnf
#> ls /var/lib/rh-mariadb101/

Feedback required!

Now, the most important thing -- the reason I shared this post here.. I want to hear feedback from current Software Collections users, whether this idea helps to overcome the Software Collection design issues and whether there are some more ideas.

For feedback, use ideally our mailing list sclorg@redhat.com.

How we've fixed upgrade path for CentOS 6 Software Collections

Short meesage for those who don't have time:
Software Collectoins for CentOS 6 are ready for upgrade from older rebuilds.

Now full story for those who care. Btw. this all is related to work done by SCLo SIG group that is part of CentOS (read more at http://wiki.centos.org/SpecialInterestGroup/SCLo).

A bit of history for beginning. Shortly after first RHSCL 1.0 release, CentOS rebuilds were prepared and since then they are available under:
http://mirror.centos.org/centos/6/SCL/x86_64/

However, keeping these rebuilds in sync with RHSCL content hasn't been easy task. With introduction of Java packages in collections, this task became even more tricky, which means these collections were not updated for long time. With that said, someone would expect there won't be problem with upgrade path, in other words that the new RPMs, that the SCLo SIG group is about to release, will update the older RPMs smoothly.

Well, not always. The original RPMs used ".el6.centos.alt" as %dist tag, while new builds use just ".el6" and that evolves in cases where python27-python-bson-2.5.2-4.el6.centos.alt.x86_64 > python27-python-bson-2.5.2-4.el6.x86_64, even if those packages have same Release tag in RPM SPEC. That obviously means the packages won't update smoothly.

Solution is quite simple in this case -- use higher Release in RPM SPEC. In some packages, this was already done, because some of the packages received update since original inclusion. In other cases we solve it by adding ".scX" (X is number) suffix to the Release tag. The ".scX" was chosen deliberately since ".scX.el6" is higher (alphabetically) than .el6.

Btw. for cases we need to build package more times before final build (bootstraping), we use suffix ".bsX", which means we can build package without any Release suffix in the end, because ".bsX.el6" < ".el6".

Anyway, this post was meant to let you know that upgrading of el6 packages from originally built RPMs is something we care about.

To verify it works, I've installed all the packages from original repository, then ran "yum update" and that evolved in proper update of all packages. I took that as proof it should work fine in your case as well. If there are still some issues, let us know.

Enjoy Software Collections on CentOS!

Happy New Year Fedora and what to do when upgrading to Heisenbug doesn't work as supposed

The old year 2013 was about to end and I realized I didn't want to start he new one with an old Fedora. Having Fedora 19 still on my personal and also on my working computer, I began. These are my hints and experiences.

My fails but finally successful with fedup

First, on my personal computer, I used dnf (a new, much faster successor-to-be of yum; I recommend it to try it out) to update all packages to the latest F19 version and then ran fedup utility with "--network 20 --nogpgcheck" options. Turning-off GPG checking was the fastest way how to work-around some signature issues. fedup reported some un-available repositories, so I just disabled those for the sake of upgrade.

After downloading packages and installing new fedup drucut I restarted computer and it proceeded without problems. And it still works fine.

So, to sum it up, on my personal computer it went very well, only some un-available repositories and GPG issues were encountered during the whole upgrade, but those were easily worked-around by using appropriate options.

What to do if fedup is not an option

The day after I proceeded with the same steps on my working computer, which is Lenovo T510. I saw the same issues with un-available repositories but not GPG issues any more. So I disabled the repositories and restarted computer after all packages had been downloaded, the same as I did on my personal comuputer.

When fedup dracut started I saw kernel panic after 7s (similar to RHBZ #873459, but I can't reproduce anymore, so I'll try it again on one of my virtual machines -- well quite paradox that I got kind of a heisenbug in a process of getting Fedora Heisenbug) so basically in a very early state. What is more, the same happened when I tried to boot with the newest kernel-3.12.x. But I was still able to boot with older kernel-3.11.x. So I wasn't screwed totally, but I was not able to finish fedup upgrade even repeatedly.

My solution was to go on with another way of upgrading -- upgrading using yum. I put dnf aside, since I wasn't sure if it is able to deal with upgrade (now I regret that I didn't try it) and using distribution-synchronisation option it went very well. After some hours (yeah, I have too many packages) I ran restorecon on whole root to be sure SELinux will work fine and rebooted. A bit surprised, I saw a new Fedora Heisenbug successfully booted and didn't noticed any issues.

Now I'm happy I could welcome the New Year with the bleeding-edge features that are served by Fedora 20 Heisenbug and I would like to wish all (not only) Fedora maintainers to have many solved issues, more interesting challenges and a lot satisfied users.

Guys, thank you all for your contribution in 2013!