Few notes from pre-Fosdem CentOS Dojo 2017

As it is already a tradition, Friday before Fosdem there is a meet-up of CentOS folks in Brussels. This time it was funny because we were in the same venue as PgDay was held in. Anyway, it was again great to talk to people and see progress in CentOS ecosystem (I really cannot call it system any more).

How it was with CentOS and Red Hat

The day with two parallel trackes was started by Karanbir, the Mr. CentOS. He talked about recent history, when first talk with people from Red Hat happened, where they talked about joining forces for the first time. That gave a great insight at how the relation looks now, which I think was really important, since especially active contributors need to understand this unique relation.

And what the relation is? Obviously, CentOS wouldn't be here without RHEL, but on the other hand, RHEL benefits from CentOS, which is not a downstream any more. Many projects like Gluster, OpenStack or Software Collections provide place for wider community to develop tools that are simply not good fit for Fedora. We can call it upstream or just community projects, important is that it works.

Then he gave several numbers, when all guesses from audience were mostly wront (lower) -- since numbers about CentOS base image downloads (tens millions) or current size of the mirror (254GB from what 150GB is not upstream RHEL) is simply astonishing.

On the negative part of the present state, CentOS folks (the official guys paid by RH for working FT on CentOS) lack resources, people resources, but with big community it still works amazingly, thanks to collaboration.

Now, what next? It's good to understand, that CentOS still can make easier everything you need to do with CentOS -- may it be modularity related, OpenStack, or NodeJS related. What is for example happening is that NodeJS master branch is tested in CI CentOS to see whether it works correctly on CentOS.

Long term they want to have CI that is available for everybody in opensource, and generally focus beyond CentOS ecosystem.

What EL6 user should pay attention in EL7, CI and really big data in Cern

Next talk by Bert Van Vreckem was more targeted to users of el6, who recently moved or decided to move soon to el7. He presented several commands any admin should know about. Exploring his github repo I found also nice collection of articles about writing readable and kinda safe Bash scripts, so I recommend checking https://t.co/F1JkYaOMQV.

Brian Stinson, the main CentOS CI guy, shared some basic info about current CI abilities, while it's more than awesome that many open-source projects, even outside of CentOS ecosystem can levarage this cool thing . It's especially cool because we do not speak about VMs, but about bare metal machines, that one can connect to and get a unique testing infrastructure for free. For the future we might find more architectures -- currently there is quite big demand for ppc64le and 64bit arm, which seems to be the future of the cloud soon.

The last talk of that day I attended was by David Abdurachmanov from Cern, who shared crazy numbers from their practice. They need to process huge amount of data that are often created in less than a second. 350PB data on disk, 800PB carried over net, 650k intel cores or 300Gbps network, that is the reality that helps humans in exploring the smallest particles in the Universe.

I hope at least slides will be soon available at https://wiki.centos.org/Events/Dojo/Brussels2017, but if not, I guess speakers will be more than happy to share them if you contact them. See you next year!

Notes and thoughts from Developer Conference 2016

This post is mostly intended to remind myself in the future, what DevConf 2016 looked like for me, but maybe somebody else find it useful as well.

Many of the talks and workshops are now uploaded to the youtube, so feel free to find the full recording: https://www.youtube.com/playlist?list=PLjT7F8YwQhr--08z5smEEJ-m6-0aFjgOU

The keynote

Tim Burke started the DevConf 2016 by a keynote full of tips how to become a rock star in open-source world. From "not being a troll", through "not doing what you don't like", to be a real team player, because a rock star is not an individual. Passion was identified as a way how to enjoy your work. See the full keynote at https://www.youtube.com/watch?v=Jjuoj2Hz03A.

Docker versus systemd

Dan Walsh talked about systemd and docker integration. He mentioned pitfalls they have already solved and which they still need to solve. Dan himself staying in the middle of docker upstream and systemd guys, who both don't like accepting compromises. He mentioned these issues:

• sd_notify that should eventually help notifying about app readiness
• logs being grabbed by journald so the logs are available after container is removed
• running systemd in docker container - all PRs closed by docker
• with docker 1.10 it will work with some options specified during container start
• machinectl, working with all VMs running on machine
• eventually libcontainer will be hopefully replaced by its clone runc, from OCSpec

All Flavors of Bundling

Vit talked about practical examples of bundling he met during ruby maintenance work, mentioning bundler not only because it helps bundling stuff, but also because it bundles a lot of stuff. He mentioned that there might be some reasons to bundle, sometimes not. All that was triggered by recent change in fedora guidelines, that start to allow bundling. Vit also went through various specifics in different languages, like javascript, c++, go, ... interesting point about generators, he said that we basically bundle the code because if they make mistake, you get problem.
Q: some example of bundling that succeeded? Bundler project learned that bundling is the correct way and the only way.

Open source distributed systems at Uber by Marek Brysa:

HA is really crucial, because payment transactions and tracks are done by Uber.
More services than transporting people - food, stuff and even kittens
Technologies used: Ubuntu, Debian, docker, python, nodejs, go, kafka, redis, cassandra, hadoop
Every project meant to be open-sources by default, except exceptions
Also contributions
Umber of micro-services grew from 0 to 700 micro-services in last two years
Ringpop:

• consistent hashing for sharding, membership protocol - SWIM membership protocol, using direct and indirect pings to get state of an instance, to prevent random network issues
• Apparatus called gossiping says something about other instances when sending a message
• Infection style of dissemination, currently 1k instances, 2.5k tested, in the future 10k?
• App level middleware

TChannel:

• Soa oriented replacement of http, which turned to be slow in some cases
• Multiplexing, performance, diagnostic, HA forwarding
• JSON and Thrift for serialization

Hyperbahn:

• Service discovery and request forwarding
• Give clients one hyperbahn instance and bootstraping starts automatically
• Services are connected to ring of routers, every service is connected to few routers

CI/CD with Openshift and Jenkins:

It is not only about tools
Containers as cows, we replace one if dies.
Containers make people think about what happens when a container dies
Openshit CI/CD wants to be generalized to pipeline that may be used by other projects
Example of running Jenkins in OpenShift
S2I used as configuration tool
https://github.com/arilivigni/openshift-ci-pipeline  - 3.3 openshift release roadmap

Is it hard to build a docker image?

Tomas asked and had also answer that it is..
Squashing, cache, secrets, adding only (metadata), usage message, evolution is rapid
Conclusion is that docker is young

Remi's PHP 7:

Reason for skipping 6 was existence of books about development version 6
New API brings binary incompatibility for binary extensions
Change in size_t and int64 only on windows
Abstract syntax tree, native TLS, expectations (assert finally usable), throwable interface
Extensions porting process still in the middle, some won't ported at all, MongoDB for instance instead of mongo
Performance increased twice for common applications, comparing number of pages served
Scalar types possible to be defined in functions declaration, strict_types option makes strong typed language from php
We now can catch parse and type errors, with keeping backward compatibility of exceptions still working
Removed extentions, change in expressions containing variable names in other variables
Fedora will eventually drop incompatible extensions
We need scls in fedora, that would be the best thing for php7

Security: Everything is on fire!

Josh Bressers talking about security and whether the situation is really that desperate. It is not yet, but there is work to be done. What is happening is people earn real money on security issues, press makes money on newspaper selling, so they make up things to sound interesting.
Where do we start? Communication is key. Security guys should listen.
Security is not here for solving problems, it is part of the problem.

Centos pipeline meet-up:

Couple of people in one room had an initial chat about existing components that should be usable for CentOS Container Pipeline and decided to use Openshift, which sounds like good way to go because it includes already the missing pieces.

Fedora and RHEL:

Denise was speaking about how fedora is important for Red Hat.
Matt then presented a lot of graphs about downloads stats of fedora from various views. The impression was that it is not that bad.

Changing the releng landscape

Denise Gilmore about releng in fedora:
Koji 2.0 still in the beginning, should be build with copr backend somehow, to allow more flexible builds
Et/bodhi alligment
Rpmdiff and license scanning done internally shoud be done in fedora as well.

Re-thinking Linux Distributions by Langdon:

Rings concept did not work
It was too complicated when working out the details
Modularity should work better
Think about applications, not packages sets
We need minimize dependencies on other applications and on OS
Give separate channel with metadata, that's what rpms were invented for
Atomic app, nulecule, rolekit, xdg-app mentioned as way
E&s is where the definition should take place, not necessarily place to code it
Q: will 10 versions of library do a mess in systems? Let's make computers track that for users

Q&A with council:

included question that cannot be missed in any similar session - fedora and proprietary drivers. Josh mentioned that the problem is not only getting the drivers installed, but also not breaking the whole video once kernel is updated. Everybody understands the politic cannot be easily changed, but atleast the problem with breaking the video might be better soon. Another question questioned matt's graphs, there was a question about possible kerberos inclusion instead of generating certificates on server, where there is btw a privat key, which doesn't belong there. Generally the session was very positive.

Closing quiz:

The last session, the quiz, which full room was participating in, was funny and interesting end of the conference.

CentOS Dojo 2016 Notes after lunch

This is a second part of my personal notes from this year's CentOS Dojo before Fosdem, first part includes notes from talks until lunch. So, after lunch, we returned to our chairs and heard these interesting talks.

4. Quickstart. Contributing packages to a CentOS Special Interest Group

Brian talked about basics in centos build system, how to jojn SIGs and how to build a package in CBS for a particular SIG. Really a "must seen" for every newbie in centos SIG.

5. Path from Software Collections to Containers for OpenShift

My talk about experiences with creating Containers for Open Shift included two dozens of tips from various fields. We closely looked how to create a nice, Open Shift friendly container image (yes, it was about Docker) for PostgreSQL and Python. These two examples covered the most important information that one needs to create any similar database or application builder image. Later I went quickly through list of images that are already out there, made by Red Hat or CentOS and that are based on Software Collections packages. In the end I shortly introduced the concept of Nulecule and what this project is intended for.

6. Getting started with kubernetes

Kubernetes was described the same as other orchestration systems, even condor which development started already in 1987. What makes the technologies different from the PoV of potential developer folks is language chosen to be written in.
Mesos was secretly influenced by Borg, a group run by Google. It means guys creating k8s know what they do, because we may see k8s as a new version of Mesos.
Basics of Kubernetes explained clearly and on simple examples -- what pods, services, replication controller do.
Atomic was presented as the solution to use k8s on CentOS.
For learning k8s use gh.c/skippbox/...8s
Terraform plan for deploying k8s on AWS with atomic host and flannel.
A demo showed automatically created k8s nodes and let them scale in AWS.

7. Atomic Developer Bundle - Containerized development made easy

Atomic developer bundle, guys showed why there is something like ABD, stating problems devels face today during application development, all on real user stories. They showed Vagrant devel environment, running docker secured by TLS, user being able to connect from host machine by evaling 'vagrant adbinfo' output that defines devel environment on host.
Another example showed Eclipse running on the host, connecting to remote docker, which is a scenario that might work from any OS, even windows. Although the demo did not work and we could see the live-demo Murphy's law in practice again, we got the point and I'm sure it worked fine just before the talk.
ADB supports the OpenShift and other orchestrations technologies as well.
Why centos? Because of community, that might give the needed feedback. In the end the list of available links were mentioned and community was called to action.
The future is so bright, I gotta wear shades.
Architecture is still a thing to be changed, they plan to make vagrantfiles easier.
Landrush does not work and some help is needed..
In the end guys tried the demo again but with poor Internet connectivity and Murphy's law working better, we saw only one step further.

Description of the talks and hopefully soon also slides and recording available at: https://wiki.centos.org/Events/Dojo/Brussels2016

CentOS Dojo 2016 Notes until lunch

This post is meant mostly as notes from the CentOS Dojo before Fosdem 2016, but maybe someone else finds it useful as well.

1. State of the CentOS Project

Well, the Dojo didn't begin very well, my phone decided to turn off during night and since I arrived after midnight, I easily overslept and thus missed the first talk, where Karanbir talked about where CentOS is today, so hopefully I'll see it from recording.

2. Relax-and-Recover simplifies Linux Disaster Recovery

Rear presentation about "relax and recovery" solution, recently also included in RHEL 7, presented by its author, Gratien, who supports these tools for living. It allows to solve recovery scenarios easily, but it is not a backup solution. Live demo showed a recovery on a virtual machine in less than 3 minutes. Interesting stuff even for people without admin experiences.

3. Desktop security, keeping the keys to the castle safe

Michel Scherer talked about security treats of various types, from stealing a computer and putting its RAM into a different computer (coldboot), through stealing a password by various ways to firewire DMA attack.

Big portion of the talk was about protecting the operating system, while many tips were given to protect various specific things. Phishing, password managers, firewall and other technologies were described from interesting point of view, mostly wrapped by a statement that they must be used properly to work properly.

What surprised me was that virus scanners were found insecure themselves, because all tested scanners could be cracked by a file send to be scanned and the fact that they usually run with pretty big privileges makes them quite dangerous.

From desktop world, few technologies were mentioned, but most focus was given to browsers. Chrome mentioned as good at some points like separating processes, but generally taken as proprietary thing by Michael, so not very good from security PoV. Firefox, better integrated alternative, seems to be better alternative for those who believe Mozzila Foundation, as Michael does, but with keeping some rules, like removing Flash, not only disabling it. Same for Java, except where really necessary. No Javascript with noscript module, which makes web faster, but also often broken.
Remove CAs not trusted.

Think about privacy in connection of surveilance. Adblock and cookiemaster, maybe even using Tor or trail...

Local attacks mean a need to protect the laptop from not only colleages, by screensaver with password, not leaving root shell opened, use credential expiration, disable ptrace by SElinux. Use password on SSH keys, use smartcards to store keys, like yubikey.

Server side security is about auditing, making hard/slow to delete data, machine learning on events may help to prevent attacks that are suspicious from its form, like very fast root session, which is always suspicious.
Ideally disable direct access to data at all, use backup, IDS is a lot of work and has same issues as anti viruses. Read-only OS like OStree may work, but update may be hard.
 After this talk we moved to the lobby, where we found a nice refreshments.

Description of the talks and hopefully soon also slides and recording available at: https://wiki.centos.org/Events/Dojo/Brussels2016

See also the notes after lunch.

How we've fixed upgrade path for CentOS 6 Software Collections

Short meesage for those who don't have time:
Software Collectoins for CentOS 6 are ready for upgrade from older rebuilds.

Now full story for those who care. Btw. this all is related to work done by SCLo SIG group that is part of CentOS (read more at http://wiki.centos.org/SpecialInterestGroup/SCLo).

A bit of history for beginning. Shortly after first RHSCL 1.0 release, CentOS rebuilds were prepared and since then they are available under:
http://mirror.centos.org/centos/6/SCL/x86_64/

However, keeping these rebuilds in sync with RHSCL content hasn't been easy task. With introduction of Java packages in collections, this task became even more tricky, which means these collections were not updated for long time. With that said, someone would expect there won't be problem with upgrade path, in other words that the new RPMs, that the SCLo SIG group is about to release, will update the older RPMs smoothly.

Well, not always. The original RPMs used ".el6.centos.alt" as %dist tag, while new builds use just ".el6" and that evolves in cases where python27-python-bson-2.5.2-4.el6.centos.alt.x86_64 > python27-python-bson-2.5.2-4.el6.x86_64, even if those packages have same Release tag in RPM SPEC. That obviously means the packages won't update smoothly.

Solution is quite simple in this case -- use higher Release in RPM SPEC. In some packages, this was already done, because some of the packages received update since original inclusion. In other cases we solve it by adding ".scX" (X is number) suffix to the Release tag. The ".scX" was chosen deliberately since ".scX.el6" is higher (alphabetically) than .el6.

Btw. for cases we need to build package more times before final build (bootstraping), we use suffix ".bsX", which means we can build package without any Release suffix in the end, because ".bsX.el6" < ".el6".

Anyway, this post was meant to let you know that upgrading of el6 packages from originally built RPMs is something we care about.

To verify it works, I've installed all the packages from original repository, then ran "yum update" and that evolved in proper update of all packages. I took that as proof it should work fine in your case as well. If there are still some issues, let us know.

Enjoy Software Collections on CentOS!