Fedora Loves Containers and You Can Now Build Them Officially

Docker containers have been around for some time already, Red Hat not only ships dozens of them in the Container Catalog, but popularity of Open Shift proves that the containers are the future, not only for the DevOps use cases. Recently, the containers started to finally be main topic in Fedora distribution as well, which is proofed by new Container Guidelines that are still being developed.

So what to do to create own container and make it available for other community users in Fedora? First, you need to know how to prepare the Dockerfile and supporting files. For that, I recommend to follow the Best Practices and of course the Fedora Container Guidelines mentioned above. The container should do something sane on start, it should not be just RPMs installed in the container. For example daemons like databases should run the daemon, they should also initiate the datadir etc.

On the other hand, we can expect different behaviour from containers like Python or NodeJS, where those will probably be used only as a base for other container images. Especially for those kind of containers I really recommend to use the standardized tool called source-to-image, that takes a builder image and an application source as input and combines them to a new image as output. Although the source-to-image concept was originally designed for containers like Python or NodeJS (basically runtime of various languages), it may be used for daemon images as well -- in case of HTTPD server the application is composed from static files, in case of databases it can be configuration or initial data. In all those cases the end user will probably need to do some own changes, that require creating an adjusted container (ideally a thin layer on top of what distribution provides).

Ok, let's say we have a Dockerfile prepared, what next when we want to build a container in Fedora? Then we can continue similarly as with RPMs -- we need to create a Container Review bug in Bugzilla, similar as some already exist.

After another packager provides the flag, which says the container follows the guidelines, you can build the container in the koji. Yes, the build tool koji supports docker containers building now, and the tool to make this happen is the fedpkg tool, the same tool that is used to build RPMs as well. We only need to use container-build command, instead of just build. After that, container will be available in the Fedora registry under registry.fedoraproject.org and image can be obviously pulled using the docker client, as usually.

Some more insight about which containers are already available can give listing all the components under docker namespace in the Fedora dist-git.

That's all for the quick update about container in Fedora. So, what container are you working on already?

Notes from Prague PostgreSQL Developers Day 2017

Thursday in the middle of February means that there was another year of p2d2 conference. This year, it's already 10th p2d2 meet-up of PostgreSQL users. Although it might seem interesting, it's nothing special, 16th will be better opportunity to celebrate.

Beating 1 billion rows per second

Billion rows in any database is nice portion of data, even more interesting it is in relation database, but as Hans-Jürgen Schönig from Cybertec explained, it is possible do it in 1s. It all began when Oracle published an article saying they can do it, Hans decided he can beat it with PostgreSQL as well.

From the real beginning, it was clear that it must be done by parallelism. Traditional one-core execution could help in parallelism when running more queries at a time, while pl/proxy could handle parallelism using sharding on procedures level. Parallelism from scratch is simplu not something feasible these days.

Hans began with a simple scenario with 3 columns, but this can be scaled to real data using more memory and more processors (more machines), theoretically to any level, so we can take this simplifying as acceptable. So, what can one server offer?

With 9.6 parallelism, which is not complete yet, optimizer determines number of cores that will be processing a query, small tables are processed by one core only. Option parallel_workers can be set to overcome guessing and make number of workers higher.

Benchmark must be done in memory only, to not do disc benchmark instead. It turned out, that one server can scale linearly up to 16 cores, even when running on 32 core box, because that box is actually 32 thread box, hyper-threading eventually degrades it to 16 core, though.

So, at this point we have 40M rows. Next thing is partition tables. And we can do this to needed extend. In the end, 32 billion rows were processed in less than 3s, which means the goal was reached, only few patching was used on top 9.6, many of them will be already in 10.

PostgreSQL for OLAP alias how the parallelism works in PostgreSQL

Tomas continued with an overview around processes and connections. Traditionally no threads and no forks are done in PostgreSQL when processing a query. That is ideal for OLTP applications, where apps do many small chunk of work jobs. Sharing processes via connection pooling is one thing how to utilize one process more, but it is not interesting for making query processing more effective.

OLAP applications use usually less connections, so some real multi-processing is more needed there. Gather process can run more processes in parallel, gather data from them and every process can do something on their own. Quite a lot of restrictions so far, like shared locks don't allow to run data modifying queries. There is also only one level of parallelism, but for selects it might cover pretty nice chunk of use cases.

In version 10 there will be some improvements in bitmap scan, hash joins or aggregations. Users will need to set flag whether the PL function is parallel safe though. Using TPC-H, standardized benchmark, many tests were faster at least twice, comparing to single process queries.

Effectiveness of parallelism (difference from theoretical limit) depends on core number, we saw it goes from 100% (1 core) to 80% for 32 cores.

Currently, refreshing materialized views is not parallelized, but it would make great sense. Same for parallel sort, although gather sort might be done in 10.

David Turoň from LinuxBox began with a bit of joke, explaining how miners in Ostrava region start to program instead of mining.. Anyway, he talked about partman_to_cstore, which is a plugin living in gtihub currently, so user needs to build it first. It might be good for: table with rotations (timestamp/timestamptz), senzor data, log of events; generally big tables that does not change. It's nice that in most cases partman_to_cstore might be used without changes in application. Cron may run function and store data to cstore.

Roman Fišer from nangu.tv continued with sharing their problems when delivering a lot of multimedia content to many users. From the tools they use, let's name Consul, Patroni, or HAProxy. As for the approaches they chose, they need to be tolerant for latences between nodes, etc. For data collection they use collectd, store logs to RRD files, Collection3/Cacti as a frontend, InfluxDB/Grafana for back-end, and they think about Prometheus. New PgSQL version integration takes them approx. 6 months.

Containers everywhere

My talk then focused on containers in distributions, of course giving all of the examples from PostgreSQL world. Describing mostly basic stuff I wanted to bring containers a bit closer to users that typically don't like them much. Mentioning OpenShift, Flatpack, or systemd-nspawn, except traditional Docker, people could start trying it on their own.

After presentation there were quite a few questions:

• Network - could IP tables (firewall) inside the container?
• Data in volumes shared in COW mode, so they can share the common part on disc - could volume-mounted data be on zfs?
• Several questions about user namespaces, what they are good for and whether it works as people expect.
• Sharing libraries in memory was an interesting question we couldn't answer for 100%, but from further look-up, any memory sharing would be against containers' concept of isolating processes, so nobody will probably implement it.

Machine learning using PostgreSQL

Lightning talk by Hans-Jurgen about Machine Learning focused on detecting fraudulent logins on a machine. Support Vector Machine is used to separate good users from bad, it's basically about creating model that is used for decision. PL/Python function to load data and use history data for learning itself. In the end the decision is also done in SQL and that all of course in PostgreSQL..

Pavel Stehule talked about \g, \gset and \gexec commands in the second and also last lightning talk. Those commands were added recently in psql, all help in faster work in psql console.

What does it mean to hack PostgreSQL?

Pavel Stehule closed the conference with a talk about experiences with implementing XMLTABLE for parsing XML, which was missing in PostgreSQL. It was possible to use the XML functions before, but with XMLTABLE it is much more obvious, which is the reason why SQL looks like it looks. It was interesting excursion to the world of PostgreSQL hacker.

When somebody reads ANIS SQL, it does not need to be really nice reading, details are missing and existing implementations are slightly different from standard, so what to do? In PostgreSQL, which implements the XMLTABLE later than DB2 and Oracle, author needs to take the functionality that is already existing. Generally, when implementing some new functionality, we might either consider patching the code, or writing an extension. If the feature can be done using extension, it's much easier, it can live anywhere and people are happy and prosper. In case patching of PostgreSQL is needed, it takes time and developer will spend a lot of energy. Especially tricky it is when using some library like libxml2, which, same as PostgreSQL, uses own memory management, uses different types of data and a conversion needs to be done.

In case of XMLTABLE implementation, some code change needs to happen in parser and executor. Here it comes handy to know well how the execution pipeline works in PostgreSQL -- parser, rewriter, planner, optimizer, executor. Also, SQL needs to be translated to some other language, so changing behaviour is basically changing of translator. On the other hand, this SQL translation won't be the bottle neck. Patching of parser (bison) needs to be clean, no conflict warnings will be acceptable during the review. Implementation is either an SRF function or executor patch.

When writing patch, first iteration in community is to write something dirty, to show that it is possible and conversation about benefits can begin. Next phase is complete patch, with documentation and regressions tests. The last part is commitfest patch, that is used for cleaning the code, documentation, extending tests, ...

The whole patch writing is very hard, one will learn assertiveness and respect in the first place. Some features take even 10 years from first patch to commit released. hstore for example helped Ruby on Rails and that helped PostgreSQL to grow. Without use case it is not very probable that change will be accepted.

So that was the 10th p2d2, I'm already looking for the next year!

CentOS Dojo 2016 Notes after lunch

This is a second part of my personal notes from this year's CentOS Dojo before Fosdem, first part includes notes from talks until lunch. So, after lunch, we returned to our chairs and heard these interesting talks.

4. Quickstart. Contributing packages to a CentOS Special Interest Group

Brian talked about basics in centos build system, how to jojn SIGs and how to build a package in CBS for a particular SIG. Really a "must seen" for every newbie in centos SIG.

5. Path from Software Collections to Containers for OpenShift

My talk about experiences with creating Containers for Open Shift included two dozens of tips from various fields. We closely looked how to create a nice, Open Shift friendly container image (yes, it was about Docker) for PostgreSQL and Python. These two examples covered the most important information that one needs to create any similar database or application builder image. Later I went quickly through list of images that are already out there, made by Red Hat or CentOS and that are based on Software Collections packages. In the end I shortly introduced the concept of Nulecule and what this project is intended for.

6. Getting started with kubernetes

Kubernetes was described the same as other orchestration systems, even condor which development started already in 1987. What makes the technologies different from the PoV of potential developer folks is language chosen to be written in.
Mesos was secretly influenced by Borg, a group run by Google. It means guys creating k8s know what they do, because we may see k8s as a new version of Mesos.
Basics of Kubernetes explained clearly and on simple examples -- what pods, services, replication controller do.
Atomic was presented as the solution to use k8s on CentOS.
For learning k8s use gh.c/skippbox/...8s
Terraform plan for deploying k8s on AWS with atomic host and flannel.
A demo showed automatically created k8s nodes and let them scale in AWS.

7. Atomic Developer Bundle - Containerized development made easy

Atomic developer bundle, guys showed why there is something like ABD, stating problems devels face today during application development, all on real user stories. They showed Vagrant devel environment, running docker secured by TLS, user being able to connect from host machine by evaling 'vagrant adbinfo' output that defines devel environment on host.
Another example showed Eclipse running on the host, connecting to remote docker, which is a scenario that might work from any OS, even windows. Although the demo did not work and we could see the live-demo Murphy's law in practice again, we got the point and I'm sure it worked fine just before the talk.
ADB supports the OpenShift and other orchestrations technologies as well.
Why centos? Because of community, that might give the needed feedback. In the end the list of available links were mentioned and community was called to action.
The future is so bright, I gotta wear shades.
Architecture is still a thing to be changed, they plan to make vagrantfiles easier.
Landrush does not work and some help is needed..
In the end guys tried the demo again but with poor Internet connectivity and Murphy's law working better, we saw only one step further.

Description of the talks and hopefully soon also slides and recording available at: https://wiki.centos.org/Events/Dojo/Brussels2016