Diverse technical topics from OSCON 2018

The O'Reilly Open Source Conference (OSCON) returned to Portland, Oregon in July for its 20th meeting. Previously, we covered some retrospectives and community-management talks that were a big part of the conference. Of course, OSCON is also a technology conference, and there were lots of talks on various open-source software platforms and tools.

$ sudo subscribe today

Subscribe today and elevate your LWN privileges. You’ll have access to all of LWN’s high-quality articles as soon as they’re published, and help support LWN in the process. Act now and you can start with a free trial subscription.

An attendee who was coming back to OSCON after a decade would have been somewhat surprised by the themes of the general technical sessions, though. Early OSCONs had a program full of Perl, Python, and PHP developer talks, including the famous "State of The Onion" (Perl) keynote. Instead, this year's conference mostly limited the language-specific programming content to the tutorials. Most of the technical sessions in the main program were about platforms, administration, or other topics of general interest, some of which we will explore below.

IBM, blockchain, and quantum computing

IBM had two keynotes at OSCON, one of them on the multiple IBM-led blockchain projects, and a more interesting one on quantum computing. IBM is a leading sponsor of the Hyperledger Foundation, a project of the Linux Foundation that is dedicated to applications of blockchain ideas and the "decentralized web". Christopher Ferris quickly reviewed the ten projects that are hosted by the Hyperledger Foundation, focusing on Hyperledger Fabric, which is a developer framework that is intended to make it simple for software developers to include Hyperledger functionality. This keynote was followed by many talks in the main program featuring Hyperledger and blockchain technologies; indeed, the blockchain seemed to be the major technical theme of this year's conference.

The next day included a more exciting "surprise" keynote, where IBM scientist Jay Gambetta announced Qiskit, a project from IBM aimed at making it easy for researchers and programmers to make use of quantum computing. In his keynote, Gambetta showed off Jupyter notebook access to quantum-computing calculators and results. For those able to use IBM Q hardware, Qiskit plans to make quantum computing just another software library to include in researcher's projects.

The Qiskit project will contain four sub-projects using the Latin names of the four elements. Terra (earth) is the base library for controlling the hardware and transmitting its results. Aqua (water) is the framework for programming quantum-computing algorithms. Ignis (fire) is error control and noise management, which is a major issue in quantum computing. Finally, Aer (air) is a set of emulators, simulators, and debuggers, particularly software tools for making regular computers mimic quantum computation sufficiently well to provide a development platform. This last "element" is already available as a Python 3 library.

Since access to IBM Q hardware is very limited, the Qiskit website includes an interactive learning portal, running in the Jupyter notebook viewer, so that interested developers can try it out right now. Sadly, the conference did not include any deep-dive sessions on the technology, but hopefully future events will.

Container security standards

Like most recent open-source conferences, OSCON included talks about Linux containers and container orchestration. One of these, "TL;DR: NIST container security standards," was presented by CoreOS staff Elsie Phillips and Paul Burt. In 2017, the US government's National Institute of Standards and Technology (NIST) released a report titled "NIST Guidance on Application Container Security." This report was issued with uncharacteristic alacrity by the usually slow-moving government agency, which generally advises on technologies well after they've become mainstream. NIST staff apparently wanted to make recommendations to curb what they saw as an alarming ignorance of Linux container security requirements.

The 63-page report, according to Phillips, is useful to anyone working with software applications packaged as containers. While she summarized it for the audience, she urged them to read it in its entirety. "Yes, it's long, but it's totally worth reading," she said.

The major source of problems, as explained in the report, is that mainstream security tools and practices aren't prepared to handle either the statelessness or the rapid deployment of containerized applications. The agency went through all of the risks that an organization takes on when it shifts to a container-based infrastructure from more traditional packaging and deployment. This includes risks associated with container image packaging, risks created by use of container registries, risks caused by use of orchestrators, and vulnerabilities in the infrastructure of shared hardware nodes running many containers.

Examples of the risks and problems that Burt and Phillips explained include:

• the requirement to rebuild many images whenever a vulnerability is found in a shared base image;
• accidental inclusion of secure credentials in the binary layers of a container image;
• insecure container registry connections;
• use of untrusted or out-of-date container images;
• poorly segregated network traffic in container orchestrator "virtual networks"; and
• the large attack surface offered by the shared Linux kernel between the host and the containers.

Since the guide is intended to help remedy these problems, it contains pages of advice on how to ameliorate them. First, it recommends using newer security tools designed for container orchestration environments, to make sure that users can enforce policies in a rapid-deployment, stateless environment. The report also provides many other recommendations, such as:

• only use container images from a verified, signed source;
• always use an encrypted connection to container registries;
• regularly purge old container images from registries;
• segregate an orchestrator network into separate networks for each sensitivity level; and
• run containers with a read-only root, or as an unprivileged user.

Phillips and Burt said that the guide had much more advice than they could cover, and that the audience should explore it on their own.

Message brokers past and present

For developers who work in the enterprise applications space, Suresh Pandey of Capital One presented a capsule history of message brokers. These systems, a central part of large enterprise software architectures, allow asynchronous communication between various separate software systems, such as between the loan-processing software and the credit-check software. Lately, the message broker concept is being overhauled, leading to new software like Kafka and Kinesis.

The message broker became a distinct type of software with the publication of the Java Message Service (JMS) API by Sun Microsystems in 2001. JMS popularized a number of concepts that are still in use for messaging, including producer and consumer, publish and subscribe (often abbreviated "pub/sub"), message queues, and topics. The last term describes a mechanism for publishing messages that can be read by many recipients, as opposed to queues, whose messages conventionally are "taken" when they are read.

The original protocol compatible with JMS was ActiveMQ. It worked well for Java clients, but required a separate broker service to interpret between a Java client and those written in any other programming language, using the intermediate Simple (or Streaming) Text Oriented Messaging Protocol (STOMP). This made it difficult for programmers in other languages to make use of message brokers, and limited their spread.

Advanced Message Queuing Protocol (AMQP), developed from 2003 to 2011, changed that through the introduction of interoperable formats. This allowed producers and consumers (or publishers and subscribers) to communicate without caring about the code base on the other end of the message. It allowed, for example, a Java client using RabbitMQ to talk to a Ruby subscriber using Qpid. Suddenly message brokers became a part of all large-scale infrastructures.

RabbitMQ was the most popular broker of this era, particularly because it implemented clusters for load balancing and avoiding single points of failure. RabbitMQ clients send messages to "exchanges", which then serve queues to subscribers. It can mirror queues for high availability.

One change in the most recent generation of message brokers is the shift away from "smart producer/dumb consumer", where the publisher is a heavyweight broker service that is expected to track what messages have been delivered to which subscribers. The new model is "dumb producer/smart consumer", meaning that the publisher simply offers up messages and topics, and the subscriber is expected to track which ones it wants and what messages it has already received.

The remainder of his talk was a comparison between what Pandey sees as the current leading message brokers: Apache Kafka and Amazon Kinesis. Kafka is an open-source project that you install and manage yourself, whereas Kinesis is a pay-by-usage Amazon service. Kinesis is good for AWS users who just want to pay for uptime and scalability; Kafka requires a knowledgeable team in order to install, configure, and maintain it.

Kafka uses topics rather than queues, and allows users to partition topics across multiple servers in a cluster for scalability. Messages are removed based on a time limit, rather than whether or not they are read. To increase the throughput of your Kafka cluster, you need to add new hardware nodes and then migrate partitions to them. As with any transactional service, Kafka's performance is bounded by the performance of its durable storage (such as hard drives).

Kinesis, on the other hand, scales invisibly to the user by using Amazon's infrastructure. Data replication between Amazon availability zones is offered as an option, and it supports "shards" that work exactly like Kafka's partitions. One surprising difference Pandey mentioned was latency: on a well-tuned Kafka cluster, message delivery latency is sub-second, but Kinesis averages between one and five seconds.

With their greater utility and scalability, Pandey expects to see more service infrastructures being built with Kafka and Kinesis in the future.

OSCON 2019

There were lots of other presentations that we weren't able to cover, including tutorials on Kubernetes, Istio, bash, Rust, and Tensorflow; and talks on GraphQL, Nomad, React, and Swift. Overall, OSCON is a mix of the technical, the pragmatic, and the trendy, kind of like open source itself. Add a splash of community management, and it's likely what we can expect from the conference next year as well.

OSCON will be returning to the Oregon Convention Center from July 16 to 19 in 2019.

[Josh Berkus is an employee of Red Hat.]