The Future of Open Source

Enterprise Tech London was very pleased to collaborate with SUSE and Orrick for a private discussion on the future of Open-Source with Gerald Pfeifer (CTO at SUSE), Natasha Ahmed (Partner at Orrick) and several key open-source consumers from the financial services sector.

Open-Source is very much a key aspect of how most, if not all, organisations consume software, so the notes below provide an overview of this important discussion that was led by Gerald, Natasha with a  number of industry-leading attendees in the room.

The Proof’s in the Numbers…

Based on the numbers, you could say that Open-Source has already won. Data from IDC in 2023(1) shows us that:

    • 65% of hardware-based servers run Linux, while this is 83% for virtual servers (windows is not really strong for virtual server environments).
    • 95% of cloud runs on Linux, even Azure where 70% of what is run directly on Azure as IaaS is based on Linux.
    • 90% of containers are orchestrated using Kubernetes.

The distributed environment operating system world (excluding mainframes) has become somewhat of a two-horse race, dominated by Linux and Windows – other players such as HP/UX, Solaris, AIX have all gone by the wayside.

Pretty much all the underlying infrastructure used in distributed computing is based on Open-Source – for example, if you run an SAP HANA environment, you will be using Linux as HANA exclusively runs on that platform.

The Many Faces of Community

Talking about “The” Open-Source community is too vague, as the ecosystem is much more diverse.  Some projects have become very professional, while others are still community-based.

There are always things for people to do within Open-Source projects, such as filing bug reports, reproducing errors, etc… This is especially the case for projects that are not super-technical, where you don’t need a PhD in Computer Science.

As an immediate example, there are:

Single Project Vendors

There are many single-project vendors, such as Docker and MariaDB (MySQL) – these can cause problems because they often struggle on a commercial basis, as it is really hard to make money with this type of setup.

Part of the challenge is that if the project becomes successful then it is relatively easy for that functionality to be adopted into bigger multi-project vendors.  Conversely, often customers don’t want to end up integrating many single projects into an end-to-end solution. This can end up making a highly successful technical project commercially challenging for the company which supports it.

Single Vendor Sourced Projects

While the project fits the legal definition of what Open-Source is, related to licencing, single vendor projects lack diversity of community to fall back on.  This creates the risk of “Throw over the fence” situations, where providers do the bare minimum to support a project.

The other more practical challenge is that the market will always catch-up, so if I create a feature that is held in this type of single vendor Open-Source model, competitors may collaborate to bring the same feature to market. They will often catch-up quicker and produce something better because they are working together.  Ultimately, you end up giving up and switching your version over to the market-driven version.

A better way to differentiate in this type of situation is through intelligence, tooling and processes for supporting customers well, rather than through the code itself.

These situations show the importance of trust and consistency in Open-Source projects, as vendor lock-in still exists and some vendors are immature in terms of stepping in/out of projects and engaging in bad behaviour.

For example, it takes a 3-digit quantity of developers and serious money to run a fully featured Linux distribution – this is something that most organisations are never going to do.

There are exceptions where the community can exert power, such as Open-Tofu (OpenTF). These forks don’t usually diverge terribly from the original, as the forking parties typically require similar functionality and if the fork diverges too much then it won’t fit the original required use case.

Also, one of the great merits of the more popular Open-Source projects is that vendors can support each other’s products – For example, SUSE decided to support Red Hat Linux several years ago – giving the customer options.

Where Control Sits and the Importance of Competition

Open-Source essentially relies on the Subscription model and this is the one that the key original players like Red Hat and SUSE have established in the ecosystem.

Together, they are the biggest investors in Linux distributions – so they share the costs and compete commercially with their own products, but they both make a point of maintaining standards and not differentiating unless absolutely necessary.  If it is done, then there is a concerted effort to bring the proprietary code upstream.

This moral authority potentially changes as we move into a world of cloud, where Hyper-scalers are of such a size that they can make proprietary changes to an Open-Source distribution without needing to recontribute/redistribute it. This is essentially forking the code and creating an unnecessary lock-in for their customers.

This allows for a change in the balance of control between the provider and customer, as you move away from on-premise to cloud and API based infrastructure.  While competition should be important in the market, some customers care more than others.

This is potentially the greatest downside consequence of the Open-Source subscription model crossing over with hyper-scaler offerings, especially if which could become a major issue if things become more monopolistic or hyperscale vendors abuse their market position.

Contributing and IP

SUSE employees are allowed to contribute in their spare time on any project and also during their workday, if relevant to their role. This isn’t the case with corporate entities and can cause conflicts between employees and their organisations.

For engineers, contributing can be an important part of their brand and having a body of public contributions is often something that more technical companies will look for when hiring engineering talent.

Legally speaking, under English Law any intellectual property rights you create in the course of your employment is seen as being owned by your employer.  This can make side-projects and Open-Source contributions tricky legally speaking – so it is important to establish where the boundaries sit at the outset of any employer relationship. It is worth noting that these things can often be managed through conflicts of interest processes, such as Outside Business Interests declarations.

Open-Source and Risk

There are always going to be risks, especially looking at how immature software engineering is as a practice relative to the complexity of the problems it is looking to solve – for instance, enterprise Linux is a billion lines of code and this creates a whole set of challenges.

Also, with the ubiquity of Linux usage globally, in terms of scale and risk profile, there are concerns about how to understand the background and motivation of contributors.  DARPA has recently looked to address this(2) though understanding geopolitical affiliations of contributors.

Supply Chain is one of the biggest challenges of our time and Open-Source can help by providing the ability to look inside solutions, to give more visibility through transparency of code and also contributor information.

As with anything, diligence and provenance of the code are key – the degree of depth for this diligence will often be dependent on the criticality of the system.

Where will Open-Source go Next?

Traditionally, Open-Source has been strongest in the platform arena, such as operating systems, container orchestration and orchestration & management of applications, but the key question is whether there will be opportunities for it to extend beyond those historically familiar use cases?

AI will definitely play a role in the Open-Source ecosystem, but right now it is a little unclear how soon this will be.  There are immediately valuable use cases, mostly around the process aspects, but not code committing yet – SUSE doesn’t take “Cold” code contributions from unknown/unvetted sources and that includes AI.

However, there is some value in the case of crashes and bug reporting or parsing of log files, where AI could be involved sooner. A word of caution around style versus substance though, as good information that’s poorly finished is always better than a slick output that’s rubbish. For instance, there was a recent case of some bug reports that were filed where the output was very polished and it looked well finished, but the actual data was rubbish because it came from an AI hallucination.

More broadly, longer term the challenge for open source will probably not come from code development, but from the risk of disempowerment where the underlying code/platform is Open-Source but the parts that really add value higher up the stack, like data and models, are proprietary – technically it’s open-source, but it’s not useful without the proprietary AI parts.

Some Closing Tips

 A couple of pointers, based on painfully learned experience:

Rely on Quality & Reputation

A good principle is to look for meta projects like CNCF, Linux Foundation, Apache Foundation – there are certain approved licenses, trademarks, governance and oversight that they require – this isn’t a must-have, but definitely something that is carefully considered.

When SUSE picks software for their own use they are very careful about which projects, vendors and licenses they use, as these are often long-term relationships. The same applies for when SUSE looks to Open-Source a product.

Stick with the Pack

Even though it is tempting, don’t invent your own Open-Source license structures. This is not a good idea!

There are existing licenses where people are already familiar with their function and the legal aspects are known. More importantly, the interactions are more clearly understood.  If you have 17 licenses that have been aggregated, will the end product be distributable?

For example, SUSE decided to collaborate with Oracle and CIQ when creating the Open ELA initiative – which is source code that is Red Hat compatible in a fully Open-Source manner. Anyone can use this, so it provides options for users to migrate.

The goal was to put it under a proper foundation, with appropriate governance, and then invited Oracle.  While some eyebrows might be raised by this partnership, Oracle are actually good contributors to the Linux ecosystem and have some great engineers contributing.

 

References:

  • “Worldwide Server Operating System Environments Market Shares, 2022: Steady Growth Persists” by Greg Macatee – IDC, July 2023 (#US51038623)
  • “The US Military wants to understand the most important software on Earth” by Patrick Howell O’Neill – MIT Technology Review, July 2022