Yesterday, the Linux Foundation (LF) announced the new OpenSearch Software Foundation, with Amazon transferring the 3½ year old open-source project to LF (RT). This outcome ensures the long term viability of this technology in a vendor-neutral way, under the most enterprise-friendly open-source Apache License v2. It is the result of the work of hundreds of people, but is also something I am personally very proud of, because I worked on the 6-page proposal to move OpenSearch to a neutral foundation at Amazon, and then my team and I sat at a table across from Adam Selipsky, then AWS CEO who took time to carefully read the doc, opened the floor for some tough questions from the AWS executive leadership team, then gave it a clear yes.
This post is about what it took to get here, what I worked on since my last status update, why I was originally opposed to this move, and the reasons why I changed my mind.
Two years ago I posted A Year Working on OpenSearch (2.0). At the time many things were going very well. We had operated a large open-source fork and built a release pipeline that enabled three automated releases over one week-end during the log4j security vulnerability fiasco. I’ve worked on areas of extensibility, was plotting a cloud-native future of the product, and helped pave the way for our first external maintainer in OpenSearch core. I also struggled with challenges of open-source being an “upstream” activity for many developers at Amazon, with some code still written behind closed doors to (never) be open-sourced in the future.
Since then, we’ve made 16 OpenSearch 2.x releases adding many features, bug fixes and performance improvements. The most interesting innovations to me are in the vector space. OpenSearch has had differentiated k-nn capabilities before the fork as part of OpenDistro for Elasticsearch (ODFE), and so combined with new machine learning capabilities it became one of the most useful tools for generative A.I. I’ve tried a number of other vector databases and found OpenSearch to be one of the easiest to get started with doing vector search. I also gave a chalk talk at re:Invent 2023 where we built a RAG pipeline, live in 20 minutes. Finally, this race is not over! In just the latest minor 2.16 release OpenSearch added vector compression automation for byte-precision vector quantization, binary vector support, sort and split search processors, application batch inference support for AI connectors, and native integration of AI/ML providers in the search path. I am not very deeply involved in this area, but I love to see it.
Most of the code I, myself, wrote was in the areas of extensibility and clients, continuing on my path of being an IC (I switched from being a CTO of a series D startup to IC in 2019). I still do write a lot of code, with 625 merged pull requests in opensearch-project over the last two years, and maintain about a dozen repos, including core, engaging on issues, and reviewing and merging countless pull requests.
I also lived through several changes at Amazon OpenSearch, including organizational ones. The massive push into Generative AI meant having to shift work and make difficult choices.
I was hoping to enable OpenSearch plugins to ship independently through the Extensibility project by now, which would have greatly simplified the development of OpenSearch plugins and enabled the community to write many new ones. This technology demonstrated roughly a 30% cost reduction for a 36-node cluster that performed high cardinality anomaly detection in one case, and the ability to write extensions in Python in another. Extensibility remains an experimental feature and is looking for someone in the community to pick this work up.
We also had to get smarter and more efficient about where to invest our energy in OpenSearch language clients. These enable the OpenSearch Project to meet developers where they are, in their programming language of choice. Developer experience matters a lot to me, starting with clients needing support for all OpenSearch APIs, or ways to make raw JSON requests for unsupported ones. We started a new OpenAPI spec project in opensearch-api-specification, which now covers 51% of APIs and the majority of clients are generated from it. As an example, the recent release of opensearch-py 2.7.0 added support for ML and SQL APIs without having to write any Python code at all. I am also glad to see that a third of maintainers on clients don’t work for Amazon.
Finally, the move to a foundation was a big idea that existed since the fork, but we needed to make a thousand decisions before prioritizing public governance. That said, the project was setup with public input and participation in mind right away, and many steps were taken to open it further to active participants.
- We established project partnerships with alternative OpenSearch service providers such as Oracle, Aiven, and InstaClustr.
- Began holding bi-weekly community meetings since May 2021. Today triage, release, and roadmap meetings are all public.
- Added the first maintainer not employed by Amazon in May 2022. Today 20% of repo maintainers (49/257) do not work for Amazon.
- Created a process for managing security issues, including a pre-disclosure list, in September 2022.
- Started a public Slack to transparently collaborate with the community in April 2023.
- Opened up our release process so that non-Amazon contributors could participate, in May 2023.
- Established a Leadership Committee (LC) comprising a mix of Amazon and non-Amazon employees in December 2023.
I had a hand in many of these things. Each required not only public agreement, but approvals inside Amazon. For example, the LC recommended we move to LF in early April, but we still needed approval from Amazon to transfer the trademark. And, from early conversations inside the company it became clear that there were many things to take into consideration, and Amazon had to get comfortable that the move would not negatively impact its OpenSearch customers. I, personally, had several reasons to dislike this idea, too.
- I was under the (wrong) impression that Amazon leadership was unlikely to say yes to moving anything to a neutral foundation given the lack of precedent.
- I was born in the Soviet Union, so I am skeptical of community-driven efforts by default. I was comfortable with the benevolent dictatorship of a commercial entity that did open-source because it made good business sense. My experiences with large open source projects, such as JNA, further confirmed that the benevolent dictatorship governance model could be sustainable and successful for decades.
- As an Amazon employee working full time on OpenSearch I had full visibility and privileged access to any behind-closed-doors decisions, and none of the barriers someone else may have had adopting or contributing to the project. I wasn’t sufficiently exposed to, or appreciating the challenges the community may have been having in contributing. I (wrongly) concluded that public governance would slow the project down.
Despite being opposed to the idea, I was asked to help, so I decided to start fresh, and created a blank new document for the proposal on December 12, 2023. I collaborated with a group of very smart people and we approached the problem in the most impartial way possible, doing our best to find reasons to do such a move, as much as to find reasons not to. It took a village to research foundations, learn about successful and failed projects that moved to foundations, and argue over every word to make sure the risks and benefits of such a move were not over- or underestimated. The final draft was a well-balanced, crisp 6-pager (a type of document that we use at Amazon to make decisions), and provided both the LC and Amazon leadership with the necessary data and the options to continue governing the project as is, or to move the project to a neutral foundation.
In a big surprise to myself, I changed my mind and recommended the move, which was not unusual when spending so much time collecting data for a 6-pager. I saw from talking to customers that OpenSearch had to be part of a neutral foundation to permanently regain customer trust after several licensing changes, and was impressed by the growth stories of projects such as PyTorch and Apache Spark. The move of these projects to a foundation resulted in a significant acceleration in participation and mindshare. Several VP-level reads of the proposal at Amazon went well with mostly questions and limited to no opposition. Finally, unlike for OpenSearch, Amazon decided to support the formation of ValKey (created as a result of re-licensing of Redis) as part of LF, immediately at the time of the fork, and we had already decided that our recommended destination was LF.
In April 2024 we received the approval we sought, and I am so happy to be in Vienna today to be part of the announcement of the new OpenSearch Software Foundation at the Open Source Summit Europe. I’ll also be a giving a talk today on innovating in open-source in your Enterprise. I hope to see you there!
Finally, the next steps for me are to help on the Technical Steering Committee that I have been grandfathered into for 2 years before my seat is up for re-election. I’m excited to continue to work on OpenSearch and with our growing community.