StreamNative Cloud for Kafka: A Cloud-Native Managed Kafka Service Without the Limits of Kafka

Creators of Pulsar introduce turn-key managed solution to overcoms current limitations of existing Kafka clusters

StreamNative, the cloud-native messaging and streaming platform founded by the original developers of Apache Pulsar, announced StreamNative Cloud for Kafka, a managed solution that supports Kafka protocol but operates Apache Pulsar under the hood.

“Kafka is a widely-adopted technology and organizations have been able to build a broad number of streaming use cases on top of it. But, it doesn’t come without challenges. For example, for use cases where data needs to be segmented into tens of thousands of granular topics, Kafka often struggles with poor performance. Or, for teams with a large number of microservices, the lack of native multi-tenancy often leads to a large number of Kafka clusters, typically with low utilization. StreamNative Cloud for Kafka helps to remove current Kafka limitations without having to change your application code,” Addison Higham, Head of Product, StreamNative.

Marketing Technology News: Mimecast Opens Office in Singapore to Continue Expansion in APAC

Pulsar was originally developed by the StreamNative founders, back when they were at Yahoo!, to improve on the Kafka architecture and add new capabilities to take advantage of the cloud. New features include built-in multi-tenancy and geo-replication to support global shared deployments and the ability to support hundreds of thousands of partitions and topics with low latency. Additionally, Pulsar was built with the ability to have worker queues and enable a broad set of messaging patterns. These capabilities are noteworthy as they address some of the constraints teams typically face as they grow their Kafka usage. Now, with StreamNative Cloud for Kafka, organizations can tap into the benefits of Pulsar without a major investment.

Pulsar engagement and adoption have accelerated over the past few years. In fact, in 2021 Apache Pulsar ranked as a top 5 Apache Software Foundation project in terms of engagement. Perhaps even more surprising, it has averaged more monthly active contributors than Apache Kafka for most of the past 18 months. If you are building microservices, moving to an event-driven architecture, or building data-intensive applications on Kubernetes, you are probably considering Pulsar.

Marketing Technology News: MarTech Interview with Jamie Hill, CEO at adMarketplace

But for the many organizations who have already adopted Kafka, they must weigh the costs to migrate to Pulsar. StreamNative Cloud for Kafka helps to make this easier. Organizations can now use a fully-compliant implementation of the Kafka protocol to alleviate Kafka pain points and to leverage the capabilities of Pulsar, one use case or team at a time.

In the longer-term, new applications, or applications that may not be an ideal for Kafka, can be moved to Pulsar to eliminate time spent building Kafka workarounds and to increase developer productivity. Additionally, the move to Pulsar, with its native multi-tenancy and geo-replication, enables different teams and different applications to safely share clusters, reducing infrastructure complexity.

StreamNative Cloud for Kafka is built on the open-source Kafka on Pulsar (KoP) technology. Tencent, the largest provider of Internet products and services in China, is both an adopter of KoP, and a contributor to the project.

“We are the team responsible for Tencent Big Data MQ and data integration infrastructure, which support nearly all of Tencent’s products and process trillions of messages per day. Using Kafka, we faced a number of pain points, such as complex and costly operations. We discovered Pulsar and have been transitioning from Kafka to Pulsar in order to build one, cloud-native architecture. Kafka-on-Pulsar (KoP) is a powerful tool that has streamlined the migration process for us. KoP allows our team to focus on leveraging the powerful features of Pulsar, instead of trying to build workarounds for Kafka. We are also excited to be an actively contributing team to the KoP project, with two current maintainers.” – Chao Zhang, Head of MQ & InLong Team, Tencent Big Data, Apache InLong PPMC.

Marketing Technology News:  Virtual and Hybrid Events to Take Center Stage in 2022