Kafka Use Surges

August 1, 2017

A report demonstrates a surge in the use of Kafka for a broadening set of business and technical objectives. Many companies are implementing the distributed streaming platform for more accurate and faster decision making, reduced operating costs, improved customer experiences and reduced risk. 1 in 4 respondents (26%) work for organizations with more than $1 billion in annual sales, illustrating how quickly this open source technology has gained traction across large enterprises. In fact, more than 15% of respondents are processing more than a billion messages a day.

The problem of managing data in enterprises is an exercise in complexity. As companies add new applications, integrate and modernize existing systems, change to a microservices architecture, and embrace a factor increase in data-producing endpoints, their infrastructure, and the engineers who support it, face an existential question: to pursue each new data project as a unique, discrete effort, or find an approach with greater scalability?

For many, Kafka has become the answer to this question. It transforms an ever-increasing number of new data producers and consumers into a simple, unified streaming platform at the center of their organization. It allows any team to join the platform, enables a central team to manage the service, and scales to trillions of messages per day while delivering those messages in real time. In many cases, Kafka can replace or augment an existing system to make data more consistently available, faster and less costly to deliver.

Key findings from the survey include:

•Kafka use is experiencing a surge: 86% of respondents reported that the number of their systems that use Kafka is increasing and a fifth (20%) reported that the number is “growing a lot!” A majority (52%) of organizations have at least 6 systems running Kafka with over a fifth (21%) having more than 20. According to last year’s report, only 41% of organizations had at least 6 systems running Kafka and only one-tenth (10%) had more than 20.

•Kafka is broadly used in the cloud: Kafka is used by organizations in some combination of virtual private clouds (34%), public clouds (52%), and on premises (57%). Nearly one-third (32%) of respondents who use Kafka in the cloud have at least 6 Kafka applications in the cloud.

•Putting Apache Kafka to use: this year, organizations are using Apache Kafka in many ways, far beyond pub-sub: two-thirds (66%) use it for stream processing and three out of five (60%) use it for data integration. The most common use case for Kafka is data pipelines (81%), while half (50%) are already using it for microservices.

“The results from this year’s Apache Kafka survey show a lot of momentum for Kafka adoption in the community and industry, even in just the past year. We’re thrilled to see so many companies taking advantage of streaming data and using Kafka as the streaming data backbone,” said Neha Narkhede, co-founder and CTO at Confluent. “We’re looking forward to hearing many more stories of working with Kafka at this year’s Kafka Summit NYC and how companies around the world are using streaming platforms.”

For this report, Confluent surveyed over 350 organizations from 47 countries and a wide variety of industries to understand the evolving Apache Kafka user base. Those surveyed range in position from developers and architects to technical management and engineers.

Terms of Use | Copyright © 2002 - 2017 CONSTITUENTWORKS SM  CORPORATION. All rights reserved. | Privacy Statement