Kafka Use Surges
August 1, 2017
report demonstrates a surge in the use of Kafka for a broadening set of
business and technical objectives. Many companies are implementing the
distributed streaming platform for more accurate and faster decision
making, reduced operating costs, improved customer experiences and
reduced risk. 1 in 4 respondents (26%) work for organizations with more
than $1 billion in annual sales, illustrating how quickly this open
source technology has gained traction across large enterprises. In fact,
more than 15% of respondents are processing more than a billion messages
The problem of managing data in enterprises is an exercise in
complexity. As companies add new applications, integrate and modernize
existing systems, change to a microservices architecture, and embrace a
factor increase in data-producing endpoints, their infrastructure, and
the engineers who support it, face an existential question: to pursue
each new data project as a unique, discrete effort, or find an approach
with greater scalability?
For many, Kafka has become the answer to this question. It transforms an
ever-increasing number of new data producers and consumers into a
simple, unified streaming platform at the center of their organization.
It allows any team to join the platform, enables a central team to
manage the service, and scales to trillions of messages per day while
delivering those messages in real time. In many cases, Kafka can replace
or augment an existing system to make data more consistently available,
faster and less costly to deliver.
Key findings from the survey include:
•Kafka use is experiencing a surge:
86% of respondents reported that the number of their systems that use
Kafka is increasing and a fifth (20%) reported that the number is
“growing a lot!” A majority (52%) of organizations have at least 6
systems running Kafka with over a fifth (21%) having more than 20.
According to last year’s report, only 41% of organizations had at least
6 systems running Kafka and only one-tenth (10%) had more than 20.
•Kafka is broadly used in the cloud:
Kafka is used by organizations in some combination of virtual private
clouds (34%), public clouds (52%), and on premises (57%). Nearly
one-third (32%) of respondents who use Kafka in the cloud have at least
6 Kafka applications in the cloud.
•Putting Apache Kafka to use: this
year, organizations are using Apache Kafka in many ways, far beyond
pub-sub: two-thirds (66%) use it for stream processing and three out of
five (60%) use it for data integration. The most common use case for
Kafka is data pipelines (81%), while half (50%) are already using it for
results from this year’s Apache Kafka survey show a lot of momentum for
Kafka adoption in the community and industry, even in just the past
year. We’re thrilled to see so many companies taking advantage of
streaming data and using Kafka as the streaming data backbone,” said
Neha Narkhede, co-founder and CTO at Confluent. “We’re looking forward
to hearing many more stories of working with Kafka at this year’s Kafka
Summit NYC and how companies around the world are using streaming
For this report, Confluent surveyed over 350 organizations from 47
countries and a wide variety of industries to understand the evolving
Apache Kafka user base. Those surveyed range in position from developers
and architects to technical management and engineers.