Datadog Observability Pipelines Debuts

June 20, 2022

Datadog Observability Pipelines is a new product that enables organizations to take greater control of their data so they can reliably scale their observability practices. Datadog Observability Pipelines is powered by Vector, an open source and high-performance framework for building telemetry pipelines.

Organizations today are tasked with managing many applications, which are powered by hundreds of services and result in the creation of trillions of logs, metrics and traces per month. The vast amount of data involved and the resulting complexity at both the application and organization levels can lead to reliability issues, technology lock-in, security risks and runaway costs.

Datadog Observability Pipelines provides customers with a unified view to control and monitor the flow of all their infrastructure and application metrics, logs and traces. Users can now seamlessly collect, enrich, transform, sanitize and route observability data from any source to any destination—before data leaves their environment. This unified view gives enterprises enhanced visibility into how much they are spending and where, what tools they are leveraging and who has access to what data. This enables them to more precisely manage costs, reduce technology lock-in, improve compliance and standardization of data quality and ultimately scale their observability practices.

"We want to empower our teams to use all of our systems and tools while staying within budget. Observability Pipelines gives us flexibility over how our data is processed and where it ends up, so our teams are taking action on the right information as we continue to scale," said Andreas Kasprzok, Observability Tech Lead at Blockfi.

"As the amount of telemetry continues to grow at an organization, teams are often completely overwhelmed—if they're not blind to—the runaway costs, reliability and compliance risks that come from a lack of visibility into infrastructure and application data," said Zach Sherman, Senior Product Manager at Datadog. "We built Datadog Observability Pipelines to give organizations a powerful way to take back control of their data, without compromising visibility for engineering, security and SRE teams."

Datadog Observability Pipelines helps IT and security teams in their goals to affordably manage and scale observability with complete flexibility and control over how their logs, metrics and traces are collected, transformed and routed.

This helps organizations:

Control Costs: Aggregate, filter and route all observability data based on use case without compromising visibility

Simplify Migrations and Reduce Lock-In: Orchestrate and monitor data processing from any source to any destination in one unified view

Protect Sensitive Data: Filter, redact and monitor sensitive data before it leaves your network in order to better meet compliance requirements

Enforce Data Quality: Standardize the format of logs, metrics and traces to improve observability across teams

Scale with Confidence: Scale seamlessly with a product powered by Vector, a vendor-agnostic, open source project and engaged community with millions of monthly downloads that is deployed in production by enterprises processing petabytes of data every month

Easily Collect and Route Data: Observability Pipelines comes with more than 80 out-of-the-box integrations so organizations can quickly and easily collect and route data to any of the tools their teams already use, without disrupting existing workflows