HPE Buys Pachyderm For AI At Scale Capabilities

January 13, 2023

Hewlett Packard Enterprise expanded its AI-at-scale offerings with the acquisition of Pachyderm, a startup that delivers software, based on open-source technology, to automate reproducible machine learning pipelines that target large-scale AI applications.

Reproducing a machine learning pipeline enables use of the same dataset to achieve the same results each time to increase transparency, trustworthiness, and accuracy in predictions while optimizing time and resources. It is critical to successful AI-at-scale initiatives, which represent the next revolutionary step in realizing AI’s potential to increase the accuracy of predictions and achieve results faster. To attain these outcomes, organizations need to adopt technologies to efficiently build and train larger machine learning models that require a high volume of complex data.

“As AI projects become larger and increasingly involve complex data sets, data scientists will need reproducible AI solutions to efficiently maximize their machine learning initiatives, optimize their infrastructure cost, and ensure data is reliable and safe no matter where they are in their AI journey,” said Justin Hotard, executive vice president and general manager, HPC and AI, at HPE. “Pachyderm’s unique reproducible AI software augments HPE’s existing AI-at-scale offerings to automate and accelerate AI and unlock greater opportunities in image, video, and text analysis, generative AI, and other emerging large-language-model needs to realize transformative outcomes.”

HPE expands AI at-scale portfolio with addition of Pachyderm

HPE unlocks AI-at-scale opportunities for its customers by bringing together its leading supercomputing technologies that are foundational for optimized AI infrastructure, and the HPE Machine Learning Development Environment, a machine-learning software that enables users to rapidly develop, iterate, and scale high-quality models from proof-of-concept to production. The combined solution already helps users train more accurate AI models faster, and at scale, on several of the world’s fastest supercomputers that have been purpose-built for demanding AI workloads.

Building on to these solutions, HPE will integrate Pachyderm’s reproducible AI capabilities in one integrated platform to deliver an advanced data-driven pipeline that automatically refines, prepares, tracks, and manages repeatable machine learning processes used throughout the development and training environment.

Delivering an end-to-end machine learning software platform to enable production AI at-scale

AI-at-scale capabilities advance popular use cases involving natural language processing, computer vision, and video and image processing that are growing across industries such as transportation, life sciences, defense, financial services, and manufacturing.

By integrating Pachyderm’s machine learning pipeline capabilities with its existing AI offerings, HPE will enable faster development and deployment of more accurate and performant large-scale AI applications with the following benefits:

Data lineage – Visibility on the origin of the data and where it moves over time during the machine learning lifecycle and analytics process to easily trace errors back to the root cause.

Data versioning – Ability to track different versions of data to understand when data was created or changed at any point in time, to increase efficiency in making any changes.

Efficient incremental data processing – As data changes over time, only incremental data needs to be processed to update AI applications. Pachyderm makes incremental data processing automatic and efficient.

Lockheed Martin deploys HPE’s AI at-scale solutions for mission-critical applications

Lockheed Martin’s AI Factory, an open architecture approach to AI-at-scale, integrates Pachyderm’s software, the HPE Machine Learning Development Environment, and other modular solutions as part of their foundational AI ecosystem. Leveraging these capabilities allows Lockheed Martin to increase trust, maximize performance, and standardize AI technologies across a broad range of contested environments in support of national security missions.

This acquisition builds on HPE’s February 2022 investment in Pachyderm through its venture capital arm, Hewlett Packard Pathfinder, to speed time-to-market for AI innovation at lower data processing and operating costs. The transaction is not subject to any regulatory approvals and is expected to close this month.

Product integration and availability

Pachyderm’s software is available today to integrate with HPE’s existing supercomputing and AI software solutions. Additionally, HPE plans to integrate Pachyderm with upcoming versions of the HPE Machine Learning Development System, which eliminates the complexity and cost to build and train models with a complete, ready-to-use solution.