Pachyderm

Pachyderm

Learn More


Pachyderm is a Data Lake -- a place to dump and process gigantic data sets. Pachyderm is inspired by the Hadoop ecosystem but shares no code with it. Instead, we leverage the container ecosystem to provide the broad functionality of Hadoop with the ease of use of Docker.

Features and Benefits


Version Control for Data

Pachyderm also version controls all data using a commit-based distributed filesystem (PFS), similar to what git does with code. Version control for data has far reaching consequences in a distributed filesystem. You get the full history of your data, it's much easier to collaborate with teammates, and if anything goes wrong you can revert the entire cluster with one click!

Containerized Pipelines

To process data, you simply create a containerized program which reads and writes to the local filesystem. You can use any tools you want because it's all just going in a container! Pachyderm will take your container and inject data into it by way of a FUSE volume. We'll then automatically replicate your container, showing each copy a different chunk of data. With this technique, Pachyderm can scale any code you write to process up to petabytes of data


Website
http://www.pachyderm.io
Repository
https://github.com/pachyderm/pachyderm
Primed For
OpenShift Container Platform 3.3
Primed On
2016-06-23