Oxen.ai

Oxen.ai is a data version control system built from the ground up for AI and machine learning workflows. Unlike Git, which struggles with large binary files, Oxen uses Merkle trees, block-level deduplication, and compression to handle repositories with millions of files at terabyte scale.

The tool works a lot like Git. You init a repo, add files, commit changes, push to a remote, and create branches. But everything is optimized for data — CSVs with millions of rows, image folders, video collections, audio files. It indexes hundreds of thousands of files in seconds and keeps storage efficient through smart deduplication.

Oxen also includes built-in tools for data exploration. You can compare DataFrames between commits, run model inference directly on your data, search embeddings with nearest neighbor lookup, and edit data through the OxenHub web interface. The whole system is written in Rust with Python bindings, so it stays fast even at scale.