Deep LakeData Lake for Deep Learning
Data infrastructure optimized for computer vision. Deep Lake is the fastest data loader for PyTorch.
The open-source community
enabling the future of data
Trended #1 in Python
5.2kGithub Stars
+10%
90+Contributors
+31%
1.1K+Community members
Meet Tensie. Tensie's lit. She likes
optimizing datasets & fire puns.Just like a vanilla data lake.
With a twist for deep learningPlus, we're
open-source!Deep Lake maintains the benefits of a vanilla data lake, such as time traveling, SQL queries, ingesting data with ACID transactions, & visualizing terabyte-scale datasets. Deep Lake comes with one key difference. With Deep Lake, complex data, such as images, audio, videos, annotations, & tabular data is stored as tensors and rapidly streamed to (a) query, (b) in-browser visualization engine, or (c) ML models without sacrificing GPU utilization.
Dive into Deep Lake
Iteration speed of images against other data loaders
Yale University Research Spotlight: Deep Lake is the Fastest Data Loader for PyTorch
In this paper, Ofeidis et al. (2022) explore the current landscape of PyTorch libraries that allow data scientists to load datasets into their models. Deep Lake obtained "remarkable" results (only a 13% increase in time compared to loading from a local disk). Deep Lake also outperformed all data loaders on networked loading.
Read More
How Deep Lake fits in a machine learning loop?
Ship AI products faster. We'll handle the complex infrastructure
Features
Visualize Your Datasets
Semantically visualize, seamlessly explore, and visually interact with audio, video, & image datasets right in your browser. Overlay metadata, & explore distributions
Rapidly Query Your Datasets
Use Tensor Query Language, our engine capable of querying terabyte-scale datasets to instantly. Run advanced queries with built-in NumPy-like array manipulations
Stream to ML frameworks
Stream the dataset to PyTorch or TensorFlow with one line of code. Our data loader efficiently streams data from remote storage to the GPUs while models are being trained
Dataset Version Control
Git for data. Modify dataset elements across versions & switch between them. Work with datasets of any size, overcome the limitations of file-based systems, instantly visualizes changes in-browser, & trace data lineage
Team and Access Management
Keep your datasets private, share them with your organization or anyone on the web. Have multiple data scientists working on the same data? We can handle that, too
Load Data from Anywhere
Deep Lake works locally, on Google Cloud, MinIO, AWS S3, Azure, Google Drive as well as Activeloop storage (no servers required). Directly stream datasets from cold storage to ML workflows. It's that fast
Visualize, query, version, & stream datasets
Deep Lake datasets are visualized right in your browser or Jupyter notebook. Instantly retrieve different versions of your data, materialize new datasets via queries on the fly, and stream them to PyTorch or TensorFlow.
- Rapidly visualize different versions of your data
- Understand your data and improve its quality
- Query, train, & edit datasets with data lineage
- Evaluate model performance
125+ open datasets, one click away
- Explore verified datasets
by team Activeloop...
- ... public organizations
(like Google!) ...
- ... and our open-source
community!
Simple Python API for data(If you use Deep Lake)
1import deeplake
2from PIL import Image
3
4ds = deeplake.load('hub://activeloop/mnist-train') # deeplake Dataset
5
6# Display an image
7Image.fromarray(ds.images[0].numpy())Deep Lake is revolutionizing Deep Learning. Dive into it.
Drive revenue growth by shipping AI products faster, saving money by saving on GPUs, increasing data scientists’ focus on core business problems, & eliminating failed ML project risk due to the lack of a solid data foundation.
> pip install deeplake
Dive into
Deep LakeCreate
an accountDeep Lake open source. Join the community
Stay in the loop