Train ML models,
don't mess with data
Fast and simple framework for building and scaling data pipelines for machine learning
Next-gen Cloud Computing
Too much time is spent on setting up the data
With well-designed data pipelines, rapid iterations of machine learning experiments will result in models with superhuman accuracy.
> pip install hub
Arrays
Create an array
Create a large array that you can read and write from anywhere. When you write one slice of the array, it automatically syncs to the cloud. You can lazy-load an existing array on-demand or connect to any other storage.
- Arrays
- Dataset
- Pipeline
- Train
import hub
import numpy as np
# Create a large array that you can read/write from anywhere.
datahub = hub.fs('./data').connect()
bigarray = datahub.array('your_array_name',
shape=(100000, 512, 512, 3),
chunk=(100, 512, 512, 3),
dtype='int32'
)
# Writing to one slice of the array. Automatically syncs to cloud.
image = np.random.random((512, 512, 3))
bigarray[0, :, :, :] = image
# Lazy-Load an existing array from cloud on-demand
bigarray = datahub.open('your_array_name')
bigarray[0, :, :, :].mean()Features
Concentrate on what really matters
Activeloop's Data Pipelines allow you to skip time-consuming setup procedures and start training on your data instantly.
Generate datasets using plug-and-play data pipelines
Using the python-native framework to seamlessly build data pipelines for feature extraction, machine learning and deep learning. Automatically ingest, clean and transform your raw data as new data comes in.
Test locally, then scale to the cloud with no code change
Activeloop enables building streamable data pipelines which work locally, and can be simply scaled to thousand machines on the cloud. No need to configure cloud infrastructure anymore.
Leverage most cost-efficient hardware on the cloud with the support of preemptible/spot instances.
Collaborate with your team
Data versioning and synchronization protocol implemented for you to be accessed across teams. User access management with encryption at rest and in transit. Access your data from anywhere.
Visualize data at any step
View results with our visualization engine deployed on premise or on cloud. Preview slices of data with no load time and keep track of feature engineering pipeline.
Connectors
Connect to the storage service of your choice
Connect your pipelines to any type of structured and unstructured data in the Powerful Cloud-Native Array Data Warehouse.
Amazon S3
Google Cloud Storage
Blob
Redshift
Snowflake
BigQuery
MySQL
PostgreSQL
Kubernetes
Our package can be seamlessly deployed and managed at scale on multiple clusters orchestrated with Kubernetes.
Security
With our integrated authentication and encryption protocols you never have to worry about your data’s security and integrity.