freshcrate
Skin:/
Language:English/简体中文
Home > Frameworks > fluid

fluid

Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)

ai-frameworkalluxiobig-datadata-abstractiondistributed-cachegokubernetes
Why this rank:Strong adoptionHealthy release cadenceRelease freshness

Description

Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)

README

License CircleCIBuild Status codecov Go Report Card Artifact HUB OpenSSF Scorecard CII Best Practices Leaderboard CNCF Status

📅 Community Meeting
The Fluid project holds bi-weekly community online meeting. To join or watch previous meeting notes and recordings, please see meeting schedule and meeting minutes.

What is Fluid?

Fluid is an open source Kubernetes-native Distributed Dataset Orchestrator and Accelerator for data-intensive applications, such as big data and AI applications. It is hosted by the Cloud Native Computing Foundation (CNCF) as an incubating project.

For more information, please refer to our papers:

  1. Rong Gu, Kai Zhang, Zhihao Xu, et al. Fluid: Dataset Abstraction and Elastic Acceleration for Cloud-native Deep Learning Training Jobs. IEEE ICDE, pp. 2183-2196, May, 2022. (Conference Version)

  2. Rong Gu, Zhihao Xu, Yang Che, et al. High-level Data Abstraction and Elastic Data Caching for Data-intensive AI Applications on Cloud-native Platforms. IEEE TPDS, pp. 2946-2964, Vol 34(11), 2023. (Journal Version)

Fluid

English | 简体中文

notification What is NEW!
🎉 January 8, 2026: Fluid has been promoted to CNCF Incubating status! The CNCF Technical Oversight Committee (TOC) officially accepted Fluid as an incubating project, a major milestone in its maturity and community adoption.
v1.0.8 Release: Oct. 31st, 2025. Fluid v1.0.8. Please check the CHANGELOG for details.
v1.0.7 Release: Sep. 21st, 2025. Fluid v1.0.7. Please check the CHANGELOG for details.
v1.0.6 Release: Jul. 12th, 2025. Fluid v1.0.6. Please check the CHANGELOG for details.
Apr. 27th, 2021. Fluid accepted by CNCF! Fluid project was accepted as an official CNCF Sandbox Project by CNCF Technical Oversight Committee (TOC) with a majority vote after the review process. New beginning for Fluid! .

Features

  • Dataset Abstraction

    Implements the unified abstraction for datasets from multiple storage sources, with observability features to help users evaluate the need for scaling the cache system.

  • Scalable Cache Runtime

    Offers a unified access interface for data operations with different runtimes, enabling access to third-party storage systems.

  • Automated Data Operations

    Provides various automated data operation modes to facilitate integration with automated operations systems.

  • Elasticity and Scheduling

    Enhances data access performance by combining data caching technology with elastic scaling, portability, observability, and data affinity-scheduling capabilities.

  • Runtime Platform Agnostic

    Supports a variety of environments and can run different storage clients based on the environment, including native, edge, Serverless Kubernetes clusters, and Kubernetes multi-cluster environments.

Key Concepts

Dataset: A Dataset is a set of data logically related that can be used by computing engines, such as Spark for big data analytics and TensorFlow for AI applications. Intelligently leveraging data often creates core industry values. Managing Datasets may require features in different dimensions, such as security, version management and data acceleration. We hope to start with data acceleration to support the management of datasets.

Runtime: The Runtime enforces dataset isolation/share, provides version management, and enables data acceleration by defining a set of interfaces to handle DataSets throughout their lifecycle, allowing for the implementation of management and acceleration functionalities behind these interfaces.

Prerequisites

  • Kubernetes version > 1.16, and support CSI
  • Golang 1.18+
  • Helm 3

Quick Start

You can follow our Get Started guide to quickly start a testing Kubernetes cluster.

Documentation

You can see our documentation at docs for more in-depth installation and instructions for production:

You can also visit Fluid Homepage to get relevant documents.

Quick Demo

Demo 1: Accelerate Remote File Accessing with Fluid
Demo 2: Machine Learning with Fluid
Demo 3: Accelerate PVC with Fluid
Demo 4: Preload dataset with Fluid
Demo 5: On-the-fly dataset cache scaling

Roadmap

See ROADMAP.md for the roadmap details. It may be updated from time to time.

Community

Feel free to reach out if you have any questions. The maintainers of this project are reachable via:

DingTalk Group:

WeChat Group:

WeChat Official Account:

Slack:

  • Join in the CNCF Slack and navigate to the #fluid channel for discussion.

Contributing

Contributions are highly welcomed and greatly appreciated. See CONTRIBUTING.md for details on submitting patches and the contribution workflow.

Adopters

If you are interested in Fluid and would like to share your experiences with others, you are warmly welcome to add your information on ADOPTERS.md page. We will continuously discuss new requirements and feature design with you in advance.

Open Source License

Fluid is under the Apache 2.0 license. See the LICENSE file for details. It is vendor-neutral.

Report Vulnerability

Security is a first priority thing for us at Fluid. If you come across a related issue, please send an email to fluid.opensource.project@gmail.com. Also see our SECURITY.md file for details.

Code of Conduct

Fluid adopts CNCF Code of Conduct.

Release History

VersionChangesUrgencyDate
v1.0.8## New Features 1. **Support for Native Sidecar Mode** In FUSE Sidecar deployments, the Kubernetes Native Sidecar injection scheme can be enabled to resolve lifecycle management, startup sequencing, and resource isolation limitations of traditional Sidecars. 2. **Fluid Dashboard KubeSphere Extension** Adds Web dashboard for Fluid, including dataset, runtime, and dataload management. 3. **Open ThinRuntime PodMetadata Configuration** Supports dynamic configuration of PodMetadatLow10/31/2025
v1.0.7## New Features 1. **JindoRuntime** - Default Jindo version upgraded to 6.9.1, improving read-bandwidth utilization. - File-prefetcher version now automatically matches the JindoFS image tag (#5235). - Added support for DLS-prefix URLs (#5158). 2. **ThinRuntimeProfile** - Users can now define custom Pod lifecycle hooks (#5123). 3. **Sidecar mode** - Multiple sidecar-mounted Pods can coexist on the same node without mount-point conflicts; webhook automatically allocates uniLow9/20/2025
v1.0.6#### New Features 1. **Prefetch Sidecar** Accelerates model file loading for inference services and improves bandwidth utilization. 2. **JuiceFSRuntime: Fuse Seamless Upgrade** Automatically cleans up idle pods during Fuse image updates (enabled via `cleanPolicy: onFuseChange`). 3. **JuiceFSRuntime: Preheating Parameter Customization** Supports advanced preheating parameters in `Dataload` (refer to [JuiceFS Documentation](https://juicefs.com/docs/zh/community/command_referenLow7/12/2025
v1.0.5## New Features - **Enhanced Binding Efficiency in Kubernetes Clusters** Optimized ThinRuntimeController’s binding performance for large-scale Dataset deployments, reducing resource scheduling latency in Kubernetes clusters. - **Integration with Third-Party Storage via HostPath** Enabled Dataset to mount third-party storage systems (e.g., NFS, Ceph) through node-local HostPath volumes, extending compatibility with heterogeneous data sources. - **Distributed Fault Tolerance fLow2/26/2025
v1.0.4## New Features - Added support to configure the `maxConcurrentReconciles` parameter for the `ThinRuntimeController`, allowing for improved concurrency management. - Introduced support for resource configuration within the Fluid webhook, enabling more precise control over resource allocation. - Added support for configuring `Burst` (default 30) and `Qps` (default 20) settings for client interactions with the APIServer through the Fluid Webhook, enhancing request handling capabilities. - ImplLow1/1/2025
v1.0.3## New Features 1. Dynamic mount now supports exposing mount status updates to users. 2. Supports configuring current data flow operations based on the affinity of previous data operations to optimize processing efficiency. 3. Enables data plane monitoring of JindoRuntime using Prometheus. 4. Allows configuration of imagePullSecret in Runtime level. ## Bug Fixes. 1. Fixed an issue where configuring affinity/nodeSelector/schedulerName/imagePullSecrets parameters in CronDataload caused pLow10/30/2024
v1.0.2## New Features * Support setting resources requests and limits for DataProcess * Add dataset in used info to mutated app pods * Support hierarchical imagePullSecrets configuration in AlluxioRuntime * Support subpath in dynamic-mount ## What's Changed * Bump golang.org/x/net from 0.26.0 to 0.27.0 by @dependabot in https://github.com/fluid-cloudnative/fluid/pull/4200 * Prepare 1.0.2 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/4207 * Delete the unused rbac roles for vLow8/30/2024
v1.0.1## New Features - Support data flow affinity scheduling. - Support dynamic updates for dataset mount points. - Support data acceleration for Fluid Persistent Volume Claim in sidecar mode ## What's Changed * Prepare 1.0.1 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3886 * fix jindo rbac remove secret get by @frankleaf in https://github.com/fluid-cloudnative/fluid/pull/3974 * support parallel tasks for cron datamigrate by @xliuqq in https://github.com/fluid-cloudnative/Low7/12/2024
v1.0.0## Highlights - Introduced a flexible, configurable, tiered data locality scheduling capability to optimize the affinity between application pods and data cache. - Provided DataProcess for custom data operations, and add three trigger strategies for data operations for all the data operations – once, onEvent, Cron. - Provided a Python SDK with dataflow capabilities for data scientists. - Implemented A new runtime for sharing in-memory immutable data: VineyardRuntime is supported in Fluid. -Low5/14/2024
v0.9.3## What's Changed * Prepare 0.9.3 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3638 * enhance: remove jindoruntime's fsGroup (#3632) by @frankleaf in https://github.com/fluid-cloudnative/fluid/pull/3641 * Remove fsgroup from alluxio runtime (#3635) by @TrafalgarZZZ in https://github.com/fluid-cloudnative/fluid/pull/3654 * cherry-pick to 0.9: del metadata sync && del duplicate metrics (#3380) by @TrafalgarZZZ @Hexilee in https://github.com/fluid-cloudnative/fluid/pull/3738 Low3/13/2024
v0.9.2## What's Changed * Prepare 0.9.2 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3374 * enhancement for 0.9: remove juicefs fuse readiness probe (#3435) by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/3436 * Cherry-pick to 0.9: check path existence and lock in NodeUnpublishVolume (#3284) by @TrafalgarZZZ in https://github.com/fluid-cloudnative/fluid/pull/3450 * fail fast with wrong kubelet rootdir (#3331) by @wangshli in https://github.com/fluid-cloudnative/fluLow10/1/2023
v0.8.7## What's Changed * Update to 0.8.7 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3223 * Change reinvocationPolicy to IfNeeded by @lizzzcai in https://github.com/fluid-cloudnative/fluid/pull/3226 * Cherry-pick bugfix for setting fuse args when using Alluxio 2.8+ by @TrafalgarZZZ in https://github.com/fluid-cloudnative/fluid/pull/3231 * fix default master journal type setting (#3281) by @VincentLeeMax in https://github.com/fluid-cloudnative/fluid/pull/3291 * Mount kubelet.coLow7/25/2023
v0.9.1## What's Changed * Prepare 0.9.1 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3263 * Fix default Alluxio writetype config (#3262) by @zhang-x-z in https://github.com/fluid-cloudnative/fluid/pull/3265 * bugfix: fix csi plugin loop mount bug (#3287) by @wangshli in https://github.com/fluid-cloudnative/fluid/pull/3290 * Build docker image for release 0.9.1 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3294 **Full Changelog**: https://github.com/fluid-clLow6/24/2023
v0.9.0# v0.9.0 ### Breaking Changes - Change matching pod requests mode of webhook from namespaceSelector to objectSelector ### Features - Add thinRuntime to simplify integration with third-party storage systems - Addon component for Fluid's open source CubeFS, NFS - Support for accessing data across namespaces - Support for subDataset - Native acceleration system EFCRuntime for distributed file systems NFS, GPFS - Support dataMigrate for data migration operations (currently only suppLow5/26/2023
v0.8.6## What's Changed * Add approvers for release 0.8 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2864 * Update fluid version to 0.8.6 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2843 * add reinvocationPolicy by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/3047 * Cherry-pick fix worker volumes in alluxio chart (#2330) by @wangshli in https://github.com/fluid-cloudnative/fluid/pull/3191 * Fix rbacs and limit CSI Plugin's node related access byLow5/8/2023
v0.8.5## What's Changed * Update fluid version to 0.8.5 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2623 * [juicefs] fix worker cache when set option (https://github.com/fluid-cloudnative/fluid/pull/2563) @zwwhdls * fix multi cache dir (https://github.com/fluid-cloudnative/fluid/pull/2639) @zwwhdls * [Enhancement]CSI plugin checks mount point liveness before binding mount points (https://github.com/fluid-cloudnative/fluid/pull/2703) @TrafalgarZZZ * Prettify error messages foLow3/27/2023
v0.8.4## What's Changed * Origin/init release 0.8.4 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2616 * Cherry-pick update fuse container as root user in sidecar mode (#2501) by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/2618 * fix NPE issue of cleaning juicefsruntime (#2505) by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/2619 * Cherry-pick Juicefs/fix secret key (#2509) by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/2620 * CheLow2/18/2023
v0.8.3## What's Changed * Update fluid version to 0.8.3 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2482 * Cherry-pick update fuse default image (#2118) by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/2484 * update resource/privileged of juicefs (#2130) by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/2485 * cherry-pick juicefs for 0.8.3 by @zwwhdls in https://github.com/fluid-cloudnative/fluid/pull/2487 * Build docker image for v0.8.3 by @cheyanLow1/9/2023
v0.8.2## What's Changed * Update fluid version to 0.8.2 by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2193 * Automated cherry pick of #2185: support image pull secrets from fluid helm charts to alluxio runtime by @allenhaozi in https://github.com/fluid-cloudnative/fluid/pull/2194 * Feat/fix parse image pull secret (#2190) by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2195 * Automated cherry pick of #2236: Change alluxio podManagementPolicy from OrderedReady ParLow11/8/2022
v0.8.1## v0.8.1 ### What's Changed * Fix no matched controller blocking dataset deletion (#2161) by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2163 * Fix no interval resync if the dataset is created without runtime by @cheyang in https://github.com/fluid-cloudnative/fluid/pull/2167 * Add the toleration to fuse (#2158) by @wang-mask in https://github.com/fluid-cloudnative/fluid/pull/2168 **Full Changelog**: https://github.com/fluid-cloudnative/fluid/compare/v0.8.0...v0.8Low10/12/2022
v0.8.0## v0.8.0 ### Features - Lifecycle management of Serverless Job with fluid sidecar support - Enabling Runtime Controller on demand - Arm64 support with JuicefsRuntime - Container Network with short-circuit read support - Leader election support for Controllers and Webhook - Automatic CRD upgrader - Restrict Pod scheduling to dataset cache nodes - Tens of thousands of nodes support - Image pull secrets support - GCS support for Alluxio Runtime ### Refactorings - Port Allocation Low9/4/2022
v0.7.0## v0.7.0 ### Breaking Changes - Update Kubernetes v1.20.12 dependencies and use Go 1.16 - Update CustomResourceDefinition to apiextensions.k8s.io/v1 - Update MutatingWebhookConfiguration to admissionregistration.k8s.io/v1 - Update CSIDriver to storage.k8s.io/v1 ### Features - Support fuse sidecar auto injection for all the runtimes，it’s helpful for no CSI environment - Support fuse auto recovery and upgrade - Support lazy fuse mount mode - Support New cache runtime: JuiceFLow3/2/2022
v0.6.0## v0.6.0 ### Features - Support dataset cache autoscaling and cronscaling - Add dataset mounting point dynamically update feature - Enhance dataset cache aware Pod scheduling - Enhance HA support for cache Runtime - Support new cache Runtime：GooseFS ### Bugs - Fix [if alluxioruntime is nonroot, databackup will fail](https://github.com/fluid-cloudnative/fluid/issues/745) - Fix [Node labels exceeds maximum length limit for long namespace and name](https://github.com/fluid-cloudnLow8/11/2021
v0.5.0### Features - Support on-the-fly dataset cache scale out/in - Add Metadata backup and restore operation - Support Fuse global mode and toleration - Enhance Prometheus monitoring support for AlluxioRuntime - Support new Runtime：JindoFS - Support HDFS configuration ### Bugs - [Fix compatibility issue of K8s 1.19+](https://github.com/fluid-cloudnative/fluid/issues/603) Please check [docs](https://github.com/fluid-cloudnative/fluid/blob/master/docs/en/TOC.md) to learn how to use FLow3/16/2021
v0.4.0### Features - Prefetch Dataset automatically before using it - Support managing 4,000,000 small files - Support deploying multiple dataset in the same node - Support access dataset by using the HCFS Access Endpoint instead of Fuse ### Bugs - [MountVolume.SetUp failed in GKE](https://github.com/fluid-cloudnative/fluid/issues/222) - [fuse.csi.fluid.io not found in the list registered CSI drivers when node restart in k8s 1.15.1](https://github.com/fluid-cloudnative/fluid/issues/220) Low11/6/2020
v0.3.0* Accelerate data access in Host-path mode in K8s * Accelerate data access in Persistent Volume mode in K8s * Support the underlay storage which is configured only with non-root * Make the Alluxio Runtime’s settings optimized by default Please check [docs](https://github.com/fluid-cloudnative/fluid/blob/master/docs/zh/TOC.md) to learn how to use Fluid. **Compatible Alluxio Version**: Commit: https://github.com/cheyang/alluxio/commit/e0feba36cba1b57a3a9a0b893f7a477556e4955c Branch: httLow10/1/2020
v0.2.0* Add code coverage badge * Update chart v0.2.0 * Update docs * Refactor the package name Please check [docs](https://github.com/fluid-cloudnative/fluid/blob/master/docs/en/TOC.md) to learn how to use Fluid.Low9/5/2020
v0.1.0Initial Release of the Fluid Controller Manager. Low8/30/2020

Dependencies & License Audit

Loading dependencies...

Similar Packages

FluidVCL🎨 Enhance your Delphi applications with FluidVCL, a modern set of high-performance VCL components offering sleek visuals and full customization.main@2026-06-01
einoThe ultimate LLM/AI application development framework in Go.v0.9.4
suricataType-safe AI agents for Go. Suricata combines LLM intelligence with Go’s strong typing, declarative YAML specs, and code generation to build safe, maintainable, and production-ready AI agents.0.0.0
PocketFlow-Zig🚀 Build LLM-powered workflows with PocketFlow-Zig, a lightweight framework leveraging Zig's features for efficient, type-safe programming.main@2026-06-06
ai-dev-assistant-frameworkA plug-and-play framework for AI-assisted software development, enhancing context-aware collaboration in complex codebases. Perfect for tools like GitHub Copilot. 🐙✨main@2026-06-06

More in Frameworks

langchainThe agent engineering platform
deer-flowAn open-source long-horizon SuperAgent harness that researches, codes, and creates. With the help of sandboxes, memories, tools, skill, subagents and message gateway, it handles different levels of ta
tqdmFast, Extensible Progress Meter
simBuild, deploy, and orchestrate AI agents. Sim is the central intelligence layer for your AI workforce.