Friday, June 20, 2008

Disconnected - Cached Read/Write - Striped - Failsafe - Distributed File System

I'd like to push forward a distributed file system that follows all of the ideas below.

Striped - Fail-safe:
Striped for performance, and fail-safe up to N nodes, depending on the configuration. Something similar to RAID 6.

Disconnected - Cached Read/Write:
     Local processes would write locally to the disks and the DFS would cache the data, intelligently distributing it to the filesystem as need be. If the node becomes disconnected, the DFS will cache the data until the node becomes connected again.

The applications of this are extremely wide. The striping would allow for applications in high performance clusters. The the cached read/write would allow for "traveling" nodes, similar to version control systems such as Git/Mercurial/Bizaar, that become disconnected for periods of time.

My intended application would be to deploy the DFS nodes as cheap virtual private servers from a multitude of companies in order to attain reliability and not have to worry about an IT department and hardware costs. Even over the internet the striping from multiple nodes should allow for "torrential" throughput.

If you're interested in discussing this more thoroughly, drop me a line.

No comments: