Disconnected - Cached Read/Write - Striped - Failsafe - Distributed File System

I'd like to push forward a distributed file system that follows all of the ideas below.

Striped - Fail-safe:

Striped for performance, and fail-safe up to N nodes, depending on the configuration. Something similar to RAID 6.

Disconnected - Cached Read/Write:

     Local processes would write locally to the disks and the DFS would cache the data, intelligently distributing it to the filesystem as need be. If the node becomes disconnected, the DFS will cache the data until the node becomes connected again.

The applications of this are extremely wide. The striping would allow for applications in high performance clusters. The the cached read/write would allow for "traveling" nodes, similar to version control systems such as Git/Mercurial/Bizaar, that become disconnected for periods of time.

My intended application would be to deploy the DFS nodes as cheap virtual private servers from a multitude of companies in order to attain reliability and not have to worry about an IT department and hardware costs. Even over the internet the striping from multiple nodes should allow for "torrential" throughput.

If you're interested in discussing this more thoroughly, drop me a line.

Awesome (OS X Server Slices)

A company has finally started slicing OS X Server

http://www.mediatemple.net/labs/xv/

Goodbye SVN Hello GIT

So I've given up subversion for git lately, and I'm not looking back. Git, gets out of my way, is easier to use, saves tons of disk space, is distributed, and generally not as big of a headache as svn is.

Migrating from SVN to GIT was easy (~10mins to migrate a grok and python project)

Some good resources:

Tutorial: 

http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html

Another: 

http://www.vimeo.com/369095

GIT vs SVN Command Comparison: 

http://git.or.cz/course/svn.html

A great git repository (not that it's needed): 

http://www.github.com

Migration guide: 

http://www.daniellucraft.com/blog/2007/11/migrating-from-subversion-to-git/

Linus addressing googlers: 

http://www.youtube.com/watch?v=4XpnKHJAok8

Syncing with subversion: 

http://www.kernel.org/pub/software/scm/git/docs/git-svn.html

Best Practices:

http://www.jdl.com/papers/BestCollabGit.pdf

Annoying Grok Problem

When creating a grokproject for the first time or running it's buildout, have you ever encountered this?

Develop: '/home/xkenneth/bb/work/buildoutbuilder/src/buildoutbuilder.grokapp/trunk/.' Develop: '/home/xkenneth/bb/work/buildoutbuilder/src/buildoutbuilder.grokapp/trunk/../../buildoutbuilder.managers/trunk/' 

While:   

Installing.   Getting section app.   Initializing section app.   

Loading zc.buildout recipe entry zc.zope3recipes>=0.5.3:application.  

An internal error occured due to a bug in either zc.buildout or in a recipe being used:  ImportError: 

No module named schemaless

Well there's an easy solution.

Create a new virtual environment with the --no-site-packages option, and rerun the buildout/grokproject script from there.

Don't know what virtual environment is?

http://pypi.python.org/pypi/virtualenv

Don't know what buildout is?

http://pypi.python.org/pypi/zc.buildout

An excellent video introduction to buildout:

http://rhodesmill.org/brandon/buildout/

OpenWITSML

I've created yet another project on openplans.org, OpenWITSML. WITSML is an XML schema defined by Energistics for transferring information in the drilling/energy industry. After attending the public seminar here in Houston, I've decided that an open source implementation of the client and server might be a worthwhile venture. If you're interested in the project, please sign up, the more people that get involved, the quicker this will become a reality.

OpenSurfaceSystem

I've just created a new project on openplans.org called OpenSurfaceSystem. This is an idea I've had for a while now, and I'd like to see if anyone else has interest in it.

Google Summer of Code Project Accepted!

My proposal for the Google Summer of Code was accepted! This post is a bit late as I've already been coding for a few weeks now. If you're interested in the Buildout Builder, feel free to check it out.