So I'm doing some testing with glusterfs 3.0.5 before we look into deploying it into production. I basically have to do this as a rolling deployment, since servers will need to have their OSes reloaded, and some will need their RAID arrays redone. Basically, I'll have to reload the first server, setup glusterfs on it as a standalone, move some data on to it. Then, reload another server, and set it up for replication. Finally, reload 2 more servers that replicate each other, and distribute from the first 2 nodes. On 4 centos 5.4 test machines, I've gotten this scenario to work fine, for the most part. The problem is getting the files redistributed after adding the last 2 nodes. I've gotten the defrag.sh and scale-n-defrag.sh scripts from here http://ftp.gluster.com/pub/gluster/glusterfs/misc/defrag/ and edited the scale-n-defrag.sh script to reflect the proper mount point. When I run the script, it creates plenty of log files for each file, saying they were successfully relocated, but when I look in the /data/export directory of the second 2 nodes, I don't see any of the files replicated to either of them. New folders get distributed correctly, but the stuff existing before does not get redistributed, even after running the script. If anyone has any ideas, I'd love to hear them. There seems to be almost zero documentation on these scripts, and this site seems to have far more docs and info on glusterfs than the gluster site does. I can post the config files if they seem relevant, but I do already have unhashed-sticky-bit and lookup-unhashed on for the distribute volume in glusterfs.vol.
