Making yet another run at a secure, redundant, distributed backup system

[Jody Harris]

I've brought this up several times in the past. I've actually attempted to
establish a grid and had it fail last year, however....

I'm still interested in a secure, redundant, distributed backup system, and
of the systems out there, Tahoe-LAFS (http://tahoe-lafs.org) seems to be the
one that's making the most progress.

VolunteerGrid2 has been initiated, and we have established some guidelines
and have some nodes online. If you're interested, you can read about the
grid background on our wiki at http://bigpig.org/

jody

----
- Think carefully.

[Gabriel Ortiz]

Sounds interesting. I sent a subscription request to the mailing list.

Post by Jody Harris
I've brought this up several times in the past. I've actually attempted to
establish a grid and had it fail last year, however....
I'm still interested in a secure, redundant, distributed backup system, and
of the systems out there, Tahoe-LAFS (http://tahoe-lafs.org) seems to be the
one that's making the most progress.
VolunteerGrid2 has been initiated, and we have established some guidelines
and have some nodes online. If you're interested, you can read about the
grid background on our wiki at http://bigpig.org/
jody
----
- Think carefully.
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Jody Harris]

Great! Did you read the rules posted? What did you think of them?

jody
----
- Think carefully.

Post by Gabriel Ortiz
Sounds interesting. I sent a subscription request to the mailing list.

Post by Jody Harris
I've brought this up several times in the past. I've actually attempted

to

Post by Jody Harris
establish a grid and had it fail last year, however....
I'm still interested in a secure, redundant, distributed backup system,

and

Post by Jody Harris
of the systems out there, Tahoe-LAFS (http://tahoe-lafs.org) seems to be

the

Post by Jody Harris
one that's making the most progress.
VolunteerGrid2 has been initiated, and we have established some

guidelines

Post by Jody Harris
and have some nodes online. If you're interested, you can read about the
grid background on our wiki at http://bigpig.org/
jody
----
- Think carefully.
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Gabriel Ortiz]

They sound totally reasonable and my first worry- that nodes were
going to be of less-than-usable size was quickly assuaged. Is there
any facility for offline data transfer? It seems that if we had a lot
of people locally who wanted to participate, it would be kind of nice
to have a data transfer party to get a baseline data set shared over
fast local connections.

Post by Jody Harris
Great! Did you read the rules posted? What did you think of them?
jody
----
- Think carefully.

Post by Gabriel Ortiz
Sounds interesting. I sent a subscription request to the mailing list.

Post by Jody Harris
I've brought this up several times in the past. I've actually attempted to
establish a grid and had it fail last year, however....
I'm still interested in a secure, redundant, distributed backup system, and
of the systems out there, Tahoe-LAFS (http://tahoe-lafs.org) seems to be the
one that's making the most progress.
VolunteerGrid2 has been initiated, and we have established some guidelines
and have some nodes online. If you're interested, you can read about the
grid background on our wiki at http://bigpig.org/
jody
----
- Think carefully.
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Jody Harris]

That's actually an interesting idea..... Even thought we have a general rule
against lurkers, several of the developers have been encouraged to eves drop
on our conversation to see how their product is being used in the wild.
Posting that question to the VolunteerGrid2 list might get you an answer,
but posting it to the tahoe-dev list would have a better chance.

Because a redistribution of shares algorithm is not in place, the pre-load
you did would probably remain on the machines you "data partied" with until
either redistribution was implemented, or there was some node loss among the
original storage nodes.

It is an interesting thought.

jody
----
- Think carefully.

Post by Gabriel Ortiz
They sound totally reasonable and my first worry- that nodes were
going to be of less-than-usable size was quickly assuaged. Is there
any facility for offline data transfer? It seems that if we had a lot
of people locally who wanted to participate, it would be kind of nice
to have a data transfer party to get a baseline data set shared over
fast local connections.

Post by Jody Harris
Great! Did you read the rules posted? What did you think of them?
jody
----
- Think carefully.

Post by Gabriel Ortiz
Sounds interesting. I sent a subscription request to the mailing list.

Post by Jody Harris
I've brought this up several times in the past. I've actually

attempted

Post by Jody Harris

Post by Gabriel Ortiz

Post by Jody Harris
to
establish a grid and had it fail last year, however....
I'm still interested in a secure, redundant, distributed backup

system,

Post by Jody Harris

Post by Gabriel Ortiz

Post by Jody Harris
and
of the systems out there, Tahoe-LAFS (http://tahoe-lafs.org) seems to

be

Post by Jody Harris

Post by Gabriel Ortiz

Post by Jody Harris
the
one that's making the most progress.
VolunteerGrid2 has been initiated, and we have established some guidelines
and have some nodes online. If you're interested, you can read about

the

Post by Jody Harris

Post by Gabriel Ortiz

Post by Jody Harris
grid background on our wiki at http://bigpig.org/
jody
----
- Think carefully.
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Michael W. Folsom]

Jody:

I'm curious - how many gig/terabytes are you looking to backup? .. how
many systems?

I've dealt with a couple of backup systems and we are quickly getting to
a point where boxes have file systems that are so big its really not
possible to back them up unless you have a ton of $'s -


Mike

Post by Jody Harris
I've brought this up several times in the past. I've actually
attempted to establish a grid and had it fail last year, however....
I'm still interested in a secure, redundant, distributed backup
system, and of the systems out there, Tahoe-LAFS
(http://tahoe-lafs.org) seems to be the one that's making the most
progress.
VolunteerGrid2 has been initiated, and we have established some
guidelines and have some nodes online. If you're interested, you can
read about the grid background on our wiki at http://bigpig.org/
jody
----
- Think carefully.
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Jody Harris]

At this point, we're all looking at fairly small backups -- on the order of
tens to hundreds of gigabytes.

You also have the bandwidth limitations, so this isn't the end-all, be-all
backup solution.

I do know that one of the top-dogs at Shutterfly was experimenting with
building an in-house tahoe grid for storing client images on. I don't know
what became of that in the end, but it was an amazing physical design.

He was using one of these as the hardware:
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

http://tahoe-lafs.org/pipermail/tahoe-dev/2010-February/003889.html


----
- Think carefully.

Post by Michael W. Folsom
I'm curious - how many gig/terabytes are you looking to backup? .. how
many systems?
I've dealt with a couple of backup systems and we are quickly getting to a
point where boxes have file systems that are so big its really not possible
to back them up unless you have a ton of $'s -
Mike
I've brought this up several times in the past. I've actually attempted to
establish a grid and had it fail last year, however....
I'm still interested in a secure, redundant, distributed backup system,
and of the systems out there, Tahoe-LAFS (http://tahoe-lafs.org) seems to
be the one that's making the most progress.
VolunteerGrid2 has been initiated, and we have established some
guidelines and have some nodes online. If you're interested, you can read
about the grid background on our wiki at http://bigpig.org/
jody
----
- Think carefully.
_______________________________________________
NMLUG mailing list
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Aaron Birenboim]

I have a program which is trying to write several dozen
0.25 to 8 MByte files/sec, when recording diagnostics.

My machine gets into a high ioWait situation, and "browns-out".
The machine barely runs for several minutes.

The test code can complete, and the high ioWait continues
for several minutes.

After the program is done, I can do "sudo sync",
which will take several minutes to complete,
then the machine is OK.

I have repeated this on several machines, same results.
I haven even tried writing to NFS... the client machine
browns-out, the server is fine.

I am not talking about some amazingly massive amount of data here.
If things worked, I would be recording about 1G in about 2 sec.
I realize that I should only be able to expect (at worst)
about 50MBytes/sec to disk, hence my 2-second run (no diagnostic
file writes) should take about 20 sec with the diagnostic data dump.
However, this run takes more like 10 minutes, and my machine
goes wanky.

Any idea what might do this?

In general, I am new'ing a 1-2 MB buffer, filling it with
data (2D arrays of floats), then writing it to a file,
then delete[] 'ing the buffer ...
as fast as I can, for some 300 files or so.

Is it not reasonable for me to be able to
create these mid-sized files with a bandwidth
of around (at least) 50MBytes/sec to SATA-II, running ext4?

aaron

[Matthew Bowie]

Are they created one at a time, or are you creating and writing to all
of them simultaneously? If the latter, it's possible that the seek
times are what are killing you. Are you doing anything that would
force writes (like fsync) and prevent the OS caching from working?
What kind of write rates do you see writing to a single file using dd?

Matthew Bowie
Programmer/IT Consultant

Post by Aaron Birenboim
I have a program which is trying to write several dozen
0.25 to 8 MByte files/sec, when recording diagnostics.
My machine gets into a high ioWait situation, and "browns-out".
The machine barely runs for several minutes.
The test code can complete, and the high ioWait continues
for several minutes.
After the program is done, I can do "sudo sync",
which will take several minutes to complete,
then the machine is OK.
I have repeated this on several machines, same results.
I haven even tried writing to NFS... the client machine
browns-out, the server is fine.
I am not talking about some amazingly massive amount of data here.
If things worked, I would be recording about 1G in about 2 sec.
I realize that I should only be able to expect (at worst)
about 50MBytes/sec to disk, hence my 2-second run (no diagnostic
file writes) should take about 20 sec with the diagnostic data dump.
However, this run takes more like 10 minutes, and my machine
goes wanky.
Any idea what might do this?
In general, I am new'ing a 1-2 MB buffer, filling it with
data (2D arrays of floats), then writing it to a file,
then delete[] 'ing the buffer ...
as fast as I can, for some 300 files or so.
Is it not reasonable for me to be able to
create these mid-sized files with a bandwidth
of around (at least) 50MBytes/sec to SATA-II, running ext4?
           aaron
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Aaron Birenboim]

Post by Matthew Bowie
Are they created one at a time, or are you creating and writing to all
of them simultaneously? If the latter, it's possible that the seek
times are what are killing you.

one at a time, serially.

Post by Matthew Bowie
Are you doing anything that would
force writes (like fsync) and prevent the OS caching from working?

I have put sync() calls in there, and it made little difference.
Instead of my program exit in 2-3 minutes, and lock the system
for the next 10 minutes, with sync in there it might take
10 minutes to run, and lock the system for 5 minutes afterwards.
I would think that fsync() should help.... If I could throttle
my process down to wait for 50MB/sec disk writes, I'd be thrilled.
It is more like 1MB/sec now... and my machine semi-locks up while
it is doing this, and for 10 minutes after I'm done.

Post by Matthew Bowie
What kind of write rates do you see writing to a single file using dd?
sudo sync;dd if=/dev/zero of=t bs=1024 count=100000;sudo sync

100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.547869 s, 187 MB/s

Printed output almost immediately, but took 50 sec for
the second sudo sync to complete!
Who else has this?

Most of my systems are Debian or Ubuntu.

On my fileserver, I got:

sudo sync;dd if=/dev/zero of=t bs=1024 count=100000;sudo sync
[sudo] password for aaron:
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.238117 s, 430 MB/s

returned almost immediately. hmmmmm.....

What might be different about my workstations?
(I have tried three of them)

aaron

[Matthew Bowie]

Post by Aaron Birenboim
What might be different about my workstations?
(I have tried three of them)

File server might be using faster disks or disks with write cache
enabled. I *believe* that a sync will just force the data out of the
OS cache to the drive, but if it has write caching enabled then it may
not actually force it to the platter, but not sure about this.

Is the fileserver using the same filesystem as the workstations?

Matthew Bowie

[Aaron Birenboim]

Post by Matthew Bowie

Post by Aaron Birenboim
What might be different about my workstations?
(I have tried three of them)

File server might be using faster disks or disks with write cache
enabled. I *believe* that a sync will just force the data out of the
OS cache to the drive, but if it has write caching enabled then it may
not actually force it to the platter, but not sure about this.
Is the fileserver using the same filesystem as the workstations?

Not quite. The fileserver I tested was ubuntu server 10.04.
My home workstation is lucid (I have no idea what number that is. could
be 9.something)

I have debian servers I can try at work...
they will run dd, but not my code.

aaron

[Aaron Birenboim]

Looking for a junior programmer with strong math background.
Mostly C++. Perhaps other languages.
Must be US citizen, and/or able to obtain a clearance at some point in
the future.

Send me a resume if interested.

aaron

[J. Marsden DeLapp]

Post by Aaron Birenboim
Looking for a junior programmer with strong math background.
Mostly C++. Perhaps other languages.
Must be US citizen, and/or able to obtain a clearance at some point in
the future.

Aaron,

You might want to reconsider the wording on your job advertisement.
Advertising a "junior" position could be considered age discrimination and
subject you to an EEOC complaint.

Mars
--
=============================================================
J. Marsden DeLapp, PE
President
DeLapp & Associates, Inc. dba DeLapp Engineering.
Providing lighting and power planning, design and analysis services
for commercial, industrial and large residential facilities.
1190 Harrison Road Ste 3a
Santa Fe NM 87507
[phone]
http://DeLapp.com
=============================================================

[Matthew Bowie]

LOL. Junior in this context is pretty well understood to refer to
skill/experience level, not age.

Post by J. Marsden DeLapp

Post by Aaron Birenboim
Looking for a junior programmer with strong math background.
Mostly C++.  Perhaps other languages.
Must be US citizen, and/or able to obtain a clearance at some point in
the future.

Aaron,
You might want to reconsider the wording on your job advertisement.
Advertising a "junior" position could be considered age discrimination and
subject you to an EEOC complaint.
Mars
--
=============================================================
J. Marsden DeLapp, PE
President
DeLapp & Associates, Inc. dba DeLapp Engineering.
Providing lighting and power planning, design and analysis services
for commercial, industrial and large residential facilities.
1190 Harrison Road Ste 3a
Santa Fe NM 87507
[phone]
http://DeLapp.com
=============================================================
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Pete Zaitcev]

On Thu, 11 Aug 2011 11:41:05 -0600

Post by J. Marsden DeLapp
You might want to reconsider the wording on your job advertisement.
Advertising a "junior" position could be considered age discrimination and
subject you to an EEOC complaint.

Oh please. "Junior" means "works for little pay", even if the person
in question is 100 years old.

-- Pete

[James Bunnell]

Post by Pete Zaitcev
On Thu, 11 Aug 2011 11:41:05 -0600

Post by J. Marsden DeLapp
You might want to reconsider the wording on your job advertisement.
Advertising a "junior" position could be considered age discrimination and
subject you to an EEOC complaint.

Oh please. "Junior" means "works for little pay", even if the person
in question is 100 years old.
-- Pete

implies intern, child, teen, youth, etc.....

[Chris McMahon]

Post by James Bunnell
implies intern, child, teen, youth, etc.....

FWIW, I'm an experienced, expert, well-known software tester and every once
in a while I get an urge to move to the dev side of the house. I would
answer a "Junior Programmer" ad, and I'm 47 years old.

Back to lurking...

-Chris