bash script: move large LV to another machine

[Ed Heron]

I've copied part of a large (100G) logical volume from one virtual
machine server to another, but it broke. I'm looking for a way to
verify it copied and complete the copy in a way that would survive
another interruption. rsync is cool, but doesn't work on devices.

Prior to running this script, I generated a ssh key and copied it to
the remote machine to enable automatic ssh login.

This isn't working. Can anybody see why? I suspect the last dd
command, but I'm not seeing what is wrong.

src_dev='/dev/vg/lv'
dst_addr='192.168.1.2'
dst_dev='/dev/vg/lv'
# sync_bs is the size of my logical extents
sync_bs='32M'
# dev_size is number of logical extents
dev_size='3200'

skip=0
while [ $skip -lt $dev_size ]
do
echo -n " $skip"
src_md5=`dd if=$src_dev bs=$sync_bs skip=$skip count=1 2>/dev/null |
md5sum`
dst_md5=`ssh $dst_addr "dd if=$dst_dev bs=$sync_bs skip=$skip count=1
2>/dev/null | md5sum"`
if [ "$src_md5" == "$dst_md5" ]
then
# this block matches, move to next block
echo -n "+"
let seek=seek+1
else
# this block doesn't match, so send. don't increment so we test.
echo -n "-"
dd if=$src_dev bs=$sync_bs skip=$skip count=1 2>/dev/null | ssh
$dst_addr "dd of=$dst_dev bs=$sync_bs skip=$skip 2>/dev/null"
fi
done

[Ed Heron]

Hi Ed,

Post by Ed Heron
I've copied part of a large (100G) logical volume from one virtual
machine server to another, but it broke. I'm looking for a way to
verify it copied and complete the copy in a way that would survive
another interruption. rsync is cool, but doesn't work on devices.

What I sometimes do is attach the disk image of the broken virtual machine
as a second drive of another (working) VM. Then you can run rsync or
whatever you like. Also, you can attach a block device directly to a
virtual machine as a disk image, if you're willing to run kvm as root.
Geoff

Thanks. I've done the same when both virtual disks are on the same
server. In this case, my servers are in different physical locations.

My virtual disk is for a windows machine.

I'm really trying to do a device sync to a remote location. DRBD
isn't reliable unless you pay their license & support fees for their
commercial buffering solution. They want about 18K per year for my 8
locations.

If I can sync the device changes, I could snapshot the windows
machines and sync it over.

Ideally, I'd get that device copy patch for rsync working, but I'm
having difficulty with it.

[Ed Heron]

... I suspect the last dd
command, but I'm not seeing what is wrong.
... dd if=$src_dev bs=$sync_bs skip=$skip count=1 2>/dev/null | ssh
$dst_addr "dd of=$dst_dev bs=$sync_bs skip=$skip 2>/dev/null"
...

OK. skip may be the wrong parameter on the write dd. I changed it to
seek. Doesn't appear to significantly improve the function.

I notice seek talks about breaking the file at that point which can't
really be done with a block device....

I'm starting to get the feeling that I'm trying to use dd in a way
that wasn't intended...

[Ed Heron]

Hi Ed,

skip may be the wrong parameter on the write dd. I changed it to seek.

Definitely.

Thanks.

I notice seek talks about breaking the file at that point which can't
really be done with a block device...

You could use the "notrunc" argument to avoid this. You might also try
the "fdatasync" argument.

I'll take a look at those.

I'm starting to get the feeling that I'm trying to use dd in a way that
wasn't intended...

I think you're on the right track; it should work. Someone has tried
http://lists.samba.org/archive/rsync/2010-June/025164.html ... He used
perl to read and write, but I think dd should work just as well or better.

I've avoided Perl, so far. I know it's unreasonable, but looking at
Perl syntax gives me flashbacks to smooshed APL code.

The thoughts behind that thread are interesting. They say that even
if rsync is patched, it does more work than necessary to synchronize
block devices.

I wouldn't want to cache checksums because I wouldn't want to trust
that the file hasn't been modified without changing modification date.
I'm paranoid...

Geoff

[Ed Heron]

Post by Ed Heron
...

You could use the "notrunc" argument to avoid this. You might also try
the "fdatasync" argument.

I'll take a look at those.

I added conv=notrunc,fdatasync to my write dd without change. I
removed the 2>/dev/null from it as well so I could see what results it
might be reporting.

0+1 records in
0+1 records out
32768 bytes (33 kB) copied, 0.00047 seconds, 69.7 MB/s

I have little idea why it is not writing the entire 32M block.

I tried, on the source, putting the first block that is different in a
local file, copying that file to the destination server, then trying to
use dd to write it in the correct place. It seems to go well.

[***@src ~]# dd if=/dev/vg/lv of=vg.lv.1037 bs=32M skip=1037 count=1
1+0 records in
1+0 records out
33554432 bytes (34 MB) copied, 0.371395 seconds, 90.3 MB/s
[***@src ~]# scp vg.lv.1037 dst:.
vsrv2b.tmp.1037 100% 32MB 32.0MB/s 00:01

[***@dst ~] # dd conv=notrunc,fdatasync if=vg.lv.1037 of=/dev/vg/lv \
bs=32M seek=1037 count=1
1+0 records in
1+0 records out
33554432 bytes (34 MB) copied, 0.498312 seconds, 67.3 MB/s

now that block matches. but my script doesn't work. weird. I'm going
to change my script to create a temp file, copy the file and dd the
piece. I wonder if dd puts any info into the data stream that is
messing with the destination dd.

[Ed Heron]

... I'm going
to change my script to create a temp file, copy the file and dd the
piece. ...

My script appears to work...

--------------------
src_dev='/dev/vg/lv'
dst_addr='192.168.2.2'
dst_dev='/dev/vg/lv'
src_size=3200

src_blk=0
while [ $src_blk -lt $src_size ]
do
echo -n " $src_blk"
src_md5=`dd if=$src_dev bs=32M skip=$src_blk count=1 2>/dev/null |
md5sum`
dst_md5=`ssh $dst_addr "dd if=$dst_dev bs=32M skip=$src_blk count=1
2>/dev/null | md5sum"`
if [ "$src_md5" == "$dst_md5" ]
then
echo -n "+"
let src_blk=src_blk+1
else
echo -n "-"
dd if=$src_dev of=vd-sync.tmp bs=32M skip=$src_blk count=1
2>/dev/null
scp vd-sync.tmp ${dst_addr}:. >/dev/null
ssh $dst_addr "dd conv=notrunc,fdatasync if=vd-sync.tmp of=$dst_dev
bs=32M seek=$src_blk count=1"
fi
done
----------

I'm very annoyed that creating a temp file works but piping the read
dd to the write dd doesn't. Obviously, my knowledge has a hole in it.

[Ed Heron]

Hi Ed,

Post by Ed Heron
I have little idea why it is not writing the entire 32M block.

I wonder if the problem is in the way the socket is breaking up the 32M
write into packets for the internet. Perhaps if you set ibs=32M on the
read side and obs=32M on the write side (don't set bs=32M), then the
default 512 byte blocks would not need to be broken up. You could try
different sizes, e.g. obs=1K on the read side and ibs=1K on the write
side. I bet this will fix it.
Geoff

Interesting thought. I don't want to do checksums in 512 byte pieces,
but I could just change the copy section to:

let lcl_blk=src_blk*65536
dd if=$src_dev ibs=512 skip=$lcl_blk count=65536 | ssh $dst_addr "dd
conv=notrunc,fdatasync of=$dst_dev obs=512 seek=$lcl_blk count=65536"

It appears to work! Yay!

So, should I expand it to make it more generically usable? Something
like dev-sync [<src_addr>:]<src_dev> [<dst_addr>:]<dst_dev>

[Ed Heron]

Post by Ed Heron
dd if=$src_dev ibs=512 skip=$lcl_blk count=65536 | ssh $dst_addr "dd
conv=notrunc,fdatasync of=$dst_dev obs=512 seek=$lcl_blk count=65536"

Note that the "count=65536" on the receiving side assumes ibs=512, but you
don't strictly need to specify this count. You could as well use
dd if=$src_dev ibs=32M obs=1K skip=$src_blk count=1 | ssh $dst_addr "dd
conv=notrunc,fdatasync of=$dst_dev ibs=1K obs=32M seek=$src_blk"
That way, dd can still read the source device and write the destination
device in 32Mb chunks. Smaller chunks may be better, though, in order to
overlap i/o and communication; it's just a matter of optimization.
Geoff

I guess the gymnastics to change block sizes for the transfer aren't
significant and retain script context. I'll try it.

If this were the only thing running on the machines, it wouldn't make
much difference. As a working virtualization host, I'm not sure which
would be nicer to the VMs; 1 big read or lots of little reads.

[Ed Heron]

Post by Ed Heron

Post by Ed Heron
dd if=$src_dev ibs=512 skip=$lcl_blk count=65536 | ssh $dst_addr "dd
conv=notrunc,fdatasync of=$dst_dev obs=512 seek=$lcl_blk count=65536"

Note that the "count=65536" on the receiving side assumes ibs=512, but you
don't strictly need to specify this count. You could as well use
dd if=$src_dev ibs=32M obs=1K skip=$src_blk count=1 | ssh $dst_addr "dd
conv=notrunc,fdatasync of=$dst_dev ibs=1K obs=32M seek=$src_blk"
That way, dd can still read the source device and write the destination
device in 32Mb chunks. Smaller chunks may be better, though, in order to
overlap i/o and communication; it's just a matter of optimization.
Geoff

I guess the gymnastics to change block sizes for the transfer aren't
significant and retain script context. I'll try it.
If this were the only thing running on the machines, it wouldn't make
much difference. As a working virtualization host, I'm not sure which
would be nicer to the VMs; 1 big read or lots of little reads.

Unfortunately, it didn't work. The source read 1 record and sent
32768 records. The destination read 32767+2 records and wrote 1+0
record. A subsequent checksum test said the pieces didn't match.

1+0 records in
32768+0 records out
33554432 bytes (34 MB) copied, 2.30826 seconds, 14.5 MB/s
32767+2 records in
1+0 records out
33554432 bytes (34 MB) copied, 2.52882 seconds, 13.3 MB/s

Since it just keeps testing and sending the same piece, I notice the
receiver reports records in all over the place. ifconfig on both sides
say there aren't any errors, but this is starting to look like data
corruption to me.