[lbackup-discussion] volume paths confusion

[Scott Haneda]

On Aug 22, 2009, at 2:16 AM, unicode <unicode at connect.homeunix.com>  
wrote:

> Hi Again,
>
>> I am not looking for bootable, I am just looking to clone the main  
>> drive to a second location.  I am cloning to a directory, which  
>> would never be able to be bootable.  I just want to have all the  
>> files that are on the boot volume, in case of catastrophe, I can  
>> either clone back to another source, and bless to make it bootable,  
>> or pick and chose what I need.
>>
>> There are too many things for me, since these are servers, in /etc / 
>> var /opt and many other places that I would rather just copy the  
>> entire / and know I have it.
>
> Okay sure. This makes sense. I have a better idea of what you are  
> trying to achieve. Please be aware that some items will not be  
> backed up correctly. There are certain items such as devices which  
> are not correctly handled by the hard linking process. Just check  
> the files which are important and ensure that they are backed up to  
> your satisfaction and you should fine.

There are many more rsync fails on. It takes so long to learn what  
they are and start over. Each time the session.inprogress file has to  
be deleted, taking a long time as well.

Maybe if there only exists one session.inprogress file you could not  
delete it, but use it as a merge point. Rsync being set to delete from  
destination if source file does not exist. This would speed up testing  
greatly, and should cause no harm.

I had to exclude /Volumes /net /home, and some others. I'll provide a  
list when I get access to my notes. Lots of cache files, iTunes  
caches, /tmp and others just never need to be in a / backup, even if  
bootable is the goal.

The thing is, rsync needs to be fixed. One could argue you don't need  
those directories/files since the OS makes the files new again on boot.

Since CCC can make bootable volumes I assume they just exclude those  
from rsync. As far as I know rsync is the backbone of CCC. Is this not  
true?

That being said, there are legitimate reasons a user may make fifo/ 
devices and those need to be backed up.

Rsync does not backup finder flags and some other things as well.  
Resource forks may still be an issue. I know I used it to back up some  
mail store files that have resources and the backups lost their icon.  
I will md5 and poke into the resource and data fork to confirm their  
integrity later.

I think my main question is, how does CCC pass the backup bouncer test  
100%? Lbackup should try to meet that goal as well.

There seems to be a small sense of false backup security with rsync.

Please take no offense. Lbackup does what it claims excelently, which  
is rotational backups. This is all about rsync. I would have thought  
rsync, in three years time since I last used it, would be 100% with  
regard to integrity of data.

What does Time Machine use, how does it at least get resources and  
meta data perfectly? Time Machine does fail on using a method of  
copying entire changed files.

Maybe Snow will have ZFS, which should make this much easier.

>
>> I used to just use rsyc with hfs+ patches in the past, and issue  
>> via cron a command to backup.  This worked, and was very fast on  
>> second run.  I wanted incremental, which lbackup offers, which is  
>> why I am trying this out.  I will use launchd this time around.
>>
>> Am I doing something wrong, or is this a bug?
>
> Good question. I believe that if the -x (or --one-file-system "don't  
> cross filesystem boundaries") argument is passed to rsync then this  
> would not be an issue. Perhaps adding an option to the configuration  
> file to enable and disable this as an option would be a good idea?  
> Your thoughts would be warmly welcomed regarding this topic.

I seem to remember -ahfsx or similar was what I used. I'm guessing;  
the best I can remember 3+ years ago :)

I don't think the one option in the config is the best idea. I would  
love to see either
ARGS=
Where if any are added they overide any defaults and solely become the  
new rsync arguments. This may get confusing with the exclusions file,  
which you would want to append back in.

I personally would like to keep it out of the config and allow ./ 
lbackup -xyz as an option.

Or heck, allow both. It's easy enough to grab $1 and pass it into ARGS.

> As a short term option either adding /Volumes to the excludes list  
> or adding the -x option to rsync when run for the lbackup script  
> would solve this issue. I am interested in your thoughts regarding  
> this matter.

Maybe we should work to build a great exclusions file for  
distribution. .DS_Store files, fs events, /Volumes , /net and whatever  
else.

The ultimate goal in my very strong opinion is to have rsync work to  
keep the integrity of all files.

Do you know how CCC is making this happen? But in the meantime, at the  
very least, the recursion issue is huge.

I got lucky, my Internet died and ssh died with it. Had I done my  
usual workflow of using screen and detaching, I would have woken up to  
a possibly broken system, maybe corruption, but at least a bunch of  
clients calling to tell me their email was down.

>
>> I take it that copying a hard link always copies the true source?
> The way I understand hard links is that they are exactly the same  
> data. They are a simply a pointer to the same data. They do require  
> an additional inode. In essence, they are another entry in the file  
> system to the same data. I am not a hard link expert and different  
> systems may implement hard links differently.

And that only applies to the same volume? If I hard link, as lbackup  
does, from src to dest, where src is / and dest is /Volumes/backup and  
it is really just a link, then I don't have a backup.

What happens if I unmount / and try to access a file on /Volumes/ 
backup ? If it really is no more than pointer plus inode, I don't have  
a backup at all, but a bunch of 0 byte links.

>
>>> LBackup is a great way for some people to backup various changes  
>>> to non-user-data directories, between full system backups.
>>
>> I am not sure I understand this statement.
>
> Bascially, I know that many people use a tool such as carbon copy  
> cloner to make a full backup. Then  they also have LBackup creating  
> backups of certain areas. LBackup seems to be a good fit in this  
> kind of situation.

But lbackup could be used for both. CCC is great, but there needs to  
be better backup tools for servers and network based backups. Rsync is  
the only way I know to do so efficiently. Time machine is just not  
server friendly, nor bootable.

Using CCC to point to a remote sparse disk image is not the best.  
Mounting validation alone can take a long time.

Servers run headless and logged out. I like how lbackup is so easy to  
script via launchd. It's a perfect way to do backups on a server.

I could just use rsync alone, which gets me the gain of only copying  
the bits that change. Whereas most other backups copy the entire file.  
Databases and large binary files where only a few bits change are  
massively efficient with rsync.

But I want rotations, not rsync scheduled on daily and weekly.

I was going to work on pre and post lbackup scripts for server admins,  
as well as a launchd and an lbackup interactive config generator.  
Maybe even a GUI.

I can't really go down that road until rsync works better. Any insight  
on how to make that happen, I would appreciate it. CCC seems to have  
solved it, I will ping him.

>
>> I downloaded it, but I can not find a change log.
> Yes it is kind of hidden once LBackup is installed :
>
> The command below will display the change log :
> cat /usr/local/libexec/lbackup/Version\ History

Thanks. Any chance you can just include that in the root of the DMG?

Regarding your screencasts. Can I offer you to host them? I have the  
bandwidth to spare. I also think maybe convert them to HD YouTube.

Do you have the source files? There are audio and video glitch issues  
I would like to solve.

Actually, if someone can get a comp of screencast app, I would love to  
take a shot at expanding on them. Too expensive for my limited use.

> Please see the current situation listed below between the beta and  
> the latest alpha :
>
>    Version 0.9.8r (as of alpha release 7)
>
>        Added the lcd command.
>
>        Set more appropriate default settings.
>
>        Additional parameters are now provided to the pre and post  
> hook scripts.
>
>        Added additional logging facilities.

Why do you set a PATH in each script? Would a full path to command not  
be more reliable. I use alias in bash a lot, and often worry I may  
have a conflict.

>
>> Are you aware the mailing list archives are not working?
> Yes. I noticed this earlier today. I am looking into what has caused  
> this problem. However, thank you very much for mentioning this in  
> the email.

Sure. Mailman is nice. The archives, not so much. Would you mind if I  
just subscribe an account to nabble? They offer list archiving in a  
really nice way.

Any way to redeliver the past messages to get them into nabble? If so,  
I can give you the address of the account to deliver to.

> Again, if you have any further questions please let me know.

I get a constant warning when calling lbackups mail command. A warning  
that mail is set to local. Any ideas?

I still would love to see STDOUT=YES so I can just ./lbackup -xyz  
main.conf >> $date-log.txt | mail -s "backup done" foo at example.com.  
*that's probably invalid, iPhone is not the best way to get an email  
out :)

Thank you. You have answered all my questions. I understand the way  
lbackup works better. I have some suggestions I will post to the list  
when I have time. Mainly, I think the config file format could be  
simplified. I got a little confused at a point.

If you have any details about rsync and OS X, and if it is possible to  
get a true bit for bit backup, that would be a great start.

-- 
Scott