[lbackup-discussion] Backup Error Emails - Update

[henri]

> I have just heard back from my email providers, and it was being  
> blocked because the domain in the return path didn't exist (since  
> this was a test server it wasn't using a proper domain). When I  
> changed the domain that the email server was authoritative for to a  
> real domain the email worked fine. However there are still two  
> problems: 1. The email address isn't being used in the config file  
> 2. The return path is set to root at domainoftheemailserver.com instead  
> of the email address in the config file (which is where the problem  
> started).
>
> Any ideas would be most appreciated.

You have identified a bug within the lmail command.  Congratulations!

An alpha version (which hopefully resolves this issue) is available  
from the following URL :
http://www.lucidsystems.org/download/utilities/LBackup_v0.9.8r2-alpha1.dmg

This alpha1 version file will be removed from the download server at  
some point in the future. Please let me know if this alpha1 release  
resolves the issues you are having related to the from email address.


> On another note, I would be keen to have the option of having  
> LBackup emailing the logs for the backup on both successful and  
> failed backups. This way I could tell by the absence of any email  
> that the server didn't bother to backup.

This is a very good suggestion. As a plan for the future this is  
certainly on the drawing board. One reason it has not been added as a  
feature (and still may not be) is because this could mean that you  
easily end up with your in box being filled with messages regarding  
backups. As an example, if you have just 5 backups running 20 times  
each day. You could end up with 100 emails each day in your in box.  
However, more importantly the lmail command also performs archiving of  
the log file and it is yet to have a locking feature added.

The current implementation of the built in email error reporting is  
still quite basic. My focus recently has been in in improving the  
multi-backup reporting, which provides a current view of the  
backup(s). If you require every day reporting. I would suggest that  
looking at the the how multi-backup reporting is a good idea. I am  
happy to answer questions regarding the multi-backup and assist you to  
get it working. The advantage is that the code used in the multi- 
backup reporting is looking at the most recent backup. It gives you a  
look at the current stats of one or more backups. As apposed to the  
lmail command which will search the lbackup log for errors anywhere  
within the log.

This current design and configuration is certainly not set in stone.

Essentially, please understand that at present the lmail command is  
geared for detecting errors anywhere witin an lbackup log file,  
delivering the log file as via email and then archiving the log file.  
With that said. I still see that adding a feature to automatically run  
lmail after the backup is a good suggestion. This is quite possibly a  
useful feature in certain situations.

The key point is that the multiple reporting system is geared for  
providing up to the moment information regarding the backup state. As  
such it will not archive the log. As a result it requires no locking  
and certainly offers more flexibility with regards someone who would  
like to look at the backup logs every day or even every hour.

If you are only managing a single backup which is run once per day, I  
certainly see the sense in having this available as a option. However,  
the multi-backup scanning scripts are also very handy. I would be  
happy to assist you with putting to gether a post script which builds  
an email along the lines of what you suggested using the multi-backup  
scanning scripts.

All that out of the way, right now if you would like to have the log  
emailed to you each day. There are a number of options available. I  
have listed just some of these below.

One possible solution is to to add a post hook script which emails the  
log after the backup has finished.  If you would like to go down this  
path and would like me to put something together then let me know. A  
basic post script will not take long to put together.

I am actually inclined to look at a way of using the lmail command to  
reference a template in the backup directory or which is generated on  
the fly by a post script or separate script. Doing this will certainly  
requires some more thought.

Another option is to make a wrapper script which will execute two  
commands, something like the following in a file called  
"run_server_backup.bash" containing the following :

#!/usr/bin/env bash
/usr/local/sbin/lbackup path_to_backup_configuration_file
/usr/local/sbin/lmail path_to_mail_configuration_file
exit 0

Assuming this file is within your current directory, is called  
"run_server_backup.bash" and is executable, then it is possible to run  
this script by issuing the following command :

./run_server_backup.bash

Effectively, this will run this custom script which will in turn run  
lbackup followed by run lmail.

If you do not want to call this kind of custom wrapper script you  
could alternatively, run something similar to the following command :

/usr/local/sbin/lbackup <path_to_backup_config> ; /usr/local/sbin/ 
lmail <path_to_mail_config>

Placing the ";" character between two commands within a BASH shell  
will result in  one command executing after the other. As an example  
you could try the typing the following into a terminal :

echo -n "hello " ; echo "there"

Another option is to write a mail command which will send the mail  
exactly as you require.

Okay one more option is to schedule lmail to run once per day when you  
are quite sure that the backup has finished. It is a good idea to  
avoid running lmail while a backup is in progress. A feature for the  
next release will be lock detection for lmail.

There are also various other ways to get this to work. However, even  
with all these options, this feature is a serious possibility for a  
future version/alpha-version.  I can certainly understand that there  
are circumstances where this feature could be very useful.

I will think about this feature. If you decide you are feed up with me  
thinking about it and just need this and put together a patch then  
please submit you patch back to the lbackup project and add you self  
to the contributors page.

Once I have come to a conclusion regarding this feature I will be in  
touch.  I think that this will probably be a feature for a future  
release. As I mentioned it could be useful in some situations.  
However, I am not sure that using lmail is the way to go just yet.

Your feed back regarding this feature is just great. If anyone else  
has thoughts regarding the use of lmail or this feature then please  
let me know.


> It would also be handy to have the subject of the email read  
> "Successful" or "Failure" so that it was obvious whether further  
> investigation was required.

Again, this is a very good idea.  Currently, if delivery of lbackup  
log fails then the priority of the mail is set to high and the subject  
includes some word regarding the error.

However, if there is only an an error found within the backup log,  
then the subject of the email currently reports no error. This is  
because the reporting of the lbackup log was successful and any  
potential error in the log could have been from some time ago. If  
errors are detected within the log being mailed then the priority of  
the message is set to high. Essentially the priority of the mail is  
telling you that an issue within the backup log has been detected and  
it would be a good idea to check this out. However, the issue may not  
have been the most recent backup attempt. This is because previous  
reporting may have failed or not even been attempted due to any number  
of failure conditions, such as network disconnects, power failures  
etc. The idea behind the email notification feature is to provide  
information at specific times. Again, this is why I am hesitant to  
just add the feature without further discussion.

The idea is that you can setup the lmail command to run once a week,  
once a monte, even once per day or multiple times each day. It will  
then provide you with some feed back regarding the backup status. The  
idea is that if you use email for reporting then you will have some  
sort of script checking for them at a specific time and then notify  
you in the event that an expected email has not arrived or if it was  
delivered/sent late.

If this is a feature you are very interested in then, you may also be  
interested in the lsync project which currently supports an expert  
system for dealing with backups : <http://www.lucidsystems.org/tools/lsync 
 >

If you are looking into how the emails are generated then the  
following information will be helpful. The subject of an error email  
is formated by the following template scripts :

/usr/local/libexec/lbackup/message_templates/mail_error.sh
/usr/local/libexec/lbackup/message_templates/ 
mail_logerror_attachhment.sh

As a quick solution, it is possible to edit these template scripts so  
that the subject is modified to your requirements. Just keep in mind  
that an update to LBackup will overwrite these changes and that  
because of the way lmail detects errors at present the most recent  
backup may have been successful even when these templates are used.

Let me know if you are interested in working on an option to select a  
template with some modified subject lines. I am happy to help you out  
with making changes so that this is possible. I do not think it will  
be very complicated. The way the emails use templates ensures this is  
quite simple.

I agree that probably the subject line should change based upon  
whether there is an error in the very last backup. However, at this  
point in time backup failures detected by lmail are not necessarily in  
the most recent backup. Essentially, it is scanning multiple backup  
sessions in the log looking for errors which may have occurred just so  
that you know they were a problem. The current version of lmail is  
using the subject line to alert you to problems such as there being no  
log available. As far as lmail is concerned if it is able to send an  
email with the log file attached, then it has done a good job and  
should report this proudly in the subject line regardless of any  
issues that may have actually occurred with the backups. It will set  
the priority of the email to high priority.

I will keep these points in mind. Again, I would like to think about  
this further. As the email reporting is geared to a low maintenance  
backup. Where you just want information when there is a problem. You  
can then scan the attached report and see what happened if it says  
there was a problem.


> Another great little feature would be to know how much space the  
> backup destination had free upon the completion of the backup, so  
> that you could see if the backup was starting to run full or not.

It could be addressed with a custom mail script or a post hook.  I am  
happy to sort out the disk usage. This should be very trivial to  
implement so that the information in in the log. I think that this is  
also a good feature to add to the mail templates as well. Again, this  
could be an option in the mail configuration file.

If anyone has any further thoughts regarding these features then  
please let me know.

I think these are all really good ideas. Essentially, I would like to  
take some time to think about all of them further.

If you would like any help with the multi-backup report scanning then  
please let me know. Further details regarding these scrips is  
available form the following url : http://connect.homeunix.com/lbackup/monitoring_multiple_backup_logs

Thank you again for all of your great suggestions. I am thinking about  
which have the hights priority with regards my development time.

Again, great job on reporting the bug within the lmail! Hopefully, it  
is resolved.