Backup to S3?

[netconstructor]

I am curious to better understand what type of backup files are created here and if its feasible to have all emails backed up to Amazon S3.

Further more - if one has this running on a continuous basis would it recognize a change in labels on previously backed up emails?

CH

[gburca]

There are 3 types of files created:

1. A file containing folder/label metadata
2. A file containing email message metadata
3. One file per email message (similar to maildir)

The first 2 file types have optional backups (similar to how logrotate deals with system log files). There's no reason why the backup can't be done to S3, and I might add that option in the future.

The script detects "change in labels", but it never deletes anything on the server. So when you restore your backup, it goes through each label and uploads messages that are missing from it. It doesn't delete messages in a folder/label if the local backup does not contain that message.

Imagine this scenario. A backup is done while message-1 has label-A. After the backup, message-1 label is changed to label-B. If you now do a restore, BaGoMa will re-attach label-A to message-1 (in addition to the already existing label-B) so the message will have 2 labels. If you've lost all your email prior to doing the restore (or if you're uploading the backup to another account), message-1 will only have label-A. Obviously if you do a new backup after you've removed label-A and added label-B, the restore will only attach label-B to the message.

[gburca]

I found a number of projects that allow you to mount an S3 bucket locally as a file system, so I don't think I'll be adding S3 support directly to BaGoMa. The --dir option can be used to direct the backup to one of the S3 backed file systems, to a Dropbox folder, to a Google Drive folder, or to any other cloud storage solution.

[netconstructor]

First of all - thanks for the clear explanation!

Yes, I am aware of those 3rd party scripts which allow for S3 connectivity and even incremental backups.

I guess I should have better explained my objectives so here it goes and if you could provide any insight it would help out a great deal.

So, what I am attempting to do is create a type of email "archive" where this archive is actually accessible via IMAP and where an actual copy of each of the archived email messages are stored on S3.

This email archive would be read only and my logic here was that a tool like yours based on its logic could be utilized to create a type of standard email index containing the headers for each email message.

My assumption here is that by maintaining an index of all of the archived email messages an imap connection would quickly allow a user to search the index. Of course this would mean that if an actual message needs to be retrieved it would have to be retrieved individually from the S3 archive.

When you have a sec maybe you could let me know your thoughts on this.

One final thing --- I was also thinking about the use of some type of NoSQL database to store this index which would be ideal but the connection between NoSQL and IMAP I think is my biggest area where I have no idea on how to set this up.