devot:eeDevoted to ExpressionEngine

Hey Eric,

Hope all’s good.

Using v.1.9.3 on EE 2.5.5.

Having a problem with the Files backup where there are large numbers of ‘items’ involved.

The progress bar all seems to work fine but when it hits the end it bumps up the total by one and thus never reaches the end and never triggers the final actions that should happen afterwards (backup to Amazon S3 etc). A local copy is created though and reported under ‘View Backups’.

So, when the file backup started all the way through it reports a target total. In my last run it was 44190. But at the end that gets incremented so it never completes. Leaving me with the following:

Total Items: 44190 of 44191

So that’s one problem.

Another one is the varying nature of the items reported there and size of the backup created. The previous time I ran the same backup on the same files I sended up with

Total items: 44181 of 44182

Something’s changed…. But they should be the same.

That time the end zip archive under view backups was over 3GB. The former time with the final items count of 44191 it ended up being 2.15GB.

Looks like it’s missing something somwhere…. I’d expect it to be more consistent.

Look forward to hearing your thoughts.

Hi again.

Any thoughts on this Eric? Seems as though the logic stalls because of that final incrementation.

Cheers,

Aidann

Eric,

Could you take a look at this again please?

Thanks,

Ben (Aidann’s associate)

Eric,

Could you review this, please?

Thanks,

Ben

Eric,

Could you review this, please?

Thanks,

Ben

I encountered this same issue when syncing to S3 and I’m running EE2.7.3 and BP1.9.4. I only encountered it with larger backups; i.e. 2Mb database backups shot up to S3 no worries but 300Mb+ file backups would stall or just not make it to S3 at all or I would hit the problem with inconsistent backup file sizes you’ve had.

I found that switching my S3 buckets to a local region (Sydney) rather than North America seemed to solve the issue.  I had to exercise some patience with this as it takes 10+ minutes for backups of this size to sync to S3, all the while of course you’re looking at the Backup Pro ‘don’t refresh page’ etc warning.

Also, is there any reason a file should show up as a different size in S3? i.e. I Have one file backup which is 67.5Mb in BP but S3 is saying 64.3Mb. Another is 395Mb in BP and 376.9Mb in S3.

Be good to see a reply from Eric on whats happening with this.

Hi everyone,

First, the good news: like 90% of all the Backup Pro issues, this is all caused by either PHP memory limits or execution time limits. Increasing either, or both depending on which is the actual issue will resolve this issue.

To figure out which, take a read of the How to Debug Backup Pro thread and use the PHP error to dictate your next move:
http://devot-ee.com/add-ons/support/backup-pro/viewthread/13253

As to the discrepancy in file size, I imagine, though I’m not 100% here, it’s due to S3 compressing the files further on their end. Backup Pro uses the Zip format for compression, which is really good for interoperability and cross platform usage (Mac, Windows, Linux can all open with ease) but not an ideal compression format in that it only compresses around 85%. My gut tells me S3 is compressing things further, but with it already being a binary file, the gains are minimal.

Eric

OK, thanks for the info.

I ran the debug process a couple of times - no issues with the db backup but on file backup got this (it took a while):

Database backup = 979kb
File Backup = 400Mb

Both backups are there in CP. Database is in S3, Files aren’t.

Checking php info:

max_execution_time = 300
memory_limit = 512Mb

If this is related to the max_execution_time can you suggest a suitable value?

Explanation of S3 compressing further is pretty much what I figured was going on.

Oh yeah, that’s a rather large backup and, judging from your max_execution_time, 5 minutes to compress and upload to S3 is a bit low. I’m honestly not sure what to increase it to though since bandwidth is variable, you know? Were it me though, I’d increase it to 20 minutes (12000 seconds) and see how it goes.

You could also make use of the exclude directory setting to keep certain large files that don’t require constant backing up from the archive.

Eric

Thanks Eric. I’ll give the tech guy’s at Nexcess a buzz and see what they reckon would be more suitable. I don’t think its a mile out as the backups have been going (although it has crept up slightly in size since the last one I did successfully) - only to Sydney-region S3 though. Perhaps doubling it would do the trick.

Not sure there’s a lot I can exclude from the backups and still maintain the integrity of it - I prefer to have a full carbon-copy good to go if needs be. Prior to BP I used to sync all files down to local and export the db.

I’ll report back when I’ve had the execution time extended.

Hi again,

Nexcess increased the max_execution_time to 12000 (I checked it in EE’s php info) as suggested but no dice.

The file backup appears to complete in the CP after approx. 5 minutes - its there locally but the files never turn up in the S3 bucket.

I’m still getting the same debug message when I run the debugging.

DB backups are still working as normal.

The issue is 100% related to file size - I have another site on the same server with exactly the same configuration & EE/add-on version numbers but its just my blog hence the file backup is on 9.7Mb and it flies over to S3, no problems.

Nexcess are being good enough to trigger some backups & watch whats going on at their end - see if they can shed any light on whats going on. I’ll be back…