Root index.html deleted unpredictably

[ghost]

The static index.html and index.html.gz are deleted unpredictably. This happens when the Cache rebuild option is checked and it appears to be triggered by posting to a php-file in the root directory.

There are several mentions in the support forums.

[ghost]

Below is an extract from the debug file. It shows what happens when wp-cron.php is running and when a user is signing in.

17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 wp_cache_get_cookies_values: return:
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 In WP Cache Phase 2
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 Setting up WordPress actions
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 Created output buffer
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 wp_cache_get_cookies_values: return: 
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 supercache dir: /home/xxxxxxxx/domains/xxxxxxxxxx/public_html/wp/wp-content/cache/supercache/xxxxxxxxxx/
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 Not caching POST request.
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 wp_cache_ob_callback clearing rebuilt files in /home/xxxxxxxxxx/domains/xxxxxxxxxx/public_html/wp/wp-content/cache/supercache/xxxxxxxxxx/
17:12:25 815364 /wp/wp-cron.php?doing_wp_cron=1450458745.0950250625610351562500 wp_cache_maybe_dynamic: returned $buffer

18:05:45 946399 /wp/wp-login.php wp_cache_get_cookies_values: return: 
18:05:45 946399 /wp/wp-login.php In WP Cache Phase 2
18:05:45 946399 /wp/wp-login.php Setting up WordPress actions
18:05:45 946399 /wp/wp-login.php Created output buffer
18:05:45 946399 /wp/wp-login.php wp_cache_get_cookies_values: return: 
18:05:45 946399 /wp/wp-login.php supercache dir: /home/xxxxxxxx/domains/xxxxxxxxxx/public_html/wp/wp-content/cache/supercache/xxxxxxxxxx/
18:05:45 946399 /wp/wp-login.php Not caching POST request.
18:05:45 946399 /wp/wp-login.php wp_cache_ob_callback clearing rebuilt files in /home/xxxxxxxx/domains/xxxxxxxxxx/public_html/wp/wp-content/cache/supercache/xxxxxxxxxx/
18:05:45 946399 /wp/wp-login.php wp_cache_maybe_dynamic: returned $buffer

To me, the clearing of rebuilt files should not be happening in both cases shown above.
(In the extract of the log file part of supercache directory and domain have been replaced by x'es on purpose).

[Sophist-UK]

See https://wordpress.org/support/topic/root-indexhtml-is-deleted-unpredictably-after-the-last-update?replies=6 for a potential fix:

In the plugin file wp-cache-phase2.php, the method that's doing the delete is: function wp_cache_ob_callback. The code block down on line 317 is being entered:

if ( is_array( $do_rebuild_list ) && false == empty( $do_rebuild_list ) ) {

I temporarily updated my code to skip that rebuild logic when the current page is one that's flagged to not cache. so my line 317 became:

if ( is_array( $do_rebuild_list ) && false == empty( $do_rebuild_list ) && wp_cache_is_rejected($_SERVER['REQUEST_URI']) == false ) {

Don't yet know if this will have any negative side-effects (such as stranding old cache page copies), but I'm preloading the cache regularly so those cache page copies will still update with each preload run.

[Sophist-UK]

P.S. I am having this problem so interested in a fix.

[Sophist-UK]

Looking at the code, there is a similar check setting $cache_this_page = false a few lines earlier.

It is unclear to me why this is the correct fix. Because I think that for home page, $_SERVER['REQUEST_URI'] is either '' or '/' and this is a trivial match to any of the strings in $autorejected in wp_cache_is_rejected.

As we can see from the debug log, it is a post request which is setting $cache_this_page to false (which is correct - we should not cache post request pages), but that is no reason to delete the existing cache.

It seems to me that there may be two problems here:

1. If it is a POST request you should just return the buffer and do nothing with cache.
2. When we check wp_cache_is_rejected( $wp_cache_request_uri ) a few lines earlier, when $wp_cache_request_uri == '' or '/' then it will match by default which is incorrect - so we either need to and checks for this on that line - or more correctly fix this in wp_cache_is_rejected itself.

What do others think?

[Sophist-UK]

For issue 1, my proposed fix is from:

    // All the things that can stop a page being cached
    $cache_this_page = true;
    if ( defined( 'DONOTCACHEPAGE' ) ) {
        wp_cache_debug( 'DONOTCACHEPAGE defined. Caching disabled.', 2 );
        $cache_this_page = false;
    } elseif ( $wp_cache_no_cache_for_get && false == empty( $_GET ) && false == defined( 'DOING_CRON' ) ) {
        wp_cache_debug( "Non empty GET request. Caching disabled on settings page. " . json_encode( $_GET ), 1 );
        $cache_this_page = false;
    } elseif ( $_SERVER["REQUEST_METHOD"] == 'POST' || !empty( $_POST ) || get_option( 'gzipcompression' ) ) {
        wp_cache_debug( 'Not caching POST request.', 5 );
        $cache_this_page = false;
    } elseif ( $wp_cache_object_cache && !empty( $_GET ) ) {
        wp_cache_debug( 'Not caching GET request while object cache storage enabled.', 5 );
        $cache_this_page = false;
    } elseif ( isset( $_GET[ 'preview' ] ) ) {
        wp_cache_debug( 'Not caching preview post.', 2 );
        $cache_this_page = false;
    } elseif ... ...

to:

    // Post requests do not invalidate existing cache
    if ( $_SERVER["REQUEST_METHOD"] == 'POST' || !empty( $_POST ) ) {
        wp_cache_debug( 'POST request - cache unaffected.', 5 );
        return $buffer;
    }
    // All the things that can stop a page being cached
    $cache_this_page = true;
    if ( defined( 'DONOTCACHEPAGE' ) ) {
        wp_cache_debug( 'DONOTCACHEPAGE defined. Caching disabled.', 2 );
        $cache_this_page = false;
    } elseif ( $wp_cache_no_cache_for_get && false == empty( $_GET ) && false == defined( 'DOING_CRON' ) ) {
        wp_cache_debug( "Non empty GET request. Caching disabled on settings page. " . json_encode( $_GET ), 1 );
        $cache_this_page = false;
    } elseif ( get_option( 'gzipcompression' ) ) {
        wp_cache_debug( 'Not caching gzipped site', 5 );
        $cache_this_page = false;
    } elseif ... ...

Note: Not yet tested this.

[Sophist-UK]

My proposed fix for issue 2 is from:

function wp_cache_is_rejected($uri) {
    global $cache_rejected_uri;

    $auto_rejected = array( '/wp-admin/', 'xmlrpc.php', 'wp-app.php' );
    foreach( $auto_rejected as $u ) {
        if( strstr( $uri, $u ) )
            return true; // we don't allow caching of wp-admin for security reasons
    }
    if ( false == is_array( $cache_rejected_uri ) )
        return false;
    foreach ( $cache_rejected_uri as $expr ) {
        if( $expr != '' && @preg_match( "~$expr~", $uri ) )
            return true;
    }
    return false;
}

to:

function wp_cache_is_rejected($uri) {
    global $cache_rejected_uri;

    if ( $uri == '' || $uri == '/') return false;

    $auto_rejected = array( '/wp-admin/', 'xmlrpc.php', 'wp-app.php' );
    foreach( $auto_rejected as $u ) {
        if( strstr( $uri, $u ) )
            return true; // we don't allow caching of wp-admin for security reasons
    }
    if ( false == is_array( $cache_rejected_uri ) )
        return false;
    foreach ( $cache_rejected_uri as $expr ) {
        if( $expr != '' && @preg_match( "~$expr~", $uri ) )
            return true;
    }
    return false;
}

[Sophist-UK]

I am not going to have time to test these for a few days. If someone else can test them then I will submit a PR if they work.

[Sophist-UK]

As you can see I submitted PRs so that they can be evaluated by plugin owners.

[M0shi]

I'm facing the same problems plus an other : the precaching does not generate cache files for mobile.

If i test with a smartphone, the file is generated but deleted after 2 min.

Do u think it's link ?
(I'll try your solution waiting an update for the pluggin)

[Sophist-UK]

Is it just mobile pages?

(My sites are all responsive - so no special pages for mobile. So I have no experience of special mobile pages.)

[Sophist-UK]

See also Automattic/wp-super-cache#98 for a discussion about doing cache page invalidation for pages and posts.

[M0shi]

The cache is generated for all files, except the mobiles ones and the index.
The mobiles cache files are not generated at all.

I have a responsive site too, but the responsive (mobile) cache files are very welcome too.

I try your fixes and it seems to work, at least for the 2 min delete of index.
Now i'm precaching and it seems to work but still not for the mobile files :/

Thx you for the fix.

[Sophist-UK]

The point about a responsive site is that the HTML is the same for both desktop and mobile screens - and it adapts depending on the screen width. So there are no special cache files for mobile screens on a responsive site.

[M0shi]

I know, but the mobile files where generated before and used by mobile.
Now, even with a responsive theme, the mobile cache files are not generated by precaching, but they are when people (or me) are accessing the website with a smartphone.

Or maybe, it was generated because i had wptouch installed (but disabled) and this is now the correct way to work with a responsive theme ?

But i still don't understand, if it is, why the plugin don't use the "normal" files and still generate a xxxx-mobile.html / gz files.

[ghost]

Finally I had some time to do a little testing. With the modifications the root index.html-file is no longer deleted when a user logs in to the WP dashboard.

I did not test the behaviour for issues Automattic/wp-super-cache#102 and Automattic/wp-super-cache#103 yet, but will do if I find some time.

[ghost]

In addition to my previous comment, it appears the cache rebuild function is also functioning properly again. Posted a comment on of the posts on the dev site and all cache files affected are marked '.needs-rebuild'. Although I am testing on a very small scale, I really like what I see :-).

[WiredWonder]

Is it just me or does release 1.4.8 still have this bug? I have been trying to work it out all day and just found this issue log.

[ghost]

Yes, 1.4.8 still has this bug, but the fixes proposed by Sophist-UK are solving the bug for me. The fixes aren't yet in the official release.

[neilmorton]

Hi. Seeing the exact same issue on 3 sites.
Would be extremely useful if this was resolved with a code fix and plugin update.
The fact this is one of the most used caching plugins for WP, and it is removing what will usually be the page you want cached the most is a little odd.
Can the developer perhaps give users an inclination of ETA for a fix?

Thanks.

[marcomarsala]

Seems enabling direct cache files for home page (/) solves the issue.

But another different issue arises: #25558

[PieterChristiaens]

Hello! It seems that a year later the bug is still in the latest version. My objective is to solve the issue of the index cache being made every 2 minutes.

I changed the code that Sophist-UK made for issue 1 and 2.

Should I also implement the first change on line 317? It says it's temporary so I'm not sure... And the code is like Chinese to me.

Thanks for the help.

[donnchawp]

I think Automattic/wp-super-cache#186 fixes this. Can you try moving the line "$do_rebuild_list[ $dir ] = 1;" around like in that PR?

[rdmitry]

I think Automattic/wp-super-cache#186 fixes this.

Yes, everything is fine now.

[marcomarsala]

Not fixed. Every time I log into my wp-admin, the homepage cache is cleared.

[RobP-git]

Issue is still in latest version 1.4.9. I finally got around to debugging this issue for sites/clients and can confirm that the 2 fixes above fix the issue. It would be great to get this committed.

Before:
04:47:39 5964 /wp-cron.php?doing_wp_cron=1498884459.7230710983276367187500 Not caching POST request.
04:47:39 5964 /wp-cron.php?doing_wp_cron=1498884459.7230710983276367187500 wp_cache_ob_callback clearing rebuilt files in /var/www/....
04:47:39 5964 /wp-cron.php?doing_wp_cron=1498884459.7230710983276367187500 wp_cache_maybe_dynamic: returned $buffer

After:
05:23:13 9591 /wp-cron.php?doing_wp_cron=1498886593.3331980705261230468750 wp_cache_get_cookies_values: return:
05:23:13 9591 /wp-cron.php?doing_wp_cron=1498886593.3331980705261230468750 supercache dir: /var/www/....
05:23:13 9591 /wp-cron.php?doing_wp_cron=1498886593.3331980705261230468750 POST request - cache unaffected.

[cwdv]

The fix referenced above in Automattic/wp-super-cache#186 only appears to be a partial fix from what I can tell:

When I select the "Preload Cache Now" option in the Preload tab, the fix referenced in Automattic/wp-super-cache#186 above appears to correct the problem of the initial root index.html page being deleted/removed. However, when the preloaded cached files are refreshed for the 1st time the root index.html file is not refreshed because the date/time stamp doesn't change. Shortly after the refresh is complete the original root index.html file gets deleted.

The 2nd preload refresh recreates the root index.html cache file, and it was not removed until shorlty after the 3rd refresh.

It appears that when the preloaded cached files are refreshed it will not refresh/recache the root index.html file if there is an existing one available, and then it deletes the existing shortly after the preload refresh is complete. The next refresh will created the root index.html file again and it will stick around until the next refresh where it will be deleted shortly after that refresh is complete. It appears this cycle continues to get repeated.
Any thoughts? Thanks....

[cwdv]

In addition to my previous post, when I receive traffic to my sites home page and a preloaded index.html page is not available, one is created. However, the next preload refresh will not refresh that cache. Instead that cache will stick around for a while and then it gets removed/deleted at random, and a new one will be generated when my home page receives another traffic hit. It doesn't appear the garbage collector is removing the home page cache because the garbage collector email I get always states "deleting 0 files and directories".

[donnchawp]

There seems to be two issues here. One to do with preload and the other to do with logging in/POSTing to files in the root directory (like wp-login.php).

Have you tried the latest code here in master? Logging in results in an attempt to rebuild ....../wp-login.php/ but there are several extra checks there now including a call to realpath() and a check if it's a directory.

realpath() converts the /wp-login.php/ into "/" and that then causes the rebuild to fail as well.

I tested a preload and the root index.html wasn't deleted either now. Please try master but read this post first about the changes as there have been many. I want to get a release out in the next few days so it's very stable.

[cwdv]

I downloaded the master zip file(wp-super-cache-master.zip), and uploaded via Wordpress > Plugins > Add New. Upon activation I received some errors and noticed some problems in the admin menu area. I ended up changing the name of the plugin folder from (wp-super-cache-master) to (wp-super-cache) and everything appears to function normally after a refresh. Did I go about this the right way?

[donnchawp]

I would have just copied the files from the master directory into the existing wp-content/plugins/wp-super-cache/ directory but that works too. The important bit is the name of the plugin directory which, as you found out, has to be wp-super-cache. Hopefully testing will go well. :)

[cwdv]

I will continue to test, but currently I have noticed a couple of things:

1. In the Preload Tab when I check "Preload tags, categories and other taxonomies" and then select "Update settings" it always removes the check mark. However, it is caching my sites category though. Before it would actually cache 2 and sometimes 3 of my sites categories. However 2 of those have no content in them as I believe they are created by my theme. So basically it is caching the correct category now, and no longer caching the other 2 theme created categories with no content. As far as I can tell that is a good thing!

2. The home page/root index.html cache did not get removed after the initial preload. However, when I went to visit my site to check the "Cache Status Message" in the source code, I noticed that the time stamp for the home page and the category page was the current time. I went to my hosting account to check the time stamps for the caches and I noticed that the Home page and Category pages had been re-cached when I went to visit them. The others had the original times yet.
   When the Preload refreshed it did not refresh those 2, and the garbage collector isn't removing them either. The garbage collector is currently set to run every 600 seconds. I should note that I have the "Make known users anonymous so they’re served supercached static files" option selected in the advanced tab.
   The site I am testing on currently receives very little traffic, a few bots every now and then. Had I not visited the Home and Category page I'm not sure what would have happened. Maybe the Preload would have refreshed them like it did the others.

Hope that makes sense,
Thanks again!

[cwdv]

First thing I did this morning was check to see what cache files were available via cpanel file manager, and I noticed:

1. There was no home page cache
2. There was no category folder or cache(As referenced above, when you select the "Preload tags,
   categories and other taxonomies" option the check mark gets removed after you select "Update
   Settings"
3. There were no cache files in any of the other folders(9 total)
4. There was a "wp-json" folder(I should note, I have seen this folder occasionally before the switch to
   Master)
   Not sure why there were no caches available first thing this morning. Emails confirm that caches were refreshed throughout the night. The preload was set to refresh every 30 minutes.

After the 1st scheduled Preload Refresh I noticed:

1. It was not caching my sites category page(preload refresh emails confirm this)
2. The "wp-json" folder was removed
3. All other caches were available including the home page cache
4. I visited my site and noticed that the home page cache time stamp did NOT change this time, but a
   category cache was created from my visit. Note: This does differ from my message yesterday in that
   the home page time stamp did not change today, and yesterday I remember the category cache being
   created during the preload refresh, and then refreshing the cache from my visit.

After the 2nd scheduled Preload Refresh I noticed:

1. The home page cache did not refresh and shortly after the Preload refresh was complete the home
   page cache was removed.
2. The category cache was not refreshed or removed. Note: This category cache was originally created
   when I visited my site as referenced above.

Throughout my testing today I also noticed:

1. Sometimes I have to try a few times to get the preload to start when I select "Cancel Cache Preload"
   and then "Preload Cache Now". Not sure why but the cron event(wp_cache_full_preload_hook) will
   time out and disappear, but it does not start caching my site.
2. Sometimes the Preload will start but after caching a couple pages it stops
3. Sometimes it will cache the category page, although typically it doesn't
   Thanks...

[donnchawp]

Is it possible these files are being removed because of garbage collection? Are you using the debug log to track what the plugin is doing. You'll be able to find out why they're being deleted.

Also, the front page is not generated by the preload.

Oh, in Automattic/wp-super-cache#289 I fixed the tags and categories checkbox, so grab the updated master and copy the files in that over your current wp-super-cache install.

[cwdv]

The garbage collection email always states "deleting 0 files and directories". I tried to access the debug file but was given the following error:
Not Acceptable!
An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.

I am confident the preload is caching my home page as referenced above. I should note that I have a static home page. If I have a static home page is it plausible then that the preload is caching it?

I have updated to the latest master copy.
Thanks...

[cwdv]

I should also note that the debug file error I received was when I attempted to access the file through the plugin link. I was able to to find the debug file on my hosting account, and the file is accessible and readable via Cpanel.

I also double checked the preload cache of my home page. The preload refresh will create a cache of the home page, and the next preload refresh will NOT create a new cache and shortly after it removes/deletes the cache that was made from the previous preload refresh.

I have confirmed that the index.html file is my home page preload cache. The preload generates the cache and if I visit my homepage and check the debug message at the end of the page source, the time stamps do match. If I then delete this index.html file and revisit my site it will generate a new index.html file with a new current time stamp.

[cwdv]

I monitored my cache files via cpanel, and a preloaded home page cache was available at the time. After a few minutes a preload refresh was performed at 16:15 and it did NOT create a new cache of the homepage. At 16:16 the homepage cache from the previous preload was removed.

I checked all my garbage collector emails in that time frame and they all stated "deleting 0 files and directories".

I am more than willing to send you a copy of the debug log, for the time frames referenced above, to review.
Thanks again

[omikron]

Any chance on a fix for this soon? Homepage seems most important to have cached.

[Ciantic]

Hmm, I'm having issue where index.html (homepage) is being deleted all the time. Like every second. (1.4.9)

I've tried to debug, but it seems like it creates the index.html, but something deletes it. Maybe same issue as in this thread.