Magento Internals: Cache Purging and Cache Tags

You buy a Megamenu extension from the Marketplace. The extension works perfectly fine on the local system. You now put this extension on the staging and content team starts making changes. They flush cache several times and they report that their changes are not reflecting. What’s wrong?

You have tested this extension on local system and ask yourself, “What is different between staging environment and local system.” Because we are talking about cacheing you say,”Staging uses Varnish for cacheing!”. Now, to take this investigation forward, you login to admin panel and start checking Varnish settings. Next, you go to command line and enable debug logging. Are cache purge requests going to Varnish? Is Magento able to connect to varnish for cache purge requests?

You run

bin/magento cache:flush

and check var/log/debug.log

You see, a cache purge request to localhost server. You now know the problem. You realize that Varnish is installed on a seperate instance. Magento is making purge requests to wrong location. To fix this problem you run

php bin/magento setup:config:set --http-cache-hosts=<IP of varnish>

Now you, you login to varnish server to check if Purge request from Magento are reaching this server.

Yes!! These purge request are reaching the varnish server. You are happy and you go ahead and try making a change to the Megamenu.

It didn’t work!! You are intrigued because varnish is receiving purge requests as expected. You try changing something else about the homepage and you flush cache. It works!! Then why is it not working in case of Megamenu?? You check database and clearly your changes are being saved.

Then you remember!! Isn’t Megamenu included using Varnish Edge side includes? You may also know this feature as “ESI comment tags in HTML”.

ESI tag used for megamenu.

You can also see this on your local machine by Going to Store->Configurations->Advanced->System->Full Page Cache

On your local machine, default option is “Built-In”. Change this to Varnish and save the settings. Then clear the cache.

Now if you go to your storefront, you will see, your navigation is gone. If you inspect the page, you will see an ESI tag similar to Tag shown above.

What is this ESI rendered block and why is Navigation implemented that way?

Navigation is included on almost all pages. Any change to navigation, would require a full cache blow up. This means, clearing all category and product pages from cache. This is inefficient. A better way is to store Navigation block as a separate block in varnish and then tell varnish to add this block to page before sending final response. This way, you don’t have to blow up full cache.

This is how it works for navigation. Let’s say Varnish has not cached anything yet. You now, request home page.

1. Varnish first requests home page from Magento
2. Varnish looks for ESI tags in your page html.
3. It find a ESI tag with url for rendering navigation menu.
4. Varnish requests Megamenu by hitting /page_cache/block/esi/block/block_id` URL.
5. Varnish composes received content with rest of the page and sends you the final response.
6. Varnish stores Mengamenu separately from rest of the page.

Next time when you change Megamenu, only Megamenu cache is cleared.

Back to our bug now. Why was Megamenu cache not being cleared?

Before we Answer this, we must understand how Magento stores cache in varnish and how does it instruct varnish to clear cache.

Each Magento page also sends X-Magento-Tag header

x-magento-tags header

As you can notice, for home page, Magento collects all the blocks to be rendered and tags associated with them. It then sends these tags as part of x-magento-tags header.

Now if you go to the Admin panel and change block with id 84, Magento will send a purge request to Varnish. Following is an example.

curl -X PURGE -H 'X-Magento-Tags-Pattern: cms_b_84' -H 'Host: example.com' <your varnish url>

If you do a full_page cache flush then the purge request look like below

curl -X PURGE -H 'X-Magento-Tags-Pattern: .*' -H 'Host: example.com' <your varnish url>

If you look at how this is configured in Varnish, you will see following in Varnish vcl config.

if (req.method == "PURGE") {
    if (client.ip !~ purge) {
        return (synth(405, "Method not allowed"));
    }
    # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
    # has been added to the response in your backend server config. This is used, for example, by the
    # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
    if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
        return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
    }
    if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }
    if (req.http.X-Pool) {
      ban("obj.http.X-Pool ~ " + req.http.X-Pool);
    }
    return (synth(200, "Purged"));
}

Focus on the following part

if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }

We are searching for objects with given tag pattern and removing them.

Now, what if your request doesn’t send any X-Magento-Tags header in the response. Then that response object will not be removed from Varnish Cache.

This is exactly what happened with our Megamenu Bug. On hitting following ESI url manually with curl

curl -I /page_cache/block/esi/blocks/%5B%22catalog.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyJd/

I received a response which did not have X-Magento-Tags header. Why did that happen? It happened because the block used by extension for Megamenu did not implement `Magento\Framework\DataObject\IdentityInterface`.

Magento’s default top navigation implements getIdentities in following manner.

/**
 * Return identifiers for produced content
 *
 * @return array
 */
public function getIdentities()
{
    return [\Magento\Catalog\Model\Category::CACHE_TAG, \Magento\Store\Model\Group::CACHE_TAG];
}

Hence the simple bug fix was to implement the Interface.

IF you don’t implement the interface then following code fails at

\Magento\PageCache\Controller\Block\Esi::execute

if ($blockInstance instanceof \Magento\Framework\DataObject\IdentityInterface) {
    $response->setHeader('X-Magento-Tags', implode(',', $blockInstance->getIdentities()));
}

Happy Debugging!!