Amasty research: Full Page Cache vs Varnish Cache

After the release of Full Page Cache Magento extension many clients started asking us what makes our extension better than Varnish Cache and whether it is better at all. At first we didn’t consider conducting such research, but the amount of our clients interested in this issue encouraged us to knock these two together to learn which of them provides the best page load time. Get inside the review to know who has become the winner!

Indeed, if you need to speed up your Magento store, just ‘faster’ is not enough to describe the performance of your website. A fair contest of two Magento cache extensions with detailed speed results is at your service.

Since a new version of Magento 1.9.0.0 has been released we have changed the test configuration a little bit in comparison to our previous Full Page Cache test.

 Hardware

  • CPU: Intel Xeon L5520 2.27GHz (16 core)
  • RAM: 16 GB
  • Storage: SSD INTEL SSDSC2BB480G4

Software

  • Magento Community 1.9.0.0
  • Magento Sample Data 1.9.0.0
  • Apache 2.2.22
  • Varnish 3.0.2
  • PHP 5.4.4 (mod_php + Zend OpCache 7.0.3)
  • MySQL 5.5.35

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. It is installed in front of any server that speaks HTTP.

Varnish Cache itself is not designed for working with Magento directly, so we took Nexcess Turpentine module for Magento and Varnish integration for the speed test. Basing on the recommendations of Turpentine module producer, we added «-p esi_syntax=0×2 -p cli_buffer=16384» to Varnish startup options. As a result Varnish startup options looked as follows:

DAEMON_OPTS=”-a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m -p esi_syntax=0x2 -p cli_buffer=16384″

To prevent Apache and Varnish conflict Apache was switched to port 8080.

For testing we used the list of 14 URLs of Category and Product pages: http://our-test-site/

  • http://our-test-site/women.html
  • http://our-test-site/women/new-arrivals.html
  • http://our-test-site/women/new-arrivals/tori-tank-460.html
  • http://our-test-site/women/new-arrivals/elizabeth-knit-top-484.html
  • http://our-test-site/women/new-arrivals/lafayette-convertible-dress.html
  • http://our-test-site/women/tops-blouses.html
  • http://our-test-site/women/tops-blouses/nolita-cami-484.html
  • http://our-test-site/women/tops-blouses/black-nolita-cami.html
  • http://our-test-site/women/pants-denim.html
  • http://our-test-site/women/pants-denim/dumbo-boyfriend-jean.html
  • http://our-test-site/women/dresses-skirts.html
  • http://our-test-site/women/dresses-skirts/essex-pencil-skirt-527.html
  • http://our-test-site/women/dresses-skirts/ludlow-sheath-dress.html

Before every test launch Magento cache was cleared and a single visit to all URLs was performed to “warm the cache up” before taking the test results. After the warm-up test for 10, 20, 30, 40 and 50 simultaneous visitors was launched.

Comparison test results (in ms) are in the table as follows:

10 visitors 20 visitors 30 visitors 40 visitors 50 visitors
Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine
Main page 10 89 35 156 44 228 71 258 109 328
Home / Women 7 46 9 69 14 58 13 63 16 81
Home / Women / New Arrivals 7 44 11 71 8 89 16 80 15 51
Home / Women / New Arrivals / Tori Tank 7 43 8 43 12 46 14 43 21 42
Home / Women / New Arrivals / Elizabeth Knit Top 7 40 8 40 10 40 15 40 18 34
Home / Women / New Arrivals / Lafayette Convertible Dress 7 40 8 40 10 39 15 38 17 34
Home / Women / Tops & Blouses 7 41 9 41 13 40 11 40 16 37
Home / Women / Tops & Blouses / NoLIta Cami 7 40 8 40 15 39 15 39 14 34
Home / Women / Tops & Blouses / Black Nolita Cami 7 40 9 40 8 39 13 39 14 33
Home / Women / Pants & Denim 7 40 9 40 8 40 9 42 18 35
Home / Women / Pants & Denim / DUMBO Boyfriend Jean 7 40 8 48 16 39 11 38 17 34
Home / Women / Dresses & Skirts 7 40 8 41 8 41 11 39 16 37
Home / Women / Dresses & Skirts / Essex Pencil Skirt 7 40 9 40 12 40 13 38 17 34
Home / Women / Dresses & Skirts / Ludlow Sheath Dress 7 40 8 40 8 39 9 41 11 34
TOTAL 8 45 10 54 13 58 17 60 23 61

 
Full Page Cache vs Varnish Cache reviewTo make the results easier to comprehend and let you vividly see the page load speed each Magento extension provides we have built some diagrams.

Full Page Cache vs Varnish Cache speed review

Full Page Cache vs Varnish Cache extensions

Full Page Cache vs Varnish Magento extensions

Magento Full Page Cache Varnish Cache comparison

Magento cache performance comparison

As you can see, the speed of Full Page Cache exceeds combination of Varnish Cache and Turpentine in 2-4 times. The biggest difference can be seen during the first visit of the website – at this moment both caches create a session for the user and have to initialize Magento. Since Full Page Cache initializes at the early stage, it can give the cached result quickly and Turpentine uses a lot of time in vain on extra initialization. This is an important point, as its impact deals with first visits and users’ impression of your website.

According to Kissmetrics, a 1 second delay in website load speed can decrease conversions by 7%. In fact, you might even want to count the money you lose using a Magento cache extension providing worse page load time.

This way Amasty Team recommends you using Full Page Cache Magento extension for providing the minimal page load time possible.