Amasty research: Full Page Cache vs Varnish Cache

Amasty research: Full Page Cache vs Varnish Cache

After the release of Full Page Cache Magento extension many clients started asking us what makes our extension better than Varnish Cache and whether it is better at all. At first we didn’t consider conducting such research, but the amount of our clients interested in this issue encouraged us to knock these two together to learn which of them provides the best page load time. Get inside the review to know who has become the winner!

Indeed, if you need to speed up your Magento store, just ‘faster’ is not enough to describe the performance of your website. A fair contest of two Magento cache extensions with detailed speed results is at your service.

Since a new version of Magento 1.9.0.0 has been released we have changed the test configuration a little bit in comparison to our previous Full Page Cache test.

 Hardware

  • CPU: Intel Xeon L5520 2.27GHz (16 core)
  • RAM: 16 GB
  • Storage: SSD INTEL SSDSC2BB480G4

Software

  • Magento Community 1.9.0.0
  • Magento Sample Data 1.9.0.0
  • Apache 2.2.22
  • Varnish 3.0.2
  • PHP 5.4.4 (mod_php + Zend OpCache 7.0.3)
  • MySQL 5.5.35

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. It is installed in front of any server that speaks HTTP.

Varnish Cache itself is not designed for working with Magento directly, so we took Nexcess Turpentine module for Magento and Varnish integration for the speed test. Basing on the recommendations of Turpentine module producer, we added «-p esi_syntax=0×2 -p cli_buffer=16384» to Varnish startup options. As a result Varnish startup options looked as follows:

DAEMON_OPTS=”-a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m -p esi_syntax=0x2 -p cli_buffer=16384″

To prevent Apache and Varnish conflict Apache was switched to port 8080.

For testing we used the list of 14 URLs of Category and Product pages: http://our-test-site/

  • http://our-test-site/women.html
  • http://our-test-site/women/new-arrivals.html
  • http://our-test-site/women/new-arrivals/tori-tank-460.html
  • http://our-test-site/women/new-arrivals/elizabeth-knit-top-484.html
  • http://our-test-site/women/new-arrivals/lafayette-convertible-dress.html
  • http://our-test-site/women/tops-blouses.html
  • http://our-test-site/women/tops-blouses/nolita-cami-484.html
  • http://our-test-site/women/tops-blouses/black-nolita-cami.html
  • http://our-test-site/women/pants-denim.html
  • http://our-test-site/women/pants-denim/dumbo-boyfriend-jean.html
  • http://our-test-site/women/dresses-skirts.html
  • http://our-test-site/women/dresses-skirts/essex-pencil-skirt-527.html
  • http://our-test-site/women/dresses-skirts/ludlow-sheath-dress.html

Before every test launch Magento cache was cleared and a single visit to all URLs was performed to “warm the cache up” before taking the test results. After the warm-up test for 10, 20, 30, 40 and 50 simultaneous visitors was launched.

Comparison test results (in ms) are in the table as follows:

 

10 visitors 20 visitors 30 visitors 40 visitors 50 visitors
Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine Full Page Cache Varnish & Turpentine
Main page 10 89 35 156 44 228 71 258 109 328
Home / Women 7 46 9 69 14 58 13 63 16 81
Home / Women / New Arrivals 7 44 11 71 8 89 16 80 15 51
Home / Women / New Arrivals / Tori Tank 7 43 8 43 12 46 14 43 21 42
Home / Women / New Arrivals / Elizabeth Knit Top 7 40 8 40 10 40 15 40 18 34
Home / Women / New Arrivals / Lafayette Convertible Dress 7 40 8 40 10 39 15 38 17 34
Home / Women / Tops & Blouses 7 41 9 41 13 40 11 40 16 37
Home / Women / Tops & Blouses / NoLIta Cami 7 40 8 40 15 39 15 39 14 34
Home / Women / Tops & Blouses / Black Nolita Cami 7 40 9 40 8 39 13 39 14 33
Home / Women / Pants & Denim 7 40 9 40 8 40 9 42 18 35
Home / Women / Pants & Denim / DUMBO Boyfriend Jean 7 40 8 48 16 39 11 38 17 34
Home / Women / Dresses & Skirts 7 40 8 41 8 41 11 39 16 37
Home / Women / Dresses & Skirts / Essex Pencil Skirt 7 40 9 40 12 40 13 38 17 34
Home / Women / Dresses & Skirts / Ludlow Sheath Dress 7 40 8 40 8 39 9 41 11 34
TOTAL 8 45 10 54 13 58 17 60 23 61

To make the results easier to comprehend and let you vividly see the page load speed each Magento extension provides we have built some diagrams.

Full Page Cache vs Varnish Cache review

Full Page Cache vs Varnish Cache speed review

Full Page Cache vs Varnish Cache extensions

Full Page Cache vs Varnish Magento extensions

Magento Full Page Cache Varnish Cache comparison

Magento cache performance comparison

As you can see, the speed of Full Page Cache exceeds combination of Varnish Cache and Turpentine in 2-4 times. The biggest difference can be seen during the first visit of the website – at this moment both caches create a session for the user and have to initialize Magento. Since Full Page Cache initializes at the early stage, it can give the cached result quickly and Turpentine uses a lot of time in vain on extra initialization. This is an important point, as its impact deals with first visits and users’ impression of your website.

According to Kissmetrics, a 1 second delay in website load speed can decrease conversions by 7%. In fact, you might even want to count the money you lose using a Magento cache extension providing worse page load time.

This way Amasty Team recommends you using Full Page Cache Magento extension for providing the minimal page load time possible.

Andrey Tataranovich

Andrey Tataranovich

Andrei believes that only at work people can fully express themselves, so he does his best to implement all his knowledge while in the office. Having made computer science his hobby, Andrei is constantly developing his professional skills. But despite such a busy schedule he always finds time to read science fiction books and communicate with new interesting people.

You may also like...

30 Responses

  1. Wes says:

    Hi Andrew,
    Can you use FPC + Varnish + Turpentine together?

  2. Wes says:

    Thanks Dave and Vera

  3. Addison says:

    Varnish means light speed. I used Lesti which provides about 62 ms with 100 visitors, 10000 requests. But with Varnish same benchmark this page loads in 6 ms. Varnish can process 2000 req/sec in my 4 core 3.4 GHz VM, 4 Gb RAM, SSD. I would prefer FPC because it is not creating trouble as Varnish. If you don’t know VCL syntax your are blind. Also there are a lot of block that need to be set to no cache. Still get trouble with Community Pole. Nexcess says they solved this issue but I still have it in my 1.8 and 1.9 version out of the box, just testing. You vote and block remains unchanged. Vote again and again and nothing happening. If you go to other page you will see vote results. A new refresh will hide the block for that session (which is normal). Also there is a lack of support for Turpentine. Page Cache also provide a free version without ESI support (which is the most wanted). My conclusion Varnish is faster than any FPC on the market, the only problem is configuration, especially when you have AJAX in your theme.

    • Ksenia Dobreva Ksenia Dobreva says:

      First of all, thanks for your comment!

      It is true that Varnish requires some work to be set properly so it gives max results, but that’s not exactly the point of the article. There are users that have time and knowledge to work on configuration, and there are users that need an extension out of the box. That’s why we did our tests like an average user would have done; we installed two products and compared their performance.

      However, it is great that you managed to get better results with Varnish.

  4. Den says:

    you dont understand the main difference between VARNISH FPC and other non-Varnish FPC.
    this is all about resources needed to process your requests. with Varnish you need 0 (zero) resources,
    with non-varnish fpc you will get to the limit very fast, more visitors/buyers you have – faster your server will die… speed is not so important as server capacity…

  5. Den says:

    I am sure that your test is invalid, the varnish response time is never dependent on the number of requests. 10 or 1000 requests – you will always get the same response time. and varnish even will be faster than any cache which is completely dependent on the database and php. Varnish is independent cache, directly from memory. even if you use ESI, there will be low impact.

    probably you misconfigured your test environment.

    i just tested many different non-varnish caching modules for magento, and they are awful, server is dying after 150-200 simultaneous users…Varnish keeps working.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hi Den, and thanks for your comment.
      Although, it looks like you’re not taking into account the integration module and various ESI operating modes in Varnish. And in this particular case Varnish performance depended on Turpentine integration module.
      Hope it clears things a bit.

  6. Gershon says:

    Hello,
    Want to share my experience:
    I have 3 sites on the same server
    1) Use Amasty Full Page Cache
    musicgate.co.il (5000 products)
    2) Varnish as fronted for static content and LETSI FPC (600 products)
    smoke4u.co.il
    3) Varnish as fronted for static content and extendware FPC (1500 products)
    il-cosmetics.com

    Amasty Full Page Cache , work best out of the box.
    LESTI and Extendware with warnish get closed result.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hi Gershon,

      thanks for your comment and sharing your experience. Happy that you are using our extension and happy with the results.

  7. Leon Poole says:

    You should clarify in the article that you’re testing against your Amasty FPC extension and not Magento Enterprise FPC native – this confused me while reading.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hi Leon,

      sorry if it confused you. In fact, we are working to publish a new article on this comparison soon.

  8. Chris says:

    Hi, after reading this article and some of the comments i think that many people will get confused because of the limited information given about the difficulties and the resources needed in each case. I will try to clarify my opinion by writing some facts combined with my personal experience and other developer experiences i have read. I don’t consider myself an expert developer although i have worked in many projects during the last 20 years so before adapting my proposals please make your own research and benchmarks.

    Varnish can work on a separate server or even on many servers with load balancing, FPC solutions cant. Varnish operates in Virtual Memory where it stores all static content while FPC uses slow filesystem. If there is enough RAM and a page doesn’t have any dynamic parts Varnish can serve it with no delay (2ms, just little slower than serving a static file which needs 0.7ms). Varnish is written in C and is massively multi-threaded while FPC depents on the webserver, PHP implementation and opcode cache. The amount of requests Varnish can handle on a load balanced multiserver environment where each dedicated server has lot of ram cant be compared to any other non custom solution i know. So i think i made it clear enough that in my opinion Varnish shine for websites with huge traffic allowing them to serve static content in a scalable ultra fast optimized way. But what about Magento dynamic parts, https, clearing cache and ram consumption? Well there is where pain starts, it is hard even for experienced developers using Varnish integration on Magento to configure non caching/lazy caching of custom dynamic parts, for example Turpentine vendor can’t configure correct behavior of the default poll module even after trying for some time. There are no current plans to add HTTPS support to Varnish. Clearing cache can be a pain too, there are a bugs in Turpentine’s integration considering cache refreshing and clearing and i am pretty convinced from both personal experience and countless posts that in all popular integrations cache will just keep growing until a manual delete (which will of course need time for cache recreation). As long as Varnish cache keeps growing the response time and RAM consumption will keep growing too so i would never use it on a server that hosts more websites and there is limited RAM. Personally my experience of using Varnish for websites that have a lot of dynamic parts is limited but people who manage large e shops claim that in this case Turpentine Varnish is slower than FPC, i am sure this is true because those developers are experienced and try hard for optimum results but i am also convinced that with a better Varnish integration this can change. Support is also another factor, with Varnish integrations support is needed a lot but because of the complexity there is none or limited while on paid FPC modules there is excellent one, although it wont be needed in most cases. So to summarize using Varnish needs root access on server so you can install it, exceptional skills to configure dynamic parts and correct integration bugs concerning cache refreshing and clearing and a lot of RAM. FPC on the other hand just need one click on Magento connect and you will have great results and one easy way to configure what custom parts will be cached and what wont be.

    For my needs a server with SSD, Apache with PHP-FPM, Zend opcode cache, Mysql, and FPC on Magento is fast enough but if i ever need to serve more requests i would use Redis for session management and cache backend. For even more requests i would find the bottleneck and depending on it and the costs of human and hardware resources i would consider taking advantage of Nginx with HHVM ( i made some tests and the results where really great) or more servers using Varnish.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Chris,

      thanks so much for your time and effort – you have shared great insights on this question! I’m sure our readers will benefit from your knowledge. Very impressive.

    • ADDISON74 says:

      Please check the latest version of Turpentine. The Poll problem is solved and many others. There is a new Turpentine team who is working great to improve this extension. Also Magento 2 will have support for Varnish 3 and 4 out of the box. I am using Turpentine latest + Varnish 3 + Magento 1.9.2 in production, one dedicated server, 16 RAM, SSD’s, 8 cores, and I can say with a huge traffic daily this kick the ass. We tried before a lot of FPC’s solutions and they are great for small websites with a less traffic. By the way Varnish will get https support officially, or you can use Pound in front of it which is easy to configure. I am working to find a way to get analyzing results correctly, because Varnish is hit first not the webserver (awstats takes info from apache logs). Nginx + HHVM could be a solution, it is very fast, or just HHVM doing the whole thing. In the last 2 years I configure many solutions, but always come back to basic because I encountered many issues. Even Lesti FPC has a problem with Poll side module (which is now solved in Turpentine), probably Amasty has it too.

  9. Kenan SALTIK says:

    As visitor number increases the response times of varnish decreases. What will If your concurrent visitor number will reach thousands….

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Kenan,

      thanks for your comment! In case it’s a 100% static page, performance will be defined by the web server and hardware. But if the page has dynamic elements, neither our FPC nor Varnish will be of great use in the case you described.

  10. Francis Kim says:

    Having fine tuned Magento instances to the max before, I’d say Varnish can be a bit overrated and its real life usage don’t necessarily correlate to what some load tests may lead you to believe.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Francis,

      thanks for sharing your experience. So will you share some figures you got on real life projects?

  11. Anabolika says:

    Thanks Dave and Vera

  12. Danish says:

    hi

    can we use apc+memcached+redis / varnish+memcached+redis to speed up the performance..

    or please suggest something to speed up the site. we have a marketplace with multi-seller.

Leave a Reply

Your email address will not be published. Required fields are marked *