Amasty Research | Full Page Cache vs. Varnish Cache

Table of Content

Amasty research: Full Page Cache vs Varnish Cache
Posted in: Development Hacks

Amasty Team has recently issued a new extension Full Page Cache, aimed to speed up Magento store pages, known for their huge weight. Full Page Cache is a great success, it considerably reduces page load time and offers a number of other cool features. But actions speak louder than words. And statistics speak even louder. Check Amasty's new research on how fast can Magento be with Full Page Cache?

→ Have already switched to Magento 2? Try the Speed Optimization extension we’ve recently released. Improve your site performance by numerous showings.


How fast can Magento be with Full Page Cache?

TEST CONFIGURATION

HARDWARE

  • CPU: Intel Xeon L5520 2.27GHz (16 core)
  • RAM: 16 GB
  • Storage: SSD INTEL SSDSC2BB480G4

SOFTWARE

  • Magento Community 1.8.1.0
  • Magento Sample Data 1.2.0
  • Apache 2.2.22
  • PHP 5.4.4 (mod_php + Zend OpCache 7.0.3)
  • MySQL 5.5.35

For testing Apache jMeter with the following scenario was used:

  1. A visitor goes to the website home page;
  2. Searches for “shirt” in the search field;
  3. The result is filtered by price ($0.00 — $99.99);
  4. After that the result is filtered by color (white color);
  5. As a result, only 1 Product is left, the user opens its page;
  6. The Product is added to the cart at the Product Page;
  7. After that, the user goes to the Category Page «Electronics / Cameras / Digital Cameras»;
  8. The user reloads the Category Page «Electronics / Cameras / Digital Cameras»;
  9. At the Category Page camera Canon PowerShot A630 is added for comparison;
  10. At the Category Page camera Kodak EasyShare C530 is added for comparison;
  11. The Products Comparison Page is opened;
  12. From the Products Comparison Page, the camera Canon PowerShot A630 is added to the cart.

The test was performed for 10, 20, 30, 40, and 50 simultaneous visitors in the variants “Only Magento cache on” and “Magento cache and FPC on”. Before every launch, the Magento cache was cleared and a test for 10 simultaneous visitors was launched, but its results were not taken into account (this is done to prevent a difference in the results connected with the increasing amount of cache during the successive test launches).

In order to compare all possible variants, we have additionally tested a variant when Magento has caching turned off. The most impatient readers can see its result in the diagram below (the result for the Product Page with 50 simultaneous visitors).



Full Page Cache - Magento research

To prevent the influence of the data transfer speed on the Internet the test was launched in the mode “Remote start”. In this case in the server where the tested Magento is installed Apache jMeter is launched in the server mode, which receives the test from the managing client, performs it, and sends the results back to the client. The client part of Apache jMeter is launched on the desktop, it manages only the server part and shows the results. This way we have prevented the influence of the Internet channel load on the test results.

Every test was performed 10 times to provide the correct calculation of the average value of the test results. This was for 50 simultaneous users jMeter will perform the test scenario 500 times. The test scenario contains 12 requests to the server (download of CSS page resources/JS/images are not taken into account in the test results), this way during the test with 50 simultaneous visitors 6000 requests are sent to the server.

The test results in ms come in the table as follows:

Full Page Cache - Magento research

You have probably noticed that Category Page opens twice. This can be put to the fact that when a product is added to the cart the status of the dynamic blocks, the content of which should be reloaded during the next request to them, is changed. In this case, the first Page impression from the cache is slower than the successive ones. To see the difference in the load speed (when the dynamic blocks are already reloaded) the test goes to the Category page one more time. And the page is downloaded completely from the cache.

According to the table data, the following diagrams have been created for a better visual assessment of the difference in the server reply speed.

Full Page Cache - Magento research

Full Page Cache - Magento research

Full Page Cache - Magento research

Full Page Cache - Magento research

Full Page Cache - Magento research

Full Page Cache - Magento research

Full Page Cache - Magento research

The last diagram shows the average results in all the mentioned categories. Usage of the Full Page Cache Magento extension lets us on average increase the speed by 5 times. In some tests, the difference is even bigger.

But what makes our extension better than Varnish Cache and whether it is better at all?

Magento Varnish Cache vs. Full Page Cache: comparison test

If you need to speed up your Magento store, just ‘faster’ is not enough to describe the performance of your website. A fair contest of two Magento cache extensions with detailed speed results is at your service.

→ Check how to configure varnish cache in Magento 2

TEST CONFIGURATION

We have changed the test configuration a little bit in comparison to our previous Full Page Cache test.

Hardware

  • CPU: Intel Xeon L5520 2.27GHz (16 core)
  • RAM: 16 GB
  • Storage: SSD INTEL SSDSC2BB480G4

Software

  • Magento Community 1.9.0.0
  • Magento Sample Data 1.9.0.0
  • Apache 2.2.22
  • Varnish 3.0.2
  • PHP 5.4.4 (mod_php + Zend OpCache 7.0.3)
  • MySQL 5.5.35

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. It is installed in front of any server that speaks HTTP.

Varnish Cache itself is not designed for working with Magento directly, so we took the Nexcess Turpentine module for Magento and Varnish integration for the speed test. Based on the recommendations of the Turpentine module producer, we added «-p esi_syntax=0×2 -p cli_buffer=16384» to Varnish startup options. As a result, Varnish startup options looked as follows:

DAEMON_OPTS=”-a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m -p esi_syntax=0x2 -p cli_buffer=16384″

To prevent Apache and Varnish conflict Apache was switched to port 8080.

For testing we used the list of 14 URLs of Category and Product pages: http://our-test-site/

  • http://our-test-site/women.html
  • http://our-test-site/women/new-arrivals.html
  • http://our-test-site/women/new-arrivals/tori-tank-460.html
  • http://our-test-site/women/new-arrivals/elizabeth-knit-top-484.html
  • http://our-test-site/women/new-arrivals/lafayette-convertible-dress.html
  • http://our-test-site/women/tops-blouses.html
  • http://our-test-site/women/tops-blouses/nolita-cami-484.html
  • http://our-test-site/women/tops-blouses/black-nolita-cami.html
  • http://our-test-site/women/pants-denim.html
  • http://our-test-site/women/pants-denim/dumbo-boyfriend-jean.html
  • http://our-test-site/women/dresses-skirts.html
  • http://our-test-site/women/dresses-skirts/essex-pencil-skirt-527.html
  • http://our-test-site/women/dresses-skirts/ludlow-sheath-dress.html

Before every test launch, the Magento cache was cleared and a single visit to all URLs was performed to “warm the cache up” before taking the test results. After the warm-up test for 10, 20, 30, 40, and 50 simultaneous visitors was launched.

Comparison test results (in ms) are in the table as follows:

 

To make the results easier to comprehend and let you vividly see the page load speed each Magento extension provides we have built some diagrams.

Full Page Cache vs Varnish Cache speed review

Full Page Cache vs Varnish Cache extensions

Full Page Cache vs Varnish Magento extensions

Magento Full Page Cache Varnish Cache comparison

Magento cache performance comparison

As you can see, the speed of Full Page Cache exceeds the combination of Varnish Cache and Turpentine in 2-4 times. The biggest difference can be seen during the first visit of the website – at this moment both caches create a session for the user and have to initialize Magento. Since Full Page Cache initializes at the early stage, it can give the cached result quickly and Turpentine uses a lot of time in vain on extra initialization. This is an important point, as its impact deals with first visits and users’ impression of your website.

According to Kissmetrics, a 1-second delay in website load speed can decrease conversions by 7%. You might even want to count the money you lose using a Magento cache extension providing worse page load time.

This way Amasty Team recommends you using the Full Page Cache Magento extension for providing the minimal page load time possible.

June 5, 2014
June 12, 2014
April 1, 2014
Comments
Wes
June 6, 2014
Hi Andrew, Can you use FPC + Varnish + Turpentine together?
Reply
Vera Rabkina
June 6, 2014
Yes, sure.
Dave
June 18, 2014
No, https://github.com/nexcess/magento-turpentine/wiki/Compatibility
Vera Rabkina
June 18, 2014
Dave is right, my bad. Sorry for the wrong answer. FPC and Varnish can't be used together. Thank you, Dave.
Wes
June 19, 2014
Thanks Dave and Vera
Reply
Addison
September 3, 2014
Varnish means light speed. I used Lesti which provides about 62 ms with 100 visitors, 10000 requests. But with Varnish same benchmark this page loads in 6 ms. Varnish can process 2000 req/sec in my 4 core 3.4 GHz VM, 4 Gb RAM, SSD. I would prefer FPC because it is not creating trouble as Varnish. If you don't know VCL syntax your are blind. Also there are a lot of block that need to be set to no cache. Still get trouble with Community Pole. Nexcess says they solved this issue but I still have it in my 1.8 and 1.9 version out of the box, just testing. You vote and block remains unchanged. Vote again and again and nothing happening. If you go to other page you will see vote results. A new refresh will hide the block for that session (which is normal). Also there is a lack of support for Turpentine. Page Cache also provide a free version without ESI support (which is the most wanted). My conclusion Varnish is faster than any FPC on the market, the only problem is configuration, especially when you have AJAX in your theme.
Reply
Ksenia Dobreva
September 3, 2014
First of all, thanks for your comment! It is true that Varnish requires some work to be set properly so it gives max results, but that’s not exactly the point of the article. There are users that have time and knowledge to work on configuration, and there are users that need an extension out of the box. That’s why we did our tests like an average user would have done; we installed two products and compared their performance. However, it is great that you managed to get better results with Varnish.
Francesco Haymar d\'Ettory
February 22, 2018
Actually you worked on some systems configuration that was not out of the box, as switching Apache port, of course. I think you should underline mostly that your module does not need effort, while an additional server does need tuning and maintenance. An average user will not install it at all for this reason, while a system engineer will want to use it even with your module (yes, the incompatibility can be fixed) to improve performance even more.
Den
October 2, 2014
you dont understand the main difference between VARNISH FPC and other non-Varnish FPC. this is all about resources needed to process your requests. with Varnish you need 0 (zero) resources, with non-varnish fpc you will get to the limit very fast, more visitors/buyers you have - faster your server will die... speed is not so important as server capacity...
Reply
Ksenia Dobreva
October 2, 2014
Den, thanks for reading and commenting. I see your point, but in this article we tested the exact case, not the theory in general. We were interested in the results on a given website. Still, I can't agree on your comment about speed. Speed isn't less important than server capacity. It works for SEO in a great way (site speed is an important ranking factor), what is more, it directly influences customer behavior (here's just one of the hundreds of articles on website speed influence studies webperformancetoday.com/2014/04/09/web-page-speed-affect-conversions-infographic/) In other words, you can save money on your server, but lower site speed takes even more money from you. Cheers!
Rick Mangles
October 21, 2014
No, speed is not an IMPORTANT SEO FACTOR! That is a myth!!!! It's a factor...but not an important one.
Ksenia Dobreva
October 22, 2014
Hi Rick, thanks for your comment. We'd be grateful if you share some proof over that statement. If to add to the link shared above, correlation studies show that site speed is important. https://www.searchmetrics.com/knowledge-base/ranking-factors/
Rick Mangles
November 18, 2014
"Correlation does not imply causation."
Ksenia Dobreva
November 21, 2014
Indeed, Rick. But Google officially confirmed site speed as a ranking factor four years ago. https://webmasters.googleblog.com/2010/04/using-site-speed-in-web-search-ranking.html And from my experience it has become a stronger factor throughout 2014.
Den
October 2, 2014
I am sure that your test is invalid, the varnish response time is never dependent on the number of requests. 10 or 1000 requests - you will always get the same response time. and varnish even will be faster than any cache which is completely dependent on the database and php. Varnish is independent cache, directly from memory. even if you use ESI, there will be low impact. probably you misconfigured your test environment. i just tested many different non-varnish caching modules for magento, and they are awful, server is dying after 150-200 simultaneous users...Varnish keeps working.
Reply
Ksenia Dobreva
November 6, 2014
Hi Den, and thanks for your comment. Although, it looks like you're not taking into account the integration module and various ESI operating modes in Varnish. And in this particular case Varnish performance depended on Turpentine integration module. Hope it clears things a bit.
Den
November 6, 2014
LOL
Gershon
October 14, 2014
Hello, Want to share my experience: I have 3 sites on the same server 1) Use Amasty Full Page Cache musicgate.co.il (5000 products) 2) Varnish as fronted for static content and LETSI FPC (600 products) smoke4u.co.il 3) Varnish as fronted for static content and extendware FPC (1500 products) il-cosmetics.com Amasty Full Page Cache , work best out of the box. LESTI and Extendware with warnish get closed result.
Reply
Ksenia Dobreva
October 16, 2014
Hi Gershon, thanks for your comment and sharing your experience. Happy that you are using our extension and happy with the results.
Leon Poole
October 22, 2014
You should clarify in the article that you're testing against your Amasty FPC extension and not Magento Enterprise FPC native - this confused me while reading.
Reply
Ksenia Dobreva
October 22, 2014
Hi Leon, sorry if it confused you. In fact, we are working to publish a new article on this comparison soon.
Chris
April 24, 2015
Hi, after reading this article and some of the comments i think that many people will get confused because of the limited information given about the difficulties and the resources needed in each case. I will try to clarify my opinion by writing some facts combined with my personal experience and other developer experiences i have read. I don't consider myself an expert developer although i have worked in many projects during the last 20 years so before adapting my proposals please make your own research and benchmarks. Varnish can work on a separate server or even on many servers with load balancing, FPC solutions cant. Varnish operates in Virtual Memory where it stores all static content while FPC uses slow filesystem. If there is enough RAM and a page doesn't have any dynamic parts Varnish can serve it with no delay (2ms, just little slower than serving a static file which needs 0.7ms). Varnish is written in C and is massively multi-threaded while FPC depents on the webserver, PHP implementation and opcode cache. The amount of requests Varnish can handle on a load balanced multiserver environment where each dedicated server has lot of ram cant be compared to any other non custom solution i know. So i think i made it clear enough that in my opinion Varnish shine for websites with huge traffic allowing them to serve static content in a scalable ultra fast optimized way. But what about Magento dynamic parts, https, clearing cache and ram consumption? Well there is where pain starts, it is hard even for experienced developers using Varnish integration on Magento to configure non caching/lazy caching of custom dynamic parts, for example Turpentine vendor can't configure correct behavior of the default poll module even after trying for some time. There are no current plans to add HTTPS support to Varnish. Clearing cache can be a pain too, there are a bugs in Turpentine's integration considering cache refreshing and clearing and i am pretty convinced from both personal experience and countless posts that in all popular integrations cache will just keep growing until a manual delete (which will of course need time for cache recreation). As long as Varnish cache keeps growing the response time and RAM consumption will keep growing too so i would never use it on a server that hosts more websites and there is limited RAM. Personally my experience of using Varnish for websites that have a lot of dynamic parts is limited but people who manage large e shops claim that in this case Turpentine Varnish is slower than FPC, i am sure this is true because those developers are experienced and try hard for optimum results but i am also convinced that with a better Varnish integration this can change. Support is also another factor, with Varnish integrations support is needed a lot but because of the complexity there is none or limited while on paid FPC modules there is excellent one, although it wont be needed in most cases. So to summarize using Varnish needs root access on server so you can install it, exceptional skills to configure dynamic parts and correct integration bugs concerning cache refreshing and clearing and a lot of RAM. FPC on the other hand just need one click on Magento connect and you will have great results and one easy way to configure what custom parts will be cached and what wont be. For my needs a server with SSD, Apache with PHP-FPM, Zend opcode cache, Mysql, and FPC on Magento is fast enough but if i ever need to serve more requests i would use Redis for session management and cache backend. For even more requests i would find the bottleneck and depending on it and the costs of human and hardware resources i would consider taking advantage of Nginx with HHVM ( i made some tests and the results where really great) or more servers using Varnish.
Reply
Ksenia Dobreva
April 24, 2015
Hey Chris, thanks so much for your time and effort - you have shared great insights on this question! I'm sure our readers will benefit from your knowledge. Very impressive.
ADDISON74
July 30, 2015
Please check the latest version of Turpentine. The Poll problem is solved and many others. There is a new Turpentine team who is working great to improve this extension. Also Magento 2 will have support for Varnish 3 and 4 out of the box. I am using Turpentine latest + Varnish 3 + Magento 1.9.2 in production, one dedicated server, 16 RAM, SSD's, 8 cores, and I can say with a huge traffic daily this kick the ass. We tried before a lot of FPC's solutions and they are great for small websites with a less traffic. By the way Varnish will get https support officially, or you can use Pound in front of it which is easy to configure. I am working to find a way to get analyzing results correctly, because Varnish is hit first not the webserver (awstats takes info from apache logs). Nginx + HHVM could be a solution, it is very fast, or just HHVM doing the whole thing. In the last 2 years I configure many solutions, but always come back to basic because I encountered many issues. Even Lesti FPC has a problem with Poll side module (which is now solved in Turpentine), probably Amasty has it too.
Kenan SALTIK
June 2, 2015
As visitor number increases the response times of varnish decreases. What will If your concurrent visitor number will reach thousands....
Reply
Ksenia Dobreva
June 3, 2015
Hey Kenan, thanks for your comment! In case it's a 100% static page, performance will be defined by the web server and hardware. But if the page has dynamic elements, neither our FPC nor Varnish will be of great use in the case you described.
Francis Kim
November 9, 2015
Having fine tuned Magento instances to the max before, I'd say Varnish can be a bit overrated and its real life usage don't necessarily correlate to what some load tests may lead you to believe.
Reply
Ksenia Dobreva
November 9, 2015
Hey Francis, thanks for sharing your experience. So will you share some figures you got on real life projects?
Anabolika
January 2, 2016
Thanks Dave and Vera
Reply
Danish
August 31, 2016
hi can we use apc+memcached+redis / varnish+memcached+redis to speed up the performance.. or please suggest something to speed up the site. we have a marketplace with multi-seller.
Reply
Ksenia Dobreva
September 6, 2016
Hey there, normally we recommend Zend Opcache + Redis + Amasty FPC for the best performance. Cheers!
Thiago
October 14, 2017
You guys have the best cache extension. Hands down. But I keep having session and cart quantity, customer login problems. Any tips or specific documentation? Thanks and congratulations.
Reply
Alina Bragina
January 22, 2018
Hi Thiago, thanks for the great feedback! Pity you have the issue. Please, feel free to send a word to our support team at [email protected], they will be happy to see how they can help you with that. Thanks in advance!
Leave your comment

Your email address will not be published

This blog was created with Amasty Blog Pro

This blog was created with Amasty Blog Pro

© 2009-2024 Amasty. All Rights Reserved.