Magento 1 vs Magento 2 performance comparison: speed test results

Table of Content

Magento 2 vs Magento 2 performance comparison: speed test results
Posted in: SEO Optimization

 One of the most important buzz features of Magento 2 is the built-in full page cache. What is more, Magento 2 FPC supports Varnish as backend. While Magento 2 is declared to be a much faster platform in comparison with the first version, we decided to check Magento 2 vs Magento 1 performance with various cache settings to see which combination gives the best results.

In spite of all those vanilla Magento 2 features like FPC and Varnish - the performance won't be perfect without magic sorcery. We won't offer you to speed up your website with a magic wand but with our new Magento 2 Speed Optimization extension. Asynchronous indexing; HTML, CSS, JS minification; lazy load and images optimization can help you gain higher scores in Google Page Speed Optimizer in just a few days. We guarantee that moving your JS to footer will never be a problem anymore. Render your pages faster, attract visitors with speedy performance.

We set up two servers, with 6 CPU cores and 8GB RAM each. Both servers are using SSD for data storage and correspond to Amasty recommendations regarding server configuration for Magento, as well as minimal recommended Magento system requirements.

Server setup

#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30, MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6

#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14, MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2

In Magento 2, the full page cache settings are changed to use Varnish as backend when needed.

Speed test tools

In the previous articles we performed various tests and used our own Apache Jmeter scenarios.

People asked us why we weren’t using Magento Performance Kit for testing. The thing is, Magento Performance Toolkit does not support Magento CE 1.9.2.2 and is designed for Magento EE, so it isn’t suitable for our case.

Starting from Magento 2.x, the toolkit supports CE, but it is different from Magento 1.x version, and it would be incorrect to compare them. What is more, the toolkit measures only the server response time without taking AJAX into account, which is actively used in Magento 2.x.

To correctly test a dynamic website on Magento 2.x, we chose Selenium Webdriver plugin for Apache JMeter. This plugin allows using a fully functional browser, such as Google Chrome, for performance tests. In this case, we are able to measure the full page loading time, including AJAX.

Performance test scenario

The tests were performed using the following scenarios:

  • The guest visits the home page
  • The guest goes to the category
  • The category is filtered by color
  • The category is filtered by material/purpose
  • Search by ‘shirt’ keyword is performed
  • The guest goes to the product page
  • The product is added to cart

Each test has been performed 100 times, and then we counted the average value for each stage. As the web driver uses the real browser, the number of simultaneous tests for a single computer is limited by the number of CPU cores. I used Intel Core i5 with 4 cores and ran only 5 simultaneous tests so the CPU performance wouldn’t influence the test results.

Please note that this experiment is not a load test. It shows the time difference between how fast the page is loaded in the browser with full page cache variants and without it.

In our previous tests we measured the server response time, and today we’re focusing on page load time in the browser. This time consists of server response generation, network data transfer, page rendering by the browser and AJAX queries execution time.

This kind of test is very close to what a user actually sees on your website. In reality, the user sees the page before it fully loads, and the data we have received in this performance test can be treated as the worst case scenario. The results will also be similar to what webpagetest.org or pingdom.com will show.

Performance test results

Magento 1.9.2.2, no FPC:

Performance test results for Magento 1.9 without full page cache

Magento 1.9.2.2 , Amasty FPC on:

Magento 1.9 + Amasty FPC page load time

 

Magento 2.0.0, no FPC:

Magento 2 with disabled FPC speed test results

Magento 2.0.0, FPC on:

Magento 2 + FPC performance results (page load speed)

Magento 2.0.0 , FPC (Varnish) on:

Magento 2 + FPC (Varnish) performance test results

We use the average value here ( see ‘Average’ column). The difference between sample data for Magento 1.9.2.2 and Magento 2.0.0 can be neglected, because the browser uses its own local cache after the first query.

Let’s see the difference between page load time for Magento 1.9.2.2 and Magento 2.0.0 with FPC off. Magento 1.9.2.2 doesn’t have a built-in FPC, so we count it as ‘off’ (all results in ms).

Magento 1.9 and Magento 2 performance test result without FPC

 

Then, we compared different variants of FPC usage in Magento 2.0.0:

  • FPC off
  • FPC on, Varnish off
  • FPC on, Varnish on

Magento 2 page load performance results

FPC with Varnish shows up to 30% increase of page load speed if compared to the variant with FPC off, and around 3% increase if compared to the variant with FPC on, but no Varnish.

Thus, FPC in Magento 2.0.0 gives at least 27% increase for page load in the browser. It’s a great result for a free built-in feature!

Now, let’s compare the page load speed in Magento 1.9.2.2 + Amasty FPC and in Magento 2.0.0 + FPC + Varnish (ms).

Magento 1.9 and Magento 2 performance test result with FPC

Takeaway

FPC for Magento 2 significantly increases page load in the browser. The difference between Varnish on/off combinations is insignificant with low load and comes as 3%, as our test has been using only 5 simultaneous users.

With higher load, Varnish will give you a bigger advantage by lowering the server load, but with smaller traffic the margin is quite thin.

Do you have any questions about the performance test results? We’re happy to answer!

21.02.2016 update

A couple of weeks ago we published this article to compare Magento 1 and Magento 2 performance. We were contacted by Magento representatives regarding the results and the settings that can influence the speed of the installation. It’s great that we have real communication here, and it’s a unique experience for our team to work with the new product and to receive info straight from the source.

At the same time doing maximal optimization only for Magento 2 seemed uneven, so we did our best to apply the improved settings to Magento 1 test installation as well.

Here’s the configuration of the test servers we used:

#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30 (Zend Opcache), MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6

#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14 (Zend Opcache), MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2

Also, on both servers JavaScript, CSS, XML, etc compression was turned on, and Expires were added. These settings guarantee that the browser receives the content fast and caches it. We ran 5 simultaneous tests at each server and performed 100 iterations total.

Magento 1.9.2.2

For Magento 1.9.2.2 we turned compilation on at System / Tools / Compilation. Minification and merging of JavaScript/CSS were off, because the server minifies the data while delivering it, and Expires is set up as well. In case of Expires, the browser downloads CSS/JavaScript files only once and uses them from the cache afterwards.

Magento 1.9.2.2 no FPC performance

Magento 1.9.2.2 FPC performance

Magento 1.9.2.2 no FPC/Amasty FPC performance comparison (ms)

Magento 1.9.2.2 performance chart

Here FPC gives 50% more speed everywhere but the Add to Cart step. I wanted to point out once again that now we are measuring page load time, not server response time. It is the time that the browser needs to receive the data, render the page and execute javascript scenarios.

Magento 2.0.0 new performance tests

For Magento 2, we applied the recommended list of optimizations:

  • php bin/magento setup:static-content:deploy
  • php bin/magento setup:di:compile
  • Install the production mode for .htaccess (SetEnv MAGE_MODE production).
  • Turn Javascript Bundling on (Stores / Configuration / Advanced / Developer / JavaScript Settings / Enable Javascript Bundling).

Still, we didn’t use JavaScript/CSS minification and merging, because it doesn’t help with using compression while transferring the data and Expires, which was set up correctly.

Magento 2 no FPC performance test

Magento 2 in production mode with Varnish FPC performance test

Magento 2 developer mode with Varnish FPC performance test results

Magento 2 no FPC/Varnish FPC performance tests (ms)

Magento 2 no FPC vs Varnish FPC production mode performance tests

Magento 1.9.2.2 vs Magento 2.0.0 performance comparison

Magento 1.9 vs Magento 2 performance comparison

Takeaway:

  • Using Magento 2 with the turned off FPC is a bad idea.
  • Production mode is much faster than Developer mode.
  • Magento 2 doesn’t cache search results. We tried to turn on query_cache in MySQL, but that didn’t influence the result at all. We suppose that using Solr instead of MySQL would give a better result, but Solr for Catalog Search is available only for the enterprise version of Magento 2.
December 16, 2015
December 18, 2015
December 9, 2015
Comments
Andrey Zhuk
December 16, 2015
Hello! Have you tried the similar test with installed bunch of 3rd extensions on the stores? Thanks.
Reply
Ksenia Dobreva
December 16, 2015
Hi there, thanks for the question and your interest. No, we didn't do such a test, because we believe these results are applicable for the particular store and can be used for optimization of this very installation. Each store has its own set of extensions of different quality, functionality and by different vendors. Even a single poor quality extension can slow down the whole installation, and a bunch of well-developed modules will hardly give any changes if compared to the clean installation. So, the test you're talking about is more about extensions, than Magento itself, if we're speaking about such general checks for public interest. But we will possibly be testing some popular themes for Magento 2 when they appear.
Dmytro Kvashnin
December 16, 2015
Hi, As you are using WebDriver for benchmarks, you need to perform several additional actions before measuring performance for Magento2. Here is a list of actions, that should be performed in order to switch M2 instance into production (most performance) mode. I highly recommend to pass this flow prior to testing: 1. Activating of client side performance features (is not the reason of slow server response, but recommended) a. JS bundling b. JS minification c. CSS minification d. HTML minification 2. Running static content deployment tool (to avoid real time less compilation, themes fallback, etc) php bin/magento setup:static-content:deploy 3. Running code compiler (to avoid real time configuration and code analysis) php bin/magento setup:di:compile 4. Setting production mode Set $MAGE_MODE production (nginx) SetEnv MAGE_MODE production (apache) 5. Performing re-index of catalog php bin/magento indexer:reindex 6. Cleaning cache to apply all settings php bin/magento cache:clean php bin/magento cache:flush For more information please refer to http://devdocs.magento.com/guides/v2.0/howdoi/deploy/deploy-to-prod.html#deploy-prod *instead of running magento deploy:mode:set it's better to perform 2-4p from the list, as deploy command runs compilation form multitenant solution, while you have only single tenant application.
Reply
Ksenia Dobreva
December 17, 2015
Hey Dmytro, thanks so much for your attention and for your suggestions. We will definitely test it once again and will post an update on the results!
Rico
December 17, 2015
It's funny how much stuff you need to do in order to make a humpy M2 running smoother … 10 steps … well … I'd like to mention that you should turn on these features in M1 as well: Compiler Mode JS/CSS merger Running all pages at least twice to measure the cached performance Reindex everything. Rico
Ksenia Dobreva
December 21, 2015
Thanks for your interest, Rico. We will also take these things into account when performing tests once again. Happy holidays!
Francis Kim
January 22, 2016
Great results! The mode change really makes a huge difference.
Reply
Ksenia Dobreva
January 22, 2016
Yes, and we're sure that in the future with the new updates the results will get even better.
Bryan
January 29, 2016
This enticed me to buy Amasty FPC and try it in my Magento 1.9.2.3 site. :D
Reply
Ksenia Dobreva
February 1, 2016
Ha-ha, very happy to hear it, Bryan =)
Aurélien
January 30, 2016
Results between magento 1.9.2 and magento 2 discourage me to upgrade some of the shops i maintain... If we read between the lines, don't expect to improve performance of your magento site if you don't use Varnish.... compare to 1.9.x versions... it's the point which can make switch off to other ecommerce solutions...
Reply
Ksenia Dobreva
February 1, 2016
Hey Aurélien, thanks for your comment! I think that it's too early to judge - because the tests reflect the current state of the platform. Meanwhile, Magento 2 is and will be updated, and I'm sure that the updates will be affecting performance, too. While ver 1 has been on the market for years, and the developers had plenty of time to polish everything. We'll be keeping our eye on the M2 updates, maybe even doing more tests for some future versions.
Mummy Ninja
January 30, 2016
How about enabling the External Full Page Cache in Magento 1? Does that help speed up Magento 1 more? The default one at System > Configuration > System > External Full Page Cache > Zend Full Page Cache. :D
Reply
Ksenia Dobreva
February 1, 2016
Hi there and thanks for your comment! Zend Pagecache is a component of Zend server, and it's a paid solution. At this moment we're not likely to test it because honestly we have never met a single client using this tool =)
Israel Guido
February 29, 2016
I'm using Vagrant to make coding locally, it is very slow. Any tips that can help me streamline my tests? Setting the current VM: * 2GB memory; * Two processor cores; * PHP 5.6 * MySQL 5.6 I used the script "MySQL Tunner" to see if improved income, and still remains too reading.
Reply
Ksenia Dobreva
March 3, 2016
Hey Israel, thanks for your question. Which Vagrant backend are you using (virtualbox, vmware, etc)? Also, did you turn on the hardware visualization in BIOS? I'm asking because the second thing is extremely important for the virtual machine performance.
Israel Guido
March 3, 2016
Thank you for Ksenia answer, I am using Parallels on these options BIOS, I will give a searched for using the MAC.
Ksenia Dobreva
March 3, 2016
Unfortunately, we can't help you with MAC settings because we haven't enough experience with it. But, depending on the backend type, you can play with your disk caching for better results.
Israel Guido
March 3, 2016
Thanks ! =)
Andrii
March 15, 2016
Why does you not use php 7 for Magento 2?
Reply
Ksenia Dobreva
March 16, 2016
Hey Andrii, thanks for your question. At the time the article was in process, PHP 7 was still pretty much 'raw'. Now 7.0.4 is out, where ZendOpcache issues were fixed, and we don't know if there will be any critical fixes in the nearest future. Yep, we also know that Magento 2 is faster on PHP 7 (as for Magento 1, too). We're looking at the situation and the PHP 7 updates and are considering extending this topic in the future blog posts.
Andrii
March 16, 2016
Thanks for answer, will wait for future posts. I have more questions. Pages response time for page in varnish is ~ 10ms. And this time should not depend on Magento 2 mode because request is not processed by php. Can you explain why your measurements show a different behavior?
Ksenia Dobreva
March 17, 2016
Sure! 10ms are actually results of synthetic tests, when both the test and Varnish are run on the same machine. Our test is not synthetic, we performed it under real life conditions. And the reality is that 30-50ms are added to each query because of the Internet connection. Also, in our test the result is the page load time in the browser, not only the Varnish response. If you have more questions about this topic, we will be happy to answer them as well!
A Man bored with Varien 2.0
April 2, 2016
Magento answered by a good april Joke: "no no, we assure you add to cart is faster on Magento2". https://magento.com/resources/white-paper-magento-2-site-performance-and-scalability-optimizations It's just manipulation of numbers, and way of testing things... It's just a shame. Long live magento 1.x
Reply
Ksenia Dobreva
April 4, 2016
Hey there, thanks for sharing the link. We haven't scanned the doc yet, but we will do this in the nearest time.
Jon
November 16, 2016
Hi, Your article has not been updated for a long time. Maybe you can test it with php7 Magento 1.9.3.1 and Magento 2.1.2
Reply
Jon
November 16, 2016
Or Magento EE 1.14.3.1, Magento-EE-2.0.10, Magento-EE-2.1.2. If you need Magento EE contact me to e-mail.
Ksenia Dobreva
November 18, 2016
Hey there and many thanks for your interest. Indeed, we have a plan to test those versions in the nearest future. Not sure with the date, but most probably it'll be in 2016. Thanks for your suggestion about the EE - I'll check with our admin who will write the post, and if he has plans on including EE on the list, I'll definitely contact you. Cheers!
Jehzeel Laurente
December 27, 2016
Do you have a comparison between Magento CE 1.9.3.1 vs Magento CE 2.1.3. Maybe the results are now different. ?
Reply
Ksenia Dobreva
December 27, 2016
Hey Jehzeel, thanks for your interest. We will be posting some new tests right after the New Year. Stay tuned!
Matt
January 20, 2017
Are those new tests still coming?
Ksenia Dobreva
January 20, 2017
Matt, feel free to see the post we've recently published: https://amasty.com/blog/magento-1-2-performance-speed-php-5-php-7/
Joshua
January 23, 2017
I thought one of the benefits of Magento 2 was speed? how come it is slow! also, I do not like to rely fully on FPC for fast page loads. The application itself should be fast too.
Reply
Ksenia Dobreva
January 30, 2017
Hey Joshua, this test was performed more than a year ago. I’m pretty sure right now Magento 2 is much faster than before. We’re planning to do the similar tests, and also you’re welcome to check this post as well: https://amasty.com/blog/magento-1-2-performance-speed-php-5-php-7/
Alex Morco
September 11, 2018
Thanks for sharing this article, Your efforts are shown here about your research and findings to get the best result to your readers, I was confused in selecting version of Magento because many said to go with Magento 1 and many said to go with Magento 2, but after reading this post and another article (cloudways.com/blog/magento-1-vs-magento-2-survey/), I have now selected to go with Magento 2 .
Reply
Guest
October 29, 2019
0.5
Reply
BZ
November 8, 2019
We've been developing our magento 2 site for nearly a year, and the site is an exact replica of the current m1 site, simply designed to comply with the m2 phase-out. We use a professional developer that we've worked with for 7 years, professional dedicated server hosting, and primarily amasty extensions. Unfortunately from a performance perspective, m2 lags behind m1 in most meaningful categories such as paints. This is not simply an issue of metrics, as we noticed the performance issues visually, not simply through performance tools, and the differences were stark, even though the amount of data is similar between the sites. For example, after much work, last painted hero on product pages goes from 1.9 seconds m1 to 8 seconds m2, even after the update that supposedly fixed the gallery loading issue. Visually our product pages appeared to render in around 0.25 seconds on m1, and take up to 10 seconds on m2 (we have that down to 4-5 seconds, still poor) This is a major problem. First content paint goes from 1.2 to 2.0, .load time from 3 to 7.5. We are finding that the user experience of m2 is inferior to m1, and a lot of work is needed to be done by extension developers and magento core to address performance. Attempting to fix this on our side results in an incredible amount fo work and cost, with the overall result still far behind the m1 site.
Reply
Polina Litreyeva
November 11, 2019
Hello! Pity you’ve come up with the speed optimization troubles. Unfortunately, it's a common issue. You can take a look at our <a href="https://amasty.com/google-page-speed-optimizer-for-magento-2.html">Google Page Speed Optimizer</a>, that we have developed to solve this problem. If you have questions, feel free to contact our support team at <a href="mailto:[email protected]" rel="nofollow">[email protected]</a>.
Leave your comment

Your email address will not be published

This blog was created with Amasty Blog Pro

This blog was created with Amasty Blog Pro

© 2009-2024 Amasty. All Rights Reserved.