Magento 1 vs Magento 2 performance comparison: speed test results

Magento 2 vs Magento 2 performance comparison: speed test results

 

One of the most important buzz features of Magento 2 is the built-in full page cache. What is more, Magento 2 FPC supports Varnish as backend. While Magento 2 is declared to be a much faster platform in comparison with the first version, we decided to check Magento 1 and 2 performance with various cache settings to see which combination gives the best results.

We set up two servers, with 6 CPU cores and 8GB RAM each. Both servers are using SSD for data storage and correspond to Amasty recommendations regarding server configuration for Magento, as well as minimal recommended Magento system requirements.

Server setup

#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30, MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6

#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14, MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2

In Magento 2, the full page cache settings are changed to use Varnish as backend when needed.

Speed test tools

In the previous articles we performed various tests and used our own Apache Jmeter scenarios.

People asked us why we weren’t using Magento Performance Kit for testing. The thing is, Magento Performance Toolkit does not support Magento CE 1.9.2.2 and is designed for Magento EE, so it isn’t suitable for our case.

Starting from Magento 2.x, the toolkit supports CE, but it is different from Magento 1.x version, and it would be incorrect to compare them. What is more, the toolkit measures only the server response time without taking AJAX into account, which is actively used in Magento 2.x.

To correctly test a dynamic website on Magento 2.x, we chose Selenium Webdriver plugin for Apache JMeter. This plugin allows using a fully functional browser, such as Google Chrome, for performance tests. In this case, we are able to measure the full page loading time, including AJAX.

Performance test scenario

The tests were performed using the following scenarios:

  • The guest visits the home page
  • The guest goes to the category
  • The category is filtered by color
  • The category is filtered by material/purpose
  • Search by ‘shirt’ keyword is performed
  • The guest goes to the product page
  • The product is added to cart

Each test has been performed 100 times, and then we counted the average value for each stage. As the web driver uses the real browser, the number of simultaneous tests for a single computer is limited by the number of CPU cores. I used Intel Core i5 with 4 cores and ran only 5 simultaneous tests so the CPU performance wouldn’t influence the test results.

Please note that this experiment is not a load test. It shows the time difference between how fast the page is loaded in the browser with full page cache variants and without it.

In our previous tests we measured the server response time, and today we’re focusing on page load time in the browser. This time consists of server response generation, network data transfer, page rendering by the browser and AJAX queries execution time.

This kind of test is very close to what a user actually sees on your website. In reality, the user sees the page before it fully loads, and the data we have received in this performance test can be treated as the worst case scenario. The results will also be similar to what webpagetest.org or pingdom.com will show.

Performance test results

Magento 1.9.2.2, no FPC:

Performance test results for Magento 1.9 without full page cache

Magento 1.9.2.2 , Amasty FPC on:

Magento 1.9 + Amasty FPC page load time

 

Magento 2.0.0, no FPC:

Magento 2 with disabled FPC speed test results

Magento 2.0.0, FPC on:

Magento 2 + FPC performance results (page load speed)

Magento 2.0.0 , FPC (Varnish) on:

Magento 2 + FPC (Varnish) performance test results

We use the average value here ( see ‘Average’ column). The difference between sample data for Magento 1.9.2.2 and Magento 2.0.0 can be neglected, because the browser uses its own local cache after the first query.

Let’s see the difference between page load time for Magento 1.9.2.2 and Magento 2.0.0 with FPC off. Magento 1.9.2.2 doesn’t have a built-in FPC, so we count it as ‘off’ (all results in ms).

Magento 1.9 and Magento 2 performance test result without FPC

 

Then, we compared different variants of FPC usage in Magento 2.0.0:

  • FPC off
  • FPC on, Varnish off
  • FPC on, Varnish on

Magento 2 page load performance results

FPC with Varnish shows up to 30% increase of page load speed if compared to the variant with FPC off, and around 3% increase if compared to the variant with FPC on, but no Varnish.

Thus, FPC in Magento 2.0.0 gives at least 27% increase for page load in the browser. It’s a great result for a free built-in feature!

Now, let’s compare the page load speed in Magento 1.9.2.2 + Amasty FPC and in Magento 2.0.0 + FPC + Varnish (ms).

Magento 1.9 and Magento 2 performance test result with FPC

Takeaway

FPC for Magento 2 significantly increases page load in the browser. The difference between Varnish on/off combinations is insignificant with low load and comes as 3%, as our test has been using only 5 simultaneous users.

With higher load, Varnish will give you a bigger advantage by lowering the server load, but with smaller traffic the margin is quite thin.

Do you have any questions about the performance test results? We’re happy to answer!

21.02.2016 update

A couple of weeks ago we published this article to compare Magento 1 and Magento 2 performance. We were contacted by Magento representatives regarding the results and the settings that can influence the speed of the installation. It’s great that we have real communication here, and it’s a unique experience for our team to work with the new product and to receive info straight from the source.

At the same time doing maximal optimization only for Magento 2 seemed uneven, so we did our best to apply the improved settings to Magento 1 test installation as well.

Here’s the configuration of the test servers we used:

#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30 (Zend Opcache), MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6

#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14 (Zend Opcache), MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2

Also, on both servers JavaScript, CSS, XML, etc compression was turned on, and Expires were added. These settings guarantee that the browser receives the content fast and caches it. We ran 5 simultaneous tests at each server and performed 100 iterations total.

Magento 1.9.2.2

For Magento 1.9.2.2 we turned compilation on at System / Tools / Compilation. Minification and merging of JavaScript/CSS were off, because the server minifies the data while delivering it, and Expires is set up as well. In case of Expires, the browser downloads CSS/JavaScript files only once and uses them from the cache afterwards.

Magento 1.9.2.2 no FPC performance

Magento 1.9.2.2 FPC performance

Magento 1.9.2.2 no FPC/Amasty FPC performance comparison (ms)

Magento 1.9.2.2 performance chart

Here FPC gives 50% more speed everywhere but the Add to Cart step. I wanted to point out once again that now we are measuring page load time, not server response time. It is the time that the browser needs to receive the data, render the page and execute javascript scenarios.

Magento 2.0.0 new performance tests

For Magento 2, we applied the recommended list of optimizations:

  • php bin/magento setup:static-content:deploy
  • php bin/magento setup:di:compile
  • Install the production mode for .htaccess (SetEnv MAGE_MODE production).
  • Turn Javascript Bundling on (Stores / Configuration / Advanced / Developer / JavaScript Settings / Enable Javascript Bundling).

Still, we didn’t use JavaScript/CSS minification and merging, because it doesn’t help with using compression while transferring the data and Expires, which was set up correctly.

Magento 2 no FPC performance test

Magento 2 in production mode with Varnish FPC performance test

Magento 2 developer mode with Varnish FPC performance test results

Magento 2 no FPC/Varnish FPC performance tests (ms)

Magento 2 no FPC vs Varnish FPC production mode performance tests

Magento 1.9.2.2 vs Magento 2.0.0 performance comparison

Magento 1.9 vs Magento 2 performance comparison

Takeaway:

  • Using Magento 2 with the turned off FPC is a bad idea.
  • Production mode is much faster than Developer mode.
  • Magento 2 doesn’t cache search results. We tried to turn on query_cache in MySQL, but that didn’t influence the result at all. We suppose that using Solr instead of MySQL would give a better result, but Solr for Catalog Search is available only for the enterprise version of Magento 2.
Andrey Tataranovich

Andrey Tataranovich

Andrei believes that only at work people can fully express themselves, so he does his best to implement all his knowledge while in the office. Having made computer science his hobby, Andrei is constantly developing his professional skills. But despite such a busy schedule he always finds time to read science fiction books and communicate with new interesting people.

You may also like...

28 Responses

  1. Andrey Zhuk says:

    Hello!

    Have you tried the similar test with installed bunch of 3rd extensions on the stores?

    Thanks.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hi there, thanks for the question and your interest.

      No, we didn’t do such a test, because we believe these results are applicable for the particular store and can be used for optimization of this very installation. Each store has its own set of extensions of different quality, functionality and by different vendors.

      Even a single poor quality extension can slow down the whole installation, and a bunch of well-developed modules will hardly give any changes if compared to the clean installation. So, the test you’re talking about is more about extensions, than Magento itself, if we’re speaking about such general checks for public interest.

      But we will possibly be testing some popular themes for Magento 2 when they appear.

  2. Dmytro Kvashnin says:

    Hi,

    As you are using WebDriver for benchmarks, you need to perform several additional actions before measuring performance for Magento2.

    Here is a list of actions, that should be performed in order to switch M2 instance into production (most performance) mode. I highly recommend to pass this flow prior to testing:

    1. Activating of client side performance features (is not the reason of slow server response, but recommended)
    a. JS bundling
    b. JS minification
    c. CSS minification
    d. HTML minification
    2. Running static content deployment tool (to avoid real time less compilation, themes fallback, etc)
    php bin/magento setup:static-content:deploy
    3. Running code compiler (to avoid real time configuration and code analysis)
    php bin/magento setup:di:compile
    4. Setting production mode
    Set $MAGE_MODE production (nginx)
    SetEnv MAGE_MODE production (apache)
    5. Performing re-index of catalog
    php bin/magento indexer:reindex
    6. Cleaning cache to apply all settings
    php bin/magento cache:clean
    php bin/magento cache:flush

    For more information please refer to
    http://devdocs.magento.com/guides/v2.0/howdoi/deploy/deploy-to-prod.html#deploy-prod
    *instead of running magento deploy:mode:set it’s better to perform 2-4p from the list, as deploy command runs compilation form multitenant solution, while you have only single tenant application.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Dmytro,

      thanks so much for your attention and for your suggestions. We will definitely test it once again and will post an update on the results!

    • Rico says:

      It’s funny how much stuff you need to do in order to make a humpy M2 running smoother … 10 steps … well …

      I’d like to mention that you should turn on these features in M1 as well:
      Compiler Mode
      JS/CSS merger
      Running all pages at least twice to measure the cached performance
      Reindex everything.

      Rico

      • Ksenia Dobreva Ksenia Dobreva says:

        Thanks for your interest, Rico. We will also take these things into account when performing tests once again.

        Happy holidays!

  3. Francis Kim says:

    Great results! The mode change really makes a huge difference.

  4. Bryan says:

    This enticed me to buy Amasty FPC and try it in my Magento 1.9.2.3 site. 😀

  5. Aurélien says:

    Results between magento 1.9.2 and magento 2 discourage me to upgrade some of the shops i maintain…
    If we read between the lines, don’t expect to improve performance of your magento site if you don’t use Varnish…. compare to 1.9.x versions…

    it’s the point which can make switch off to other ecommerce solutions…

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Aurélien,

      thanks for your comment!

      I think that it’s too early to judge – because the tests reflect the current state of the platform. Meanwhile, Magento 2 is and will be updated, and I’m sure that the updates will be affecting performance, too.

      While ver 1 has been on the market for years, and the developers had plenty of time to polish everything.

      We’ll be keeping our eye on the M2 updates, maybe even doing more tests for some future versions.

  6. Mummy Ninja says:

    How about enabling the External Full Page Cache in Magento 1? Does that help speed up Magento 1 more? The default one at System > Configuration > System > External Full Page Cache > Zend Full Page Cache. 😀

    • Ksenia Dobreva Ksenia Dobreva says:

      Hi there and thanks for your comment!

      Zend Pagecache is a component of Zend server, and it’s a paid solution. At this moment we’re not likely to test it because honestly we have never met a single client using this tool =)

  7. Israel Guido says:

    I’m using Vagrant to make coding locally, it is very slow. Any tips that can help me streamline my tests?
    Setting the current VM:
    * 2GB memory;
    * Two processor cores;
    * PHP 5.6
    * MySQL 5.6
    I used the script “MySQL Tunner” to see if improved income, and still remains too reading.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Israel,

      thanks for your question.

      Which Vagrant backend are you using (virtualbox, vmware, etc)? Also, did you turn on the hardware visualization in BIOS?

      I’m asking because the second thing is extremely important for the virtual machine performance.

      • Israel Guido says:

        Thank you for Ksenia answer, I am using Parallels on these options BIOS, I will give a searched for using the MAC.

        • Ksenia Dobreva Ksenia Dobreva says:

          Unfortunately, we can’t help you with MAC settings because we haven’t enough experience with it. But, depending on the backend type, you can play with your disk caching for better results.

  8. Andrii says:

    Why does you not use php 7 for Magento 2?

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey Andrii, thanks for your question.

      At the time the article was in process, PHP 7 was still pretty much ‘raw’. Now 7.0.4 is out, where ZendOpcache issues were fixed, and we don’t know if there will be any critical fixes in the nearest future.

      Yep, we also know that Magento 2 is faster on PHP 7 (as for Magento 1, too). We’re looking at the situation and the PHP 7 updates and are considering extending this topic in the future blog posts.

      • Andrii says:

        Thanks for answer, will wait for future posts.

        I have more questions.
        Pages response time for page in varnish is ~ 10ms. And this time should not depend on Magento 2 mode because request is not processed by php. Can you explain why your measurements show a different behavior?

        • Ksenia Dobreva Ksenia Dobreva says:

          Sure!
          10ms are actually results of synthetic tests, when both the test and Varnish are run on the same machine. Our test is not synthetic, we performed it under real life conditions. And the reality is that 30-50ms are added to each query because of the Internet connection.
          Also, in our test the result is the page load time in the browser, not only the Varnish response.
          If you have more questions about this topic, we will be happy to answer them as well!

  9. A Man bored with Varien 2.0 says:

    Magento answered by a good april Joke: “no no, we assure you add to cart is faster on Magento2”.
    https://magento.com/resources/white-paper-magento-2-site-performance-and-scalability-optimizations

    It’s just manipulation of numbers, and way of testing things… It’s just a shame.

    Long live magento 1.x

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey there, thanks for sharing the link. We haven’t scanned the doc yet, but we will do this in the nearest time.

  10. Jon says:

    Hi,
    Your article has not been updated for a long time.
    Maybe you can test it with php7 Magento 1.9.3.1 and Magento 2.1.2

    • Jon says:

      Or Magento EE 1.14.3.1, Magento-EE-2.0.10, Magento-EE-2.1.2.
      If you need Magento EE contact me to e-mail.

    • Ksenia Dobreva Ksenia Dobreva says:

      Hey there and many thanks for your interest.
      Indeed, we have a plan to test those versions in the nearest future. Not sure with the date, but most probably it’ll be in 2016. Thanks for your suggestion about the EE – I’ll check with our admin who will write the post, and if he has plans on including EE on the list, I’ll definitely contact you. Cheers!

Leave a Reply

Your email address will not be published. Required fields are marked *