Wednesday, February 22, 2012

Apache 2.4 vs Nginx Benchmark Showdown

Disclaimer: This test was highly unscientific, with a +/- 10% fabricated margin of error.  No maths were performed to gather proper averages and statistics.  No effort was made to ensure consistency of the two Apache builds and no legitimate effort was made with regard to proper scientific rigor during this test.  This test does not take into account memory usage, responsiveness of the server under load, or any other relevant metric that would be of more use than this test.  I highly encourage you to do your own testing and draw your own conclusions.

How the tests were performed: 
I wanted to simulate a VPS environment similar to a basic Linode.  A base install of Ubuntu Server 10.04 was installed in VMWare with an older/slower 7200RPM sata drive for storage.  This drive was not in use by any other system during the test, nor were any other VMs active on the host.  The guest was given 512MB of RAM and 1 CPU core of the host's 8 cores.  The host's CPUs are dual Xeon X5365 @ 3Ghz.

Apache 2.2 was installed along with nginx 0.7.65, later Apache 2.4 was compiled on this same system.

Testing was performed using Apache JMeter on an 8 core Xeon workstation running Windows 7 and 32GB of RAM  with a 1Gb/s link to the VMWare host.  Requests per second were determined by rounding off the throughput displayed in the Summary Report listener.  Each test was run until the requests per second stabilized.

The Apache 2.2 server was only tested with the Prefork MPM, while the Apache 2.4 server was tested with both Prefork and Event.  Apache's KeepAlive setting was on throughout the testing and set at 2 seconds.

Update:
I received several requests to post memory usage statistics so I've updated the Jquery test with memory results.  Again, care was not taken to keep the Apache builds consistent.  It concerns me that the Prefork build of Apache 2.4 was using so much memory compared to the other Apache builds.  Take these results with a grain of salt, but trust that Nginx definitely uses significantly less memory than Apache.

Update 2 - Nginx 1.0.12:
I received some flak for using an older version of Nginx, so I tested with Nginx 1.0.12 and it was around 4% slower than the results shown here.

Test 1 - 21KB text file
HTTP Server Req/s
Apache 2.2 Prefork 2220
Apache 2.4 Prefork 2250
Apache 2.4 Event 2300
Nginx 2600


Test 2 - 2B text file consisting of a single period.
HTTP Server Req/s
Apache 2.2 Prefork 4400
Apache 2.4 Prefork 4700
Apache 2.4 Event 4810
Nginx 6650


Test 3 - jquery.min.js (92KB)

HTTP ServerReq/sMemory Usage
Apache 2.2 Prefork65012MB
Apache 2.4 Prefork77072MB
Apache 2.4 Event82020MB
Nginx10002MB


Test 4 - PHP output of phpinfo()
HTTP Server Req/s
Apache 2.2 Prefork 525
Apache 2.4 Prefork 575
Nginx FastCGI 450


8 comments:

MassiveScale.net said...

On 1 core it makes no sense. Run the tests on many CPUs, because this is what Apache's supposed to be good at: scalability as opposed to simple efficiency.

MTecknology said...

I find it rather interesting how so many people have run these benchmarks, come up with basically the same results, explained in detail ranging from none to extremely detailed, and continue to get picked on for some asinine reason.

Every benchmark that I've seen or run against the latest Apache and Nginx shows rather clearly that Nginx is still winning in terms of performance for both large and small systems.

In this particular test, you covered most all of your bases and yet you still get someone saying something to discredit all of your work.
(zomg, apache lost? well u didnt done it rite lol! o u did? well dat cuz ur puter wrong puter, get better buter)

Really?

This is definitely one of the better benchmark blog posts. I'll have to discredit your PHP test purely based on lack of information about how any of it was configured. Any chance you could try with php-fpm and a static number of children and set it to 10 (a common number for a linode vps)? I don't know if you can really mess with the way php works behind apache2.4. Something like that would be very nice to see.

Anyway... thanks for the coherent benchmark. Aside from PHP, I'd say you covered your backing for the benchmarks very well. I also like how you clarified what your use case in the benchmark was for.

Spookerton said...

"this is what Apache's supposed to be good at: scalability as opposed to simple efficiency"

No. Apache's scalability is so incredibly poor it's laughable when compared with lighthttpd, nginx, and even IIS.

Apache's strength is its market share. Everything was made to work with it, so it's highly compatible. That's all it has going for it.

Unknown said...

Great stuff, Andy.

If would be perfect if you could include Nginx 1.2 benchmarks, though.

Steve Ramage said...

I look at the comments and it appears that people are thinking hat nginx is ahead in the performance department in this benchmark but this looks to me to be the opposite for what I am looking for in performance... here's why:

In the Environments I am working in PHP processing is much more of a bottle neck than static file serving especially if 75% of the static files (js,css,pngs,gifs) are already cached by the browser.

The Concurrency for the environments I am working in is between 50 & 200 requests per second so not too high, but there is a lot of php processing going on in the system.

Mass Media file serving appears faster in nginx, there are not many dynamic images, no video or other files in my environments.

In the environments I am evaluating to upgrade from apache 2.2 to either apache 2.4 or nginx, I don't think nginx will be an improvement based on these benchmarks.


Arguments can be made for tweaking either nginx or apache but if the compiled options for php are similar between apache and nginx then apache is ahead for the purposes I need it for.

I would like to see a benchmark between (nginx + php-fpm) vs (apache + statically compiled php) as I would assume the statically compiled apache php is supposedly the fastest.

So far I haven't seen a good case for switching to nginx for my purposes yet as it doesn't look like performance will improve at all, so I am leaning towards apache 2.4.

In the past I have tried lighttpd and for serving media files and it works well but there was no noticeable performance increase over apache 2.2 when using php.

Has anybody seen any other good php benchmarks comparing nginx 1.2 and apache 2.4?

Francesco Corriga said...

Obviously Apache/mod_php is faster than Nginx+php-fpm... if you just look at pure php performance. However websites are not just php. If you look at a more realistic situations, you'll see nginx outperforms apache in any application, be it an ecommerce or a CMS or whatever.
I've choosen nginx over apache when setting up my webserver coz i made my own benchmarks with both and nginx was really much faster even if my application depends heavily on PHP. It allowed me to halve the budget for my production webserver (cloud based) at the cost of spending few hours RTFMing coz i know nothing about nginx.
Thanks to dotdeb.org i now run the latest version of nginx, php and mysql on my debian server and i'm completely satisfied by the performance.

yh said...

could you update the benchmark with apache 2.2 + fastCGI + php-fpm?

Ontananza said...

At this scenario, the right way could be.

Use nginx in the front as proxy, and serving all static content (css, images, js) and leaving the php process to Apache.