July marks the fourth month of our ongoing project measuring the performance of web application frameworks and platforms. We've just posted Round 6, which includes several more developer community-provided framework test implementations: Beego, Dart, Hapi, Jester, Luminus, Nancy, Yaf, Plack, Play-Slick, and Undertow.

View Round 6 results

The results web site has been improved with test-type and hardware-type navigation, allowing you to share links to a specific results chart, such as Round 6, Fortunes on EC2.

By popular demand, Round 6 introduces a plaintext test that uses HTTP pipelining, implemented in 14 frameworks so far. Previously, the most fundamental test was JSON serialization of a very small object. The new plaintext test demonstrates the extremely high request routing throughput possible on very high-performance platforms.

View Round 6 results now.

Round 6 notes and observations

  • HTTP pipelining is the star of the new plaintext test. With pipelining enabled, the capacity of gigabit Ethernet is much more efficiently used as compared to plain HTTP Keep-alives. The wrk benchmark tool reports a peak of approximately 95 megabytes per second transfer rate with pipelining and 35 megabytes per second without.
  • Tiny variations in response payload can appear as significant variations in requests per second in the new pipelining plaintext test. We caution all readers to review the source code of each test when interpreting the plaintext numbers. We will work to further normalize the implementations' response headers for Round 7.
  • The final plaintext data was captured from wrk configured to send 16 requests per pipeline. In preliminary tests, we experimented with more requests per pipeline but doing so caused socket write buffer overflows. The wrk tool was later enhanced to allow larger write buffers, but we ultimately decided to retain a 16 requests per pipeline test configuration.
  • All of the other tests (JSON, Fortunes, database read and write tests) are still run without HTTP pipelining. However, we spot-tested the JSON serialization test with pipelining enabled. Performance was essentially identical to the plaintext tests, confirming earlier suspicions that the impact of small JSON serialization workloads on high-performance platforms is trivial compared to the HTTP network traffic. (Incidentally, a later test will exercise larger JSON workloads.)
  • Also in response to popular demand, we ran the plaintext test with higher client-side concurrency levels than other tests. Where other tests are run at levels up to 256, the plaintext test runs at 256, 1024, 4096, and 16384 concurrency. However, most servers were already CPU or network limited at 256 concurrency, so it is not surprising that the results are fairly even across these higher concurrency levels.
  • The results web site now includes a "framework overhead" chart inspired by comments from Hacker News user goodwink. This new chart type compares frameworks versus their underlying platform (e.g., Unfiltered versus Netty; Spring versus Servlet; Rails versus Rack). The theoretical ideal is 100%, meaning the framework imparts no performance penalty compared to its platform. However, in some cases due to custom core components in a framework or implementation particulars, a framework may exceed 100%. For example, in the multiple query test, Revel currently considerably exceeds Go. This seems implausible on the surface, but it's repeatable. We look forward to hearing from Revel and Go experts with explanations and pull requests to apply for Round 7.
  • If you've enjoyed this project so far, we'd like your opinion on what test to include next. Take a look at the list of test ideas we have collected to date.

Thanks!

Huge thanks to Will Glozer who created a special branch of his Wrk benchmarking tool with HTTP pipeline support for use in this project. Not only is Wrk an excellent tool; @wg is an awesome developer whose contributions we greatly appreciate!

In addition to test implementations in new frameworks, the community continued to improve previous tests. In particular, a great deal of effort was provided in reviewing the ASP.NET MVC tests. Special thanks to MalcolmEvershed and kppullin for their contributions here.

Thanks to several others for many contributions (in no particular order): robfig (Revel and Go), pdonald (additional Windows and ASP.NET work), jrudolph (Spray), sirthias (Spray), oberhamsi (Ringo), methane (Python), nraychaudhuri (Play), dom96 (Jester), xaxaxa (CPPSP), JulienSchmidt (Go), moxford (Erlang), gjuric (Symfony2), stuartwdouglas (Undertow), avaly (Express and Hapi), yogthos (Luminus), vividsnow (Perl), amarsahinovic (Beego), LekisS (Kohana), cmircea (ServiceStack), bradfitz (Go), fruit (YAF).

If you have questions, comments, criticism, or would like to contribute a new test or an improvement to an existing one, please join our Google Group or visit the project at Github.

About TechEmpower

We provide web and mobile application development services and are passionate about application performance. Read more about what we do.