Blog

You are viewing entries tagged frameworks and may want to check out the most recent entries.

December 17, 2013

Framework Benchmarks Round 8

Merry Christmas web framework performance aficionados! What better way to celebrate the holidays than by cheering on your favorites as they race through a variety of application fundamentals in the biggest web platform grudge match of the season? We certainly can't think of anything more festive.

Now at 90 frameworks and 230 permutations (variations on configuration), Round 8 has something for everyone. And if it doesn't have what you want, you can join the party! We have fruitcake and egg nog. Or maybe not. But we enjoy pull requests; they're almost as good as egg nog.

A veritable rainbow of holiday cheer awaits!

View Round 8 results

View Round 8 results now.

Round 8 notes and observations

  • Go, always the scrappy competitor, flexes some performance muscle and lands a razor-thin victory in JSON serialization on i7 hardware. But be aware, the highest-performance frameworks are network limited in the JSON serialization and Plaintext tests. Anything in this "200k club" is sure to keep overhead at a bare minimum, leaving maximum headroom for your custom application logic.
  • Round 7 was missing some of the Go frameworks due to configuration problems. Those problems have been resolved, and the Go frameworks have returned in Round 8 to reaffirm that Go is a viable performance rival to the JVM.
  • The HipHop PHP VM with no framework and thanks in part to the MySQL driver for PHP, yields dominion over the Updates test. HHVM is impressive in the multiple query test as well. However, hhvm trails plain PHP in the Fortunes test presently. Implementation details may be at play here. If you're interested in testing HHVM with popular PHP frameworks, we would be happy to receive a pull request.
  • Vert.x and Netty have wrestled the Plaintext crown from Undertow, but this rivalry isn't yet settled. Rumor has it they have more improvements in store for Round 9. Meanwhile, a newcomer named Plain (which may rival Go as the most in need of a more search-friendly name; though the irony of Go makes it uncontested champion) is right behind the leaders. Most interestingly, Plain demonstrates the highest Windows performance we've seen by a massive margin (reaching 611,095 pipelined plaintext requests per second on i7). Once again, bear in mind that these tests are network-bound by our gigabit Ethernet.
  • On EC2, the Netty and Vert.x upgrades have paid huge dividends with Netty now breaking 200,000 pipelined plaintext responses per second on a humble m1.large instance.
  • Grizzly performance on JSON serialization is off from its Round 7 showing, but unfortunately, we have not yet determined the cause.
  • The Plaintext test requirements were clarified. It is not necessary to copy the bytes of the small response payload per request. Using a pre-rendered byte buffer for the body is acceptable as long as that is conventional for the platform or framework being tested and response headers are composed normally.
  • The maximum query and update performance for Mongo on EC2 is substantially higher than MySQL. When looking at that chart in particular, consider filtering by your preferred data store to maintain a useful perspective. Related: a late change to the Mongo "schema" intended to replace "id" with "_id" caused some challenges. We further postponed Round 8 to re-run Mongo tests with a schema that provides both columns to allow all tests to complete. We want to normalize the implementations for Round 9.
  • We were targeting early December for Round 8 and we're off by about two weeks. There is still room for improvement toward our goal of a monthly cycle. We will target mid-January for Round 9.

Thanks!

A big thank-you to all of the contributors who have added and improved existing test implementations for Round 8.

The contributors for Round 8 are, in no particular order: @lhotari, @methane, @pseudonom, @lucassp, @aualin, @weltermann17, @kpacha, @nareshv, @martin-g, @bclozel, @ijl, @bbrowning, @sbordet, @purplefox, @stuartwdouglas, @normanmaurer, @kardianos, @hamiltont, and @julienschmidt.

If you have questions, comments, criticism, or would like to contribute a new test or an improvement to an existing one, please join our Google Group or visit the project at Github.

About TechEmpower

We provide web and mobile application development services and are passionate about application performance. Read more about what we do.

October 31, 2013

Framework Benchmarks Round 7

Happy Halloween fans of web development frameworks! After a several-month hiatus, Round 7 of our project measuring the performance of web application frameworks and platforms is available!

View Round 7 results

Round 7 includes many new framework test implementations contributed by the community. They are Falcore, Grizzly, HttpListener, PHPixie, Plain, Racket-WS, Start, Stream, and Treefrog. There are now a whopping 84 frameworks and over 200 individual test permutations.

Many preexisting frameworks' tests have been updated to include more test coverage and/or update dependencies and tune their implementation. To date, the project has processed 344 pull requests from the community. Thanks so much for your contributions. We are grateful for your continued interest!

View Round 7 results now.

Round 7 notes and observations

  • The Round 6 champion Undertow (the web server for WildFly) continues to impress with chart-dominating showings such as 180,000 plaintext requests per second on meager m1.large instances.
  • Thanks to community contributions, the C# tests have been dramatically improved, especially when querying the database. We also have some SQL Server tests in our i7 environment.
  • A contributor prepared scripts for running the benchmark suite on Windows Azure. Unfortunately, we were unable to reach the author of these scripts in the past weeks. If any Azure experts are interested in picking up that work where it exists now, please visit the GitHub repository or the Google Group for the project.
  • The high-performance tier has become significantly more crowded even during this project's relatively short history. Most interesting to us is how many frameworks can easily saturate our gigabit Ethernet with the JSON serialization and plaintext tests, even with our tests' intentionally small payloads. We do not have the hardware necessary to run 10 gigabit Ethernet tests, but if you have a 10 GBE lab and are willing to run the suite, we'd love to publish the results.
  • The benchmark toolset continues to mature gradually, but a lot of room for improvement still exists. A great deal of sanity-checking remains a manual process. If you're a Python programmer and interested in this project, let us know. We have several enhancements we'd like to make to the benchmark tool set (Python scripts), time permitting.
  • This round used a community-review model wherein project participants were able to review preliminary results we were capturing in our i7 environment and submit pull requests. The model is not perfect and will need to improve with each round, but it will help reduce the amount of time we (TechEmpower) need to allocate to each round's sanity checks, meaning quicker turn-around of rounds (see how I spun that as a good thing?).
  • Starting now, we aim to be on a monthly cycle of running official rounds. This helps reduce the perceived severity of configuration problems since they can be addressed in the next run, which is only a month away.
  • We've also pushed the display name for tests into the project, allowing contributors to assign test permutations any name they choose. E.g., "play-scala-anorm" and "aspnet-mvc-mono."
  • One particularly interesting anomaly is the dominance of Windows paired with Mongo on EC2 in the Updates test. The performance is only slightly lower than the same pairing on i7, where in most cases our i7s (2600K workstations, to be precise) and EC2 (m1.large) instances differ by a factor of seven or more. It's possible the Windows EC2 instance is running on a newer host than the Linux EC2 instance, but both are classified as m1.large.
  • Speaking of database tests, in previous rounds, we had used an SSD to host the databases. Prior to finishing Round 7, that SSD failed, so Round 7 is run with ramdisk-backed databases (excluding SQL Server). This project is not a database benchmark so we believed it would be fascinating to see the performance of the full stack when the friction of the database writes is reduced to a bare minimum. As confirmed by our previous spot checking in Round 5, database writes are about 20% to 30% faster across the board when using a ramdisk versus the Samsung 840 Pro SSD we had been using. As expected, reads are unaffected since the tests are designed to allow the database engine to fit the entire data set into memory.

Thanks!

As always, we'd like to say thank you to all of the contributors who have added test implementations for new frameworks or improved existing implementations. Round 7 was unusually long, so we also thank everyone for their patience.

The contributors for Round 7 are numerous. In no particular order: @fernandoacorreia, @kppullin, @MalcolmEvershed, @methane, @KevinHoward, @huntc, @lucassp, @dracony, @weltermann17, @kekekeks, @fwbrasil, @treefrogframework, @yogthos, @oberhamsi, @purplefox, @yz0075, @necaris, @pdonald, @Kepinator, @DavidBadura, @zznate, @nightlyone, @jeapostrophe, @astaxie, @troytoman, @grob, @torhve, @trautonen, @stuartwdouglas, and @xaxaxa. Sincere apologies if we forgot anyone!

If you have questions, comments, criticism, or would like to contribute a new test or an improvement to an existing one, please join our Google Group or visit the project at Github.

About TechEmpower

We provide web and mobile application development services and are passionate about application performance. Read more about what we do.

July 2, 2013

Frameworks Round 6

July marks the fourth month of our ongoing project measuring the performance of web application frameworks and platforms. We've just posted Round 6, which includes several more developer community-provided framework test implementations: Beego, Dart, Hapi, Jester, Luminus, Nancy, Yaf, Plack, Play-Slick, and Undertow.

View Round 6 results

The results web site has been improved with test-type and hardware-type navigation, allowing you to share links to a specific results chart, such as Round 6, Fortunes on EC2.

By popular demand, Round 6 introduces a plaintext test that uses HTTP pipelining, implemented in 14 frameworks so far. Previously, the most fundamental test was JSON serialization of a very small object. The new plaintext test demonstrates the extremely high request routing throughput possible on very high-performance platforms.

View Round 6 results now.

Round 6 notes and observations

  • HTTP pipelining is the star of the new plaintext test. With pipelining enabled, the capacity of gigabit Ethernet is much more efficiently used as compared to plain HTTP Keep-alives. The wrk benchmark tool reports a peak of approximately 95 megabytes per second transfer rate with pipelining and 35 megabytes per second without.
  • Tiny variations in response payload can appear as significant variations in requests per second in the new pipelining plaintext test. We caution all readers to review the source code of each test when interpreting the plaintext numbers. We will work to further normalize the implementations' response headers for Round 7.
  • The final plaintext data was captured from wrk configured to send 16 requests per pipeline. In preliminary tests, we experimented with more requests per pipeline but doing so caused socket write buffer overflows. The wrk tool was later enhanced to allow larger write buffers, but we ultimately decided to retain a 16 requests per pipeline test configuration.
  • All of the other tests (JSON, Fortunes, database read and write tests) are still run without HTTP pipelining. However, we spot-tested the JSON serialization test with pipelining enabled. Performance was essentially identical to the plaintext tests, confirming earlier suspicions that the impact of small JSON serialization workloads on high-performance platforms is trivial compared to the HTTP network traffic. (Incidentally, a later test will exercise larger JSON workloads.)
  • Also in response to popular demand, we ran the plaintext test with higher client-side concurrency levels than other tests. Where other tests are run at levels up to 256, the plaintext test runs at 256, 1024, 4096, and 16384 concurrency. However, most servers were already CPU or network limited at 256 concurrency, so it is not surprising that the results are fairly even across these higher concurrency levels.
  • The results web site now includes a "framework overhead" chart inspired by comments from Hacker News user goodwink. This new chart type compares frameworks versus their underlying platform (e.g., Unfiltered versus Netty; Spring versus Servlet; Rails versus Rack). The theoretical ideal is 100%, meaning the framework imparts no performance penalty compared to its platform. However, in some cases due to custom core components in a framework or implementation particulars, a framework may exceed 100%. For example, in the multiple query test, Revel currently considerably exceeds Go. This seems implausible on the surface, but it's repeatable. We look forward to hearing from Revel and Go experts with explanations and pull requests to apply for Round 7.
  • If you've enjoyed this project so far, we'd like your opinion on what test to include next. Take a look at the list of test ideas we have collected to date.

Thanks!

Huge thanks to Will Glozer who created a special branch of his Wrk benchmarking tool with HTTP pipeline support for use in this project. Not only is Wrk an excellent tool; @wg is an awesome developer whose contributions we greatly appreciate!

In addition to test implementations in new frameworks, the community continued to improve previous tests. In particular, a great deal of effort was provided in reviewing the ASP.NET MVC tests. Special thanks to MalcolmEvershed and kppullin for their contributions here.

Thanks to several others for many contributions (in no particular order): robfig (Revel and Go), pdonald (additional Windows and ASP.NET work), jrudolph (Spray), sirthias (Spray), oberhamsi (Ringo), methane (Python), nraychaudhuri (Play), dom96 (Jester), xaxaxa (CPPSP), JulienSchmidt (Go), moxford (Erlang), gjuric (Symfony2), stuartwdouglas (Undertow), avaly (Express and Hapi), yogthos (Luminus), vividsnow (Perl), amarsahinovic (Beego), LekisS (Kohana), cmircea (ServiceStack), bradfitz (Go), fruit (YAF).

If you have questions, comments, criticism, or would like to contribute a new test or an improvement to an existing one, please join our Google Group or visit the project at Github.

About TechEmpower

We provide web and mobile application development services and are passionate about application performance. Read more about what we do.