Happy Halloween fans of web development frameworks! After a several-month hiatus, Round 7 of our project measuring the performance of web application frameworks and platforms is available!
Round 7 includes many new framework test implementations contributed by the community. They are Falcore, Grizzly, HttpListener, PHPixie, Plain, Racket-WS, Start, Stream, and Treefrog. There are now a whopping 84 frameworks and over 200 individual test permutations.
Many preexisting frameworks' tests have been updated to include more test coverage and/or update dependencies and tune their implementation. To date, the project has processed 344 pull requests from the community. Thanks so much for your contributions. We are grateful for your continued interest!
Round 7 notes and observations
- The Round 6 champion Undertow (the web server for WildFly) continues to impress with chart-dominating showings such as 180,000 plaintext requests per second on meager m1.large instances.
- Thanks to community contributions, the C# tests have been dramatically improved, especially when querying the database. We also have some SQL Server tests in our i7 environment.
- A contributor prepared scripts for running the benchmark suite on Windows Azure. Unfortunately, we were unable to reach the author of these scripts in the past weeks. If any Azure experts are interested in picking up that work where it exists now, please visit the GitHub repository or the Google Group for the project.
- The high-performance tier has become significantly more crowded even during this project's relatively short history. Most interesting to us is how many frameworks can easily saturate our gigabit Ethernet with the JSON serialization and plaintext tests, even with our tests' intentionally small payloads. We do not have the hardware necessary to run 10 gigabit Ethernet tests, but if you have a 10 GBE lab and are willing to run the suite, we'd love to publish the results.
- The benchmark toolset continues to mature gradually, but a lot of room for improvement still exists. A great deal of sanity-checking remains a manual process. If you're a Python programmer and interested in this project, let us know. We have several enhancements we'd like to make to the benchmark tool set (Python scripts), time permitting.
- This round used a community-review model wherein project participants were able to review preliminary results we were capturing in our i7 environment and submit pull requests. The model is not perfect and will need to improve with each round, but it will help reduce the amount of time we (TechEmpower) need to allocate to each round's sanity checks, meaning quicker turn-around of rounds (see how I spun that as a good thing?).
- Starting now, we aim to be on a monthly cycle of running official rounds. This helps reduce the perceived severity of configuration problems since they can be addressed in the next run, which is only a month away.
- We've also pushed the display name for tests into the project, allowing contributors to assign test permutations any name they choose. E.g., "play-scala-anorm" and "aspnet-mvc-mono."
- One particularly interesting anomaly is the dominance of Windows paired with Mongo on EC2 in the Updates test. The performance is only slightly lower than the same pairing on i7, where in most cases our i7s (2600K workstations, to be precise) and EC2 (m1.large) instances differ by a factor of seven or more. It's possible the Windows EC2 instance is running on a newer host than the Linux EC2 instance, but both are classified as m1.large.
- Speaking of database tests, in previous rounds, we had used an SSD to host the databases. Prior to finishing Round 7, that SSD failed, so Round 7 is run with ramdisk-backed databases (excluding SQL Server). This project is not a database benchmark so we believed it would be fascinating to see the performance of the full stack when the friction of the database writes is reduced to a bare minimum. As confirmed by our previous spot checking in Round 5, database writes are about 20% to 30% faster across the board when using a ramdisk versus the Samsung 840 Pro SSD we had been using. As expected, reads are unaffected since the tests are designed to allow the database engine to fit the entire data set into memory.
As always, we'd like to say thank you to all of the contributors who have added test implementations for new frameworks or improved existing implementations. Round 7 was unusually long, so we also thank everyone for their patience.
The contributors for Round 7 are numerous. In no particular order: @fernandoacorreia, @kppullin, @MalcolmEvershed, @methane, @KevinHoward, @huntc, @lucassp, @dracony, @weltermann17, @kekekeks, @fwbrasil, @treefrogframework, @yogthos, @oberhamsi, @purplefox, @yz0075, @necaris, @pdonald, @Kepinator, @DavidBadura, @zznate, @nightlyone, @jeapostrophe, @astaxie, @troytoman, @grob, @torhve, @trautonen, @stuartwdouglas, and @xaxaxa. Sincere apologies if we forgot anyone!
We provide web and mobile application development services and are passionate about application performance. Read more about what we do.