The latest round of our ongoing Framework Benchmarks project is now available! Round 9 updates several test implementations and introduces a new test environment using modern server hardware.
Since the first round, we have known that the highest-performance frameworks and platforms have been network-limited by our gigabit Ethernet. For Round 9, Peak Hosting has provided a high-performance environment equipped with 10-gigabit Ethernet at one of their data centers.
With ample network bandwidth, the highest-performing frameworks are further differentiated. Additionally, the Peak test environment uses servers with 40 HT cores across two Xeon E5 processors, and the results illustrate how well each framework scales to high parallelism on a single server node.
View Round 9 results now.
Round 9 notes and observations
In previous rounds, JSON serialization results for the highest-performance frameworks on our in-house i7 hardware converged on
approximately 200,000 requests per second. Although some response headers are normalized, we believe that the small variations beyond 200,000 RPS are caused by how each framework uses the available bandwidth of our gigabit Ethernet.
For example, a larger number of response headers might mean slightly fewer responses sent per second. However, this is acceptable as we want each implementation to be production-grade and typical for the framework, not tuned to include only the response headers we require. Thorough normalization of behavior is not a goal. (More importantly, we recommend against making decisions based on such small variations. 200,000 versus 20,000 is notable; 210,000 versus 200,000 is probably not.)
Top ten frameworks at JSON serialization; i7 on left, Peak on right
With 10-gigabit Ethernet, the results from the new Peak environment show a larger spread between top-performing frameworks. Of course, other variables are at play, such as significantly more HT processor cores--40 versus 8 for our i7 tests--and a different system architecture, NUMA.
The NUMA architecture presented a variety of challenges including unexpected difficulty scaling the database servers to utilize all
available CPU cores. For example, we are using MySQL 5.5 in our database tests, but later versions of MySQL are reportedly better suited for NUMA. For this reason, we may migrate all MySQL tests to a more recent version in a future round. Having reviewed the results, we plan to use differently-equipped machines for Round 10. The hardware specifications used in Round 9 were not well-optimized for the web-app use case our project exercises.
As a result of the concessions made for NUMA, the performance delta for database tests between our i7 workstations and the the 40-core Xeon servers is not as pronounced as the non-database tests. We'd like to see this situation improve in future rounds. We would be happy to receive input and advice from readers with NUMA database deployment expertise.
Similarly, we expect that some application platforms are better suited for NUMA than others. For example, platforms that use process-level concurrency scaled to a large number of processor cores quite well. For JSON serialization, node.js performance in the Peak environment is about 3.3x that of i7. Meanwhile, Go's JSON serialization performance on Peak is only 1.6x that of i7. Even more interesting: Go's database performance is slightly lower on Peak than i7. (Yes, the Go tests use all CPU cores. Putting aside scaling as CPU cores increase, Go is quicker in the JSON serialization test than node.js in both environments.)
C++ continues to dominate virtually all tests. If you are a web developer using C++, you win this particular bragging-rights game. With 6,738,911 plaintext RPS from this single server, the remarkable 188,585 Fortunes RPS might be overlooked at first although it may be more impressive on consideration.
For Round 9, we added validation checks that confirm each implementation is behaving as we expect prior to measuring its performance. These checks are rigid--failing implementations for improper JSON response schema, for example--but ultimately a valuable tool to ensure fairness. We've fixed up many of the implementations to pass validation, but several remain to be fixed. If your favorite framework is showing up as "did not complete" at the bottom of the charts, we'd appreciate your help in correcting that for the next round. Visit the GitHub repository if you can lend a hand.
Busy schedules delayed Round 9. We apologize for the delay and thank you for your continued interest and patience! We welcome you to join in for Round 10.
As always, thank you to all of the contributes to this project, especially those who helped us address validation errors for this latest round.
An extra special thanks to Peak Hosting for providing a no-joking-around, no-holds-barred, genuine server environment. It has been a treat to see these servers set massive new records.
If you have questions, comments, criticism, or would like to contribute a new test or an improvement to an existing one, please join our Google Group or visit the project at Github.
We provide web and mobile application development services and are passionate about application performance. We are presently looking for a generalist web/mobile developer to expand our team. If this sounds interesting to you, we'd love to hear from you.