We have posted Round 5 of our ongoing project measuring the performance of web application frameworks and platforms. In this round, we're very happy to announce that a community member has contributed tests for ASP.NET running on native Windows.
Additionally, we have added a new test type focused on writes, wherein a variable number of database updates are executed per request. This test thoroughly punishes the database server to a degree not seen in any of our previous tests. The test is uniquely I/O limited where previous tests have been predominantly CPU limited, and in some circumstances, network limited.
Round 5 notes and observations
- The new Windows results, while respectable, do not match the Linux tests. We suspect there is room for improvement and we would be particularly interested in community recommendations for tuning the performance of the Windows environment. Note that the Windows configuration presently retains the Linux-hosted database server—only the web/application server is running Windows Server 2012. And yes, we would like to include Microsoft SQL Server in the future. Also, a future round will include Windows results on our dedicated hardware.
- We outfitted the early-2011 vintage i7 workstation we are using as a database server in our dedicated-hardware tests with a Samsung 840 Pro solid state disk in preparation for running our new database updates test. Previous rounds' i7 tests were run using a traditional magnetic hard drive, although the tests were read-focused and therefore the disk was scarcely involved in fulfilling requests. As a result, you'll notice the other database tests are essentially unchanged versus Round 4. Equipping the i7 workstation with a high-performance SSD has given the database updates test a large performance gulf between our physical hardware and EC2 m1.large.
- We ran a bunch of spot-tests of the database updates test with various configurations, some scarcely suitable for a production server.
- Prior to installing the SSD, the i7 hardware with traditional magnetic disks was only two to three times quicker than the EC2 instances.
- We experimented with hosting a MySQL database on a ramdisk to observe performance with I/O operations reduced to bare minimum. But surprisingly, the performance was only about 30% higher than the SSD performance.
- MySQL with MyISAM was substantially faster at writes than with InnoDB, but the official Round 5 results are using InnoDB.
- Note that our database updates test at "20 updates" is actually executing 40 total database statements: 20 queries to fetch random rows and 20 updates to persist the same rows after one small modification each.
- At the recommendation of the community, we have modified the HTTP request payload to include several commonplace headers such as a complex User-Agent and some Cookies (though none that would be considered session identifiers). One particularly interesting outcome of having done so: the plain Servlet implementation of the Fortunes test, which is implemented using JSP, suffers a significant performance penalty. We tentatively suspect this is related to optimistic parsing of the request's cookies.
- The larger HTTP request payload has normalized the peak JSON response rate on our i7 hardware somewhat. The JSON i7 leaderboard has shuffled slightly as a result. The Finagle test implementation in particular was extensively modified to use Finagle best-practices which are not oriented to eking out the highest performance at all costs, but rather on other measurements of code quality.
- Some frameworks reacted to the new request headers in unexpected ways. Kohana, for example, refused to process requests that provided Cookies it was not expecting to receive, responding with "A valid cookie salt is required. Please set Cookie::$salt." Configuration tweaks are likely required. Kohana has been temporarily hidden from view since all of its results measured 0.
- As with previous rounds, Round 5 adds some frameworks to the test suite: Spray, RestExpress, Web::Simple, Revel, and CPPSP. It also includes some updates such as Play 2.1.2-RC1, Go 1.1 final, and Python 2.7.4. If you're a maintainer of a framework we're testing and you've released a new version, please let us know so that we can get the tests updated.
- This round also includes Flask on PyPy, which is our first Python test running on PyPy.
- When you apply filters to the results view, the URL should change to capture your settings; you can share specific comparisons with friends and colleagues.
- Some other glitches affected tests in Round 5. The ASP.NET contribution included both native Windows and Linux/Mono tests, but unfortunately, the Mono tests are not yet working correctly. The Windows configuration includes the Go test, but a configuration problem prevents the database portions of the Go test from completing. On Windows, the Go database tests show 0 requests per second. The new RingoJS + Mongo test wasn't able to complete. We're optimistic these issues can be identified and resolved before the next round.
Thank you again to the developer community for the numerous contributions in this and previous rounds. The magnitude of the project is a testament to the development community's continued participation and assistance in broadening the tests, both in framework coverage and test types. We are especially delighted by the response and involvement we've observed from framework maintainers. If you are a framework maintainer, we'd love to hear from you if you have questions, recommendations, or complaints.
We provide web and mobile application development services and are passionate about application performance. Read more about what we do.