WARNING: Preliminary data!

You are about to view preliminary data for the Framework Benchmarks project. This preview is being provided to project participants in order to sanity-check results before an official run. Errors, anomalies, and other peculiarities should be expected.

For the most recent official round, please navigate to the main results site. Otherwise, click anywhere else to proceed.

Web Framework Benchmarks

Show filters panel
Showing all frameworks.
Filters
Classification
?
We classify frameworks as follows:
  • Full-stack, meaning a framework that provides wide feature coverage including server-side templates, database connectivity, form processing, and so on.
  • Micro, meaning a framework that provides request routing and some simple plumbing.
  • Platform, meaning a raw server (not actually a framework at all). Good luck! You're going to need it.
Disable all
Language
?
The principal programming language used by the framework.
Disable all
Platform
?
The platform is the low-level software or API used to host web applications for the framework; the platform provides an implementation of the HTTP fundamentals.
Disable all
Application operating system
?
The operating system hosting the application server and web server, if applicable.
Disable all
Front-end server
?
The front-end server ("web server") paired with the application framework, if applicable. Some frameworks provide a built-in web server.
Disable all
Database-server
?
The database server paired with the application framework. Not every framework has tests for every database server.
Disable all
Database operating system
?
The operating system hosting the database server.
Disable all
Object-relational mapper (ORM) classification
?
We classify object-relational mappers as follows:
  • Full, meaning an ORM that provides wide functionality, possibly including a query language.
  • Micro, meaning a less comprehensive abstraction of the relational model.
  • Raw, meaning no ORM is used at all; the platform's raw database connectivity is used.
Disable all
Implementation approach
?
Implementation approach describes the test's design disposition.
  • A realistic implementation approach uses the framework with most out-of-the-box functionality enabled. We consider this realistic because most applications built with the framework will leave these features enabled.
  • A stripped implementation approach removes or outright avoids implementing features that are unnecessary for the particulars of this benchmark exercise. This might illuminate the marginal improvement available in fine-tuning a framework to your application's use-case.
Stripped implementations are hidden by default. See the Motivation and Question section for more information.
Disable all
Framework
?
In addition to filtering frameworks using the attributes above, you may hide frameworks one-by-one by clicking their names below.
Disable all
Key
Close filters panel
Enabled
Disabled
Unavailable
Test types
Preliminary

—

WARNING: Preliminary data!
Environment: Citrine Date: 2018-05-20
Round 16 Preview 1

The preliminary results seen here are provided to project participants for review and sanity-checking. This preview is a duplicate of continuous run ae097e87-3b23-402c-8f51-1608e16dd11a and is posted as a preview to reach a potentially wider audience of project participants. Previews such as this are an opportunity to discover and correct defects before the official round is closed. However, after this brief window of time, the round will be closed and subsequent fixes will need to be deferred until the next round.

We encourage project participants to monitor the Continuous Benchmarking results dashboard for more immediate and frequent updates. The Continuous Benchmarking dashboard fulfills the same role as Previews, and we plan to cease posting Previews after Round 16.

If you are not a participant in this project, we strongly advise that you do not make decisions based on what you see here. For the latest official results, visit the main results site.

Resources:

Logs Diff: R15F vs R16p1

Requirements summary

In this test, each request is processed by fetching a single row from a simple database table. That row is then serialized as a JSON response.

Example response:

HTTP/1.1 200 OK Content-Length: 32 Content-Type: application/json Server: Example Date: Wed, 17 Apr 2013 12:00:00 GMT {"id":3217,"randomNumber":2149}

For a more detailed description of the requirements, see the Source Code and Requirements section.

Requirements summary

In this test, each request is processed by fetching multiple rows from a simple database table and serializing these rows as a JSON response. The test is run multiple times: testing 1, 5, 10, 15, and 20 queries per request. All tests are run at 256 concurrency.

Example response for 10 queries:

HTTP/1.1 200 OK Content-Length: 315 Content-Type: application/json Server: Example Date: Wed, 17 Apr 2013 12:00:00 GMT [{"id":4174,"randomNumber":331},{"id":51,"randomNumber":6544},{"id":4462,"randomNumber":952},{"id":2221,"randomNumber":532},{"id":9276,"randomNumber":3097},{"id":3056,"randomNumber":7293},{"id":6964,"randomNumber":620},{"id":675,"randomNumber":6601},{"id":8414,"randomNumber":6569},{"id":2753,"randomNumber":4065}]

For a more detailed description of the requirements, see the Source Code and Requirements section.

Requirements summary

This test exercises database writes. Each request is processed by fetching multiple rows from a simple database table, converting the rows to in-memory objects, modifying one attribute of each object in memory, updating each associated row in the database individually, and then serializing the list of objects as a JSON response. The test is run multiple times: testing 1, 5, 10, 15, and 20 updates per request. Note that the number of statements per request is twice the number of updates since each update is paired with one query to fetch the object. All tests are run at 256 concurrency.

The response is analogous to the multiple-query test. Example response for 10 updates:

HTTP/1.1 200 OK Content-Length: 315 Content-Type: application/json Server: Example Date: Wed, 17 Apr 2013 12:00:00 GMT [{"id":4174,"randomNumber":331},{"id":51,"randomNumber":6544},{"id":4462,"randomNumber":952},{"id":2221,"randomNumber":532},{"id":9276,"randomNumber":3097},{"id":3056,"randomNumber":7293},{"id":6964,"randomNumber":620},{"id":675,"randomNumber":6601},{"id":8414,"randomNumber":6569},{"id":2753,"randomNumber":4065}]

For a more detailed description of the requirements, see the Source Code and Requirements section.

Requirements summary

In this test, the framework's ORM is used to fetch all rows from a database table containing an unknown number of Unix fortune cookie messages (the table has 12 rows, but the code cannot have foreknowledge of the table's size). An additional fortune cookie message is inserted into the list at runtime and then the list is sorted by the message text. Finally, the list is delivered to the client using a server-side HTML template. The message text must be considered untrusted and properly escaped and the UTF-8 fortune messages must be rendered properly.

Whitespace is optional and may comply with the framework's best practices.

Example response:

HTTP/1.1 200 OK Content-Length: 1196 Content-Type: text/html; charset=UTF-8 Server: Example Date: Wed, 17 Apr 2013 12:00:00 GMT <!DOCTYPE html><html><head><title>Fortunes</title></head><body><table><tr><th>id</th><th>message</th></tr><tr><td>11</td><td>&lt;script&gt;alert(&quot;This should not be displayed in a browser alert box.&quot;);&lt;/script&gt;</td></tr><tr><td>4</td><td>A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1</td></tr><tr><td>5</td><td>A computer program does what you tell it to do, not what you want it to do.</td></tr><tr><td>2</td><td>A computer scientist is someone who fixes things that aren&apos;t broken.</td></tr><tr><td>8</td><td>A list is only as strong as its weakest link. — Donald Knuth</td></tr><tr><td>0</td><td>Additional fortune added at request time.</td></tr><tr><td>3</td><td>After enough decimal places, nobody gives a damn.</td></tr><tr><td>7</td><td>Any program that runs right is obsolete.</td></tr><tr><td>10</td><td>Computers make very fast, very accurate mistakes.</td></tr><tr><td>6</td><td>Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen</td></tr><tr><td>9</td><td>Feature: A bug with seniority.</td></tr><tr><td>1</td><td>fortune: No such file or directory</td></tr><tr><td>12</td><td>フレームワークのベンチマーク</td></tr></table></body></html>

For a more detailed description of the requirements, see the Source Code and Requirements section.

Requirements summary

In this test, each response is a JSON serialization of a freshly-instantiated object that maps the key message to the value Hello, World!

Example response:

HTTP/1.1 200 OK Content-Type: application/json Content-Length: 28 Server: Example Date: Wed, 17 Apr 2013 12:00:00 GMT {"message":"Hello, World!"}

For a more detailed description of the requirements, see the Source Code and Requirements section.

Requirements summary

In this test, the framework responds with the simplest of responses: a "Hello, World" message rendered as plain text. The size of the response is kept small so that gigabit Ethernet is not the limiting factor for all implementations. HTTP pipelining is enabled and higher client-side concurrency levels are used for this test (see the "Data table" view).

Example response:

HTTP/1.1 200 OK Content-Length: 15 Content-Type: text/plain; charset=UTF-8 Server: Example Date: Wed, 17 Apr 2013 12:00:00 GMT Hello, World!

For a more detailed description of the requirements, see the Source Code and Requirements section.

Comments

If you have any comments about this round, please post at the Framework Benchmarks Google Group.

Results testing

Running these benchmarks in your own test environment? You can visualize the results by copying and pasting the contents of your results.json file in the text box below.

Test duration: seconds
Visualize results