Framework Benchmarks Round 22

November 15, 2023

Nate Brady

We’re pleased to announce Round 22 of the TechEmpower Framework Benchmarks!

The TechEmpower Framework Benchmarks project celebrates its 10th anniversary, boasting significant engagement with over 7,000 stars on GitHub and more than 7,100 Pull Requests. Renowned as one of the leading projects of its kind, it benchmarks the peak performance of server-side web application frameworks and platforms, primarily using tests contributed by the community. Numerous individuals and organizations leverage the insights from The TechEmpower Framework Benchmarks to enhance their framework’s performance.

Microsoft has been steadfast in their dedication to improving the performance of their .NET framework, and has been active in the Framework Benchmark community to further this goal. With the announcement of the release of .NET 8, it is clear that performance is paramount.

Here are some updates from our contributors

@franz1981 on GitHub, @forked_franz on Twitter:

Right after Round 21 I’ve worked on the 3 projects delivering:

All these changes has improved the performance of the mentioned frameworks from 40% to 200% depending on the test

Oliver Trosien says:

I would like to use that opportunity to highlight Scala’s “new kid on the block”, Pekko, which is a fork of Akka, and currently undergoing incubation as Apache project. One of the reasons for contributing it to the Framework Benchmarks, was to verify no obvious performance regressions were introduced in the process of forking, and the results look good! Pekko is very much en-par with its legacy counterpart.


List a few of Rust’s performance optimizations.

In a real production environment, several approaches can be tried to optimize the application:

  1. Specify memory allocators
  2. Declaring static variables
  3. Putting a small portion of data on the stack
  4. Using a capacity to new vector or hash, a least capacity elements without reallocating
  5. SIMD

@fakeshadow says in response:

In general you should not take anything from tfb benchmark and simply consider it useful in real world. Context and use case determines how you optimize your code.

btw: xitca-web (bench code not including dependencies) does not do 2,3,5 and still remains competitive in micro bench can be used as a reference.


I have written a blog post about TFB and object pascal – yes, we added our object pascal framework in round 22! About how we maximize our results for TFB, we tried several ways of accessing the DB (ORM, blocking, async), reduced the syscalls as much as possible, minimized multi-thread locks especially during memory access (a /plaintext request requires no memory allocation), and made a lot of profiling and micro-optimizations. The benefit of the object pascal language is obvious: it is at the same time a high-level language (with interfaces, classes and safe ARC/COW strings), safe and readable, but also a system-level language (with raw pointers and direct memory buffers access). So the user can write its business code with high level of abstraction and safety, but the framework core could be tuned to the assembly level, to leverage the hardware it runs on. Finally, OpenSource helped a lot having realistic feedback from production, even if the project and the associated FreePascal compiler are maintained by a small number of developers, and object pascal as a language is often underestimated.


A heartfelt thank you to all our contributors and fans! We recognize the complexities involved in executing a benchmarking project accurately. While it’s challenging to meet everyone’s expectations, we are committed to continual improvement and innovation, made possible with your invaluable support and collaboration.

Round 22 is composed of:

Run ID: 66d86090-b6d0-46b3-9752-5aa4913b2e33 on our Citrine environment.

Follow us on Twitter for regular updates.


Want to learn how TechEmpower can help make your web application faster?

Framework Benchmarks Round 20

February 8, 2021

Nate Brady


Today we announce the results of the twentieth official round of the TechEmpower Framework Benchmarks project.

Now in its eighth year, this project measures the high-water mark performance of server side web application frameworks and platforms using predominantly community-contributed test implementations. The project has processed more than 5,200 pull requests from contributors.

Round 20 Updates from our contributors

In the months between Round 19 and Round 20, about four hundred pull requests were processed. Some highlights shared by our contributors:

(Please reach out if you are a contributor and didn’t yet get a chance to share your updates. We’ll get them added here.)


Thanks again to contributors and fans of this project! As always, we really appreciate your continued interest, feedback, and patience!

Round 20 is composed of:

Framework Benchmarks Round 19

May 28, 2020

Nate Brady


Round 19 of the TechEmpower Framework Benchmarks project is now available!

This project measures the high-water mark performance of server side web application frameworks and platforms using predominantly community-contributed test implementations. Since its inception as an open source project in 2013, community contributions have been numerous and continuous. Today, at the launch of Round 19, the project has processed more than 4,600 pull requests!

We can also measure the breadth of the project using time. We continuously run the benchmark suite, and each full run now takes approximately 111 hours (4.6 days) to execute the current suite of 2,625 tests. And that number continues to steadily grow, as we receive further test implementations

Composite scores and TPR

Round 19 introduces two new features in the results web site: Composite scores and a hardware environment score we’re calling the TechEmpower Performance Rating (TPR). Both are available on the Composite scores tab for Rounds 19 and beyond.

Composite scores

Composite scores

Frameworks for which we have full test coverage will now have composite scores, which reflect an overall performance score across the project’s test types: JSON serialization, Single-query, Multi-query, Updates, Fortunes, and Plaintext. For each round, we normalize results for each test type and then apply subjective weights for each (e.g., we have given Fortunes a higher weight than Plaintext because Fortunes is a more realistic test type).

When additional test types are added, frameworks will need to include implementations of these test types to be included in the composite score chart.

You can read more about composite scores at the GitHub wiki.

TechEmpower Performance Rating (TPR)

TechEmpower Performance Rating

With the composite scores described above, we are now able to use web application frameworks to measure the performance of hardware environments. This is an exploration of a new use-case for this project that is unrelated to the original goal of improving software performance. We believe this could be an interesting measure of hardware environment performance because it’s a holistic test of compute and network capacity, and based on a wide spectrum of software platforms and frameworks used in the creation of real-world applications. We look forward to your feedback on this feature.

Right now, the only hardware environments being measured are our Citrine physical hardware environment and Azure D3v2 instances. However, we are implementing a means for users to contribute and visualize results from other hardware environments for comparison.

Hardware performance measurements must use the specific commit for a round (such as 801ee924 for Round 19) to be comparable, since the test implementations continue to evolve over time.

Because a hardware performance measurement shouldn’t take 4.6 days to complete, we use a subset of the project’s immense number of frameworks when measuring hardware performance. We’ve selected and flagged frameworks that represent the project’s diversity of technology platforms. Any results files that include this subset can be used for measuring hardware environment performance.

The set of TPR-flagged frameworks will evolve over time, especially if we receive further input from the community. Our goal is to constrain a run intended for hardware performance measurement to several hours of execution time rather than several days. As a result, we want to keep the total number of flagged frameworks somewhere between 15 to 25.

You can read more about TPR at the GitHub wiki.

Other Round 19 Updates

Once again, Nate Brady tracked interesting changes since the previous round at the GitHub repository for the project. In summary:


Thanks again to contributors and fans of this project! As always, we really appreciate your continued interest, feedback, and patience!

Round 19 is composed of:

Framework Benchmarks Round 13

November 16, 2016

Nate Brady

Round 13 of the ongoing Web Framework Benchmarks project is here! The project now features 230 framework implementations (of our JSON serialization test) and includes new entrants on platforms as diverse as Kotlin and Qt. Yes, that Qt. We also congratulate the ASP.NET team for the most dramatic performance improvement we’ve ever seen, making ASP.NET Core a top performer.View Round 13 resultsThe large filters panel on our results web site is a testament to the ever-broadening spectrum of options for web developers. What a great time to be building web apps! A great diversity of frameworks means there are likely many options that provide high-performance while meeting your language and productivity requirements.

Good fortunes

As the previous round—Round 12—was wrapping up, we were unfortunately rushed as the project’s physical hardware environment was being decommissioned. But good fortune was just around the corner, thanks to the lucky number 13!

New hardware and cloud environments

For Round 13, we have all new test environments, for both physical hardware and the virtualized public cloud.

Microsoft has provided the project with Azure credits, so starting with Round 13, the cloud environment is on Azure D3v2 instances. Previous rounds’ cloud tests were run on AWS.

Meanwhile, ServerCentral has provided the project a trio of physical servers in one of their development lab environments with 10 gigabit Ethernet. Starting with Round 13, the physical hardware environment is composed of a Dell R910 application server (4x 10-Core E7-4850 CPUs) and a Dell R420 database server (2x 4-Core E5-2406 CPUs).

We’d like to extend huge thanks to ServerCentral and Microsoft for generously supporting the project!

We recognize that as a result of these changes, Round 13 is not easy to directly compare to Round 12. Although changing the test environments was not intentional, it was necessary. We believe the results are still as valuable as ever. An upside of this environment diversity is visibility into the ways various frameworks and platforms work with the myriad variables of cores, clock speed, and virtualization technologies. For example, our new physical application server has twice as many HT cores as the previous environment, but the CPUs are older, so there is an interesting balance of higher concurrency but potentially lower throughput. In aggregate, the Round 13 results on physical hardware are generally lower due to the older CPUs, all else being equal.

Many fixes to long-broken tests

Along with the addition of new frameworks, Round 13 also marks a sizeable decrease in the number of existing framework tests that have failed to execute properly in previous rounds. This is largely the result of a considerable community effort over the past few months to identify and fix dozens of frameworks, some of which we haven’t been able to successfully test since 2014.

Continuous benchmarking

Round 13 is the first round conducted with what we’re calling Continuous Benchmarking. Continuous Benchmarking is the notion of setting up the test environment to automatically reset to a clean state, pull the latest from the source repository, prepare the environment, execute the test suite, deliver results, and repeat.

There are many benefits of Continuous Benchmarking. For example:

  • At any given time, we can grab the most recent results and mark them as a preview or final for an official Round. This should allow us to accelerate the delivery of Rounds.
  • With some additional work, we will be able to capture and share results as they are made available. This should give participants in the project much quicker insight into how their performance tuning efforts are playing out in our test environment. Think of it as continuous integration but for benchmark results. Our long-term goal is to provide a results viewer that plots performance results over time.
  • Any changes that break the test environment as a whole or a specific framework’s test implementation should be visible much earlier. Prior to Continuous Benchmarking, breaking changes were often not detected until a preview run.

Microsoft’s ASP.NET Core

We consider ourselves very fortunate that our project has received the attention that it has from the web framework community. It has become a source of a great pride for our team. Among every reaction and piece of feedback we’ve received, our very favorite kind is when a framework maintainer recognizes a performance deficiency highlighted by this project and then works to improve that performance. We love this because we think of it as a small way of improving performance of the whole web, and we are passionate about performance.

Round 13 is especially notable for us because we are honored that Microsoft has made it a priority to improve ASP.NET’s performance in these benchmarks, and in so doing, improve the performance of all applications built on ASP.NET.

Thanks to Microsoft’s herculean performance tuning effort, ASP.NET—in the new cross-platform friendly form of ASP.NET Core—is now a top performer in our Plaintext test, making it among the fastest platforms at the fundamentals of web request routing. The degree of improvement is absolutely astonishing, going from 2,120 requests per second on Mono in Round 11 to 1,822,366 requests per second on ASP.NET Core in Round 13. That’s an approximately 85,900% improvement, and that doesn’t even account for Round 11’s hardware being faster than our new hardware. That is not a typo, it’s 859 times faster! We believe this to be the most significant performance improvement that this project has ever seen.

By delivering cross-platform performance alongside their development toolset, Microsoft has made C# and ASP.NET one of the most interesting web development platforms available. We have a brief message to those developers who have avoided Microsoft’s web stack thinking it’s “slow” or that it’s for Windows only: ASP.NET Core is now wicked sick fast at the fundamentals and is improving in our other tests. Oh, and of course we’re running it on Linux. You may be thinking about the Microsoft of 10 years ago.

The best part, in our opinion, is that Microsoft is making performance a long-term priority. There is room to improve on our other more complex tests such as JSON serialization and Fortunes (which exercises database connectivity, data structures, encoding of unsafe text, and templating). Microsoft is taking on those challenges and will continue to improve the performance of its platform.

Our Plaintext test has historically been a playground for the ultra-fast Netty platform and several lesser-known/exotic platforms. (To be clear, there is nothing wrong with being exotic! We love them too!) Microsoft’s tuning work has brought a mainstream platform into the frontrunners. That achievement stands on its own. We congratulate the Microsoft .NET team for a massive performance improvement and for making ASP.NET Core
a mainstream option that has the performance characteristics of an acutely-tuned fringe platform. It’s like an F1 car that anyone can drive. We should all be so lucky.


Is your webapp slow? We can help!?


Framework Benchmarks Round 1

March 28, 2013

Nate Brady

How much does your framework choice affect performance? The answer may surprise you.

Authors’ Note: We’re using the word “framework” loosely to refer to platforms, micro-frameworks, and full-stack frameworks. We have our own personal favorites among these frameworks, but we’ve tried our best to give each a fair shot.

Show me the winners!

We know you’re curious (we were too!) so here is a chart of representative results.

Whoa! Netty, Vert.x, and Java servlets are fast, but we were surprised how much faster they are than Ruby, Django, and friends. Before we did the benchmarks, we were guessing there might be a 4x difference. But a 40x difference between Vert.x and Ruby on Rails is staggering. And let us simply draw the curtain of charity over the Cake PHP results.

If these results were surprising to you, too, then read on so we can share our methodology and other test results. Even better, maybe you can spot a place where we mistakenly hobbled a framework and we can improve the tests. We’ve done our best, but we are not experts in most of them so help is welcome!


Among the many factors to consider when choosing a web development framework, raw performance is easy to objectively measure. Application performance can be directly mapped to hosting dollars, and for a start-up company in its infancy, hosting costs can be a pain point. Weak performance can also cause premature scale pain, user experience degradation, and associated penalties levied by search engines.

What if building an application on one framework meant that at the very best your hardware is suitable for one tenth as much load as it would be had you chosen a different framework? The differences aren’t always that extreme, but in some cases, they might be. It’s worth knowing what you’re getting into.

Simulating production environments

For this exercise, we aimed to configure every framework according to the best practices for production deployments gleaned from documentation and popular community opinion. Our goal is to approximate a sensible production deployment as accurately as possible. For each framework, we describe the configuration approach we’ve used and cite the sources recommending that configuration.

We want it all to be as transparent as possible, so we have posted our test suites on GitHub.


We ran each test on EC2 and our i7 hardware. See Environment Details below for more information.

JSON serialization test

First up is plain JSON serialization on Amazon EC2 large instances. This is repeated in the introduction, above.

The high-performance Netty platform takes a commanding lead for JSON serialization on EC2. Since Vert.x is built on Netty, it too achieved full saturation of the CPU cores and impressive numbers. In third place is plain Java Servlets running on Caucho’s Resin Servlet container. Plain Go delivers the best showing for a non-JVM framework.
We expected a fairly wide field, but we were surprised to see results that span four orders of magnitude.

Dedicated hardware

Here is the same test on our Sandy Bridge i7 hardware.

On our dedicated hardware, plain Servlets take the lead with over 210,000 requests per second. Vert.x remains strong but tapers off at higher concurrency levels despite being given eight workers, one for each HT core.

Database access test (single query)

How many requests can be handled per second if each request is fetching a random record from a data store? Starting again with EC2.

For database access tests, we considered dropping Cake to constrain our EC2 costs. This test exercises the database driver and connection pool and illustrates how well each scales with concurrency. Compojure makes a respectable showing but plain Servlets paired with the standard connection pool provided by MySQL is strongest at high concurrency. Gemini is using its built-in connection pool and lightweight ORM.
It’s worth pausing to appreciate that this shows an EC2 Large instance can query a remote MySQL instance at least 8,800 times per second, putting aside the additional work of each query being part of an HTTP request and response cycle.

Dedicated hardware

The dedicated hardware impresses us with its ability to process nearly 100,000 requests per second with one query per request. JVM frameworks are especially strong here thanks to JDBC and efficient connection pools. In this test, we suspect Vert.x is being hit very hard by its connectivity to MongoDB. We are especially interested in community feedback related to tuning these MongoDB numbers.

Database access test (multiple queries)

The following tests are all run at 256 concurrency and vary the number of database queries per request. The tests are 1, 5, 10, 15, and 20 queries per request. The 1-query samples, leftmost on the line charts, should be similar (within sampling error) of the single-query test above.

As expected, as we increase the number of queries per request, the lines converge to zero. However, looking at the 20-queries bar chart, roughly the same ranked order we’ve seen elsewhere is still in play, demonstrating the headroom afforded by higher-performance frameworks.
We were surprised by the performance of Raw PHP in this test. We suspect the PHP MySQL driver and connection pool are particularly well tuned. However, the penalty for using an ORM on PHP is severe.

Dedicated hardware

The dedicated hardware produces numbers nearly ten times greater than EC2 with the punishing 20 queries per request. Again, Raw PHP makes an extremely strong showing, but PHP with an ORM and Cake—the only PHP framework in our test—are at the opposite end of the spectrum.

How we designed the tests

This exercise aims to provide a “baseline” for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application’s performance can only get worse. We aim to know the upper bound being set on an application’s performance per unit of hardware by each platform and framework.

But we also want to exercise some of the frameworks’ components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it’s a reasonable starting point.

For the data-connected test, we’ve deliberately constructed the tests to avoid any framework-provided caching layer. We want this test to require repeated requests to an external service (MySQL or MongoDB, for example) so that we exercise the framework’s data mapping code. Although we expect that the external service is itself caching the small number of rows our test consumes, the framework is not allowed to avoid the network transmission and data mapping portion of the work.

Not all frameworks provide components for all of the tests. For these situations, we attempted to select a popular best-of-breed option.

Each framework was tested using 2^3 to 2^8 (8, 16, 32, 64, 128, and 256) request concurrency. On EC2, WeigHTTP was configured to use two threads (one per core) and on our i7 hardware, it was configured to use eight threads (one per HT core). For each test, WeigHTTP simulated 100,000 HTTP requests with keep-alives enabled.

For each test, the framework was warmed up by running a full test prior to capturing performance numbers.

Finally, for each framework, we collected the framework’s best performance across the various concurrency levels for plotting as peak bar charts.

We used two machines for all tests, configured in the following roles:

  • Application server. This machine is responsible for hosting the web application exclusively. Note, however, that when community best practices specified use of a web server in front of the application container, we had the web server installed on the same machine.
  • Load client and database server. This machine is responsible for generating HTTP traffic to the application server using WeigHTTP and also for hosting the database server. In all of our tests, the database server (MySQL or MongoDB) used very little CPU time; and WeigHTTP was not starved of CPU resource. In the database tests, the network was being used to provide result sets to the application server and to provide HTTP responses in the opposite direction. However, even with the quickest frameworks, network utilization was lower in database tests than in the plain JSON tests, so this is unlikely to be a concern.

Ultimately, a three-machine configuration would dismiss the concern of double-duty for the second machine. However, we doubt that the results would be noticeably different.

The Tests

We ran three types of tests. Not all tests were run for all frameworks. See details below.

JSON serialization

For this test, each framework simply responds with the following object, encoded using the framework’s JSON serializer.

{"message" : "Hello, World!"}

With the content type set to application/json. If the framework provides no JSON serializer, a best-of-breed for the platform is selected. For example, on the Java platform, Jackson was used for frameworks that do not provide a serializer.

Database access (single query)

In this test, we use the ORM of choice for each framework to grab one simple object selected at random from a table containing 10,000 rows. We use the same JSON serialization tested earlier to serialize that object as JSON. Caveat: when the data store provides data as JSON in situ (such as with the MongoDB tests), no transcoding is done; the string of JSON is sent as-is.

As with JSON serialization, we’ve selected a best-of-breed ORM when the framework is agnostic. For example, we used Sequelize for the JavaScript MySQL tests.

We tested with MySQL for most frameworks, but where MongoDB is more conventional (as with node.js), we tested that instead or in addition. We also did some spot tests with PostgreSQL but have not yet captured any of those results in this effort. Preliminary results showed RPS performance about 25% lower than with MySQL. Since PostgreSQL is considered favorable from a durability perspective, we plan to include more PostgreSQL testing in the future.

Database access (multiple queries)

This test repeats the work of the single-query test with an adjustable queries-per-request parameter. Tests are run at 5, 10, 15, and 20 queries per request. Each query selects a random row from the same table exercised in the previous test with the resulting array then serialized to JSON as a response.

This test is intended to illustrate how all frameworks inevitably will converge to zero requests per second as the complexity of each request increases. Admittedly, especially at 20 queries per request, this particular test is unnaturally database heavy compared to real-world applications. Only grossly inefficient applications or uncommonly complex requests would make that many database queries per request.

Environment Details

  • Two Intel Sandy Bridge Core i7-2600K workstations with
  • 8 GB memory each (early 2011 vintage) for the i7 tests
  • Two Amazon EC2 m1.large instances for the EC2 tests
  • Switched gigabit Ethernet
Load simulator
Operating system
Web servers
Java / JVM


  • For the database tests, any framework with the suffix “raw” in its name is using its platform’s raw database connectivity without an object-relational map (ORM) of any flavor. For example, servlet-raw is using raw JDBC. All frameworks without the “raw” suffix in their name are using either the framework-provided ORM or a best-of-breed for the platform (e.g., ActiveRecord).

Code examples

You can find the full source code for all of the tests on Github. Below are the relevant portions of the code to fetch a configurable number of random database records, serialize the list of records as JSON, and then send the JSON as an HTTP response.


View on Github

public function index() {
    $query_count = $this->request->query('queries');
    if ($query_count == null) {
        $query_count = 1;
    $arr = array();
    for ($i = 0; $i < $query_count; $i++) { $id = mt_rand(1, 10000); $world = $this->World->find('first', array('conditions' =>
            array('id' => $id)));
        $arr[] = array("id" => $world['World']['id'], "randomNumber" =>
    $this->set('worlds', $arr);
    $this->set('_serialize', array('worlds'));


View on Github

(defn get-world []
  (let [id (inc (rand-int 9999))] ; Num between 1 and 10,000
    (select world
            (fields :id :randomNumber)
            (where {:id id }))))

(defn run-queries [queries]
  (vec ; Return as a vector
    (flatten ; Make it a list of maps
        queries ; Number of queries to run
        (repeatedly get


View on Github

def db(request):
    queries = int(request.GET.get('queries', 1))
    worlds = []
    for i in range(queries):
        worlds.append(World.objects.get(id=random.randint(1, 10000)))
    return HttpResponse(serializers.serialize("json", worlds), mimetype="application/json")


View on Github

app.get('/mongoose', function(req, res) {
    var queries = req.query.queries || 1,
        worlds = [],
        queryFunctions = [];

    for (var i = 1; i <= queries; i++ ) {
        queryFunctions.push(function(callback) {
            MWorld.findOne({ id: (Math.floor(Math.random() * 10000) + 1 )})
            .exec(function (err, world) {
                callback(null, 'success');

    async.parallel(queryFunctions, function(err, results) {


View on Github

public boolean db() {
    final Random random = ThreadLocalRandom.current();
    final int queries = context().getInt("queries", 1, 1, 500);
    final World[] worlds = new World[queries];
    for (int i = 0; i < queries; i++) {
        worlds[i] = store.get(World.class, random.nextInt(DB_ROWS) + 1);
    return json(worlds);


View on Github

def db() {
    def random = ThreadLocalRandom.current()
    def queries = params.queries ?'queries') : 1
    def worlds = []

    for (int i = 0; i < queries; i++) {
        worlds.add( + 1))

    render worlds as JSON


View on Github

if (path === '/mongoose') {
    var queries = 1;
    var worlds = [];
    var queryFunctions = [];
    var values = url.parse(req.url, true);

    if (values.query.queries) {
        queries = values.query.queries;
    res.writeHead(200, {'Content-Type': 'application/json; charset=UTF-8'});

    for (var i = 1; i <= queries; i++) {
        queryFunctions.push(function(callback) {
            MWorld.findOne({ id: (Math.floor(Math.random() * 10000) + 1 )})
                .exec(function (err, world) {
                    callback(null, 'success');

    async.parallel(queryFunctions, function(err, results) {

PHP (Raw)

View on Github

$query_count = 1;
if (!empty($_GET)) {
    $query_count = $_GET["queries"];
$arr = array();
$statement = $pdo->prepare("SELECT * FROM World WHERE id = :id");
for ($i = 0; $i < $query_count; $i++) { $id = mt_rand(1, 10000); $statement->bindValue(':id', $id, PDO::PARAM_INT);
    $row = $statement->fetch(PDO::FETCH_ASSOC);
    $arr[] = array("id" => $id, "randomNumber" => $row['randomNumber']);
echo json_encode($arr);


View on Github

$query_count = 1;
if (!empty($_GET)) {
    $query_count = $_GET["queries"];
$arr = array();
for ($i = 0; $i < $query_count; $i++) { $id = mt_rand(1, 10000); $world = World::find_by_id($id); $arr[] = $world->to_json();
echo json_encode($arr);



View on Github

public static Result db(Integer queries) {
    final Random random = ThreadLocalRandom.current();
    final World[] worlds = new World[queries];

    for (int i = 0; i < queries; i++) {
        worlds[i] = World.find.byId((long)(random.nextInt(DB_ROWS) + 1));

    return ok(Json.toJson(worlds));


View on Github

def db
  queries = params[:queries] || 1
  results = []

  (1..queries.to_i).each do
    results << World.find(Random.rand(10000) + 1) end render :json => results


View on Github


final DataSource source = mysqlDataSource;
int count = 1;

try {
    count = Integer.parseInt(req.getParameter("queries"));
} catch (NumberFormatException nfexc) {
    // Handle exception

final World[] worlds = new World[count];
final Random random = ThreadLocalRandom.current();

try (Connection conn = source.getConnection()) {
    try (PreparedStatement statement = conn.prepareStatement(
            ResultSet.CONCUR_READ_ONLY)) {

        for (int i = 0; i < count; i++) {
            final int id = random.nextInt(DB_ROWS) + 1;
            statement.setInt(1, id);

            try (ResultSet results = statement.executeQuery()) {
                if ( {
                    worlds[i] = new World(id, results.getInt("randomNumber"));
} catch (SQLException sqlex) {
    System.err.println("SQL Exception: " + sqlex);

try {
    mapper.writeValue(res.getOutputStream(), worlds);
} catch (IOException ioe) {


View on Github

get '/db' do
  queries = params[:queries] || 1
  results = []

  (1..queries.to_i).each do
    results << World.find(Random.rand(10000) + 1)



View on Github

@RequestMapping(value = "/db")
public Object index(HttpServletRequest request,
                    HttpServletResponse response, Integer queries) {

    if (queries == null) {
        queries = 1;

    final World[] worlds = new World[queries];
    final Random random = ThreadLocalRandom.current();
    final Session session = HibernateUtil.getSessionFactory().openSession();

    for(int i = 0; i < queries; i++) {
        worlds[i] = (World)session.byId(World.class).load(random.nextInt(DB_ROWS) + 1);


    try {
        new MappingJackson2HttpMessageConverter().write(
        worlds, MediaType.APPLICATION_JSON,
        new ServletServerHttpResponse(response));
    } catch (IOException e) {
        // Handle exception

    return null;


View on Github

StreamResponse onActivate() {
    int queries = 1;
    String qString = this.request.getParameter("queries");

    if (qString != null) {
        queries = Integer.parseInt(qString);

    if (queries <= 0) {
        queries = 1;

    final World[] worlds = new World[queries];
    final Random rand = ThreadLocalRandom.current();

    for (int i = 0; i < queries; i++) {
        worlds[i] = (World)session.get(World.class, new Integer(rand.nextInt(DB_ROWS) + 1));

    String response = "";

    try {
        response = HelloDB.mapper.writeValueAsString(worlds);
    } catch (IOException ex) {
        // Handle exception

    return new TextStreamResponse("application/json", response);


View on Github

private void handleDb(final HttpServerRequest req) {
    int queriesParam = 1;
    try {
        queriesParam = Integer.parseInt(req.params().get("queries"));
    } catch(Exception e) {
    final DbHandler dbh = new DbHandler(req, queriesParam);
    final Random random = ThreadLocalRandom.current();
    for (int i = 0; i < queriesParam; i++) {
            new JsonObject()
                .putString("action", "findone")
                .putString("collection", "world")
                .putObject("matcher", new JsonObject().putNumber("id",
                (random.nextInt(10000) + 1))), dbh);

class DbHandler implements Handler<Message<JsonObject>> {
    private final HttpServerRequest req;
    private final int queries;
    private final List<Object> worlds = new CopyOnWriteArrayList<>();

    public DbHandler(HttpServerRequest request, int queriesParam) {
        this.req = request;
        this.queries = queriesParam;

    public void handle(Message<JsonObject> reply) {
        final JsonObject body = reply.body;

        if ("ok".equals(body.getString("status"))) {

        if (this.worlds.size() == this.queries) {
            try {
                final String result = mapper.writeValueAsString(worlds);
                final int contentLength = result
                    "application/json; charset=UTF-8");
                this.req.response.putHeader("Content-Length", contentLength);
            } catch (IOException e) {
                req.response.statusCode = 500;


View on Github

protected ResourceResponse newResourceResponse(Attributes attributes) {
    final int queries = attributes.getRequest().getQueryParameters()
    final World[] worlds = new World[queries];
    final Random random = ThreadLocalRandom.current();
    final ResourceResponse response = new ResourceResponse();
    response.setWriteCallback(new WriteCallback() {
        public void writeData(Attributes attributes) {
            final Session session = HibernateUtil.getSessionFactory()
            for (int i = 0; i < queries; i++) {
                worlds[i] = (World)session.byId(World.class)
                    .load(random.nextInt(DB_ROWS) + 1);
            try {
            } catch (IOException ex) {
    return response;

Expected questions

We expect that you might have a bunch of questions. Here are some that we’re anticipating. But please contact us if you have a question we’re not dealing with here or just want to tell us we’re doing it wrong.

  1. “You configured framework x incorrectly, and that explains the numbers you’re seeing.” Whoops! Please let us know how we can fix it, or submit a Github pull request, so we can get it right.
  2. “Why WeigHTTP?” Although many web performance tests use ApacheBench from Apache to generate HTTP requests, we have opted to use WeigHTTP from the LigHTTP team. ApacheBench remains a single-threaded tool, meaning that for higher-performance test scenarios, ApacheBench itself is a limiting factor. WeigHTTP is essentially a multithreaded clone of ApacheBench. If you have a recommendation for an even better benchmarking tool, please let us know.
  3. “Doesn’t benchmarking on Amazon EC2 invalidate the results?” Our opinion is that doing so confirms precisely what we’re trying to test: performance of web applications within realistic production environments. Selecting EC2 as a platform also allows the tests to be readily verified by anyone interested in doing so. However, we’ve also executed tests on our Core i7 (Sandy Bridge) workstations running Ubuntu 12.04 as a non-virtualized sanity check. Doing so confirmed our suspicion that the ranked order and relative performance across frameworks is mostly consistent between EC2 and physical hardware. That is, while the EC2 instances were slower than the physical hardware, they were slower by roughly the same proportion across the spectrum of frameworks.
  4. “Why include this Gemini framework I’ve never heard of?” We have included our in-house Java web framework, Gemini, in our tests. We’ve done so because it’s of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we’re proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.
  5. “Why is JRuby performance all over the map?” During the evolution of this project, in some test runs, JRuby would slighly edge out traditional Ruby, and in some cases—with the same test code—the opposite would be true. We also don’t have an explanation for the weak performance of Sinatra on JRuby, which is no better than Rails. Ultimately we’re not sure about the discrepancy. Hopefully an expert in JRuby can help us here.
  6. “Framework X has in-memory caching, why don’t you use that?” In-memory caching, as provided by Gemini and some other frameworks, yields higher performance than repeatedly hitting a database, but isn’t available in all frameworks, so we omitted in-memory caching from these tests.
  7. “What about other caching approaches, then?” Remote-memory or near-memory caching, as provided by Memcached and similar solutions, also improves performance and we would like to conduct future tests simulating a more expensive query operation versus Memcached. However, curiously, in spot tests, some frameworks paired with Memcached were conspicuously slower than other frameworks directly querying the authoritative MySQL database (recognizing, of course, that MySQL had its entire data-set in its own memory cache). For simple “get row ID n” and “get all rows” style fetches, a fast framework paired with MySQL may be faster and easier to work with versus a slow framework paired with Memcached.
  8. “Do all the database tests use connection pooling?” Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling.
  9. “What is Resin? Why aren’t you using Tomcat for the Java frameworks?” Resin is a Java application server. The GPL version that we used for our tests is a relatively lightweight Servlet container. Although we recommend Caucho Resin for Java deployments, in our tests, we found Tomcat to be easier to configure. We ultimately dropped Tomcat from our tests because Resin was slightly faster across all frameworks.
  10. “Why don’t you test framework X?” We’d love to, if we can find the time. Even better, craft the test yourself and submit a Github pull request so we can get it in there faster!
  11. “Why doesn’t your test include more substantial algorithmic work, or building an HTML response with a server-side template?” Great suggestion. We hope to in the future!
  12. “Why are you using a (slightly) old version of framework X?” It’s nothing personal! We tried to keep everything fully up-to-date, but with so many frameworks it became a never-ending game of whack-a-mole. If you think an update will affect the results, please let us know (or submit a Github pull request) and we’ll get it updated!


Let go of your technology prejudices.

We think it is important to know about as many good tools as possible to help make the best choices you can. Hopefully we’ve helped with one aspect of that.

Thanks for sticking with us through all of this! We had fun putting these tests together, and experienced some genuine surprises with the results. Hopefully others find it interesting too. Please let us know what you think or submit Github pull requests to help us out.

About TechEmpower

We provide web and mobile application development services and are passionate about application performance. Read more about what we do.