Comparing browsers for OpenUI – Octane 2 as a benchmark tool?

Recently I took Google Chrome Canary 64-bits (R39) to the test. It appeared that Google’s own disclaimer:

Be forewarned: it's designed for developers and early adopters, 
and can sometimes break down completely.

Should be taken seriously. Both Canary and the stable version (R37) should be able to run side-by-side. Even though both R37 and R39 were working flawlessly for normal browsing, my Siebel dedicated client would no longer run against R37. Uninstalling both releases and installing R37 fresh resolved the issue.

First thing I did: run Google’s own Octane2 benchmark against both browsers a couple of times. I made sure there were no other CPU intensive tasks running.

Interestingly enough R39 falls behind R37 overall.

Octane is a an open source initiative from Google. The dimensions against which Octane tests browsers are shown on the left-hand side. In the middle section contains the rough results for IE11, FF31, Chrome R37 and R39 are shown. On right-hand side performance factors are calculated.

BrowserOctane2results

R37 overall is a factor 1,1 faster than R39. On certain dimensions the differences are more than a little.

The red highlighted rows have in my personal view are important dimensions for Siebel OpenUI performance. The dimensions are pretty well explained here.

For example “Splay” and “SplayLatency” are relevant to the OpenUI framework. OpenUI depends on JQuery and executes fast amounts of script. So object creation, destruction and an efficient garbage collection process are key.

  • Splay
    Data manipulation benchmark that deals with splay trees and exercises the automatic memory management subsystem. Main focus: Fast object creation, destruction.
  • SplayLatency
    The Splay test stresses the Garbage Collection subsystem of a VM. SplayLatency instruments the existing Splay code with frequent measurement checkpoints. A long pause between checkpoints is an indication of high latency in the GC. This test measures the frequency of latency pauses, classifies them into buckets and penalizes frequent long pauses with a low score. Main focus: Garbage Collection latency.

Put it together in a visual representation…

OctaneGraphOUI

What I’m personally particularly interested in is whether Octane2 could be a valid benchmark for Siebel OpenUI in particular. Octane2 is developed by Google, so will Chrome prevail by definition over other browsers? There are other benchmarks such as Sunspider (see previous post) and Dromaeo. The nice thing about Dromaeo is that it splits up in certain categories of tests focussed e.g. on CSS selection and JQuery (used quite a bit in OUI as we know). I did not used Dromaeo so far, on the list. Dromaeo tests are more elaborate and take more time to complete.

So let’s loop back to my previous post where I build a number of increasingly heavy Siebel OpenUI views. I ran a new set of timing tests using the described mechanism.

You will see two rows (collapsed and expanded). For now – just neglect this. Will spent a separate post on my challenges to improve complex views – rather than splitting the applets across different views 😉

Collapsedexpanded

For example these test tell that Chrome R37 is on average 1,2x faster than R39. That compares to the Octane2 results. Especially if you relate it to Siebel OpenUI relevant test dimensions:

OpenUISpecificOctane2

Also the Chrome vs FF (1,4x) compares pretty well to the timings measured in my OpenUI stress test views.

But to comparing to IE it would be a very bad indicator… Where the OpenUI performance factors are 3,4x for FF vs IE and 4,3x for Chrome vs IE – Octane would respectively indicate a factor 1,8x and 2,5x.

What is clear is that the performance breakdown on IE11 follows an exponential function, where as for FF and Chrome it’s a more linear curve. Which was made clear in my previous post already.

In a next post I will dive a bit deeper into Dromaeo as well to continue my quest to improve rendering large views in OpenUI.

 

Advertisements

One thought on “Comparing browsers for OpenUI – Octane 2 as a benchmark tool?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s