Smartbench 2010 Aims to Change How We Benchmark

There are dozens upon dozens of benchmarks now available on the Android Market, so for a newcomer to stand out, it has to differentiate itself on some fundamental level. In the land of the Quadrants, Nenamark1s, and Linpacks; Smartbench 2010 is a 22-day-old benchmark by Smartphone Benchmarks that manages to distinguish itself by being a refreshingly reliable benchmark that gives two final scores: Productivity and Games.

The benchmark is made up of 6 individual tests, all of which complete fairly quickly:

• Pi – This is a predominantly CPU-based test, which calculates the mathematical constant Pi. It deals with arithmetic operations and floating point CPU performance. This is where processors like the ARM Cortex A9 in the Tegra 2 (and to a slightly lesser extent, the Scorpion in Snapdragon SoCs) are likely to be effective due to branching and elements of speculative execution.
• Mandelbrot – This test renders a couple of iterations of the Mandelbrot set. Like Pi, this is a mainly CPU-based test, which highlights higher IPC processor microarchitectures.
• GLTunnel – This predominantly GPU-bound OpenGL ES test renders a tunnel with several distinct light sources. The polygon count in this section is kept high in order to stress the GPU. This benchmark will highlight GPUs with high polygon crunching capabilities, such as the PowerVR SGX 540 in the Hummingbird SoC and the Tegra 2′s Geforce ULP.
• FileIOSmall – This test reads and writes small packets of information to your /data partition. Fast iNAND memory will yield the best results. Here is where EXT4 tweaks will come in handy… and of course also in day to day operations.
• FileIOLarge – This is basically the same as the previous test, just with larger chunks of information, which should highlight sustained reads and writes more than random reads and writes.
• GLJellyFish – This is the second GPU-bound OpenGL ES test. It is predominantly pixel fill-rate limited due to the use of large amounts of transparency in the jelly fish cubes. Here is where Tile-Based Deferred Rendering found in PowerVR chips becomes extremely useful.

Pros
This app has quite a bit going for it. It’s a fast, lightweight, and reliable benchmark that fills a gaping hole in the current state of benchmarking in Android. The major problem with most current tests is that they either are too limited in scope to be useful for anything other than bragging rights (Linpack, Nenamark1) or they are too broad and deliver a composite score that places too much emphasis on things such as I/O performance (Quadrant). Plus, I seem to get much more consistent results with this benchmark than its closest competitor, Quadrant. Normally more acquisition time equates to more consistency, so I am surprised that this benchmark can generate more precise results in a shorter time.

Cons
Clearly, this app is lacking in the polish department. While it’s incredibly functionally useful, it lacks a lot of the finishing touches of other Android Benchmarks. A quick look at the Results menu shows that while the benchmark itself runs at native resolution, the results display does not. Additionally, the application needs more reference scores. Currently there are only three in the database: HTC G2, LG Optimus One (does anyone who cares about benchmarking actually use this?), and the Nexus One running CyanogenMod 6, which coincidentally becomes the reference point for the test.

Conclusions
Overall, this is a great app with a lot of potential. It may be a little rough around the edges, but it’s a good deal more relevant than almost any other benchmark available. Oh, and did I mention that it’s FREE? So go ahead and do yourself a favor and head to the Android Marketplace to download a copy.

Update: As of version 1.1, which was released less than a day after my initial review, the flaws that I mentioned above have been corrected. The results screen text is rendered at native resolution, and there are many more devices that you can compare your score against. Now you have even more reason to grab yourself a free copy!

Source: Smartphone Benchmarks Forum