HOME | BENCHMARKS | FORUM| CONTACT |
 
DAWbench - Reference Benchmarks :.
   

PC DAW X-Scaling Shootout - Part I : Nuendo v Reaper V SONAR :.

   

Following on from my previous report regarding the issues surrounding MP Scalability in regards to DAW's - specifically in regards to Dual Quadcore Systems using Steinberg audio applications , there has been a lot of debate over the last 12 months or so in regards to the scaling capabilities of the various other audio applications on Dual Quadcore systems, and whether they were equally effected, and in not, why so. I had specific benchmarks for both Cubendo and SONAR, but no way to directly compare scaling performance between the 2 applications.

I decided to develop a Universal benchmark based on the methodology used on the existing benchmarks that could can be quickly configured for any audio application. I configure the new sessions for not only Nuendo/Cubase and SONAR, but this time around also included Reaper on the PC platform, and will expand to other applications in due course.

 
     
Preparing for Battle :.    

While doing some routine testing and basic maintenance on the Dual Quad development system, I stumbled on a surprising fix that for the greater part resolved a lot of the scaling issues being experienced at lower latencies with the Steinberg app's on the Dual Quads for over 16 months. More detail can be read Here

With Nuendo 4 scaling way better on Dual Quads than it had at anytime in the past, despite Steinberg's"official" announcement to the contrary, and with the microcode improvements also showing performance benefits for both SONAR and Reaper, it was time to place them head to head.

I developed the new DAWbench DSP-Universal test, and configured the sessions for the respective app's.

 

DAW Application Details :
Cakewalk SONAR: version 7.02
Cockos Reaper : Version 2.1.419
Steinberg Nuendo: Version 4.1.2

Single Quadcore System Detail:
Intel QX6700 Quadcore/ 2.66 GHZ
Intel i975 / 2GB DDR2-PC6400/ Nvidia Quadro NVS:

Dual Quadcore System Detail:
2 x Intel X5355 Quadcore / 2.66GHZ
Intel i5000X / 4 GB FBDIM DD2-PC5300/ Nvidia Quadro NVS:

Audio Hardware Detail:
RME : HDSP9632 : Driver 3.056
Lynx: LynxTwo-C: Driver 2.014e

* Results were cross referenced across both audio hardware*

O.S Detail: Windows XPSP2

 

 

The next stage involved selecting the plugin(s) to be used for applying the incremental and progressive load to the sessions. To ensure that I had a level playing field , I needed to select a loading plugin which would not only be on equal footing across all applications, but also be stable, flexible and resource hungry enough to stress the 8 core systems with their new found overhead.

Through a process of elimination, I narrowed the selection down to the 2 final plugin's, ReaFX ReaXComp and WaveArts MultiDynamics 5 , but not without some added hurdles to navigate, read on...

         


Results using the original ReaXComp Standalone/ReaXComp Internal:


Results using the recompiled ReaXComp Standalone:

Leveling The Playing Field:    

I initially configured the 3 sessions to use the ReaXComp , as with it being Freeware would be the one most would be using to do the personal testing. I noticed immediately that the standalone version of the plugin would not appear in Reaper. There were a few possible reasons for that, one being it may have been sharing the same plugin ID as the internal version, or the developers had simply made it unavailable to eliminate confusion.

That posed a dilemma, as there was no way of ensuring we were using identical plugin's across the 3 applications. The first graph shows the results of Nuendo/SONAR using the standalone version, and Reaper using its internal build. The Standalone performed equally across both Nuendo/SONAR , so I was confident that at least those results were consistent, but with the huge delta being shown with the Reaper result I needed to ensure we were all on the same page.

 

I contacted Justin Frankel, one of the developers of Reaper, and asked for not only clarification on whether the plugin's were in fact the same between the standalone and the internal versions, but also if it was possible for him to send my a standalone version that would appear in Reaper. That would ensure we were dealing from the same deck.

Justin was quite open in admitting that there were some differences between the 2 versions.i.e, the internal is 64 bit compatible whereas the standalone is only 32 bit, but more importantly that the internal was using a faster compiler which could be loading the deck in favor of Reaper.

He graciously supplied a new standalone version using the faster compiler, and also made it visible to Reaper, so we could ensure all app's were using the identical code.

 

Here's were it gets really interesting, as the new version showed a substantial improvement in Nuendo proving that the faster compiler had in fact been a variable in the initial result. Reaper had also shown some improvement but the delta had definitely narrowed. I was expecting SONAR to follow suit, but to my surprise the results were almost exactly the same as the previous standalone build.

So where the new compiled code had shown improvement on both Nuendo and Reaper, it had made no difference to SONAR.

I still do not have a definitive reason why SONAR had not shown similar improvement to Nuendo, where on the earlier build they were on par. It does show that test results can vary widely, depending on the loading plugin used. This why it was crucial to have a second plugin to use as a cross reference.

 

Round One : DAWbench DSP - ReaFX ReaXComp :.

With the playing field leveled in regards to the ReaXComp , I proceeded to run the session across all 3 applications on the Single Quadcore systems. I already had the results for the Dual Quad listed in the previous graphs.

With the added results of the Single Quadcore system, this would give us 2 sets of numbers to analyze, the first being the overall number of plugin's for each respective application on each system , and secondly the X-Scaling capability from 4-8 Cores

Lets look at each application individually, starting with Reaper as it performed the best overall in regards to plugin numbers.

Firstly lets measure the scalability in regards to latency.

Using the results for 256 Samples as a base result of 100%, we can then measure the scalability of the system as the latency is lowered.

Reaper @ 032 samples managed 64% on the Single Quad System while achieving 58% on the Dual Quad.

X-Scaling from 4-8 Cores.

@256 - 95%, @ 128 - 93%, @ 064 - 82% , @ 032 - 77%.

Reaper has always been renowned for its MP scalability, and the above results show the reputation was well deserved, showing an almost linear scaling at the higher latencies, and still managing and impressive 77% at the lowest latency.

 

Moving on to Nuendo , again we will first measure the scalability in regards to latency.

Nuendo @ 032 samples managed 62% on the Single Quad System while achieving 50% on the Dual Quad.

X-Scaling from 4-8 Cores.

@256 - 83%, @ 128 - 80%, @ 064 - 66% , @ 032 - 48 %.

While not as impressive as Reaper, the results are quite good considering the scalability on Dual Quad systems is still "officially" broken. It is hinting that there is still some room for improvement.

Moving on to SONAR, and this is where the figures can get a little muddy, as I will explain.

Lets get the numbers out of the way first, again measuring the scalability in regards to latency.

SONAR @ 032 samples managed 03% on the Single Quad System while achieving 5.7% on the Dual Quad.

Moving on to X-Scaling from 4-8 Cores.

@256 - 100%, @ 128 - 104%, @ 064 - 108% , 032 - 250%

The 2 sets of figures for SONAR are a little confusing to say the least when held up in direct comparison to the other 2 applications. Whereas on the previous applications there is a direct correlation to the the 2 sets of figures, with SONAR however we have a few curve balls .

 

To be fair we really do need to make a special note of the 032 results from SONAR, as the application really struggled at that latency setting. If testing the application independently, I would have not included the 032 results to be honest ,but when placed in a direct comparison with the other app's, we need to navigate the data as it is presented.

The data shows 2 distinct and almost contradictory sets of results. The overall results and scalability in regards to latency is greatly diminished in comparison to the other applications , while the x-scalability from 4 -8 cores is basically linear, which is quite an achievement.

What is causing the contradiction is SONAR's overall poor performance compared to the other applications using this specific plugin.

The poor performance was directly related to one of the cores on SONAR's internal CPU meter always reading a lot higher, which inhibits the remaining cores to be able to scale any further. Interestingly in Task Manager, the variable is nowhere near as high, its only in the SONAR meters for the CPU's

It wasn't clear why one core was spiking, and I hadn't ruled out the 3rd party plugin at this stage.

It was time to move to the WaveArts MultiDynamics 5 plugin, to not only cross reference the results, but also to alleviate the concerns if any in regards to using a plugin supplied by one of the combatants, so to speak.

         


Round Two : DAWbench DSP - WaveArts MD5 :.    

The Results with the WaveArts MultiDynamics 5 Plugin were interesting in that although the performance between the 3 applications remained consistent in regards to order from top to bottom, the delta between Reaper and Nuendo had increased measurably across the board on both the Single and Dual Quad. It was noticeably more predominant on the Dual Quad System. SONAR was still struggling with that spiking core, ruling out it being specific to the ReaXComp.

That aside, the results highlight that different plugin's can and will alter overall results when measuring scalability across applications, and also highlight the fact that certain plugin's themselves are far more efficient in MP environments than others. This is definitely another area worth exploring in future.

The whole subject of the various multithreading models within audio applications and how that relates to plugin's is beyond the scope of this article, but I will visit it at a later date.

 

O.K the numbers..

Starting with Reaper again , firstly the scalability in regards to latency.

Reaper @ 032 samples managed 48% on the Single Quad System while achieving 43% on the Dual Quad

X-Scaling from 4-8 Cores.

@256 - 80%, @ 128 - 71%, @ 064 - 71% , @ 032 - 67%.

Moving on to Nuendo , again we will
first measure the scalability in regards to latency.

Nuendo @ 032 samples managed 38% on the Single Quad System while achieving 35 % on the Dual Quad.

X-Scaling from 4-8 Cores.

@256 - 69%, @ 128 - 54%, @ 064 - 48% , @ 032 - 54 %.

 

SONAR again throws the whole thing into a loop..

SONAR @ 032 samples managed 03% on the Single Quad System while achieving 4.5 % on the Dual Quad.

X-Scaling from 4-8 Cores.

@256 - 66%, @ 128 - 85%, @ 064 - 89% , 032 - 133%.

Where as the performance with both Nuendo and Reaper remained consistent , both dropping a measured amount % wise using the MD5, SONAR's results remained unpredictable due to the variables being imposed by that spiking core.

It was time to contact the dev's to hopefully get some answers .

There were also some other curves encountered during the testing , but they were all successfully navigated, specifically in Reaper.

     


DAWbench DSP- Dual X5355 - SONAR @032 Samples - No Plugin's Loaded


DAWbench DSP - Dual X5355 - SONAR @064 Samples -72 MD5's loaded

Navigating the Curves :.        

The 2 screenshots above clearly highlight the issue with the spiking core in SONAR's internal metering, and how that spike was not correlating within the task manager.

As soon as one core hits the deep red, the engine stops, so on the first screen shot with the latency set to 032, it is clear why the results were so low, as one core is very close to peaking without any plugin's loaded at all. The second screenshot is the system fully loaded at 064 samples, but again, one core is clearly spiking way above the others, inhibiting the scalability.

I have been in contact with the Cakewalk developers in regards to the behavior, and they are looking into the exact cause of the single core being so disproportionally loaded in respect to the others. There were a few suggestions investigated and eliminated as possible causes in relation to uneven plugin loading, GUI, etc. It won't be fair to speculate any further until I have had some clear confirmation from the Cakewalk team

 

The other curve that was navigated during the testing was in respect to a major looping glitch that was being experienced with the earlier versions of Reaper : 2.1.0.6/8. When the test session was approaching the loop point, Reapers playback would severely glitch , which was making finalizing a result extremely difficult. Despite the fact that playback was clean apart from the looping issue, it was concerning to say the least.

I again contacted Justin Frankel and after detailing the issue and my concerns , he immediately suggested some changes in the disk read buffering mode.

Not wasting any time, Justin made the required changes in the next version of Reaper, 2.1.419, which on its release a few days later ,totally resolved the looping issue. There has since been another update following that version at the time of writing.

 

The immediate response to end user concerns and qualms is something that Justin and the team at Reaper are renown for. I was extremely impressed by not only the speed at which Justin responded to the concerns, but also his willingness to address the issues quickly, his openness, honesty and continuing support.

It really is a breath of fresh air in comparison to the reception from some of the other developers I have received over the years.

Its this exact same approach to their end users that has gained Justin and the team at Reaper such a great and steadily growing reputation within the community, as well as the application itself continuing on its path as being a formidable force in the DAW arena.

Conclusion :

I would like to stipulate that this is not a shootout in regards to feature sets of the respective DAW's , but purely in regards to native low latency scaling and x-scaling capabilities.

In that respect Reapers Multi Processor threading model- called "Anticipative FX Processing" is definitely proving more efficient than those being utilized by the other competing applications, at least in this series of tests.

 

Each application has its own strengths and weaknesses in regards to MP scaling , and how well the respective models deal with the added arbitration required by DSP cards,for example, so there is more to this race before its run its course.

I will update this report periodically with any added information, and will also expand the next round to include XPSP3 v Vista64 SP1.

 

This is the first of a series of reports which will be expanded to include more applications as the sessions are configured.

I am also planning to include Apple MAC / OSX based hardware / applications in a future shootout, Logic being a particular focus.

Until then.

Vin Curigliano
AAVIM Technology
April 9 2008

         

© AAVIMT 2007