- Reference Benchmarks :.
Cross Platform DAW Performance - Part IV :
last reports we covered a lot of ground in regards to the cross
platform performance on some of the better
known DAW's in the market, but without doubt the one DAW that is
more synonymous with the whole area of DAW's than any other is AVID (
formally Digidesign ) Protools. Where Protools was significantly
different to the others was its genealogy was not based in Native
DAW's, but in hardware specific DSP powered systems that have pretty
much ruled the roost at the top end for over a decade. Their Native
forays have been very restrictive in regards to not only qualified
hardware , but also in regards to the features available in the lower
tiered versions in comparison to the big brother TDM / DSP based
systems - i.e. , track counts, no ADC, limited number of I/O and
numerous other features more specific to Post/Video.
|Preparing for Battle :.|
While test system spec remained unchanged, the actual audio hardware was completely different to the previous testing due to the specific hardware requirements of the AVID DAW applications.
This was already highlighting the difficulties in trying to do head to head comparative testing due to the restrictive nature of the Protools Native platform , when even within its own product line , there were 2 very distinctly different versions that used their own specific hardware , add to that the fact that the native driver protocols were not the ones used by other DAW's - ASIO/Core Audio, but there own proprietary DAE variant.
There are respective ASIO drivers for the hardware, but the consistency was not there between the 2 , M-Audio having a far better ASIO low latency implementation on Windows for example , so testing against the other applications would have to be done with the M-Audio only.
To illustrate the state of the ASIO drivers for the
Digi 002 in Win7 x64 , these are the buffer settings and reported latencies
- 064 : 10.8ms , 128: 13.7ms , 256: 19.5ms,
which are essential at least 3 times where they should be.
The performance of LE v MP cross platform was similar, LE being a touch better across the board , but my suspicion is that it is due to some extra padding in the buffers as mentioned before . Overall the variable in the results were within a few % , so I was comfortable in going ahead with just the M-Audio from there on.
With the choice of reference hardware sorted , I knuckled down and ran thru the test sessions on the selection of the competing DAW's with the new M-Audio Hardware. Due to the fact that I needed to retest all of the other DAW's as none of the previous testing on the RME HDSPe could be used, I took the opportunity to shift the Steinberg testing back to Cubase 5.5 , which is a closer head to head market wise, and updated StudiOne to 1.6, as the developers had contacted me to inform me that some under bonnet multiprocessor work had been done since the version 1.5 that I had tested previously
Just as I was finishing up my testing , AVID released PT9 which works with not only all Digi and M-Audio hardware , but for the first time allowed the use of 3rd party hardware running under ASIO and Core Audio
Hell had officially frozen over , and to add a final cherry, they had included ADC , the lack of on the previous versions being a major gripe.
Perfect timing , as I managed to get an upgrade online and downloaded within hours of it being available and allowed me to place a nice finishing touch on this round of testing.
To be able to use my reference RME hardware at long last within Protools , will greatly simplify the comparative testing in future, and will also have a far higher consistency. That will be highlighted in Part V when we move to the ASIO only testing , this round will be the DAE used with the M-Audio interfaces in PT v ASIO in the other DAW hosts.
|Round 1 : DAWbench DSP : WaveArts MD5 Multiband Compressor : M-Audio Profire 610 :|
First cab off the rank , the WaveArts MD5 that showed itself to have closer comparable performance between OSX / Windows with the VST version , so it would be interesting if the same would be experienced with the RTAS version.
Lets take a closer look at the comparative cross platform results for all the respective DAW's.
Protools MP 8.0 : @ 064 performed better on Windows by 630% , @ 128 performance was identical, @ 256 performed better on OSX by 2%
Protools 9.0 : @064 performed better on Windows by 555%, @ 128 performance was identical, @ 256 performed better on OSX by 13%
Cubase 5.5 : @ 064 performed better on Windows by 34% , @ 128 performed better on Windows by 14%, @256 performed better on Windows by 2%
StudioOne 1.6 : @064 performed better on Windows by 70% , @ 128 performed
better on Windows by 56% , @256 performed better on Windows by 53%
For those that have been reading in on the previous reports, the most obvious and distinct change with the Protools testing is that for the first time we have OSX out performing Windows at the higher latency setting. Also @ 128 performance cross platform is identical on both PT8 and PT9 so again that is very different to the experience with the other DAW hosts.
There is a very strange anomaly at 064 where the session could only manage 18-20 instances of the plugin, and it took a few stops and starts to even get to that point - despite the internal metering being only in the low 20% area, so something very odd going on in the buffering interaction between the audio engine and this particular plugin.
On Windows the performance was far more consistent, and both PT8 and PT9 was either equal to or out performed the other other 2 DAW application in the test pool .
On OSX , apart from the anomaly @ 064, @128 and @ 256 the Protools /RTAS products easily out performed both Cubase and StudioOne.
I should also note that the PT8/9 M-Audio combination returned identical results on Windows and very close to identical on OSX- PT9 being a touch better @ 256. Overall, both behaved in close enough to an identical manner on both Windows and OSX , which confirms that the underlying driver architecture is still the previous DAE , not ASIO.
So in summary, the results show a very consistent result on Windows for the Protools products but not so with OSX, most obvious is the huge dip @ 064 sample buffers. Despite that, the cross platform performance is incredibly even in comparison to what we had witnessed with the other 3rd party DAW's in the previous testing. The results for both Cubase and StudioOne were very consistent with the previous testing , the reported improvements to the later from 1.5 to 1.6 did not eventuate on this session
Possible variables are the optimizations for the RTAS Plugin's over VST and the DAE engine over ASIO / Core Audio.
Lets see if that behavior is maintained with the other 3rd party plugin's.
|Round 3 : DAWbench DSP : URS Channel Strip Pro : M-Audio Profire 610 :|
As I mentioned in the previous report, the Channel Strip Pro was one of the newly introduced reference plugin's for DAWbench, so this was the first time I had tested it in the RTAS format cross platform.
Lets take a look at the respective results.
Protools MP 8.0 : @ 064 performed better on Windows by 47% , @ 128 performance was better on Windows by 27%, @ 256 performed better on Windows by 21%
Protools 9.0 : @064 performed better on Windows by 8%, @ 128 performance was better on Windows by 8%, @ 256 performed better on Windows by 3%
Cubase 5.5 : @ 064 performed better on Windows by 53% , @ 128 performed better on Windows by 36%, @256 performed better on Windows by 32%
StudioOne 1.6 : @064 performed better on Windows by 84% , @ 128 performed better on Windows by 83% , @256 performed better on Windows by 74%
On Windows the results were again very consistent and close to par across all of the applications , Cubase having a few more legs @ 128/256. PT8/9 RTAS again scaled very well against the VST driven hosts . Performance of PT8/9 was again identical.
On OSX, the results this time for PT8/9 were a lot more consistent , no strange anomaly's @ 064 as experienced on the MD5 , which was a relief on one hand , but only deepened the mystery further as to what actually caused it.
The performance of PT8 and Cubase was very close to par on this session, StudioOne remaining consistently behind , but where PT9 on the last session was close to PT8 , on this session it showed quite a substantial improvement , and clearly out performed all of the other DAW's.
On the previous round of testing, all of the tested DAW's did not scale as well on this particular session in OSX, and that had remained consistent in all of the testing up to PT9 on this round. That maybe hinting at some possible optimization changes under the bonnet of PT9 that has proved beneficial with the specific processing required by reference plugin.
The significant difference on this round of testing is that the cross platform performance variable for Protools is more noticeable , and leaning towards Windows, tho it should be noted that the variable is greatly reduced with Protools 9 over Protools 8.
The result @ 256 , which leaned measurably towards OSX with the MD5, has not been maintained , so some questions arise there as to whether the MD5 results are the exception or the rule.
The MD5 did perform better comparably cross platform on the last round of testing as well , so there is some consistency being maintained in what we are experiencing with the RTAS versions.
My feeling to this point is that the WaveArts plugin's are very well coded and optimised for OSX , not to say they are not well coded and optimised for Windows, but they do consistently perform better than the other plugin's in the sessions tested so far.
On to the next...
|Round 4 : DAWbench DSP : Elysia mPressor : M-Audio Profire 610|
The Elysia mPressor is
the only plugin in the test pool that has an inherent delay, and with
Protools MP 8 not being able to automatically adjust for
that , the playing field was a little uneven in regards to whether
the ADC/PDC calculations would play a part in the comparative scaling
against the other host DAW's. With the release of PT9 we were able
to not only test the scaling effects of ADC, if any between PT8/PT9 ,
but also level the playing field in regards to how each
DAW host arbitrates the ADC/PDC.
Protools MP 8.0 : @ 064 performed better on Windows by 25% , @ 128 performance was better on Windows by 22%, @ 256 performed better on Windows by 13%
Protools 9.0 : @064 performed better on Windows by 12%, @ 128 performance was better on Windows by 3%, @ 256 performed better on Windows by 3%
Cubase 5.5 : @ 064 performed better on Windows by 76% , @ 128 performed better on Windows by 59%, @256 performed better on Windows by 42%
StudioOne 1.6 : @064 performed better on Windows by 98% , @ 128 performed better on Windows by 93% , @256 performed better on Windows by 95%
The Windows results showed a measurable advantage to the VST powered application on this specific session across the board. The results for both PTMP 8 and PT9 were again identical, but what is interesting there is that the inclusion of ADC in PT9 has not had any measurable effect to scaling. That in itself is quite interesting and a definite positive for PT9.
One significant thing in regards to ADC in PT9, is how it is actually applied. Whereas in the other DAW's the PDC/ADC is applied in real time when the plugin's are activated, causing a noticeable glitch/pause as the hosts have to do the recalcs in real time, PT9 does not glitch at all when the plugin's are activated, which I have had confirmed is because the actual ADC calc's are not done in real time, instead are applied when the engine is stopped and restarted. Not a huge issue in Real World application, but something to be aware of.
On OSX the results were more consistent to the last session, no anomaly at 064 for PTMP8/PT9 as with the MD5, so that was definitely the exception not the rule , also the improved scaling of PT9 over PTMP8 has been maintained despite the inclusion of ADC , which bodes well for the ADC implementation on that platform as well.
Both PTMP8/PT9 showed the other DAW's a clean set of
An analysis of
the results maintains the variable we witnessed with the CSP between
OSX and Windows 7 , with it being consistently in favor of the later.
That's not to say that other plugin's with inherent latency will not take a resource hit when using ADC, but it was not the case in this instance.
The results with this particular plugin definitely showed performance in favor of VST2 on Windows over RTAS, much more so than any of the previous, but of course that could simply mean that it is better optimised than the RTAS variant , whereas the opposite could be said for the plugin on OSX , where the RTAS version performs measurably better .
Nothing conclusive either way , but something to keep in mind as the test pool widens.
This round did add some extra variables into the mix , RTAS v VST, DAE v ASIO/Core Audio , so it was a bit harder to do direct head to head comparatives than on previous test sessions simply because of the grey areas those variables introduced .
Some of the questions raised were, was DAE equally
optimised cross platform ,
were the plugin formats of RTAS / VST2 equally optimised even within
the respective O.S platforms , were the actual latency values for
the respective buffer settings within DAE/ASIO/CoreAudio consistent
within an acceptable range, etc, etc.
Now I do have to note that I do not have the exact latency values for DAE in respect to the buffer settings , there could well be some added padding there , but within the available testing parameters - it did hold its own against ASIO/Core Audio.
In regards to the RTAS plugin's performance, I have to admit this has been a real eye opener for me. I , like many in the past , simply bought into the mantra that RTAS was less efficient in comparison, but I never really knuckled down and did the side by side empirical testing. Granted there were technical issues in the past that stopped me from doing the testing, one being PTLE/MP for many years being track limited to under the required 45 stereo tracks of audio required for the test suite , then the extended delay of sorting Windows 7 x64. Finally with 8.04 all the ducks lined up and I could do some head to head, and the results were surprising to say the least.
Several significant things were highlighted during the testing, the first being that the RTAS plugin format was not the dog that it was made out to be, in fact on Windows it held its own against the VST variants, and on OSX it actually performed measurably better. Now lets be clear, this is for plugin's , virtual instruments may be a totally new ball game, but in this instance, it performed admirably.
The second significant factor was that DAE actually performed very well cross platform comparatively against ASIO/CoreAudio and thirdly and most importantly in respect to the previous reports, the cross platform performance between OSX and Windows was a lot closer than the other DAW hosts.
Of course that opens up a whole host of new questions, is it that RTAS as a plugin format is more consistent cross platform , does the same apply to DAE over ASIO/CoreAudio , is it simply that PT is coded equally cross platform while the the other DAW's are all equally badly coded for OSX, etc, etc.
I'll try and tackle at least one of those questions in the final chapter where I will ditch the proprietary M-Audio Hardware / DAE, and use my reference RME HDSPe / ASIO/ CoreAudio in PT9