MPEG Audio Decoder Compliance

Introduction

The purpose of this page is to share an objective analysis of various MPEG audio decoders, in particular the results of tests for decoder compliance with respect to the ISO/IEC 11172 international standard.

Decoder Compliance

Part 4 of ISO/IEC 11172 defines the compliance tests for MPEG-1 audio and video decoders. Section 2.6.3 defines the computational accuracy tests for audio decoders. It states:

To be called an ISO/IEC 11172-3 audio decoder, the decoder shall provide an output such that the rms level of the difference signal between the output of the decoder under test and the supplied reference output is less than 2−15⁄√12 for the supplied sine sweep (20Hz–10kHz) with an amplitude of −20dB relative to full scale. In addition, the difference signal shall have a maximum absolute value of at most 2−14 relative to full-scale.

Then it says:

To be called a limited accuracy ISO/IEC 11172-3 audio decoder, the decoder shall provide an output for a provided test sequence such that the rms level of the difference signal between the output of the decoder under test and the supplied reference output is less than 2−11⁄√12 for the supplied sine sweep (20Hz–10kHz) with an amplitude of −20dB relative to full scale.

Thus, for the purpose of testing computational accuracy, we shall use three categories of compliance: fully compliant, limited accuracy, and not compliant.

Designations

ISO/IEC 11172-4 designates two kinds of audio decoders. Section 2.6.3 states:

An ISO/IEC 11172-3 compliant decoder that is able to support at least one but not all combinations of the options defined in 2.4.3 such as bit rates, sampling rates and modes, will be designated as an ISO/IEC 11172-3 Layer “N” audio decoder. Decoders that support all combinations are designated as Full Layer “N” ISO/IEC 11172-3 audio decoders, where “N” indicates I, II or III.

It should also be noted that, according to section 0.2 of ISO/IEC 11172-3:

An ISO/IEC 11172-3 Audio Layer N decoder is able to decode bitstream data which has been encoded in Layer N and all layers below N.

About the Tests

The following results have been obtained in tests conducted by Underbit in accordance with Annex A of ISO/IEC 11172-4.

All measurements are carried out relative to full scale where decoder output signals are normalized to be between −1.0 and +1.0. The supplied sine sweep with an amplitude of −20dB relative to full scale has an absolute amplitude of ±0.1 and a precision of 24 bits.

As instructed, where the output of a decoder is fewer than 24 bits, the remaining least significant bits are set to zero for purposes of normalization and measurement. (The number of output bits from the decoder is noted.)

Where the compliance test data contains two channels, or in the case of Layer II where multiple compliance test streams exist, the rms level and maximum difference values are calculated separately for each, and the greatest overall value used for the result.

Calculation of RMS

The rms (root-mean-square) level of the difference signal is calculated as

Sqrt[Sum[(t[i] - r[i])^2, {i, 1, N}] / N]

where ti is the ith output sample from the decoder under test, ri is the ith sample from the reference output, and N is the number of samples.

The maximum absolute value of the difference signal is simply the greatest value of |ti − ri|.

Test Results

Decoder Bits Layer I Layer II Layer III
RMS Level Max Diff RMS Level Max Diff RMS Level Max Diff
Floating-Point Decoders
amp 0.7.6 16 failed 2.099×10−5 3.421×10−5 1.959×10−5 3.362×10−5
limited accuracy limited accuracy
Amplay 1 16 failed failed 7.860×10−2 1.487×10−1
not compliant
Apollo 37 16 failed 2.099×10−5 3.421×10−5 1.961×10−5 3.529×10−5
limited accuracy limited accuracy
Audioactive MP3 Decoder 1.12 16 failed failed 8.602×10−6 1.538×10−5
fully compliant
CoolPlayer 7224
[Xaudio]
16 8.728×10−6 1.526×10−5 9.336×10−5 3.175×10−3 8.602×10−6 1.538×10−5
fully compliant limited accuracy fully compliant
FreeAmp 2.1 beta 5 16 failed 1.408×10−1 2.259×10−1 1.410×10−1 2.003×10−1
not compliant not compliant
Jet-Audio 4.7
(64-bit float mode)
16 1.351×10−1 1.938×10−1 1.367×10−1 2.519×10−1 1.366×10−1 1.941×10−1
not compliant not compliant not compliant
LAME 3.86 beta
[mpglib]
16 failed failed 8.870×10−6 1.836×10−5
limited accuracy
maplay 1.2 16 1.738×10−5 3.052×10−5 1.850×10−5 3.052×10−5 failed
limited accuracy limited accuracy
maplay 1.2+ for Win32 1.A 16 8.728×10−6 1.526×10−5 9.996×10−2 2.659×10−1 8.602×10−6 1.526×10−5
fully compliant not compliant fully compliant
mp3blaster 2.0b17 16 1.738×10−5 3.052×10−5 6.654×10−1 1.130×100 1.783×10−5 3.088×10−5
limited accuracy not compliant limited accuracy
MP3 Decoder 1.45 16 failed 9.995×10−2 2.659×10−1 1.074×10−3 3.479×10−2
not compliant not compliant
MP3Player 1.23 16 8.728×10−6 1.526×10−5 8.759×10−6 1.526×10−5 8.602×10−6 1.538×10−5
fully compliant fully compliant fully compliant
MP3 To Wave Converter 1.14 16 5.234×10−2 1.255×10−1 9.793×10−2 2.226×10−1 9.982×10−2 2.000×10−1
not compliant not compliant not compliant
MpegDJ GoWave! 1.41 16 failed 1.854×10−5 3.779×10−5 1.780×10−5 3.064×10−5
limited accuracy limited accuracy
mpg123 0.59r 16 8.728×10−6 1.526×10−5 8.759×10−6 1.526×10−5 8.602×10−6 1.538×10−5
fully compliant fully compliant fully compliant
MusicMatch Jukebox 5.10.0149
[Xaudio]
16 8.728×10−6 1.526×10−5 8.759×10−6 1.526×10−5 8.602×10−6 1.538×10−5
fully compliant fully compliant fully compliant
NAD 0.94pb3 16 8.822×10−3 1.107×10−1 9.639×10−2 2.000×10−1 9.991×10−2 2.000×10−1
not compliant not compliant not compliant
QuickTime 4.1.2, x86 16 failed failed 8.602×10−6 1.526×10−5
fully compliant
RightClick-MP3 1.65e
[L3dec 2.72]
16 failed failed 8.602×10−6 1.538×10−5
fully compliant
Shibatch mpg123 plug-in 1.18+ for Winamp 16 failed failed 8.854×10−6 1.836×10−5
limited accuracy
Sonique 1.63 16 1.394×10−1 2.001×10−1 8.759×10−6 1.526×10−5 1.410×10−1 2.003×10−1
not compliant fully compliant not compliant
Sonique 1.808
[audioEnlightenment 4.70]
16 8.981×10−6 1.836×10−5 8.994×10−6 1.895×10−5 8.867×10−6 1.836×10−5
limited accuracy limited accuracy limited accuracy
Sonique 1.808
[audioEnlightenment 4.858 (Default Decoder)]
16 8.981×10−6 1.836×10−5 8.994×10−6 1.895×10−5 8.867×10−6 1.836×10−5
limited accuracy limited accuracy limited accuracy
splay 0.8.2 16 1.738×10−5 3.052×10−5 1.863×10−5 4.232×10−5 1.783×10−5 3.088×10−5
limited accuracy limited accuracy limited accuracy
UltraPlayer 2.00 16 8.965×10−6 1.824×10−5 1.262×10−2 1.297×10−1 8.854×10−6 1.836×10−5
limited accuracy not compliant limited accuracy
Winamp 2.65
[Nullsoft MPEG audio decoder plug-in 2.33 (486)]
16 8.980×10−6 1.836×10−5 9.020×10−6 1.907×10−5 9.066×10−6 5.877×10−5
limited accuracy limited accuracy limited accuracy
Winamp 2.65
[Nullsoft MPEG audio decoder plug-in 2.33 (Pentium)]
16 8.980×10−6 1.836×10−5 9.020×10−6 1.907×10−5 9.067×10−6 5.877×10−5
limited accuracy limited accuracy limited accuracy
Winamp 2.65
[Nullsoft MPEG audio decoder plug-in 2.33 (MMX)]
16 4.524×10−5 1.634×10−4 4.539×10−5 1.811×10−4 9.067×10−6 5.877×10−5
limited accuracy limited accuracy limited accuracy
Winamp 2.666
[Nullsoft MPEG Audio Decoder 2.666]
16 8.727×10−6 1.526×10−5 8.759×10−6 1.526×10−5 8.602×10−6 1.526×10−5
fully compliant fully compliant fully compliant
Xaudio 1.1.0, x86
[decoder library 1.3.1]
16 8.728×10−6 1.526×10−5 8.759×10−6 1.538×10−5 8.602×10−6 1.538×10−5
fully compliant fully compliant fully compliant
Integer Decoders
Intel® IPP 1.01, StrongARM 16 failed failed 8.631×10−6 1.812×10−5
fully compliant
MAD 0.11.4b, ARM 24 4.667×10−8 2.384×10−7 4.906×10−8 2.384×10−7 5.338×10−8 2.384×10−7
fully compliant fully compliant fully compliant
MAD 0.11.4b, x86 24 6.198×10−8 3.576×10−7 6.198×10−8 2.384×10−7 9.000×10−8 9.537×10−7
fully compliant fully compliant fully compliant
MAD 0.11.4b, x86
(optimized for accuracy)
24 4.667×10−8 2.384×10−7 4.906×10−8 2.384×10−7 5.555×10−8 2.384×10−7
fully compliant fully compliant fully compliant
MAD 0.11.4b, x86
(optimized for speed)
24 7.131×10−6 2.480×10−5 7.250×10−6 2.789×10−5 7.227×10−6 2.730×10−5
fully compliant fully compliant fully compliant
mpg123 0.59r-arm32 16 6.972×10−2 1.000×10−1 7.058×10−2 1.300×10−1 7.051×10−2 1.001×10−1
not compliant not compliant not compliant
splay 0.8.2-fp1 16 2.180×10−5 6.604×10−5 1.961×10−5 4.458×10−5 5.618×10−2 1.190×10−1
limited accuracy limited accuracy not compliant
Xaudio 1.1.0, ARM
[decoder library 1.3.1]
16 6.945×10−2 9.965×10−2 2.003×10−5 3.576×10−5 2.065×10−5 6.580×10−5
not compliant limited accuracy limited accuracy
Legend
fully compliant The rms level of the difference signal is less than 2−15⁄√12 (≅ 8.810×10−6) and the maximum absolute value of the difference signal is less than or equal to 2−14 (≅ 6.104×10−5). The decoder is an ISO/IEC 11172-3 Layer “N” audio decoder.
limited accuracy The rms level of the difference signal is less than 2−11⁄√12 (≅ 1.410×10−4). (The maximum absolute value of the difference signal does not matter.) The decoder is a limited accuracy ISO/IEC 11172-3 Layer “N” audio decoder.
not compliant The rms level of the difference signal is greater than or equal to 2−11⁄√12 (≅ 1.410×10−4). The decoder is not a compliant ISO/IEC 11172-3 Layer “N” audio decoder.
failed The decoder failed to produce output for the given test bitstream. The decoder is not a Full Layer “N” ISO/IEC 11172-3 audio decoder.

Notes

Full Compliance

It should be noted that a fully compliant result from these tests does not mean the decoder in question is without flaws.

These tests only verify the computational accuracy of decoders against a reference signal that spans only 20Hz–10kHz of the frequency spectrum. While this test may be enough to detect a problem with a decoder, it is not enough to certify that a given decoder will correctly reconstruct frequencies outside this range, or that the same decoder will correctly handle any valid bitstream. To obtain these kinds of assurances, further tests are necessary.

Quantization Method

It should further be noted that these test results can be affected by the method used by each decoder to quantize its PCM output. Some decoders may use dithering or noise shaping to reduce the negative effects of quantization noise and improve the perceived audio quality, while others may simply round each output sample to the nearest integral value. Unfortunately, with few exceptions, it is not known what quantization method is used by each of the tested decoders.

It is possible for an otherwise fully compliant decoder that implements dithering with 16-bit output to appear only to have limited accuracy. The decoder’s output before quantization should be the true measure of its accuracy.