Mean Opinion Score

Mean Opinion Score (MOS) is a numerical measure used in telecommunications and multimedia engineering to represent the overall quality of a stimulus or system as perceived by human evaluators. It is calculated as the arithmetic mean of individual ratings given by test subjects on a predefined scale, typically ranging from 1 (lowest perceived quality) to 5 (highest perceived quality). MOS is widely used for evaluating voice, video, and audiovisual quality in applications ranging from traditional telephony to modern text-to-speech systems and streaming media.

The methodology originated in the telecommunications industry for assessing telephone call quality and was formally standardized by the International Telecommunication Union (ITU-T) in Recommendation P.800 in 1996. Since then, it has become the gold standard for subjective quality assessment across various domains where human perception of quality is critical.

History and Development

Early Origins

The concept of Mean Opinion Score emerged in the telecommunications industry during the 1970s as telephone networks became more complex and digital transmission methods were introduced. Initially developed by the ITU, MOS provided a standardized way to assess voice transmission quality over telephone networks by aggregating human judgments of call quality.

The early methodology involved having listeners sit in controlled "quiet rooms" and score telephone call quality as they perceived it. This subjective testing approach had been in use in the telephony industry for decades before formal standardization, reflecting the industry's recognition that technical measurements alone could not capture the human experience of communication quality.

Standardization

The methodology was formally standardized in ITU-T Recommendation P.800, "Methods for subjective determination of transmission quality," approved on August 30, 1996. This recommendation established rigorous protocols for conducting subjective quality tests, including specific requirements for test environments and procedures.

The standardization specified that test subjects should be seated in quiet rooms with volumes between 30 and 120 cubic meters, reverberation times less than 500 milliseconds (preferably 200-300 ms), and room noise levels below 30 dBA with no dominant spectral peaks. These environmental controls ensured consistency and reliability in MOS evaluations across different testing facilities and organizations.

Evolution and Extensions

Following the success of P.800, the ITU-T developed additional recommendations to clarify and extend MOS methodology:

• ITU-T P.800.1 (2003, updated 2016): Established terminology for different types of MOS scores, distinguishing between listening quality subjective (MOS-LQS), listening quality objective (MOS-LQO), and listening quality estimated (MOS-LQE) to avoid confusion about the source and nature of scores.
• ITU-T P.800.2: Prescribed how MOS values should be reported, emphasizing that MOS scores from separate experiments cannot be directly compared unless explicitly designed for comparison and statistically validated.
• ITU-T P.808 (2021): Addressed crowdsourcing methods for conducting subjective evaluations, recognizing the need for scalable approaches to MOS testing in the digital age.

Methodology

Rating Scales

The most commonly used rating scale is the Absolute Category Rating (ACR) scale, which maps subjective quality ratings to numerical values:

Score	Quality Level	Description
5	Excellent	Completely natural speech; imperceptible artifacts
4	Good	Mostly natural speech; just perceptible but not annoying
3	Fair	Equally natural and unnatural; perceptible and slightly annoying
2	Poor	Mostly unnatural speech; annoying but not objectionable
1	Bad	Completely unnatural speech; very annoying and objectionable

Alternative scales may use different ranges (e.g., 1-100) or different qualitative descriptors, depending on the specific application and testing requirements.

Testing Procedures

Traditional MOS testing involves several key steps:

1. Subject Selection: Recruiting appropriate test participants, typically naive listeners without specialized training in audio quality assessment
2. Environment Control: Conducting tests in acoustically controlled environments meeting ITU-T specifications
3. Stimulus Presentation: Playing audio samples to subjects in randomized order to minimize bias
4. Rating Collection: Having subjects rate each stimulus on the chosen scale
5. Statistical Analysis: Calculating the arithmetic mean and confidence intervals for each stimulus

Modern extensions include comparative methods such as:

• Degradation Category Rating (DCR): Subjects compare processed audio to a reference
• Comparison Category Rating (CCR): Direct comparison between two stimuli

Objective Estimation

While traditional MOS relies on human evaluation, objective models have been developed to predict MOS scores automatically. Key standardized methods include:

• PESQ (ITU-T P.862): Perceptual Evaluation of Speech Quality, introduced in 200
• POLQA (ITU-T P.863): Perceptual Objective Listening Quality Assessment, approved in 201
• PSQM (ITU-T P.861): Perceptual Speech Quality Measure, the first standardized method from 1997

These algorithms analyze acoustic properties of audio signals to estimate human perceptual quality, enabling automated quality monitoring and real-time assessment.

Applications

Telecommunications

MOS remains fundamental in telecommunications for evaluating voice and video call quality. In Voice over IP (VoIP) systems, MOS scores help assess the impact of network impairments such as packet loss, jitter, and latency on user experience. The G.711 codec, commonly used in VoIP, has a maximum theoretical MOS of 4.4, serving as a benchmark for quality comparisons.

Telecommunications companies use MOS for:

• Network planning and optimization
• Codec evaluation and selection
• Service level agreement monitoring
• Competitive benchmarking

Speech Synthesis and AI

In modern artificial intelligence applications, MOS has become critical for evaluating text-to-speech (TTS) systems. As synthetic speech quality has improved dramatically with neural approaches like WaveNet, Tacotron, and VITS, MOS remains the primary method for assessing how natural and human-like synthesized speech sounds to listeners.

However, recent research has highlighted limitations of MOS for evaluating state-of-the-art speech synthesis systems. Studies have shown that as synthetic speech approaches human quality, MOS becomes less sensitive to remaining differences, leading researchers to explore complementary evaluation methods.

Multimedia and Streaming

MOS is extensively used in multimedia applications for evaluating:

• Video streaming quality
• Audio codec performance
• Compression artifact assessment
• Real-time communication platforms

Streaming services use MOS to optimize their delivery pipelines, balancing bandwidth efficiency with perceptual quality to ensure user satisfaction across diverse network conditions.

Modern Challenges and Limitations

Statistical and Methodological Issues

MOS faces several inherent limitations that researchers and practitioners must consider:

Ordinal Scale Problems: MOS ratings are based on ordinal scales where the ranking of items is known but intervals between ratings are not necessarily equal. Mathematically, calculating an arithmetic mean from ordinal data is problematic, and median values would be more appropriate. However, the practice of using arithmetic means is widely accepted and standardized.

Range-Equalization Bias: Test subjects tend to use the full rating scale during an experiment, making scores relative to the range of quality present in the test rather than absolute measures of quality. This prevents direct comparison of MOS scores from different experiments.

Contextual Dependence: MOS values are influenced by the testing context, participant demographics, and the presence of anchor stimuli (very high or low quality samples that influence perception of other stimuli).

Scalability and Cost

Traditional MOS testing is time-consuming and expensive, requiring recruitment of human evaluators and controlled testing environments. This has led to increased interest in:

• Crowdsourcing platforms for distributed evaluation
• Objective quality models that predict MOS
• Automated evaluation metrics that correlate with human perception

Limitations in Advanced Applications

As technology advances, particularly in AI-generated content, traditional MOS evaluation faces new challenges:

Ceiling Effects: When synthetic speech approaches human quality, MOS becomes less discriminative, with most systems scoring in the 4.0-4.5 range where small differences may not be statistically significant.

Missing Dimensions: MOS provides only an overall quality rating and may miss specific aspects like speaker similarity, emotional expression, or intelligibility of specific linguistic phenomena.

Cultural and Linguistic Bias: MOS scores can vary based on evaluator demographics, language background, and cultural factors, potentially limiting generalizability across diverse user populations.

Contemporary Developments

Crowdsourcing and Remote Testing

ITU-T P.808 (2021) established guidelines for conducting MOS evaluations using crowdsourcing platforms, recognizing the need for scalable testing methods. This approach enables larger-scale evaluations but introduces new challenges in quality control and participant screening.

Deep Learning Integration

Recent research explores using deep learning models for automatic MOS prediction, potentially enabling real-time quality assessment. Some approaches integrate MOS predictions into other tasks, such as fake audio detection, where predicted MOS scores help identify synthetic speech.

Multi-Modal Assessment

Modern applications increasingly require evaluation beyond audio quality alone. Research is extending MOS concepts to multi-modal scenarios, including audio-visual quality assessment and the evaluation of text-to-speech avatars that combine voice and visual synthesis.

Related Standards and Metrics

Several ITU-T recommendations work in conjunction with MOS:

• ITU-T G.107: The E-model for objective quality assessment that can be mapped to MOS scales
• ITU-T P.910: Subjective video quality assessment methods
• ITU-T P.863: POLQA objective quality measurement
• ITU-T P.862: PESQ objective speech quality assessment

Other quality metrics used alongside MOS include technical measurements like signal-to-noise ratio, mean squared error, and mel-cepstral distortion, though these objective measures often correlate poorly with human perception.

Criticism and Future Directions

The speech synthesis research community has increasingly questioned whether MOS alone is sufficient for evaluating modern high-quality systems. Critics argue that the field may have reached "the end of a cul-de-sac by only evaluating the overall quality with MOS" and advocate for developing new evaluation protocols better suited to analyzing advanced speech synthesis technologies.

Proposed alternatives and extensions include:

• Fine-grained evaluation of specific quality dimensions
• Task-specific intelligibility testing
• Comparative ranking methods that avoid absolute scaling issues
• Objective metrics that better correlate with human perception

Despite these limitations, MOS remains the most widely accepted method for subjective quality assessment and continues to evolve to meet the needs of advancing technology.

External Links

• ITU-T Recommendation P.800
• ITU-T Recommendation P.800.1
• ITU-T Recommendation P.808