Design and evaluation of real-time Voice-Over-IP (VOIP) systems with high perceptual conversational quality

VoIP;Perceptual Quality;Conversational Quality;Subjective Evaluations;Multi-dimensional Quality;Comparative Evaluation;Human Factors;Model of Pair-wise Subjective Comparisons;Statistical Scheduling;Probabilistic Preference Model;Performance Evaluation

Sat, Batu

关键词: VoIP;    Perceptual Quality;    Conversational Quality;    Subjective Evaluations;    Multi-dimensional Quality;    Comparative Evaluation;    Human Factors;    Model of Pair-wise Subjective Comparisons;    Statistical Scheduling;    Probabilistic Preference Model;    Performance Evaluation;   

Others  :  https://www.ideals.illinois.edu/bitstream/handle/2142/16809/1_Sat_Batu.pdf?sequence=2&isAllowed=y

美国|英语

来源: The Illinois Digital Environment for Access to Learning and Scholarship

【 摘 要 】

This research addresses real-time multimedia communication systems that can achieve high perceptualquality for their users. It focuses on the fundamental understanding of multiple qualityaspects perceived and their trade-offs in the design of run-time control schemes that adapt tochanging network conditions and expectations of the users of a system.One of the main contributions of this thesis is the use of adaptively scheduled off-line subjectivecomparison tests to efficiently and accurately learn users' subjective preferences among thealternative trade-offs in order to guide the run-time control schemes to achieve high perceptualquality.In order to illustrate the application of the general framework and our methodology, throughoutthis thesis we study the design of real-time VoIP (voice-over-IP) systems that can achieve highperceptual conversational quality. The trade-offs in the design and operation of a VoIP systeminvolve the design of speech codecs and strategies for network control, playout scheduling, andloss concealment.The perceptual quality of a conversation over a network connection depends on the quality of theone-way speech (listening-only speech quality or LOSQ) received and the delay incurred from themouth of the speaker to the ear of the listener (mouth to ear delay or MED). In a conversation, eachparticipant takes turns in speaking and listening, and both perceive a silence duration called mutualsilence (MS) when the conversation switches from one party to another. When the connection hasdelays, the MSs perceived by a participant consist of alternating short and long silence periodsbetween turns. As a result, listeners may experience lower perceptual quality when the MSs arehighly asymmetric, some speakers appearing to be more distant than others, or some respondingslower than others.The evaluation of conversational quality is a largely unexplored area. There are many objectivemetrics for assessing the quality of a VoIP conversation, but there is no single objective metricwhose results match well with subjective results. Indiscriminate subjective testing is not feasiblebecause it is prohibitively expensive to carry out many such tests under various conversationaland network conditions. Moreover, there is no systematic method to generalize the subjective test results to unseen conditions. To this end, there are five key innovations in this research that willaddress the limitations of existing work and improve the perceptual quality of VoIP systems.Firstly, we have developed a methodology and test-bed to objectively and subjectively evaluateVoIP systems under repeatable and fair conditions using a black-box approach. We have appliedthis methodology and test-bed on four commonly used VoIP systems; Skype, Google-Talk, WindowsLive and Yahoo Messenger. The results show that different systems operate differently incoping with network imperfections such as jitter and loss. We also observe that systems do notchange their operation as a function of the one-way delay of the connection, or of the turn-takingfrequency of the conversation. The results show thatWindows Live is preferred under a significantset of conditions but none of the systems is dominant under all conditions.We have also learned the mapping between easily obtainable objective measures and subjectivepreferences of users in order to allow any VoIP system to be comprehensively compared againstothers without expensive subjective comparisons. We later use the mapping learned to subjectivelycompare our newly designed system with the four systems already evaluated.Secondly, we have developed a general model of pair-wise subjective comparisons, based onindividually identified properties, axioms and lemmas, that models comparisons on a continuousoperating curve with a single control parameter. The model provides a basis for developing themethod to schedule adaptive off-line subjective tests and for identifying the optimal point by fusingthe information obtained from separate subjective evaluations on the same operating curve. Themodel can be used in formulating and solving any type of pair-wise comparison problem thatexhibits the same properties identified. The model is flexible to allow the existence of multipleoptimal points on an operating curve and includes a belief function framework that can guide thesearch for optimal points efficiently. Furthermore, the model is built on a statistical framework thatallows for the confidence of individual evaluation results to be represented in the conclusivenessof the combined belief function.Thirdly, we have developed a method for tackling the control design problem of finding the optimalpoint in an N-dimensional space, which includes all the metrics that affect quality. The overallproblem is transformed into two independent problems of finding the optimal point on a continuousbut one-dimensional curve, and learning the mapping on a set of curves that adequately spansthe K-dimensional (K < N) curve space, where K stands for the number of metrics characterizingthe network and conversational conditions.Fourthly, we have applied the methodology to the design of play-out scheduling control forVoIP systems by conducting subjective tests and by learning from them under a comprehensiveset of network and conversational conditions. We have also verified the accuracy and efficiency ofour methodology using exhaustive subjective tests on a subset of the network and conversationalconditions. The verification of our methodology on a real-life application also justified our modelof pair-wise subjective comparisons and our optimal algorithm to adaptively choose upcomingcomparisons using information learned from previously conducted tests.Lastly, we have generalized the learned results in the design of play-out scheduling control toconditions that are unseen at design time and observed only at run-time. Along with design choicesin other components of the architecture, this results in a VoIP system that can achieve higher andmore consistent perceptual quality than other four VoIP systems analyzed. We have also shownthat our system performs very close to a non-causal ideal system where the POS decisions aremade optimally using future information.Our model and methodologies are applicable to a wide variety of real-time multimedia communicationsystem control design problems which operate under constrained resources, communicatingover non-stationary IP networks, and for which the overall quality perceived by users ofthe system has multiple counteracting aspects which cannot be represented by a single objectivemetric.