QoS: Measurement and Evaluation of Telecommunications Quality of Service / Edition 1

by William C. Hardy

Overview

Quality of Service (QoS) is continuously growing in importance in the telecommunications industry because competition is growing fiercer by the day. By drawing on 30 years of experience, William C. Hardy explains how to examine specific tools and techniques that he has developed for the measurement and evaluation of QoS and understand the underlying analysis perspectives and methodologies.

Details the basic concepts of QoS, together with the methodologies for organizing, structuring, and carrying out analyses of QoS from scratch.
Describes the atttributes of the telecommunications service that determine user perception of quality in non-technical terms.
Discusses specific measures, measurement techniques and evaluation criteria for all of the factors that affect user perception of QoS.
Addresses user concerns including:

* Will I be able to get to the service when I want to use it?
* How long does it take before I know a connection is being set up?
* How good will voice sound over a connection?
* Includes valuable tips for QoS analysis and the perspectives vital for describing QoS in ways that are useful and operationally meaningful.

Whether you have a limited technical background or are a telecommunications professional this simple and straightforward approach will be an essential tool to understanding QoS.

Product Details

ISBN-13:	9780471499572
Publisher:	Wiley
Publication date:	07/10/2001
Pages:	248
Product dimensions:	6.14(w) x 9.17(h) x 0.76(d)

About the Author

William C. Hardy is the author of QoS: Measurement and Evaluation of Telecommunications Quality of Service, published by Wiley.

Read an Excerpt

Chapter 2: Measurement and Evaluation

2.1 Function of Measurement and Evaluation

What is suggested here, then, is that analysis is a process whose ultimate end is to produce specific answers to specific questions. This point of view is predicated on the modest assertion that:

The only good reason to measure anything is to reduce uncertainty with respect to some course of action that must be decided.

Admittedly, this statement has some of the flavor of the Caterpillar trying to tell Alice which is the right and left side of a round mushroom. However, all that is posited here is that measurement and evaluation to produce and interpret quantitative descriptions of performance, quality, or whatever other attributes are being examined, will neither be useful nor worthwhile unless the results help someone feel more comfortable about some decision as to what to do and when to do it, such as what new car to buy, what telephone services to order, how to go about correcting a recognized problem, how to recognize that a problem has emerged, or when to sell a stock. Without such an underlying need for the information gleaned from measurement and evaluation, the results will be of no more use to a decision-maker than a painstaking analysis of carefully collected data showing with great precision and confidence that the sun will nova in exactly 9 787 316 years, 3 months, and 4.7 h, evoking responses from decision-makers that the results are "interesting", or more damning, "nice-to-know", but not "actionable".

The principal value of this concept of the function of measurement and evaluation is that it readily suggests a number of questions that the analyst should address before undertaking any analysis. These include questions of:

• Audience: which decision-makers are to be supported by the results of the analysis?
• Utility: what kinds of decisions are to be facilitated? How must measurements be evaluated to produce information that can be used for those decisions?
• Concerns: what are the questions that those decision-makers are likely to want to have answered during the course of making those decisions?
• Objectives: what are the courses of action that will be decided or determined by appeal to the results of the analysis?

2.1.1 Audience and Utility

To appreciate the importance of addressing these questions at the outset, consider first the diversity of possible audiences for analyses of quality of telecommunications services. As described below, there are at least five distinct classes of decision-makers who might be responsible for actions whose efficacy depends on reliable information of likely user perception of QoS, and the evaluation of measures needed to make the results of the analysis useful to the decision-makers is in each case different.

(1) Service users. The most obvious class comprises the actual users of the service, who are continually testing its quality by placing calls. The principal uncertainties that they face are ones of how often they will encounter problems that materially impede the act of placing a call and completing the desired exchanges of information. Consequently, users will be very conscious of any difficulties experienced and will synthesize that experience over time to determine whether the incidence and severity of problems actually encountered is acceptable, thereby producing a subjective assessment of perceived quality. On the basis of that subjective assessment, a user then decides tentatively that the service is satisfactory or unsatisfactory. If it is unsatisfactory, the user will initially complain, and then later abandon the service, if the is no improvement. If the service is tentatively found to be satisfactory, the user continues its use and continues to synthesize the experience with it to verify the original subjective assessment. As long as the assessment does not change, the user remains satisfied. However, perceptible changes in the type, incidence, severity, or user's accommodation of problems with the service may result in a different assessment of perceived quality, leading the user to decide to complain about or change the service, when possible. As a possible audience for results of QoS analyses, then, users will be looking for results providing reassurances with respect to uncertainties as to what will be experienced in the unknown future. Such reassurances sought will be of one of two kinds:

• Assurances that a service that has not been experienced, such as a new offering, a less expensive substitute for an existing service of the same kind, or a similar service based on new technology is likely to be found to be satisfactory; or
• Assurances that a service that has been experienced and found to be unsatisfactory will be put right and no longer exhibit the type, severity, or incidence of problems that rendered it unsatisfactory in the first place.

Since users are the ultimate decision-makers with respect to which of possibly many competing services is to be used, the user concerns are the principal focus of QoS measurement, and the evaluation of those measures should answer the basic question:

What is the likelihood that users of a service exhibiting the value x for the QoS measure MP, will find the service to be satisfactory with respect to the attribute measured by MP?

(2) User representatives. Users of residential and small business telecommunications services usually represent themselves in such activities as selecting telecommunications services and features, choosing among competing providers of the chosen services, and negotiating prices. However, such activities are otherwise vested in a small group of people whose principal decisionmaker, whom we will call the Comm Manager, is responsible for choosing, acquiring, and maintaining services for a large body of users. Since Comm Managers are the representatives of their user communities, they must be concerned with user satisfaction with the services they select, and are therefore naturally interested in analyses of perceived quality of service as a means of reassuring their users of the validity of their decisions. However, since their role is also one of assuring their management of economy of services, their perspective on QoS will be one of trying to assess cost-benefit trade-offs, and the principal question with respect to measures of QoS will frequently be more like:

What is the smallest value x for the QoS measure MP that will keep complaints from my user community as to the quality of service with respect to the attribute measured by MP at manageable levels?

In addition, by virtue of being the principal decision-maker for a user community, the Comm Manager will be the one responsible for the assessed quality of service. The Comm Manager will therefore be much more concerned with questions of billing and customer support, and much more actively involved in trying to define and assure satisfaction of the criteria for assessed quality, than the individual user.

(3) Service provider sales and marketing personnel. On the other side of the fence, one of the major consumers of QoS analyses will be the sales and marketing personnel, who are not necessarily decision-makers, but must respond to the concerns with QoS raised by the users and Comm Managers who are their prospective customers. Because of their role in telling prospective customers about telecommunications services, they will want whatever the customer wants, but with the additional feature that the analyses must also show how quality of the services they sell compares with that of competing services offered by other providers. Because of the need to characterize, communicate, and interpret any differences in measures of QoS between the competing telecommunications services, their principal questions with respect to evaluation of QoS is usually (or by all means should be):

What does the difference between the value x for the QoS measure MP for the service we sell and the value y for a competing service really mean to users? Will it be noticeable? Will any noticeable differences be great enough to alter the users' synthesis of their experience to produce an assessment of perceived QoS?

(4) Service operations and maintenance personnel. Standing right behind the sales and marketing personnel, usually cursing them for creating unrealistic customer expectations of QoS, are the service provider's operations and maintenance personnel, who are responsible for monitoring day-to-day performance of the systems that deliver the service to assure that QoS is maintained at acceptable levels. Because they must be able to understand and act on QoS via actions taken on those aspects of operations that are within their control, their focus is necessarily on intrinsic quality of service, and their principal questions with respect to measurement and evaluation of QoS will be ones of the relationship between measures of intrinsic and perceived QoS of the form:

What values of the measure of intrinsic QoS, M;, will indicate likely user dissatisfaction with the perceived quality of the attributes of service of concern to users affected by the characteristic of operational system performance measured by Mi?

Analysis of perceived QoS, then, will be largely worthless to operations and maintenance decision-makers unless the evaluation of the results is extended to produce derived indicators of specific conditions that must be corrected in order to avoid deleterious effects on the service users' assessment of perceived quality.

(5) System architects and engineers. Last on our list of possible consumers of QoS measurement and evaluation are the persons who must make the decisions as to the technology to be employed in implementing various telecommunications services and the way various assets are to be configured to deliver particular services. Like operations and maintenance personnel, the system architects and engineers are concerned with intrinsic quality. Unlike operations and maintenance personnel, who are constrained to manage performance within the constraints of the existing system and resources, the architects and engineers are responsible for deciding the characteristics of the telecommunications system and the allocation of resources that will achieve intrinsic quality adequate to assure a high likelihood that perceived quality will be acceptable. To do this, they must have hard and fast requirements that can be used as the basis of system design and configuration. Notions of subjectivity and perception must be totally factored out of the equations, and the fuzzy indicators that might be used for operations and maintenance management must be replaced by criteria for acceptability of variations of intrinsic quality that are technical, concrete, specific and completely unambiguous. The need for such criteria, then, generates questions of the form:

What value, x, of the measure of intrinsic QoS, M;, is an upper/lower limit for what must be achieved in the system design to assure the ability to deliver acceptable perceived QoS?

Table of Contents

Preface.

Foreword.

Introduction.

BASIC CONCEPTS.

Definitions.

Measurement and Evaluation.

The Analysis Process.

Telecommunications Concepts.

EVALUATIVE CONCEPTS, MEASURES, AND QUANTIFIERS.

Overview.

Accessibility.

Routing Speed.

Connection Reliability.

Routing Reliability.

Connection Quality -
Voice.

Connection Quality -
Data.

Connection Continuity.

Disconnection Reliability.

The Other Stuff.

Afterword.

Appendix A.

Appendix B.

Appendix C.

Abbreviations.

Index.

Foreword

My involvement in analysis of quality of telecommunications services began almost by accident in June, 1967, when I started my first full-time job out of graduate school. The job was with the Operations Evaluation Group of the Center for Naval Analyses. It seems that what they happened to need the day I reported was someone to fill a slot as a communications analyst. Since I was there, I was anointed, never mind that I knew absolutely nothing about telecommunications systems, electrical engineering, or even electricity, since I had skipped that part of the college physics curriculum, and almost nothing of my graduate education in mathematics was relevant to understanding Navy tactical voice and teletype communications over radio frequency channels.

Because my career started with such a complete lack of practical experience and technical skills, my analytical efforts have never been marred or impeded by technical expertise or conventional wisdom. Rather, what I discovered was that all I really needed to do to be effective as a problem solver in this area was to:

• Imagine myself using the system I was studying;
• Decide what I would be concerned about if I were using it;
• Research the technology of the system to the extent necessary to under stand the mechanisms affecting performance of the system with respect to those concerns; and
• Formalize the relationships between system performance and user percep tion of quality of service gleaned from this drill.

When I did this, everything else needed to solve the problem would readily follow - the user view would suggest concerns; concerns would suggest measures of quality and effectiveness; understanding of the mechanisms would suggest measures of performance and their relationship to measures of quality; measures would suggest quantifiers; quantifiers would suggest data requirements; and so on, all the way down the analytical chain.

This book is based on more than 30 years experience in successfully applying this approach in analyzing issues of quality of service of telecommunications systems to produce practicable solutions to quality problems. Because of the very basic nature of the approach, this book is apt to be viewed by some as being short on technical content and long on formulation of evaluative concepts and generic measures. However, I refuse to apologize for this, because the perspectives on quality of telecommunications services that I am trying to lay out here are exactly those that I would want all of my employees to share, were I ever to become the CEO of a telecommunications company, so that, for example:

• My marketing and sales forces would know how to communicate with customers in a way that would demonstrate their understanding of customers' concerns;
• My system engineers would know how to design my networks to satisfy customer expectations, rather than simply meet industry design standards;
• My operations managers would know the comfortable levels of performance affecting quality of services that must be achieved and maintained to assure user satisfaction;
• My service technicians would know how to troubleshoot user complaints with the same competence that they identify, diagnose, and correct technical problems; and
• Everyone involved anywhere in the company would have a very good idea of exactly how their day-to-day activities affect user perception of the quality of our services.

To this end, what I have tried to present here is a treatise on the ways and means of measuring and evaluating telecommunications services that is simple and straightforward enough to be appreciated by anyone, but sophisticated enough to be informative and useful to telecommunications professionals. The only way you can judge whether I have succeeded is to turn the page...

William C. Hardy
WorldCom, USA

Editorial Reviews

"...provides a straightforward and very accessible approach to measurement and evaluation of QoS in telecommunications networks...strongly recommended for all people, either experienced professionals or graduates, involved in the area of networking..." (IEEE Communications Magazine, Vol. 40, No. 22, February 2002)

From the Publisher