Ambiguity-based evaluation of objective quality metrics for image compression

Manri Cheon, Jong-Seok Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

While results of subjective quality assessment are represented by mean opinion scores and corresponding confidence intervals, the output of an objective quality metric for a given stimulus is only a single estimated quality level. Accordingly, the performance of a metric is evaluated by measuring the accuracy of its outputs with respect to the corresponding subjective scores. However, the concept of the ambiguity interval for objective quality has been raised recently. In this paper, we propose to consider not only the accuracy but also the ambiguity of objective quality metrics for performance evaluation. In particular, we conduct benchmarking of the seven state-of-the-art image quality metrics for images compressed with JPEG and JPEG2000. It is demonstrated that the best metric in terms of accuracy may not be the best in terms of ambiguity.

Original languageEnglish
Title of host publication2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509003549
DOIs
Publication statusPublished - 2016 Jun 23
Event8th International Conference on Quality of Multimedia Experience, QoMEX 2016 - Lisbon, Portugal
Duration: 2016 Jun 62016 Jun 8

Publication series

Name2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016

Other

Other8th International Conference on Quality of Multimedia Experience, QoMEX 2016
CountryPortugal
CityLisbon
Period16/6/616/6/8

Fingerprint

Benchmarking
Image compression
Image quality
evaluation
benchmarking
performance
stimulus
confidence

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Media Technology
  • Communication

Cite this

Cheon, M., & Lee, J-S. (2016). Ambiguity-based evaluation of objective quality metrics for image compression. In 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016 [7498944] (2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/QoMEX.2016.7498944
Cheon, Manri ; Lee, Jong-Seok. / Ambiguity-based evaluation of objective quality metrics for image compression. 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016. Institute of Electrical and Electronics Engineers Inc., 2016. (2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016).
@inproceedings{251564dda73540b4bede9beaafd0f259,
title = "Ambiguity-based evaluation of objective quality metrics for image compression",
abstract = "While results of subjective quality assessment are represented by mean opinion scores and corresponding confidence intervals, the output of an objective quality metric for a given stimulus is only a single estimated quality level. Accordingly, the performance of a metric is evaluated by measuring the accuracy of its outputs with respect to the corresponding subjective scores. However, the concept of the ambiguity interval for objective quality has been raised recently. In this paper, we propose to consider not only the accuracy but also the ambiguity of objective quality metrics for performance evaluation. In particular, we conduct benchmarking of the seven state-of-the-art image quality metrics for images compressed with JPEG and JPEG2000. It is demonstrated that the best metric in terms of accuracy may not be the best in terms of ambiguity.",
author = "Manri Cheon and Jong-Seok Lee",
year = "2016",
month = "6",
day = "23",
doi = "10.1109/QoMEX.2016.7498944",
language = "English",
series = "2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016",
address = "United States",

}

Cheon, M & Lee, J-S 2016, Ambiguity-based evaluation of objective quality metrics for image compression. in 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016., 7498944, 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016, Institute of Electrical and Electronics Engineers Inc., 8th International Conference on Quality of Multimedia Experience, QoMEX 2016, Lisbon, Portugal, 16/6/6. https://doi.org/10.1109/QoMEX.2016.7498944

Ambiguity-based evaluation of objective quality metrics for image compression. / Cheon, Manri; Lee, Jong-Seok.

2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016. Institute of Electrical and Electronics Engineers Inc., 2016. 7498944 (2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Ambiguity-based evaluation of objective quality metrics for image compression

AU - Cheon, Manri

AU - Lee, Jong-Seok

PY - 2016/6/23

Y1 - 2016/6/23

N2 - While results of subjective quality assessment are represented by mean opinion scores and corresponding confidence intervals, the output of an objective quality metric for a given stimulus is only a single estimated quality level. Accordingly, the performance of a metric is evaluated by measuring the accuracy of its outputs with respect to the corresponding subjective scores. However, the concept of the ambiguity interval for objective quality has been raised recently. In this paper, we propose to consider not only the accuracy but also the ambiguity of objective quality metrics for performance evaluation. In particular, we conduct benchmarking of the seven state-of-the-art image quality metrics for images compressed with JPEG and JPEG2000. It is demonstrated that the best metric in terms of accuracy may not be the best in terms of ambiguity.

AB - While results of subjective quality assessment are represented by mean opinion scores and corresponding confidence intervals, the output of an objective quality metric for a given stimulus is only a single estimated quality level. Accordingly, the performance of a metric is evaluated by measuring the accuracy of its outputs with respect to the corresponding subjective scores. However, the concept of the ambiguity interval for objective quality has been raised recently. In this paper, we propose to consider not only the accuracy but also the ambiguity of objective quality metrics for performance evaluation. In particular, we conduct benchmarking of the seven state-of-the-art image quality metrics for images compressed with JPEG and JPEG2000. It is demonstrated that the best metric in terms of accuracy may not be the best in terms of ambiguity.

UR - http://www.scopus.com/inward/record.url?scp=84979665819&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84979665819&partnerID=8YFLogxK

U2 - 10.1109/QoMEX.2016.7498944

DO - 10.1109/QoMEX.2016.7498944

M3 - Conference contribution

T3 - 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016

BT - 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Cheon M, Lee J-S. Ambiguity-based evaluation of objective quality metrics for image compression. In 2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016. Institute of Electrical and Electronics Engineers Inc. 2016. 7498944. (2016 8th International Conference on Quality of Multimedia Experience, QoMEX 2016). https://doi.org/10.1109/QoMEX.2016.7498944