Fair-VQA: Fairness-Aware Visual Question Answering through Sensitive Attribute Prediction

Sungho Park; Sunhee Hwang; Jongkwang Hong; Hyeran Byun

doi:10.1109/ACCESS.2020.3041503

Fair-VQA: Fairness-Aware Visual Question Answering through Sensitive Attribute Prediction

Sungho Park, Sunhee Hwang, Jongkwang Hong, Hyeran Byun

College of Computing

Research output: Contribution to journal › Article › peer-review

2 Citations (Scopus)

Abstract

Visual Question Answering (VQA) is a task that answers questions on given images. Although previous works achieve a great improvement in VQA performance, they do not consider the fairness of answers in terms of ethically sensitive attributes, such as gender. Therefore, we propose a Fair-VQA model that contains two modules: VQA module and SAP (Sensitive Attribute Prediction) module. On top of VQA module, which predicts various kinds of answers, SAP module predicts only sensitive attributes using the same inputs. The predictions of SAP module are utilized to rectify answers from VQA module to be fairer in terms of the sensitive attributes with graceful performance degradation. To validate the proposed method, we conduct extensive experiments on VQA, GQA, and our proposing VQA-Gender datasets. In all the experiments, our method shows the fairest results in various metrics for fairness. Moreover, we demonstrate that our method works interpretably through the analysis of visualized attention maps.

Original language	English
Article number	9274341
Pages (from-to)	215091-215099
Number of pages	9
Journal	IEEE Access
Volume	8
DOIs	https://doi.org/10.1109/ACCESS.2020.3041503
Publication status	Published - 2020

Bibliographical note

Funding Information:
This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIT) under Grant 2019R1A2C2003760, and in part by the Institute for Information & Communications Technology Planning & Evaluation (IITP) Grant funded by the Korean Government (MSIT) (Development of framework for analyzing, detecting, mitigating of bias in AI model, and training data) (Artificial Intelligence Graduate School Program (Yonsei University)) under Grant 2019-0-01396 and Grant 2020-0-01361.

Publisher Copyright:
© 2013 IEEE.

All Science Journal Classification (ASJC) codes

Computer Science(all)
Materials Science(all)
Engineering(all)
Electrical and Electronic Engineering

Access to Document

10.1109/ACCESS.2020.3041503

Cite this

@article{ea043f312c664779a45f4c0710ac07d1,

title = "Fair-VQA: Fairness-Aware Visual Question Answering through Sensitive Attribute Prediction",

abstract = "Visual Question Answering (VQA) is a task that answers questions on given images. Although previous works achieve a great improvement in VQA performance, they do not consider the fairness of answers in terms of ethically sensitive attributes, such as gender. Therefore, we propose a Fair-VQA model that contains two modules: VQA module and SAP (Sensitive Attribute Prediction) module. On top of VQA module, which predicts various kinds of answers, SAP module predicts only sensitive attributes using the same inputs. The predictions of SAP module are utilized to rectify answers from VQA module to be fairer in terms of the sensitive attributes with graceful performance degradation. To validate the proposed method, we conduct extensive experiments on VQA, GQA, and our proposing VQA-Gender datasets. In all the experiments, our method shows the fairest results in various metrics for fairness. Moreover, we demonstrate that our method works interpretably through the analysis of visualized attention maps.",

author = "Sungho Park and Sunhee Hwang and Jongkwang Hong and Hyeran Byun",

note = "Funding Information: This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIT) under Grant 2019R1A2C2003760, and in part by the Institute for Information & Communications Technology Planning & Evaluation (IITP) Grant funded by the Korean Government (MSIT) (Development of framework for analyzing, detecting, mitigating of bias in AI model, and training data) (Artificial Intelligence Graduate School Program (Yonsei University)) under Grant 2019-0-01396 and Grant 2020-0-01361. Publisher Copyright: {\textcopyright} 2013 IEEE.",

year = "2020",

doi = "10.1109/ACCESS.2020.3041503",

language = "English",

volume = "8",

pages = "215091--215099",

journal = "IEEE Access",

issn = "2169-3536",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - Fair-VQA

T2 - Fairness-Aware Visual Question Answering through Sensitive Attribute Prediction

AU - Park, Sungho

AU - Hwang, Sunhee

AU - Hong, Jongkwang

AU - Byun, Hyeran

N1 - Funding Information: This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIT) under Grant 2019R1A2C2003760, and in part by the Institute for Information & Communications Technology Planning & Evaluation (IITP) Grant funded by the Korean Government (MSIT) (Development of framework for analyzing, detecting, mitigating of bias in AI model, and training data) (Artificial Intelligence Graduate School Program (Yonsei University)) under Grant 2019-0-01396 and Grant 2020-0-01361. Publisher Copyright: © 2013 IEEE.

PY - 2020

Y1 - 2020

N2 - Visual Question Answering (VQA) is a task that answers questions on given images. Although previous works achieve a great improvement in VQA performance, they do not consider the fairness of answers in terms of ethically sensitive attributes, such as gender. Therefore, we propose a Fair-VQA model that contains two modules: VQA module and SAP (Sensitive Attribute Prediction) module. On top of VQA module, which predicts various kinds of answers, SAP module predicts only sensitive attributes using the same inputs. The predictions of SAP module are utilized to rectify answers from VQA module to be fairer in terms of the sensitive attributes with graceful performance degradation. To validate the proposed method, we conduct extensive experiments on VQA, GQA, and our proposing VQA-Gender datasets. In all the experiments, our method shows the fairest results in various metrics for fairness. Moreover, we demonstrate that our method works interpretably through the analysis of visualized attention maps.

AB - Visual Question Answering (VQA) is a task that answers questions on given images. Although previous works achieve a great improvement in VQA performance, they do not consider the fairness of answers in terms of ethically sensitive attributes, such as gender. Therefore, we propose a Fair-VQA model that contains two modules: VQA module and SAP (Sensitive Attribute Prediction) module. On top of VQA module, which predicts various kinds of answers, SAP module predicts only sensitive attributes using the same inputs. The predictions of SAP module are utilized to rectify answers from VQA module to be fairer in terms of the sensitive attributes with graceful performance degradation. To validate the proposed method, we conduct extensive experiments on VQA, GQA, and our proposing VQA-Gender datasets. In all the experiments, our method shows the fairest results in various metrics for fairness. Moreover, we demonstrate that our method works interpretably through the analysis of visualized attention maps.

UR - http://www.scopus.com/inward/record.url?scp=85097380331&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85097380331&partnerID=8YFLogxK

U2 - 10.1109/ACCESS.2020.3041503

DO - 10.1109/ACCESS.2020.3041503

M3 - Article

AN - SCOPUS:85097380331

SN - 2169-3536

VL - 8

SP - 215091

EP - 215099

JO - IEEE Access

JF - IEEE Access

M1 - 9274341

ER -

Fair-VQA: Fairness-Aware Visual Question Answering through Sensitive Attribute Prediction

Abstract

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this