ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations

Sheo Yon Jhin; Minju Jo; Taeyong Kong; Jinsung Jeon; Noseong Park

doi:10.1145/3447548.3467419

ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations

Sheo Yon Jhin, Minju Jo, Taeyong Kong, Jinsung Jeon, Noseong Park

Graduate School

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

Neural ordinary differential equations (NODEs) presented a new paradigm to construct (continuous-time) neural networks. While showing several good characteristics in terms of the number of parameters and the flexibility in constructing neural networks, they also have a couple of well-known limitations: i) theoretically NODEs learn homeomorphic mapping functions only, and ii) sometimes NODEs show numerical instability in solving integral problems. To handle this, many enhancements have been proposed. To our knowledge, however, integrating attention into NODEs has been overlooked for a while. To this end, we present a novel method of attentive dual co-evolving NODE (ACE-NODE): one main NODE for a downstream machine learning task and the other for providing attention to the main NODE. Our ACE-NODE supports both pairwise and elementwise attention. In our experiments, our method outperforms existing NODE-based and non-NODE-based baselines in almost all cases by non-trivial margins.

Original language	English
Title of host publication	KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher	Association for Computing Machinery
Pages	736-745
Number of pages	10
ISBN (Electronic)	9781450383325
DOIs	https://doi.org/10.1145/3447548.3467419
Publication status	Published - 2021 Aug 14
Event	27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021 - Virtual, Online, Singapore Duration: 2021 Aug 14 → 2021 Aug 18

Publication series

Name	Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Conference

Conference	27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021
Country/Territory	Singapore
City	Virtual, Online
Period	21/8/14 → 21/8/18

Bibliographical note

Funding Information:
Noseong Park is the corresponding author. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University)).

Publisher Copyright:
© 2021 ACM.

All Science Journal Classification (ASJC) codes

Software
Information Systems

Access to Document

10.1145/3447548.3467419

Cite this

Jhin, S. Y., Jo, M., Kong, T., Jeon, J., & Park, N. (2021). ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations. In KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 736-745). (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). Association for Computing Machinery. https://doi.org/10.1145/3447548.3467419

Jhin, Sheo Yon ; Jo, Minju ; Kong, Taeyong et al. / ACE-NODE : Attentive Co-Evolving Neural Ordinary Differential Equations. KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2021. pp. 736-745 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

@inproceedings{c240fa6b376d474b868b633cd352a6e2,

title = "ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations",

abstract = "Neural ordinary differential equations (NODEs) presented a new paradigm to construct (continuous-time) neural networks. While showing several good characteristics in terms of the number of parameters and the flexibility in constructing neural networks, they also have a couple of well-known limitations: i) theoretically NODEs learn homeomorphic mapping functions only, and ii) sometimes NODEs show numerical instability in solving integral problems. To handle this, many enhancements have been proposed. To our knowledge, however, integrating attention into NODEs has been overlooked for a while. To this end, we present a novel method of attentive dual co-evolving NODE (ACE-NODE): one main NODE for a downstream machine learning task and the other for providing attention to the main NODE. Our ACE-NODE supports both pairwise and elementwise attention. In our experiments, our method outperforms existing NODE-based and non-NODE-based baselines in almost all cases by non-trivial margins.",

author = "Jhin, {Sheo Yon} and Minju Jo and Taeyong Kong and Jinsung Jeon and Noseong Park",

note = "Funding Information: Noseong Park is the corresponding author. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University)). Publisher Copyright: {\textcopyright} 2021 ACM.; 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021 ; Conference date: 14-08-2021 Through 18-08-2021",

year = "2021",

month = aug,

day = "14",

doi = "10.1145/3447548.3467419",

language = "English",

series = "Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

publisher = "Association for Computing Machinery",

pages = "736--745",

booktitle = "KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining",

}

Jhin, SY, Jo, M, Kong, T, Jeon, J & Park, N 2021, ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations. in KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, pp. 736-745, 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021, Virtual, Online, Singapore, 21/8/14. https://doi.org/10.1145/3447548.3467419

ACE-NODE : Attentive Co-Evolving Neural Ordinary Differential Equations. / Jhin, Sheo Yon; Jo, Minju; Kong, Taeyong et al.

KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2021. p. 736-745 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - ACE-NODE

T2 - 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021

AU - Jhin, Sheo Yon

AU - Jo, Minju

AU - Kong, Taeyong

AU - Jeon, Jinsung

AU - Park, Noseong

N1 - Funding Information: Noseong Park is the corresponding author. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University)). Publisher Copyright: © 2021 ACM.

PY - 2021/8/14

Y1 - 2021/8/14

N2 - Neural ordinary differential equations (NODEs) presented a new paradigm to construct (continuous-time) neural networks. While showing several good characteristics in terms of the number of parameters and the flexibility in constructing neural networks, they also have a couple of well-known limitations: i) theoretically NODEs learn homeomorphic mapping functions only, and ii) sometimes NODEs show numerical instability in solving integral problems. To handle this, many enhancements have been proposed. To our knowledge, however, integrating attention into NODEs has been overlooked for a while. To this end, we present a novel method of attentive dual co-evolving NODE (ACE-NODE): one main NODE for a downstream machine learning task and the other for providing attention to the main NODE. Our ACE-NODE supports both pairwise and elementwise attention. In our experiments, our method outperforms existing NODE-based and non-NODE-based baselines in almost all cases by non-trivial margins.

AB - Neural ordinary differential equations (NODEs) presented a new paradigm to construct (continuous-time) neural networks. While showing several good characteristics in terms of the number of parameters and the flexibility in constructing neural networks, they also have a couple of well-known limitations: i) theoretically NODEs learn homeomorphic mapping functions only, and ii) sometimes NODEs show numerical instability in solving integral problems. To handle this, many enhancements have been proposed. To our knowledge, however, integrating attention into NODEs has been overlooked for a while. To this end, we present a novel method of attentive dual co-evolving NODE (ACE-NODE): one main NODE for a downstream machine learning task and the other for providing attention to the main NODE. Our ACE-NODE supports both pairwise and elementwise attention. In our experiments, our method outperforms existing NODE-based and non-NODE-based baselines in almost all cases by non-trivial margins.

UR - http://www.scopus.com/inward/record.url?scp=85114910068&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85114910068&partnerID=8YFLogxK

U2 - 10.1145/3447548.3467419

DO - 10.1145/3447548.3467419

M3 - Conference contribution

AN - SCOPUS:85114910068

T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

SP - 736

EP - 745

BT - KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

PB - Association for Computing Machinery

Y2 - 14 August 2021 through 18 August 2021

ER -

Jhin SY, Jo M, Kong T, Jeon J, Park N. ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations. In KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery. 2021. p. 736-745. (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). doi: 10.1145/3447548.3467419

ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this