Training a multilayer network with low-memory kernel-and-range projection

Huiping Zhuang, Zhiping Lin, Kar Ann Toh

Research output: Contribution to journalArticle

Abstract

Recently, a learning method based on the kernel and the range space projections has been proposed. This method has been applied to learn the multilayer network analytically with interpretable relationships among the weight matrices. However, the bulk matrix based formulation suffers from a high memory demand during network learning. In this study, a low-memory resolution is proposed to address the memory demanding problem. Essentially, the bulk matrix operations are implemented by a low-memory formulation in which only one training sample is processed at a time. Such a formulation is proved to be mathematically equivalent to the original batch learning version. We also point out that the rounding errors in computing systems could hinder the performance of the proposed formulation. This formulation is then robustified by introducing a regularization technique with the cost of an additional but negligible memory usage. Our experiments show that the proposed low-memory resolution can indeed tremendously reduce the memory consumption while maintaining reasonably good performances in both regression and classification tasks.

Original languageEnglish
Pages (from-to)522-550
Number of pages29
JournalJournal of the Franklin Institute
Volume357
Issue number1
DOIs
Publication statusPublished - 2020 Jan

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Signal Processing
  • Computer Networks and Communications
  • Applied Mathematics

Cite this