Training a multilayer network with low-memory kernel-and-range projection

Recently, a learning method based on the kernel and the range space projections has been proposed. This method has been applied to learn the multilayer network analytically with interpretable relationships among the weight matrices. However, the bulk matrix based formulation suffers from a high memory demand during network learning. In this study, a low-memory resolution is proposed to address the memory demanding problem. Essentially, the bulk matrix operations are implemented by a low-memory formulation in which only one training sample is processed at a time. Such a formulation is proved to be mathematically equivalent to the original batch learning version. We also point out that the rounding errors in computing systems could hinder the performance of the proposed formulation. This formulation is then robustified by introducing a regularization technique with the cost of an additional but negligible memory usage. Our experiments show that the proposed low-memory resolution can indeed tremendously reduce the memory consumption while maintaining reasonably good performances in both regression and classification tasks.