Functional near-infrared spectroscopy (fNIRS) is a powerful clinical tool for monitoring hemoglobin concentration in brain tissues by analyzing absorption of scattered light. Since human brain is composed of multilayers including scalp, skull, and cerebral cortex, fNIRS signals need be analyzed with a multilayer tissue model. However, retrieving the optical properties of a multilayer tissue is often difficult because nonlinear fitting of absorption parameters from a scattered light signal by a tissue is ill-posed especially when the signal level is low. In this paper we introduce the cost function based masking technique for effective error minimization in the nonlinear fitting of fNIRS signals. We have shown that this method effectively reduces the influences of measurement errors with a newly defined cost function. Numerically simulated fNIRS data were generated for a two-layered tissue model and are used to extract the optical parameters of the two-layered tissue model. Accuracies of extracted parameters were compared with and without our proposed cost function.