This paper considers visual feature selection and its regression to estimate the position of a vehicle using an omnidirectional camera. The Gaussian process (GP)-based localization builds on a maximum likelihood estimation (MLE) with a GP regression from optimally selected visual features. In particular, the collection of selected features over a surveillance region is modeled by a multivariate GP with unknown hyperparameters. The hyperparameters are identified through the learning process as the corresponding MLEs and they are used for prediction in an empirical Bayes fashion. To select features, we apply a backward sequential elimination technique in order to improve the quality of the position estimation with reduced number of features for efficient GP-based localization. The excellent results of the proposed algorithm from the real-world outdoor experimental study are illustrated using different visual features.