To date, the literature on quantile regression and least absolute deviation regression has assumed either explicitly or implicitly that the conditional quantile regression model is correctly specified. When the model is misspecified, confidence intervals and hypothesis tests based on the conventional covariance matrix are invalid. Although misspecification is a generic phenomenon and correct specification is rare in reality, there has to date been no theory proposed for inference when a conditional quantile model may be misspecified. In this paper, we allow for possible misspecification of a linear conditional quantile regression model. We obtain consistency of the quantile estimator for certain "pseudo-true" parameter values and asymptotic normality of the quantile estimator when the model is misspecified. In this case, the asymptotic covariance matrix has a novel form, not seen in earlier work, and we provide a consistent estimator of the asymptotic covariance matrix. We also propose a quick and simple test for conditional quantile misspecification based on the quantile residuals.