Correction of building height effects is a critical step in image interpretation from aerial imagery in urban area. In this paper, an efficient scheme to correct building height effects from aerial color imagery using multi source data sets is presented. The following steps have been performed to remove the shadow effect from aerial color imagery. First, the shadow regions of the orthorectified aerial color image are precisely located using the solar position and the heights of ground objects derived from LIDAR (Light Detection and Ranging) data. The shadow area is composed of many different ground features. To accurately recover the original spectral information the step for accurate segmentation of shadow regions needed. This step has been performed by utilizing the existing digital map which contains surface cover information. To assign correction information corresponding to the surface features on the shadowed area, correction factor needed to be modeled. For this three basic assumptions are proposed and comparison between the context region and the same non-shadowed context region is made. Finally, the shadow-effect-corrected image is generated by using the correction factor obtained from the previous step.