This paper proposes an internal memory optimization method for the spatial scalable inter-layer prediction. The spatial scalable prediction of the H.264 Scalable Video Coding (SVC) has additional inter-layer predictions. The up-scale processing of the inter-layer prediction consists of the Wiener interpolation filter and the deblocking filter. They need internal memories for the pixel buffering. Thus, they need memory optimizing for an efficient hardware implementation. In addition, the SVC motion prediction requires a significant amount of motion vector from the previous layer prediction and an additional interpolation process. These motion vectors cause the internal memory size to increase which may result in increasing hardware cost and power consumption. To optimize internal memories, this paper proposes a memory optimized architecture for the deblocking filter and a motion vector bit compression scheme of the inter-layer motion prediction. The proposed architecture can control the internal memory easily and the proposed compression scheme reduces motion vector storing area by about 66.49% with less change in hardware size.