With increasing digital video, the user wants to grasp contents of video efficiently. We need to reduce the size of video without losing contents to satisfy with such request. In this paper, we propose the method that infers contents of video by using color and speech information of video and makes a summary automatically. Our targets for the summarization is video including consecutive story such as movie, not video such as TV news program which includes relatively clear topic boundaries. We use color frequency histograms of images as color information and Closed Caption as speech information. Generated summaries keep most of contents of original video, and compression rates of the summary are between 10% and 30%.