Sports video analysis and reconstruction, especially soccer, has received much of the attention due to its popularity as well as potential commercial value. A broad range of applications such as: content retrieval and indexing, semantic event highlight and summarization or objects recognition and tracking make building sports video analysis system a hot research area. By knowing the positions and movement trajectories of the players on the field, professionals could detect semantic events, analyze the teams tactics, strength and weakness as well as produce meaningful highlights and summarizations. The analysis of soccer video is complex due to its nature of congested, rapidly changing, and non-linear fashion of each player's movements. In this work, we present a soccer video analysis system for the Video-to-cartoon applications. The system is capable of performing detection from single and multiple camera views with robustness and efficiency, as well as generates the 3D virtual reality of the scene so users can interactively change the viewpoints while watching the match. Novel algorithms and techniques for building the system will be discussed in detail.