With the rapid progress of deep neural models and the explosion of data resources, dialogue systems that supports extensive topics and chit-chat conversations are emerging in natural language processing (NLP), information retrieval (IR), and machine learning (ML). To facilitate the development of both retrieval-based chit-chat systems and IR tasks supported by them, the authors survey chit-chat systems from two perspectives: (1) techniques to build chit-chat systems, and (2) chit-chat components in completing IR tasks. The main contributions of this survey are: surveying the deep neural models; connecting the recently resurgent chit-chat systems and task-oriented system; introducing various solutions for challenges from different perspectives, including dataside and model-side solutions and utilization of extra resources; presenting data resources and evaluation methods for building retrieval-based and generation-based chit-chat systems. The authors also analyze the main challenges, possible new exploration directions and rising trends, which will shed light on building human-like systems. This survey is intended to bridge the researchers of IR and the NLP community to move chit-chat systems forward and support more IR tasks. It will be of interest to IR or NLP researchers who want to study chit-chat from different perspectives, IR researchers who need to complete their tasks with the assistance of chit-chat systems, engineers with hands-on experience in building these systems to leverage advanced chit-chat modeling techniques, or anyone who wants keep up with the frontier of chit-chat systems or learn how to build them with deep neural architectures.