Spectrally efficient modulation formats can be used to overcome the problems associated with limited channels and bandwidth of Dense Wavelength Division Multiplexing (DWDM) optical systems. Multilevel modulation formats are considered spectrally efficient and can double the transmission capacity by transmitting more information in the amplitude, phase, polarization or a combination of all. Here we have studied and evaluated different multilevel modulation formats in a practical optical transmission system. Such modulation formats are Differential Quadrature Phase Shift Keying (DQPSK), DQPSK with Polarization Division Multiplex (PolDM), Quaternary Intensity Modulation (4-IM, QASK), Quaternary Duobinary (QDB), and Quaternary Polarization Duobinary (QPolDB). The QDB and QPolDB quaternary signals are attractive for DWDM systems applied for short reach (several kilometers) to medium reach (several hundred kilometers) transmission applications (metro applications). However, for long or ultra long haul optical fiber transmission systems, DQPSK and DQPSK PolDM modulation formats featuring better receiver sensitivities are potential candidates.