Orthogonal Frequency Division Multiplexing (OFDM) is a multicarrier modulation technique which is used for supporting high data rates. In this modulation scheme, Fourier transform is used to provide orthogonality between subcarriers. Since the channel is multi-path, a cyclic prefix having a length greater than channel delay spread is added to each OFDM symbol. The cyclic prefix removes the inter symbol interference, but, since it does not carry useful information, it decreases spectrum efficiency. In wavelet-based OFDM, there is no need for cyclic prefix. Wavelets have suitable localization, finite length and orthogonality both in time and frequency domains. Therefore, they could be used to generate suitable waveform for transmission through a fading channel. In this paper, a comparison has been made between performances of the Fourier transform and orthogonal wavelet families in OFDM-based systems in order to improve. Also, in order to select optimum filter order in these systems, the effect of increasing filter order is investigated. Simulation results show that wavelet transforms outperform the Fourier transform in terms of bit error rate (BER) and system efficiency and wavelet transforms with smaller filter orders should be used.