In satellite transponder design, both the group delay of demultiplexer and multiplexer filters and the AM/PM distortion due to the high power amplifier (HPA) nonlinearity are critical specifications, which should be in an appropriate range to ensure the required BER for having acceptable satellite service quality. Practical problems that inevitably arise in the manufacturing process of a transponder lead to a slight deviation in the fulfillment of the maximum allowable group delay and AM/PM distortion. In this paper, the acceptable range of the group delay and AM/PM distortion is investigated simultaneously so that the required BER of DVB-S2 service is guaranteed. Since there are no proper analytical relations to achieve this goal, these nonlinear effects are simulated in MATLAB SIMULINK for both the QPSK and 8PSK modulation schemes. It is inferred from the achieved results that the acceptable phase shift and the group delay of the 8PSK modulation is less than those of the QPSK modulation.