In this research, layup sequence of composite laminates was optimized by using the response surface method and Genetic Algorithms (GA) in order to increase their buckling capacity. In order to define the buckling function of composite cylinder by using response surface method, several finite element models were run, and by using least square method, two functions, namely, first and second order polynomial functions were obtained. A genetic algorithm was employed to optimize buckling capacity of the composite cylinder by changing layup sequence of the laminates. Finally, after optimization by GA, MATLAB software was employed to obtain buckling capacity of the optimum layup. In addition, in order to confirm the accuracy of the developed optimization method in this research, buckling capacity of a rectangular composite plate was obtained by the present method and was compared with the results of the other researchers.