The commonly used Least Square (LS) ellipse fitting algorithm based on minimum algebraic distance is simple and easy to implement, but it has no choice to the sample points, which leads to the fitting results are easily inaccurate due to the error points. According to this case, an improved ellipse fitting algorithm based on Letts criterion was proposed to overcome the shortage of LS algorithm. Firstly, the ellipse was fitted from the fitting curve by using the LS ellipse fitting algorithm based on minimum algebraic distance. Then, the algebraic distance of ellipse fitted by LS algorithm from the point distance on the fitting curve was set as the fitting point set. After the point set was verified to be normal distribution, the points which were greater than|3 σ|were determined to be outliers and eliminated by using Letts criterion. Then the steps above were repeated until all points were within the scope of  [-3σ ,3σ ]. Finally, the best fitting ellipse was obtained. The simulation experiment results show that the fitting error of the improved algorithm based on Letts criterion is within 1.0%, and its fitting accuracy is improved by at least 2 percentage points compared with the LS algorithm under the same condition. The simulation result and the practical application in roundness measurement of cigarette verify the effectiveness of the improved algorithm.