Abstract: Addressing the issue of nonlinearity correction for homodyne interference signals in vibration calibration, this paper describes an improved ellipse fitting algorithm. Compared to the traditional Heydeman ellipse correction method based on least squares, the proposed method makes two improvements. First, it adds constraints on the ellipse parameters in the fitted conic equation. Second, it transforms the conventional parameter calculation based on the minimum distance criterion into solving the eigenvalues of the data matrix. The proposed method ensures that even when the Lissajous figure formed by the homodyne interference signal is only a partial ellipse, a complete ellipse can still be fitted. This extends the working frequency range of the homodyne interferometer in vibration calibration while improving the measurement accuracy.