Aiming at the defects of traditional variable focal length relative orientation algorithm, such as strong dependence on initial value and poor convergence, the paper proposes a variable focal length relative orientation method based on p-h algorithm. In order to solve the defects caused by rotation matrix formed by trigonometric function, this method uses unit quaternion instead of trigonometric function to describe rotation matrix, solving algorithm parameters iteratively according to the least square principle. New method is optimized by p-h algorithm, and it established the corresponding solution model. The experimental results showed that the improved algorithm has the following characteristics: initial value weak dependence, fast convergence, adaptability for large angle, which solved multiple values, singularity problems.