There is a lot of noise interference in the welding image,and processing it is the premise of weld recognition. To deal with the image denoising problem of passive visual weld tracking system,a denoising method based on inter-frame matching is proposed. Through combining the ORB（Oriented FAST and Rotated BRIEF）algorithm with the RANSAC（Random Sample Consensus）algorithm,the single response matrix between the two frames of images is obtained,and the welding parts in the two frames are aligned by this method. By using the gray level replacement of the image between frames,the spatter area is replaced by the welding piece image to obtain the weld image without spatter. Aiming at the interference of the straight line of the edge of the welding piece in the fillet welding,three straight line fitting algorithms, least square method,hough transform and RANSAC,are compared. The results show that the RANSAC algorithm can better resist the interference of a large number of outliers and the precision can be up to 2 pixels,which can meet the requirement of weld tracking.