Optimal solution for 2 set of complimantary data

Working on image similarity between input image versus a set of known images based on their key features. Key features are measured by distance metric floating no. <1 . The smaller the distance, the more resemblance. Ideally the most similar images should have more matches of key features and smallest average distance . But is not happening due to false positives in distance.