Machine Learning-Based Analysis of Test Results

Abstract

A comprehensive understanding of the overall performance of the tests is critical in software testing to make necessary adjustments to the test schedule or conduct targeted investigations. In ABB Bryne, the software testing is processed by a test system named NAST for the IPS software for paint robots. This thesis addresses the critical challenges by investigating two key aspects: ranking frequently failing test cases and clustering test cases based on error messages.

Two models were employed for the ranking task: the conventional Analytic Hierarchy Process-Weighted Sum Model (AHP-WSM) and the Learning-to-rank model LambdaMART. The results highlight the superiority of the AHP-WSM model over LambdaMART, attributed to the influence of dataset quality and size in the machine learning technique.

Agglomerative clustering with Jaccard and cosine similarity metrics was applied for the clustering. The findings reveal that cosine similarity yielded superior outcomes due to its calculation characteristics to capture semantic relationships between texts based on overall content rather than relying on exact term matches.

This research provides valuable insights into effective methodologies for addressing the complexities associated with test case prioritisation and clustering in software testing.