A key requirement for calibrating a supervised machine learning model is the accurate identification of the target variable. For a Customer Due Diligence (CDD) model, the goal is to predict the risk rating (Low, Medium, or High) of a client file, which correlates with the perceived risk of Money Laundering and Terrorism Financing (AML risk). This task is challenging due to the absence of an observable ground truth, low agreement among Subject Matter Experts, and the presence of partial ordering in the data (input variables are typically discrete, hand-crafted rules, leading to a finite number of possible combinations, called profiles). To address these challenges, we developed a novel three-phase labelling methodology that leverages group decision-making to improve interrater agreement. Consequently, we investigated and mitigated psychological biases introduced by group decision-making. Additionally, we utilized partial ordering to compare different profiles, treating it as a strength rather than a weakness. This results in a mixed approach that combines elements of both absolute and relative labelling methodologies. Finally, understanding how the data structure leads to a label can inform the modelling process by constraining the optimization to respect the partial ordering. The results show a marked performance improvement in the calibrated model, compared to a more naïve labelling methodology previously employed.