An AI developer notices that the labeling guidelines changed during the project and asks an...

When labeling guidelines change , there is a risk of label shift —inconsistency between older and newer labels. The BEST recommendation is to relabel stratified samples across the dataset to estimate the extent of the shift and adjust datasets accordingly (D). This allows the team to quantify inconsistencies, align labels to the updated guidelines, and ensure that retraining reflects a coherent labeling policy. AAIA emphasizes managing data and label quality as core aspects of AI operations.

Adding layers (A) or training a larger model (C) does not solve inconsistent labeling; it may even entrench confusion. Increasing class weights on recent data only (B) can bias the model and does not repair inconsistent historical labels. Therefore, relabeling structured, representative samples to assess and correct label shift is the most appropriate and audit-aligned recommendation.