Tip #5 - Measuring performance on sub-groups

Liran Nahum
September 17, 2020

💡 Tip a low performance on a sub-group doesn’t mean a bad model

In many cases, you have a general model but different segments of interest that you want to measure (VIP users / different marketing channels / etc.), or you are looking for underperforming sub-groups. 

In all of these cases, you want to remember that low performance scores for sub-groups don’t necessarily mean that these groups are bad or they reduce the overall performance of the model! Some metrics are not simply additive or not additive at all, like AUC.

In this sense, group performance may look bad when considered “by itself”, but it contributes to the overall performance, when you look at the whole population.

Let’s say your model estimates the Lifetime Value (LTV) of your users, and you have two groups of interests: group_1 and group_2, and your metric is “Mean Error”.

  • group_1 mean error is +1000$
  • group_2 mean error is -1000$

Each of these groups seem to perform badly, but these two groups combined are having a perfect score of zero mean error!

Recommended Readings: