Categorical variables with lots of options (e.g., country of origin, occupation, postcodes) can be problematic when regression modelling; they have to be dummy coded and use many degrees of freedom, increasing the potential for model overfitting. The typical approaches to dealing with this are to:
- Discard the variable if it doesn’t appear it will be a good discriminator. It is sometimes hard to tell this up front when you have loads of categories.
- Roll the categories up into larger sets based on conceptual similarity. This can work for ordinal or geographic data, but is more difficult for purely nominal variables. There is also the risk that you’ll ‘average away’ some of the predictive value in the variable.
- Use a statistical technique (e.g., a decision tree) to work out groupings of categories based on their discriminative power. This may make for groupings that are hard to explain.
Another option I’ve recently come across is to convert the categorical variable to a metric-level variable using historic response data. For instance, say you’ve been collecting your customer’s postcodes for a while and are looking to employ this variable in a predictive model. Perhaps you are predicting response to a mailing offer (or something similar) which has been running for at least one learning cycle. A potential way to deal with the ‘too many categories’ problem would be to calculate the proportion of people contacted in each postcode during prior mailings who responded to the offer. Voilà! You’ve now got a metric level and continuous variable to play with. You can apply the historic response values to any new prospects you are looking to score by matching on the postcode.
There are at least a couple of caveats to consider when attempting this. One is that the proportion will be less robust when you have very few people in a specific category historically (e.g., rural postcodes). In these cases you might have to do some category roll-ups first. Another potential issue is that it assumes historic contacts were made at random, or according to some mechanism that will also be applied in future selection processes, such that you can consider the prior contacts ‘representative’ of category membership for the purposes of your modelling. Violations of the assumption would probably require some statistical adjustment to get around.
If anyone sees other potential issues with this approach, or has other alternatives they use to deal with problematic categorical variables, feel free to comment!
_____
Short URL for this post: http://wp.me/pnqr9-1R
