“If you are looking for a career where your services will be in high demand, you should find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis. So my recommendation is to take lots of courses about how to manipulate and analyze data: databases, machine learning, econometrics, statistics, visualization, and so on.”  Hal Varian, Chief Economist at Google

Me suffer from confirmation bias? Never!
_____

Short URL for this post: http://wp.me/pnqr9-2K

• Discard the variable if it doesn’t appear it will be a good discriminator. It is sometimes hard to tell this up front when you have loads of categories.
• Roll the categories up into larger sets based on conceptual similarity.  This can work for ordinal or geographic data, but is more difficult for purely nominal variables.  There is also the risk that you’ll ‘average away’ some of the predictive value in the variable.
• Use a statistical technique (e.g., a decision tree) to work out groupings of categories based on their discriminative power.  This may make for groupings that are hard to explain.

Another option I’ve recently come across is to convert the categorical variable to a metric-level variable using historic response data.  For instance, say you’ve been collecting your customer’s postcodes for a while and are looking to employ this variable in a predictive model.  Perhaps you are predicting response to a mailing offer (or something similar) which has been running for at least one learning cycle.  A potential way to deal with the ‘too many categories’ problem would be to calculate the proportion of people contacted in each postcode during prior mailings who responded to the offer.  Voilà!  You’ve now got a metric level and continuous variable to play with.  You can apply the historic response values to any new prospects you are looking to score by matching on the postcode.

There are at least a couple of caveats to consider when attempting this.  One is that the proportion will be less robust when you have very few people in a specific category historically (e.g., rural postcodes).  In these cases you might have to do some category roll-ups first.  Another potential issue is that it assumes historic contacts were made at random, or according to some mechanism that will also be applied in future selection processes, such that you can consider the prior contacts ‘representative’ of category membership for the purposes of your modelling.  Violations of the assumption would probably require some statistical adjustment to get around.

If anyone sees other potential issues with this approach, or has other alternatives they use to deal with problematic categorical variables, feel free to comment!

_____

Short URL for this post: http://wp.me/pnqr9-1R