Apache Mahout could be used to group documents into clusters of similar subject area. Mahout also includes a number of classification algorithms that can be used to assign category labels to text documents. One algorithm that Mahout provides is the Naive Bayes algorithm.
This algorithm is used for a wide variety of classification problems and is as an excellent introduction into probabilistic classification. In order to perform class assignments, the algorithms that employ probabilistic classification techniques build a model based on the probability with which document features appear for a given class.
In this article, we will use the Mahout implementation of the Naive Bayes algorithm to build a document categorizer. In this example will develop our own test corpus from data we collected from the Internet and use it to train the classifier. From there, we will demonstrate how training a classifier is an iterative process and present strategies for re-organizing training data in order to improve categorization accuracy. Finally, I will demonstrate how the document categorizer can be integrated into Solr so that documents are automatically assigned to categories when they are indexed using the based on the trained category model. Let us begin by discussing the theoretical underpinnings of the Naive Bayes classification algorithm.