- November 25, 2013
- Vasilis Vryniotis
- . 6 Comments

In the previous two machine learning tutorials, we examined the Naive Bayes and the Max Entropy classifiers. In this tutorial we will discuss the Multinomial Logistic Regression also known as Softmax Regression. Implementing Multinomial Logistic Regression in a conventional programming language such as C++, PHP or JAVA can be fairly straightforward despite the fact that an iterative algorithm would be required to estimate the parameters of the model.

**Update: The Datumbox Machine Learning Framework is now open-source and free to download. Check out the package com.datumbox.framework.machinelearning.classification to see the implementation of SoftMax Regression Classifier in Java.**

The Multinomial Logistic Regression, also known as SoftMax Regression due to the hypothesis function that it uses, is a supervised

learning algorithm which can be used in several problems including text classification. It is a regression model which generalizes the logistic regression to classification problems where the output can take more than two possible values. We should note that Multinomial Logistic Regression is closely related to MaxEnt algorithm because it uses the same activation functions. Nevertheless in this article we will present the method in a different context than we did on Max Entropy tutorial.

Multinomial Logistic Regression requires significantly more time to be trained comparing to Naive Bayes, because it uses an iterative algorithm to estimate the parameters of the model. After computing these parameters, SoftMax regression is competitive in terms of CPU and memory consumption. The Softmax Regression is preferred when we have features of different type (continuous, discrete, dummy variables etc), nevertheless given that it is a regression model, it is more vulnerable to multicollinearity problems and thus it should be avoided when our features are highly correlated.

Similarly to Max Entropy, we will present the algorithm in the context of document classification. Thus we will use the contextual information of the document in order to categorize it to a certain class. Let our training dataset consist of m (x_{i},y_{i}) pairs and let k be the number of all possible classes. Also by using the bag-of-words framework let {w_{1},…,w_{n}} be the set of n words that can appear within our texts.

The model of SoftMax regression requires the estimation of a coefficient theta for every word and category combination. The sign and the value of this coefficient show whether the existence of the particular word within a document has a positive or negative effect towards its classification to the category. In order to build our model we need to estimate the parameters. (Note that the θ_{i }vector stores the coefficients of i^{th} category for each of the n words, plus 1 for coefficient of the intercept term).

In accordance with what we did previously for Max Entropy, all the documents within our training dataset will be represented as vectors with 0s and 1s that indicate whether each word of our vocabulary exists within the document. In addition, all vectors will include an additional “1” element for the intercept term.

In Softmax Regression the probability given a document x to be classified as y is equal to:

[1]

Thus given that we have estimated the aforementioned θ parameters and given a new document x, our hypothesis function will need to estimate the above probability for each of the k possible classes. Thus the hypothesis function will return a k dimensional vector with the estimated probabilities:

[2]

By using the “maximum a posteriori” decision rule, when we classify a new document we will select the category with the highest probability.

In our Multinomial Logistic Regression model we will use the following cost function and we will try to find the theta parameters that minimize it:

[3]

Unfortunately, there is no known closed-form way to estimate the parameters that minimize the cost function and thus we need to use an iterative algorithm such as gradient descent. The iterative algorithm requires us estimating the partial derivative of the cost function which is equal to:

[4]

By using the batch gradient descent algorithm we estimate the theta parameters as follows:

1. Initialize vector θ_{j} with 0 in all elements

2. Repeat until convergence {

θ_{j} <- θ_{j} - α (for every j)

}

After estimating the θ parameters, we can use our model to classify new documents. Finally we should note that as we discussed in a previous article, using the Gradient Descent “as is” is never a good idea. Extending the algorithm to adapt learning rate and normalizing the values before performing the iterations are ways to improve convergence and speed up the execution.

Did you like the article? Please take a minute to share it on Twitter.

Latest Comments

2013-2014 © Datumbox. All Rights Reserved. Privacy Policy | Terms of Use

HI,

Liked your article very much. Between I having trouble in implementing multi-class logistic regression using softmax with IRLS. I mean I unable to find the equations to update the weights for each class. Is it possible for you to give me some guidance?

Hi,

Have a look on this implementation:

https://github.com/datumbox/datumbox-framework/blob/master/src/main/java/com/datumbox/framework/machinelearning/classification/SoftMaxRegression.java

Hello, i really enjoyed your article. Please i’m working on classifying an object into one of five classes available using multinomial logistic regression. Can u please put me through how to go abt it, especially the mathematical aspect that has to do with interpretation of odd ratio. Thanks

You don’t need to adapt anything. Softmax regression already supports multiple classes. Check out this implementation to get some insights on how you can code it:

https://github.com/datumbox/datumbox-framework/blob/master/src/main/java/com/datumbox/framework/machinelearning/classification/SoftMaxRegression.java

What is “l{y^(i)=y}”? Some kind of a delta function?

It’s an indicator function. It gets value equal to 1 if the expression within the brackets is true and 0 elsewhere. So basically this means that if the response variable of the ith record is equal to class y then it should take value 1.