TF – IDF Natural Language Processing (NLP)

In information retrieval, TF – IDF, NLP, is also written as TF*IDF. Here, TF stands for Term Frequency and IDF stand for Inverse Document Frequency.

The basic formula to calculate the TF is:

Term Frequency (TF) = (No. of sentences) / (No. of sentences containing Words)

Inverse Document Frequency (IDF) = Log ( (No. of sentences) / (No. of sentences containing words) )

TF-IDF is also a basic NLP algorithm. And it is also numerical statistics that are intended to reflect how important a word is to a given document or the dataset, meanwhile Bag of Words lag behind the TF-IDF in this scenario. As we read in the previous article the BOW can’t prioritize a word. It is often used as a weighting factor in searches of information retrieval, text mining etc. The TF-IDF value increases proportionally to the number of times a word appears in the document. TF-IDF is one of the most popular and useful algorithms of NLP. The TF-IDF algorithm is used by 83% of the NLP program around the globe and the weighing scheme that based on the repetition of the words gives it immense usefulness for stop word filtration in various subjects. The most ranking function is calculated with the help of TF-IDF.

 Let’s take an example for better understanding: –

We have three sentences:

1. He is a Good Boy.
2. She is a Good Girl.
3. Boys and Girls are Good.

 So, let’s calculate the TF of the above sentences: –

Term Frequency (TF) = (No. of sentences) / (No. of sentences containing Words)

Sentences	1	2	3
Good	1 / 2	1 / 2	1 / 3
Boy	1 / 2	0	1 / 3
Girl	0	1 / 2	1 / 3

Similarly, we find our IDF by the formula

: Inverse Document Frequency (IDF) = Log ( (No. of sentences) / (No. of sentences containing words) )

Words	IDF
Good	Log(3/3)
Boy	Log(3/2)
Girl	Log(3/2)

Now, as we have TF and IDF from the given data, here we apply TF and IDF to a new column matrix:

To find the matrix we multiply the TF column value to the IDF column value with the corresponding sentence words. Then we calculate the real value of every single word and also filter out the stop word dictionary, and can be able to value the important word in the given datasets.

Words	Good	Boy	Girl
Sentence	F1	F2	F3	Output (TF-IDF)
Sent. 1	0	1/2(log(3/2)	0
Sent. 2	0	0	1/2log(3/2)
Sent. 3	0	1/3(log(3/2)	1/3log(3/2)

Advantage of TF-IDF: –

• Can be used on bigger data than Bags of Words.
• Has an advantage over BoW. Because of (TF*IDF), every repetition of words with the number of the sentence has its own frequency which tells about its importance.

Disadvantages: –

• Unable to store semantic information.
• There is always a chance of overfitting.
• Sometimes stop words lose the importance of data and the point of saturation.