Power Law or Zipf’s Law: Word Frequency Distribution

-

This article will help you to understand the basic concepts for lexical processing for text data before using in any machine learning model. 

Working with any type of data, be it numeric, textual or images, involves following steps

  1. Explore : Performing  pre-processing of data
  2. Understand the data
As text is made up of words, sentences and paragraphs, hence exploring of text data, can be started by analyzing the words frequency distribution.

Famous linguist, George Zipf had started a simple exercise: 

  1. Count the number of times each word appear in the document
  2. Create a rank order on the frequency of each word. The most frequent word was given the rank 1, second most frequent work was given rank 2 and so on. 

He repeated this exercise on many documents and found a specific pattern in which words are distributed in the document. Basis the pattern observed he has given a principle known as “Zipf Law or Power Law”

Let’s analyze the word frequencies of following small story:
Once upon a time, there lived a shepherd boy who was bored watching his flock of sheep on the hill. To amuse himself, he shouted, “Wolf! Wolf! The sheep are being chased by the wolf!” The villagers came running to help the boy and save the sheep. They found nothing and the boy just laughed looking at their angry faces.
“Don’t cry ‘wolf’ when there’s no wolf boy!”, they said angrily and left. The boy just laughed at them.
After a while, he got bored and cried ‘wolf!’ again, fooling the villagers a second time. The angry villagers warned the boy a second time and left. The boy continued watching the flock. After a while, he saw a real wolf and cried loudly, “Wolf! Please help! The wolf is chasing the sheep. Help!”
But this time, no one turned up to help. By evening, when the boy didn’t return home, the villagers wondered what happened to him and went up the hill. The boy sat on the hill weeping. “Why didn’t you come when I called out that there was a wolf?” he asked angrily. “The flock is scattered now”, he said.
An old villager approached him and said, “People won’t believe liars even when they tell the truth. We’ll look for your sheep tomorrow morning. Let’s go home now”.

We can clearly visualize that most frequent words in the paragraphs are words like a, an, and, he, the, on, to etc. These most frequent words are called “Stop Words” or “Language Builder Words”. 
As stop words do not necessarily tell us what the document actually represents, they are not relevant for analysis.
After doing the study of various documents for word frequency, he found that most relevant words or most significant words are distributed in well known “Gaussian Distribution” or “Bell Curve”.
So when we work with textual data, we remove the stop words from the data.
Now we know that frequency of stop words are very high therefore removing stop words result in
  1. Smaller data in terms of size.
  2. Less number of features to work with 
Basis, above understanding we can do exploration of textual data and check whether word frequencies follows normal distribution or not, as most of machine learning algorithm expects data to be normally distributed.

Kindly share your queries under comment section that you would like me to address.

Please do subscribe to the blog

Comments

Post a Comment

Popular posts from this blog

Levenshtein distance concept in NLP

-
Image
In my last blog we have discussed how we can use TF - IDF method implementation using python for more details refer [ TF - IDF Implementation Using Python  ]. In this blog we will discuss how to deal with spelling correction to do the stemming or lemmatization effectively.  There is a concept known as "Edit Distance". "Edit distance is the minimum number of edits required to one string to another string". We can perform following types of operations Insertion of a letter Deletion of a letter Modification of a letter Let's take an example to understand this concept in more detail. "success" is written as "sucess". We have two strings one with length 7 [with correct spelling] and another with length 6 [ with incorrect spelling].  Step 1:   If the string of length M and N then we need to create the matrix of size (M+1) and (N+1). In our case we will create the matrix of size 7 X 8 as follows. Step 2: Initialize the first row and first column st...
Read more

Lexical Processing

-
Image
In my previous blog WordFrequency Distribution : Power Law . I explained basic concepts of lexical processing for normal distribution of data for machine learning algorithm. In this article we will go through the high level steps to process the textual data for machine learning and as part of this series I will be explaining the lexical processing of the text using different types of tokenization feature in python. In processing the textual data for machine learning following steps are performed: Lexical Processing of text:  Converting raw text into words, sentences, paragraphs etc. Syntactic Processing of text: Understanding the relationships among words used in the sentences. Semantic processing of text: Understanding the meaning of text. To do the lexical processing of text we perform: -  Tokenization and Extraction of features from text “To kenization ” is technique that is used to split the text into smaller elements. These elements can be characters...
Read more

Creating Bag-of-Words using Python

-
Image
In my last two blog we have discussed about the bag of words method for extracting features from text [refer this link  Feature Extraction - Text- Bag of word  ] and stemming and lemmatization techniques to avoid the redundant token problem [ refer this link Stemming and Lemmatization ] . Now it’s time to apply those concepts using python and see the things in action: We are going to explore below three options Let’s consider following three sentences: Kesri is a good movie to watch with family as it has good stories about India freedom fight. The success of a movie depends on the performance of the actors and story it has. There are no new movies releasing this month due to corona virus. Using above three sentences we will extract the Bag-of-words by applying the concepts of tokenization, stemming and lemmatization. So let’s get started: Step 1: Import the libraries word_tokenize for tokenization stopwords for stop words and CountVectorizer for creating bag-of words. #...
Read more