Explore and Learn Machine Learning Concept

In my last blog we have discussed about how can we create the bag of words using python [refer this link  CreatingBag-of-Words using Python  ]. Now we have seen that bag-of-word approach is purely dependent on the frequency of words. Now let’s discuss another approach to convert the textual data into matrix format, it called us TF-IDF [Term Frequency – Inverse document Frequency] and it is the most preferred way used by most of data scientist and machine learning professionals. In this approach we consider a term is relevant to document if that term appears frequently in the document and term is unique to document i.e. term should not appear in all the documents. So its frequency considering with respect to all documents should be small and term frequency for specific document should be high. TF-IDF score is calculated as follows: Term frequency of a term (t) in a document (d).   Inverse document frequency of a term   Below are formulas for calculating the and...

In my last two blog we have discussed about the bag of words method for extracting features from text [refer this link  Feature Extraction - Text- Bag of word  ] and stemming and lemmatization techniques to avoid the redundant token problem [ refer this link Stemming and Lemmatization ] . Now it’s time to apply those concepts using python and see the things in action: We are going to explore below three options Let’s consider following three sentences: Kesri is a good movie to watch with family as it has good stories about India freedom fight. The success of a movie depends on the performance of the actors and story it has. There are no new movies releasing this month due to corona virus. Using above three sentences we will extract the Bag-of-words by applying the concepts of tokenization, stemming and lemmatization. So let’s get started: Step 1: Import the libraries word_tokenize for tokenization stopwords for stop words and CountVectorizer for creating bag-of words. #...

In my previous blog WordFrequency Distribution : Power Law . I explained basic concepts of lexical processing for normal distribution of data for machine learning algorithm. In this article we will go through the high level steps to process the textual data for machine learning and as part of this series I will be explaining the lexical processing of the text using different types of tokenization feature in python. In processing the textual data for machine learning following steps are performed: Lexical Processing of text:  Converting raw text into words, sentences, paragraphs etc. Syntactic Processing of text: Understanding the relationships among words used in the sentences. Semantic processing of text: Understanding the meaning of text. To do the lexical processing of text we perform: -  Tokenization and Extraction of features from text “To kenization ” is technique that is used to split the text into smaller elements. These elements can be characters...