removeInfrequentWords

Remove words with low counts from bag-of-words model

Syntax

newBag = removeInfrequentWords(bag,count)

newBag = removeInfrequentWords(bag,count,'IgnoreCase',true)

Description

newBag = removeInfrequentWords(bag,count) removes the words that appear at most count times in total from the bag-of-words model bag. The function, by default, is case sensitive.

newBag = removeInfrequentWords(bag,count,'IgnoreCase',true) removes the words that appear at most count times in total ignoring case. If words differ only by case, then the corresponding counts are merged.

Examples

Remove Infrequent Words

Remove the words that appear two times or fewer from a bag-of-words model.

Create a bag-of-words model from an array of tokenized documents.

documents = tokenizedDocument([
    "an example of a short sentence"
    "a second short sentence"
    "another example"
    "a short example"]);
bag = bagOfWords(documents)

bag = 
  bagOfWords with properties:

        NumWords: 8
          Counts: [4×8 double]
      Vocabulary: ["an"    "example"    "of"    "a"    "short"    "sentence"    "second"    "another"]
    NumDocuments: 4

Remove the words that appear two times or fewer from the bag-of-words model.

count = 2;
newBag = removeInfrequentWords(bag,count)

newBag = 
  bagOfWords with properties:

        NumWords: 3
          Counts: [4×3 double]
      Vocabulary: ["example"    "a"    "short"]
    NumDocuments: 4

Input Arguments

bag — Input bag-of-words model

bagOfWords object

Input bag-of-words model, specified as a bagOfWords object.

count — Count threshold to remove words

positive integer

Count threshold to remove words, specified as a positive integer. The function removes the words that appear count times in total or fewer.

Version History

Introduced in R2017b