# rougeEvaluationScore

Evaluate translation or summarization with ROUGE similarity score

## Syntax

``score = rougeEvaluationScore(candidate,references)``
``score = rougeEvaluationScore(candidate,references,Name,Value)``

## Description

The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scoring algorithm evaluates the similarity between a candidate document and a collection of reference documents. Use the ROUGE score to evaluate the quality of document translation and summarization models.

example

````score = rougeEvaluationScore(candidate,references)` returns the ROUGE score between the specified candidate document and the reference documents. The function, by default, computes unigram overlaps between `candidate` and `references`. This is also known as the ROUGE-N metric with n-gram length 1. For more information, see ROUGE Score.```

example

````score = rougeEvaluationScore(candidate,references,Name,Value)` specifies additional options using one or more name-value pairs.```

## Examples

collapse all

Specify the candidate document as a `tokenizedDocument` object.

```str = "the fast brown fox jumped over the lazy dog"; candidate = tokenizedDocument(str)```
```candidate = tokenizedDocument: 9 tokens: the fast brown fox jumped over the lazy dog ```

Specify the reference documents as a `tokenizedDocument` array.

```str = [ "the quick brown animal jumped over the lazy dog" "the quick brown fox jumped over the lazy dog"]; references = tokenizedDocument(str)```
```references = 2x1 tokenizedDocument: 9 tokens: the quick brown animal jumped over the lazy dog 9 tokens: the quick brown fox jumped over the lazy dog ```

Calculate the ROUGE score between the candidate document and the reference documents.

`score = rougeEvaluationScore(candidate,references)`
```score = 0.8889 ```

Specify the candidate document as a `tokenizedDocument` object.

```str = "a simple summary document containing some words"; candidate = tokenizedDocument(str)```
```candidate = tokenizedDocument: 7 tokens: a simple summary document containing some words ```

Specify the reference documents as a `tokenizedDocument` array.

```str = [ "a simple document" "another document with some words"]; references = tokenizedDocument(str)```
```references = 2x1 tokenizedDocument: 3 tokens: a simple document 5 tokens: another document with some words ```

Calculate the ROUGE score between the candidate document and the reference documents using the default options.

`score = rougeEvaluationScore(candidate,references)`
```score = 1 ```

The `rougeEvaluationScore` function, by default, compares unigram (single-token) overlaps between the candidate document and the reference documents. Because the ROUGE score is a recall-based measure, if one of the reference documents is made up entirely of unigrams that appear in the candidate document, the resulting ROUGE score is one. In this scenario, the output of the `rougeEvaluationScore` function is uninformative.

For a more meaningful result, calcualte the ROUGE score again using bigrams by setting the `'NgramLength'` option to `2`. The resulting score is less than one, since every reference document contain bigrams that do not appear in the candidate document.

`score = rougeEvaluationScore(candidate,references,'NgramLength',2)`
```score = 0.5000 ```

## Input Arguments

collapse all

Candidate document, specified as a `tokenizedDocument` scalar, a string array, or a cell array of character vectors. If `candidate` is not a `tokenizedDocument` scalar, then it must be a row vector representing a single document, where each element is a word.

Reference documents, specified as a `tokenizedDocument` array, a string array, or a cell array of character vectors. If `references` is not a `tokenizedDocument` array, then it must be a row vector representing a single document, where each element is a word. To evaluate against multiple reference documents, use a `tokenizedDocument` array.

### Name-Value Pair Arguments

Specify optional comma-separated pairs of `Name,Value` arguments. `Name` is the argument name and `Value` is the corresponding value. `Name` must appear inside quotes. You can specify several name and value pair arguments in any order as `Name1,Value1,...,NameN,ValueN`.

Example: ```scores = rougeEvaluationScore(candidate,references,'ROUGEMethod','weighted-subsequences')``` specifies to use the weighted subsequences ROUGE method.

ROUGE method, specified as the comma-separated pair consisting of `'ROUGEMethod'` and one of the following:

• `'n-grams'` – Evaluate the ROUGE score using n-gram overlaps between the candidate document and the reference documents. This is also known as the ROUGE-N metric.

• `'longest-common-subsequences'` – Evaluate the ROUGE score using Longest Common Subsequence (LCS) statistics. This is also known as the ROUGE-L metric.

• `'weighted-subsequences'` – Evaluate the ROUGE score using weighted longest common subsequence statistics. This method favors consecutive LCSs. This is also known as the ROUGE-W metric.

• `'skip-bigrams'` – Evaluate the ROUGE score using skip-bigram (any pair of words in sentence order) co-occurrence statistics. This is also known as the ROUGE-S metric.

• `'skip-bigrams-and-unigrams'` – Evaluate the ROUGE score using skip-bigram and unigram co-occurrence statistics. This is also known as the ROUGE-SU metric.

N-gram length used for the `'n-grams'` ROUGE method (ROUGE-N), specified as the comma-separated pair consisting of `'NgramLength'` and a positive integer.

If the `'ROUGEMethod'` option is not `'n-grams'`, then the `'NgramLength'` option has no effect.

Tip

If the longest document in `references` has fewer than `NgramLength` words, then the resulting ROUGE score is `NaN`. If `candidate` has fewer than `NgramLength` words, then the resulting ROUGE score is zero. To ensure that `rougeEvaluationScore` returns nonzero scores for very short documents, set `NgramLength` to a positive integer smaller than the length of `candidate` and the length of the longest document in `references`.

Data Types: `single` | `double` | `int8` | `int16` | `int32` | `int64` | `uint8` | `uint16` | `uint32` | `uint64`

Skip distance used for the `'skip-bigrams'` and `'skip-bigrams-and-unigrams'` ROUGE methods (ROUGE-S and ROUGE-SU), specified as the comma-separated pair consisting of `'SkipDistance'` and a positive integer.

If the `'ROUGEMethod'` option is not `'skip-bigrams'` or `'skip-bigrams-and-unigrams'`, then the `'SkipDistance'` option has no effect.

Data Types: `single` | `double` | `int8` | `int16` | `int32` | `int64` | `uint8` | `uint16` | `uint32` | `uint64`

## Output Arguments

collapse all

ROUGE score, returned as a scalar value in the range [0,1] or `NaN`.

A ROUGE score close to zero indicates poor similarity between `candidate` and `references`. A ROUGE score close to one indicates strong similarity between `candidate` and `references`. If `candidate` is identical to one of the reference documents, then `score` is 1. If `candidate` and `references` are both empty documents, then the resulting ROUGE score is `NaN`.

Tip

If the longest document in `references` has fewer than `NgramLength` words, then the resulting ROUGE score is `NaN`. If `candidate` has fewer than `NgramLength` words, then the resulting ROUGE score is zero. To ensure that `rougeEvaluationScore` returns nonzero scores for very short documents, set `NgramLength` to a positive integer smaller than the length of `candidate` and the length of the longest document in `references`.

## Algorithms

collapse all

### ROUGE Score

The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scoring algorithm [1] calculates the similarity between a candidate document and a collection of reference documents. Use the ROUGE score to evaluate the quality of document translation and summarization models.

N-gram Co-Occurrence Statistics (ROUGE-N)

Given an n-gram length n, the ROUGE-N metric between a candidate document and a single reference document is given by

where the elements ri are sentences in the reference document, is the number of times the specified n-gram occurs in the candidate document and numNgrams(ri) is the number of n-grams in the specified reference sentence ri.

For sets of multiple reference documents, the ROUGE-N metric is given by

To use the ROUGE-N metric, set the `'ROUGEMethod'` option to `'n-grams'`.

Longest Common Subsequence (ROUGE-L)

Given a sentence $d=\left[{w}_{1},\dots ,{w}_{m}\right]$ and a sentence s, where the elements si correspond to words, the subsequence $\left[{w}_{{i}_{1}},\dots ,{w}_{{i}_{k}}\right]$ is a common subsequence of d and s if ${w}_{{i}_{j}^{\prime }}\in \left\{{s}_{1},\dots ,{s}_{n}\right\}$ for $j=1,\dots ,k$ and ${i}_{1}\text{​}<\cdots <{i}_{k}$, where the elements of s are the words of the sentence and k is the length of the subsequence. The subsequence $\left[{w}_{{i}_{1}},\dots ,{w}_{{i}_{k}}\right]$ is a longest common subsequence (LCS) if the subsequence length k is maximal.

Given a candidate document and a single reference document the union of the longest common subsequences is given by

where $\text{LCS}\left(\text{candidate},{r}_{i}\right)$ is the set of longest common subsequences in the candidate document and the sentence ri from a reference document.

The ROUGE-L metric is an F-score measure. To calculate it, first calculate the recall and precision scores given by

Then, the ROUGE-L metric between a candidate document and a single reference document is given by the F-score measure

where the parameter $\beta$ controls the relative importance of the precision and recall. Because the ROUGE score favors recall, $\beta$ is typically set to a high value.

For sets of multiple reference documents, the ROUGE-L metric is given by

To use the ROUGE-L metric, set the `'ROUGEMethod'` option to `'longest-common-subsequences'`.

Weighted Longest Common Subsequence (ROUGE-W)

Given a weighting function f such that f has the property f(x+y)>f(x)+f(y) for any positive integers x and y, define to be the length of the longest consecutive matches encountered in the candidate document and a single reference document scored by the weighting function f. For more information about calculating this value, see [1].

The ROUGE-W is metric given an F-score measure which requires the recall and precision scores given by

The ROUGE-W metric between a candidate document and a single reference document is given by the F-score measure

where the parameter $\beta$ controls the relative importance of the precision and recall. Because the ROUGE score favors recall, $\beta$ is typically set to a high value.

For multiple reference documents, the ROUGE-W metric is given by

To use the ROUGE-W metric, set the `'ROUGEMethod'` option to `'weighted-longest-common-subsequences'`.

Skip-Bigram Co-Occurrence Statistics (ROUGE-S)

A skip-bigram is an ordered pair of words in a sentence allowing for arbitrary gaps between them. That is, given a sentence ${c}_{i}=\left[{c}_{i1},\dots ,{c}_{im}\right]$ from a candidate document, where the elements cij correspond to the words in the sentence, the pair of words $\left[{c}_{i{j}_{1}^{\prime }},{c}_{i{j}_{2}^{\prime }}\right]$ is a skip-bigram if${j}_{1}^{\prime }<{j}_{2}^{\prime }$.

The ROUGE-S metric is an F-score measure. To calculate it, first calculate the recall and precision scores given by

where the elements ri and ci are sentences in the reference document and candidate document, respectively, is the number of times the specified skip-bigram occurs in the candidate document, and numSkipBigrams(s) is the number of skip-bigrams in the sentence s.

Then, the ROUGE-S metric between a candidate document and a single reference document is given by the F-score measure

For sets of multiple reference documents, the ROUGE-S metric is given by

To use the ROUGE-S metric, set the `'ROUGEMethod'` option to `'skip-bigrams'`.

Skip-Bigram and Unigram Co-Occurrence Statistics (ROUGE-SU)

To also include unigram co-occurrence statistics in the ROUGE-S metric, introduce unigram counts into the recall and precision scores for ROUGE-S. This is equivalent to including start tokens in the candidate and reference documents, since

where Count(unigram,candidate) is the number of times the specified unigram appears in the candidate document, and ${r}_{i}^{+}$ and ${\text{candidate}}^{+}$ denote the reference sentence and the candidate document augmented with start tokens, respectively.

For sets of multiple reference documents, the ROUGE-SU metric is given by

where ${\text{reference}}^{+}$ is the reference document with sentences augmented with start tokens.

To use the ROUGE-SU metric, set the `'ROUGEMethod'` option to `'skip-bigrams-and-unigrams'`.

## References

[1] Lin, Chin-Yew. "Rouge: A package for automatic evaluation of summaries." In Text Summarization Branches Out, pp. 74-81. 2004.