曝光台 注意防骗
网曝天猫店富美金盛家居专营店坑蒙拐骗欺诈消费者
cosine overlap 0.622 0.276 31 (64.6) 0.618 0.265 28 (58.3)
KL div inp-summ -0.628 -0.262 28 (58.3) -0.577 -0.267 22 (45.8)
topic overlap 0.597 0.265 30 (62.5) 0.689 0.277 26 (54.2)
% summary = topic wd 0.607 0.269 23 (47.9) 0.534 0.272 23 (47.9)
mult. summary prob. 0.434 0.268 8 (16.7) 0.459 0.272 10 (20.8)
unigram summary prob. 0.292 0.261 2 ( 4.2) 0.466 0.287 2 (4.2)
regression 0.736 0.281 37 (77.1) 0.642 0.262 32 (66.7)
ROUGE-1 recall 0.833 0.264 47 (97.9) 0.754 0.266 46 (95.8)
ROUGE-2 recall 0.875 0.316 48 (100) 0.742 0.299 44 (91.7)
Table 3: Spearman correlations at micro level (query focused task). Only the minimum, maximum
values of the significant correlations are reported together with the number and percentage of significant
correlations.
update input only avg. update & background
features pyramid respons. pyramid respons.
JS div -0.827 -0.764 -0.716 -0.669
JS div smoothed -0.825 -0.764 -0.713 -0.670
% of input topic words 0.770 0.709 0.677 0.616
KL div summ-inp -0.749 -0.709 -0.651 -0.624
KL div inp-summ -0.741 -0.717 -0.644 -0.638
cosine overlap 0.727 0.691 0.649 0.631
% of summary = topic wd 0.721 0.707 0.647 0.636
topic overlap 0.707 0.674 0.645 0.619
mult. summmary prob. 0.284 0.355 0.152 0.224
unigram summary prob. -0.093 0.038 -0.151 -0.053
regression 0.789 0.605 0.699 0.522
ROUGE-1 recall 0.912 0.865 . .
ROUGE-2 recall 0.941 0.884 . .
regression combining features comparing with background and update inputs (without averaging)
correlations = 0.8058 with pyramid, 0.6729 with responsiveness
Table 4: Spearman correlations at macro level for update summarization. Results are reported separately
for features comparing update summaries with the update input only or with both update and background
inputs and averaging the two.
6 Comparison with ROUGE
For manual pyramid scores, the best correlation,
0.88, we observed in our experiments was with
JS divergence. This result is unexpectedly high
for a fully automatic evaluation metric. Note that
the best correlation between pyramid scores and
ROUGE (for R2) is 0.90, practically identical with
JS divergence. For ROUGE-1, the correlation is
0.85.
In the case of manual responsiveness, which
combines aspects of linguistic quality along with
content selection evaluation, the correlation with
JS divergence is 0.73. For ROUGE, it is 0.80
for R1 and 0.87 for R2. Using higher order ngrams
is obviously beneficial as observed from the
differences between unigram and bigram ROUGE
scores. So a natural extension of our features
would be to use distance between bigram distributions.
At the same time, for responsiveness,
ROUGE-1 outperforms all the fully automatic features.
This is evidence that the model summaries
provide information that is unlikely to ever be approximated
by information from the input alone,
regardless of feature sophistication.
At the micro level, ROUGE does clearly better
than all the automatic measures. The results are
shown in the last two rows of Table 3. ROUGE-1
recall obtains significant correlations for over 95%
of inputs for responsiveness and 98% of inputs for
pyramid evaluation compared to 73% (JS divergence)
and 77% (regression). Undoubtedly, at the
input level, comparison with model summaries is
substantially more informative.
When reference summaries are available,
ROUGE provides scores that agree best with human
judgements. However, when model sum-
312
maries are not available, our features can provide
reliable estimates of system quality when averaged
over a set of test inputs. For predictions at the level
of individual inputs, our fully automatic features
are less useful.
7 Update Summarization
In Table 4, we report the performance of our features
for system evaluation on the update task. The
column, “update input only” summarizes the correlations
obtained by features comparing the summaries
with only the update inputs (set B). We
also compared the summaries individually to the
update and background (set A) inputs. The two
sets of features were then combined by a) averaging
(“avg. update and background”) and b) linear
regression (last line of Table 4).
As in the case of query focused summarization,
JS divergence and percentage of input topic signatures
in summary are the best features for the
update task as well. The overall best feature is
JS divergence between the update input and the
summaries—correlations of 0.82 and 0.76 with
pyramid and responsiveness.
Interestingly, the features combining both update
中国航空网 www.aero.cn
航空翻译 www.aviation.cn
本文链接地址：
航空资料8(94)
