Yanjun Gao's latest success stories 💯

Not one, but two papers were published this spring by our student Yanjun Gao!

The following papers were published by Yanjun Gao in LREC 2018 and BEA 2018 respectively: 

PyrEval: An Automated Method for Summary Content Analysis

The pyramid method is a content analysis approach in automatic summarization evaluation for manual construction of a content model from reference summaries, and manual scoring of unseen summaries with the pyramid model. PyrEval automates the manual pyramid method. PyrEval uses low-dimension distributional semantics to represent phrase meanings, and a new algorithm, EDUA (Emergent Discovery of Units of Attraction), to solve a set cover problem to construct the content model from vectorized phrases. Because the vectors are pretrained, and EDUA is an efficient greedy algorithm, PyrEval can apply pyramid content evaluation with no retraining, and in excellent time. Moreover, PyrEval has been tested on many datasets derived from humans and machine generated summaries, and shown good performance on both.

in the Language Resources and Evaluation Conference (LREC) 2018

Automated Content Analysis: A Case Study of Computer Science Student Summaries

Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess précis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearsons correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.

in the 13th Workshop on Innovative Use of NLP for Building Educational Applications, 

part of The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Please find more details about the papers in the links provided. We welcome any discussion or question!

Yanjun Gao at BEA 2018

We congratulate her and wish her to continue with even more success stories!

Find more about Yanjun's work here.