[Archive]. This unpublished paper summarises a part of quantitative-qualitative research carried out in the context of my Master’s thesis in Science and Technologies of Information and Communication at the Université Libre de Bruxelles (2015), dedicated to the possibilities and limits of automated news. The proposed method consisted of assessments through metrics and human-based judgments, which are used in computational linguistics (metrics are commonly used in automated translation). The judges were all journalists or writers. A corpus of twenty articles written by software was first evaluated using the most common metrics used in computational linguistics. The experience then consisted of presenting the human judges with three samples of articles written by their colleagues and produced by an algorithmic process. Without knowing the object of the experience, they were asked to evaluate the texts in terms of quality criteria as defined by Clerwall (2014). They were then asked to assign an author to each text: human or software. In two out of three cases, they did not recognise that the author was not human.
ASIA tools were used for the metrics assessments. Since then, I have developed my own tools, which encompass several readability scores, iBLEU score, and Levenshtein distance: https://ohmybox.info//linguistics/. This method is also used in the context of my PhD thesis, but it was refined to fit its purpose, which is about the uses of news automation as a tool for journalists within their investigative or daily routines.
Click here for the available download link
NLGassessments
