Weblogs for market research : Finding more relevant opinion documents using system fusion
- Authors: Osman, Deanna , Yearwood, John , Vamplew, Peter
- Date: 2009
- Type: Text , Journal article
- Relation: Online Information Review Vol. 33, no. 5 (2009), p. 873-888
- Full Text: false
- Reviewed:
- Description: Purpose - The purpose of this paper is to examine the usefulness of fusion as a means of improving the precision of automated opinion detection. Design/methodology/approach - Five system fusion methods are proposed and tested using runs submitted by the Text REtrieval Conference (TREC) Blog06 participants as input. The methods include a voting method, an inverse rank method (IRM), a linear-normalised score method and two weighted methods that use a weighted IRM score to rank the document. Findings - Mean average precision (MAP) is used as an indicator of the performance of the runs in this study. The best system fusion method achieves a 55.5 percent higher MAP result compared with the highest MAP result of any individual run submitted by the Blog06 participants. This equates to an increase in detection of 2,398 relevant opinion documents (21 percent). Practical implications - System fusion can be used to improve upon the results achieved by existing individual opinion detection systems. On the other hand, multiple opinion detection approaches can be combined into one system and fusion used to combine the results to build in diversity. Diversity within fusion inputs can increase the improvements achieved by fusion methods. The improved output from a diverse opinion detection system will then contain a higher number of relevant documents and reduce the incidence of high-ranking non-relevant documents and low-ranking relevant documents. Originality/value - The fusion methods proposed in this study demonstrate that simple fusion of opinion detection systems can improve performance.
Automated opinion detection : Implications of the level of agreement between human raters
- Authors: Osman, Deanna , Yearwood, John , Vamplew, Peter
- Date: 2010
- Type: Text , Journal article
- Relation: Information Processing and Management Vol. 46, no. 3 (2010), p. 331-342
- Full Text: false
- Reviewed:
- Description: The ability to agree with the TREC Blog06 opinion assessments was measured for seven human assessors and compared with the submitted results of the Blog06 participants. The assessors achieved a fair level of agreement between their assessments, although the range between the assessors was large. It is recommended that multiple assessors are used to assess opinion data, or a pre-test of assessors is completed to remove the most dissenting assessors from a pool of assessors prior to the assessment process. The possibility of inconsistent assessments in a corpus also raises concerns about training data for an automated opinion detection system (AODS), so a further recommendation is that AODS training data be assembled from a variety of sources. This paper establishes an aspirational value for an AODS by determining the level of agreement achievable by human assessors when assessing the existence of an opinion on a given topic. Knowing the level of agreement amongst humans is important because it sets an upper bound on the expected performance of AODS. While the AODSs surveyed achieved satisfactory results, none achieved a result close to the upper bound. © 2009 Elsevier Ltd. All rights reserved.