SemEval-2014 (Task 4 - ABSA) Current work in progress 2

With the training data released just before Christmas, I started working on this benchmark this month. For three of the four subtasks I have established a rough baseline method. I will try to keep this updated over time.

Aspect Detection

  • A relatively simple Part-of-Speech pattern algorithm that finds aspects
    (Precision: 54% Recall: 77% F1: 63%)
  • Default HMM tagger
    (Precision: 61% Recall: 30% F1: 40%)

Aspect Polarity Classification

  • A SentiWordNet method that simply aggregates by summing, while discounting words based on how far they are from the aspect words. Tested with golden aspect data as input.
    (Accuracy: 55%)
  • Simply choose always positive.
    (Accuracy: 58%)

Category Detection

  • A simple Bag-of-Words classifier based on SVMlib.
    (Precision: 74% Recall: 32% F1: 45%)
  • A method based on co-occurrences.
    (Precision: 73% Recall: 70% F1: 71%)

Category Polarity Classification

  • none yet


If you are also working on this problem, let me know in the comments! If you also have some preliminary results, then shout them out as well. It's a competition after all...


2 thoughts on “SemEval-2014 (Task 4 - ABSA) Current work in progress

  • Reply
    Kim Schouten Post author

    A default HMM tagger to find aspects is now added to overview. Precision is slightly better than PoS-pattern method, but recall much lower. I will have to check why performance is this low, and whether it can be boosted somehow.

  • Reply
    Kim Schouten Post author

    So, a while ago, the results of the benchmark were published. Although I send in some results for all four subtasks, the algorithm for category detection was most fleshed out. Unfortunately, I had a massive overfitting issue, which I was not able to resolve before having to send in the final benchmark data. Therefore, my results were disappointing. I am still in the process of analyzing the results to learn more about it.
    The final results of the benchmark can be found here. For the interested reader: my team id is COMMIT-P1WP3.

Leave a Reply

Your email address will not be published. Required fields are marked *