COSC 426 F24 Lab 9

COSC 426 F24 Lab 9

Introduction

One of the goals of this course is to train NLP Scholars who can critically reason about the relationship between evaluations of NLP system capabilities and society. For this course, you will write a 1-2 page “Society Reflection” paper that reflects on the relationship between hype culture and scientific advances in NLP. The goal of this lab is to get you started with writing this paper, and maybe even finishing it!

Provided files

NA

What to submit

A 1-2 page paper on the gradescope for your course section. Prior to writing your paper, develop answers to the following parts. Your final paper should synthesize the findings from the parts.

Part 1

Find a news article that discusses advances in NLP. Here is a (non-comprehensive) list of journalists who have written about this topic. You can look for articles they’ve written, or find your own.

If you are having trouble finding articles, or are finding articles that are behind a paywall, talk to your lab instructor

Part 2

Identify the central claims being made in the article you selected. Which of the claims adopt a positive approach to the advances, and which ones adopt a negative approach?

Part 3

What kind of evidence would you need to substantiate the claims you identified in Part 2? What evidence does the article provide? This should be concrete. If the article references a benchmark, one thing you can do is look at examples in the benchmark, possibly playing around with a model trained on the benchmark using interact mode in NLPScholar to develop intuitions about whether success on the benchmark corresponds to the capability the benchmark is meant to measure.

Part 4

Evaluate whether the claims in the article are justified. Reason about possibilities for why these claims are being made and what societal impact these claims have.