The items below are things Marco Tulio Ribeiro said in two conference talks about testing language models beyond accuracy. — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.

Visual summary of operating lessons from Marco Tulio Ribeiro.

Part 3: Behavioral Testing and CheckList

  1. On Testing Like Software: In a talk on CheckList, Ribeiro said: "let's test nlp models just like we test software". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  2. On Knowing How to Test: In a talk on CheckList, Ribeiro said: "we know how to test software". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  3. On Decoupling Tests: In a talk on CheckList, Ribeiro said: "we're saying let's decouple the tests from training and let's test behaviors with different test types". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  4. On Inherited Biases: In a talk on CheckList, Ribeiro said: "whatever biases you have in training you're going to be there in tests as well". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  5. On a Process and a Tool: In a talk on CheckList, Ribeiro said: "checklist is a process and a tool". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  6. On Not Relying on Creativity: In a talk on CheckList, Ribeiro said: "you don't have to rely just on your creativity". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  7. On an Open Source Tool: In a talk on CheckList, Ribeiro said: "it's not just a process that we're proposing there's also an open source tool". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.

Part 4: Moving Beyond Accuracy Metrics

  1. On Benchmark Shortcuts: In a talk on CheckList, Ribeiro said: "models can do really well on benchmarks accuracy benchmarks by relying on shortcuts that are specific to those data sets that don't generalize to the real world". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  2. On Accuracy Alone: In a talk on CheckList, Ribeiro said: "we can't rely on accuracy alone". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  3. On Almost Perfect Models: In a talk on CheckList, Ribeiro said: "if you just rely on accuracy you think that they're almost perfect better than humans". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  4. On Humans Versus Models: In a talk on CheckList, Ribeiro said: "so these models are failing in ways that humans would not". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.
  5. On Basic Tests: In a talk on CheckList, Ribeiro said: "bert can't pass the most basic minimum functionality tests in this case even though it can answer quite a few of wikipedia's questions". — Marco Tulio Ribeiro talk on CheckList, behavioral testing of NLP models, auto-generated captions.

Part 5: Human-AI Collaboration and Debugging

  1. On Human Steering: In a talk on adaptive testing, Ribeiro said: "the user is adapting the tree and the focus as bugs appear". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  2. On Needing No Expertise: In a talk on adaptive testing, Ribeiro said: "you don't need creativity coding or any expertise whatsoever". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  3. On Play: In a talk on adaptive testing, Ribeiro said: "it's just play anyone can do it it's not work". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  4. On Fun: In a talk on adaptive testing, Ribeiro said: "maybe the biggest advantage over checklist is that it's fun". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.

Part 6: Counterfactuals and Augmentation

  1. On Augmenting With Templates: In a talk on adaptive testing, Ribeiro said: "my answer is always the same i don't think that's a good idea". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  2. On Wasted Tests: In a talk on adaptive testing, Ribeiro said: "all the effort that went into creating the test is basically wasted because you can't use the test anymore". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  3. On Cheap Tests: In a talk on adaptive testing, Ribeiro said: "the fact that this is so quick and easy to run makes a big difference". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.

Part 7: Prompt Engineering and Language Models

  1. On Creativity and Effort: In a talk on adaptive testing, Ribeiro said: "using checklists or something similar requires a lot of creativity and effort". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  2. On Unfound Bugs: In a talk on adaptive testing, Ribeiro said: "sometimes you get something in the matrix where the error rate is zero percent but there's still bugs in there you just haven't found them". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  3. On Semantically Related Failures: In a talk on adaptive testing, Ribeiro said: "the fact that we can find so many of them that are so semantically related means that there's a bug in the model". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.
  4. On Working Everywhere: In a talk on adaptive testing, Ribeiro said: "this works for everything we've tried so far". — Marco Tulio Ribeiro talk on adaptive testing of NLP models, auto-generated captions.