Fix six instruction verifiers that mis-score correct responses#34
Open
jlu-figma wants to merge 2 commits into
Open
Fix six instruction verifiers that mis-score correct responses#34jlu-figma wants to merge 2 commits into
jlu-figma wants to merge 2 commits into
Conversation
Six check_following verifiers assign the wrong verdict; this fixes them and adds
regression tests to instructions_test.py. Full suite passes (65 tests); no new
dependencies.
False-fail (a correct response was rejected):
- repeat:repeat_span (RepeatSpanChecker): prompt says "word indices, split by
whitespace" but the checker sliced character indices, so the target was a
mid-word substring. Now slices whitespace words; description/defaults use word
indices.
- ratio:overlap (NGramOverlapChecker): nltk.ngrams(value, 3) on the raw string
produced character trigrams, not word trigrams. Now tokenizes into word
trigrams.
- count:keywords_multiple (KeywordsMultipleChecker): value.lower().count(kw)
counted substrings ("door" inside "doorway"/"indoors", any -ed/-s/-ing form).
Now counts whole words with the same \b-bounded, case-insensitive match used
by PersonNameCountChecker and IncludeKeywordChecker.
- custom:sentence_alphabet (SentenceAlphabetChecker): only lstrip() whitespace,
so a first word wrapped in a quote made the quote the "first letter". Now
strips leading punctuation.
- words:words_position (WordsPositionChecker): nltk tokens included leading and
trailing punctuation, shifting the 2nd / 2nd-to-last word positions. Now uses
the shared _word_tokens_without_punctuation helper.
False-pass (a wrong response was accepted):
- ratio:sentence_words (CharacterCountUniqueWordsChecker): never checked "all
different words". Now enforces word uniqueness across the three sentences.
Fix six instruction verifiers that mis-score correct responses
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was going through eval results and found seven verifier bugs that were causing my benchmark score to be lower.
repeat:repeat_spanPrompt example:
Copy the span of words that lies between (and including) index 3 and 6, the indices are word indices, split by whitespace!Problem: The checker slices by character indices. The checker comments even used to say "the indices are character indices"
ratio:overlapPrompt example:
Maintain a trigram overlap of 72% (±2%) with the provided reference text.Problem: Code was doing
nltk.ngrams(value, 3)on the raw string.ngramstakes in lists / sequences, so we were actually calculating character trigrams instead of word trigramscustom:sentence_alphabetPrompt example:
Tell me a 26-sentence story where each sentence's first word starts with the letters of the alphabet in order.I specifically found this annoying when going through failing samples. When a sentence starts with a quote here, we end up counting the quote as the "first letter"
words:words_positionPrompt example:
The second word in your response and the second to last word in your response should be the word whisper.Problem: Similar issue with punctuation. nltk tokens include leading / trailing punctuation. We need to account for this in the check
ratio:sentence_wordsPrompt format:
Respond with three sentences, all containing the same number of characters but using all different words.Problem: Seems like the code was missing the "all different words" check.
keywords:multiple_wordsPrompt format:
Include keyword door once in your response, keyword bread twice in your response, keyword blue three times in your response, keyword lamp five times in your response, and keyword river seven times in your response.Problem: small nit, but sometimes a word can appear as a substring of another word.
Other Issues
I know that
words:start_verbhas an issue too. The classifier sometimes will classify words likeRunorCultivateat the start of a sentence as not a verb.