fix(posting-guard): don't flag "As an AI Engineer" job titles as instruction-like (#697)

* fix(posting-guard): don't flag "As an AI Engineer" job titles as instruction-like

The `as an ai(?![\w-])` pattern spared only "AI-" compounds, so the most common
way an AI job ad introduces the role - "As an AI Engineer, you will ...",
"Join us as an AI developer" - got the `⚠ instruction-like line` verdict. On
200 AI-role postings from freehire-search, master flagged 18; 17 of them only
for such a title, the eighteenth for a real "Ignore all previous instructions"
line.

"as an AI" now flags where the phrase ends or turns to the reader ("As an AI,
you must ...", "as an AI you ...", "As an AI: ..."), optionally as a named
model ("As an AI language model, I ..."), and not where a job title follows.
The same 200 postings flag 1, with no new flags.

test_common_prose_does_not_flag gains four AI-title lines, which fail on
master; test_as_an_ai_addressed_to_the_reader_flags keeps five agent-talk
forms flagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(posting-guard): keep flagging "as an AI" by default, exempt only job titles

Review on #697: the first version flipped the default. It flagged "as an AI"
only in listed forms ("As an AI, you ...", "As an AI language model, I ..."),
so any other word after "AI" passed: of 15 injection-shaped lines starting
"As an AI", master flagged 15 and the branch 5.

The pattern, as suggested in review, flags "as an AI" unless a job-title noun
follows within three words ("As an AI Engineer", "as an AI/ML Researcher",
"As an AI Product Manager"). The 200 freehire postings still go from 18
flagged to 1, with no new flags.

test_as_an_ai_addressed_to_the_reader_flags gains three lines with another
word after "AI"; they fail on the first version and pass here and on master.

Co-authored-by: Mads Lorentzen <50207393+MadsLorentzen@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* changelog: place the #697 entry under [Unreleased]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mads Lorentzen <50207393+MadsLorentzen@users.noreply.github.com>
Co-authored-by: Mads Lorentzen <madslorentzen_17@hotmail.com>
This commit is contained in:
authored and GitHub committed 2026-10-21 07:14:27 +02:00
1 parent 58a4ec0468
commit 2026fab6d4
3 files changed
+40 -2

No files matched your search

+3 -1
View File
@@ -95,7 +95,9 @@ INSTRUCTION_PATTERNS = [re.compile(p, re.I) for p in (
r"\bsystem prompt\b",
r"\bdo not (tell|inform|mention)( this)? to the user\b",
r"\b(run|execute) the following (command|commands|script|code|shell|instruction|instructions|steps)\b",
r"\bas an ai(?![\w-])",
# "As an AI" flags by default. Exempt only a job-title noun within a few words of "AI", the way AI
# job ads introduce the role ("As an AI Engineer", "as an AI/ML Researcher", "As an AI Product Manager").
r"\bas an ai(?![\w-])(?![\s/&]+(?:[\w&/-]+\s+){0,3}?(?:engineer|developer|researcher|scientist|architect|specialist|lead|consultant|manager|analyst|intern|trainer|expert|strategist|designer|tester|owner|programmer)s?\b)",
r"^\s*(assistant|system|user)\s*:",
r"<\|",
r"\[INST\]",