Shortcut Auditing Exposes Agent-Safety Gaps

// 56m agoRESEARCH PAPER

Shortcut Auditing Exposes Agent-Safety Gaps

Hasan Demirkiran and Jens Ernstberger introduce a three-level framework for detecting shortcut reliance in agent-safety evaluations. Tests across LinuxArena, AgentDojo-Traj, and AgentHarm-Traj show that benchmark labels and guard decisions can depend heavily on artifacts such as command length, tool identity, formatting, and provenance markers.

// ANALYSIS

This paper makes a strong case that agent-safety scores are only as trustworthy as the evaluation design behind them.

–Simple cues reach 0.910 AUC on LinuxArena, while exact-tool lookup collapses from F1 .901 to .495 under domain holdout
–AgentHarm-Traj performance falls from F1 .902 to .719 when weighting interactions equally instead of individual steps
–Rewriting an untrusted-content marker changed Claude Haiku’s blocking rate by 8.33 percentage points, while opaque tool aliases shifted it by 11.46 points
–The effects were configuration-specific and did not reproduce consistently on Luna or a deterministic TS-Guard reproduction
–Developers should add group holdouts, unit-balanced metrics, paired perturbations, and formatting placebos to safety evaluations

// TAGS

shortcut-sensitivityevaluationbenchmarkguardrailssafetysecurityagent

DISCOVERED

56m ago

2026-10-16

PUBLISHED

1h ago

2026-10-16

RELEVANCE

9/ 10

AUTHOR

arnaudmercier

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

UPDATE25m ago

Claude Code Desktop Adds Nested Sessions, Environments

Anthropic’s October 14 desktop update adds nested parallel sessions, reusable environments for local, SSH, and cloud work, per-folder worktrees, and `/btw` side chats. It also improves session visibility with status groups, live to-do lists, recap cards, follow-ups, and idle compaction.

NEWS3h ago

Claude Gabagool EAP Fuels Opus 6 Speculation

WorldofAI reports that an unreleased Claude checkpoint named Gabagool EAP appeared alongside Onigiri in Artificial Analysis code, fueling speculation about a future Opus 6 or upgraded Opus model. Anthropic has not publicly confirmed the checkpoint; its Opus page documents currently released models and availability.[Claude Opus](https://www.anthropic.com/claude/opus)

NEWS3h ago

Claude Onigiri EAP surfaces in code

An unreleased Claude checkpoint, claude-onigiri-eap-2, appeared in Artificial Analysis frontend code with low, medium, high, and xhigh reasoning variants. Its Opus 4.8 fallback suggests a possible Fable 5.5 connection, but Anthropic has not confirmed the model.