Ruoqi Guo et al. present RLCDAlignBench, a new benchmark designed to evaluate the ability of Jev—a model trained via reinforcement learning for calibrated decisions (RLCD)—to detect various types of alignment failures in language models zero-shot. The study examines ten specific failure modes, including sycophancy, jailbreaks, and hallucination, across 44 benchmarks and five target models. Results indicate that a single generic question applied to Jev achieves a median AUROC of 0.886, outperforming supervised baselines in many cases while being significantly more cost-effective than using LLM-judge scorers.
- Evaluates ten failure modes: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking.
- Uses a "relational" detection strategy by varying the question wording separately from input fields to handle failures defined against external references.
- Jev costs 63x less than traditional LLM-judge scorers.
- Evaluates ten failure modes: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking.
- Uses a "relational" detection strategy by varying the question wording separately from input fields to handle failures defined against external references.
- Jev costs 63x less than traditional LLM-judge scorers.
This paper challenges the traditional "singularity" concept of a single, all-powerful AI, proposing instead that the next intelligence explosion will be plural, social, and deeply intertwined with human intelligence. The authors highlight recent advances in agentic AI, demonstrating that intelligence fundamentally involves the interaction of diverse perspectives and emerges from social organization. They present evidence of "societies of thought" within reasoning models, where internal debates and multi-agent interactions enhance accuracy. The paper draws parallels to previous intelligence explosions, emphasizing the importance of scaling not just computational power, but also the social infrastructure—institutions, norms, and protocols—that govern these systems.
This post discusses a study that finds that refusal behavior in language models is mediated by a single direction in the residual stream of the model. The study presents an intervention that bypasses refusal by ablating this direction, and shows that adding in this direction induces refusal. The study is part of a scholars program and provides more details in a forthcoming paper.
