跳到正文
Sam Altman· @sama · X·· 5 天前AI 评分38
AI 导读

很高兴看到 Nick 的这篇分享! @nickcammarata:可解释性可能是当下世界上最重要的科学问题,因为如果我们不知道自己正在构建什么,就无法解决对齐问题。我想我对此有好消息。我曾在 OpenAI 参与机械可解释性的早期工作,当时得出的结论是悲观的,因为神经网络似乎会折叠成任意纠缠的结。多年后,我启动了 OpenAI 首个大规模自动化可解释性项目,用当时原始的工具发现约 2% 的神经元是可读的。 借助现代智能体工具和更细致的观察,情况看起来完全不同。在参数量高达数万亿的模型中,绝大多数神经元都是可读的,它们的回路很优美。许多著名的纠缠神经元案例其实是被误解了。我预想的障碍大多并不存在。我现在认为,雄心勃勃的可解释性研究会在我们最需要它的时候被解决,而且可能相当快,因为自动化科学家在不断进步。这对可解释性或许是一个苦涩的教训:简单的东西,在荒谬的规模下,是有效的。 https://already-clean.goodfire.com/

正文 · 原文

Very happy to see this from Nick!

引用Nick@nickcammarata
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works https://already-clean.goodfire.com/
在 X 查看被引用的帖子

来源:Sam Altman · x.com