<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Ai-Evals on David R. Longnecker - Converting Coffee to Code</title><link>https://drlongnecker.com/tags/ai-evals/</link><description>Recent content in Ai-Evals on David R. Longnecker - Converting Coffee to Code</description><generator>Hugo</generator><language>en-US</language><lastBuildDate>Thu, 29 Oct 2026 09:00:00 -0600</lastBuildDate><atom:link href="https://drlongnecker.com/tags/ai-evals/index.xml" rel="self" type="application/rss+xml"/><item><title>Verifiers Over Vibes</title><link>https://drlongnecker.com/blog/2026/10/verifiers-over-vibes-rlvr/</link><pubDate>Thu, 29 Oct 2026 09:00:00 -0600</pubDate><guid>https://drlongnecker.com/blog/2026/10/verifiers-over-vibes-rlvr/</guid><description>&lt;p&gt;For the last three years, the generative AI industry ran on &lt;strong&gt;vibes&lt;/strong&gt;. &lt;em&gt;How does this &amp;ldquo;feel&amp;rdquo;?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;When teams wanted to align a model to be helpful, safe, or knowledgeable, they turned to Reinforcement Learning from Human Feedback (RLHF). Human evaluators compared two side-by-side AI responses, picked the one they preferred, and those comparisons trained a reward model.&lt;/p&gt;
&lt;p&gt;The industry built billion-dollar product strategies on top of what was essentially a collective popularity contest.&lt;/p&gt;</description></item><item><title>When AI Evals Move to the PM, What Moves Out?</title><link>https://drlongnecker.com/blog/2026/10/pm-role-scope-grows-who-clears-plate/</link><pubDate>Fri, 23 Oct 2026 09:00:00 -0600</pubDate><guid>https://drlongnecker.com/blog/2026/10/pm-role-scope-grows-who-clears-plate/</guid><description>&lt;p&gt;I had coffee recently with two product managers who were spending their afternoons learning to write and score evals. Not data scientists. Not machine learning researchers. Senior product managers.&lt;/p&gt;
&lt;p&gt;They weren&amp;rsquo;t poking at chat prompts for fun. They were knee-deep in trace logs, annotating failure modes, and wrestling with whether a model&amp;rsquo;s summary hallucinated an edge case or merely summarized it aggressively.
With product leaders getting deeper into the muck while working on generative AI products, evaluations have migrated from the research lab directly onto the product desk. What struck me as they described their weeks wasn&amp;rsquo;t that they were taking on the work—they were genuinely engaged by it. What struck me was that nobody had asked what they were putting down to make room for it. Earlier this week I wrote that a team can only absorb so much change at once (&lt;a href="https://drlongnecker.com/blog/2026/10/what-we-stopped-to-make-room"&gt;Capping In-Flight Change&lt;/a&gt;). This is the same constraint at the scale of a single job description.&lt;/p&gt;</description></item><item><title>Delete a Section and Watch the Agent</title><link>https://drlongnecker.com/blog/2026/10/delete-a-section-watch-the-agent/</link><pubDate>Wed, 21 Oct 2026 09:00:00 -0600</pubDate><guid>https://drlongnecker.com/blog/2026/10/delete-a-section-watch-the-agent/</guid><description>&lt;p&gt;Most agent instruction files grow the way a legacy README grows with each line or section going in after something goes wrong (a &amp;ldquo;lesson&amp;rdquo;). However, almost nobody goes back to ask whether it&amp;rsquo;s still valid. My own agent setup had the same problem and I&amp;rsquo;d guess yours does too.&lt;/p&gt;
&lt;p&gt;The usual advice is to give an agent more context, including architecture notes and conventions. Thoughtworks put the other side of that on its &lt;a href="https://www.thoughtworks.com/en-us/radar"&gt;Technology Radar&lt;/a&gt; this spring. The April 2026 edition lists Context engineering under Adopt and Agent instruction bloat under Caution. The bloat entry describes how files grow by accretion, with each addition sensible on its own while the total gets long, contradictory and easy to skim past.&lt;/p&gt;</description></item></channel></rss>