Explore top daily AI community discussions from different sources, summarized and scored in one place.

Models

/r/singularity

111

Open-weights VLA hits 80%+ task progress on 4 of 17 real-robot tasks with zero fine-tuning. Demo reel attached

Sharing this because it is an embodied AI release trying to make the pretrained checkpoint itself measurable, instead of only showing results after task-specific tuning.

The video is a reel from Wall-OSS-0.5, a vision language action model released with open-source resources. Every clip in the reel has the same "Autonomous w/o Fine-Tuning" watermark in the corner. The robot is doing things like opening a pot lid and dropping fruit inside, covering blocks with a cloth, sorting items by color, putting drinks in specific containers in a specified order, shredding paper, putting a cup to the right of a calculator. According to the release, these clips are from the pretrained checkpoint rather than task-specific fine tuning.

What is interesting compared with the usual humanoid demo cycle is the evaluation framing. They report 4 of 17 real robot tasks above 80 percent task progress at zero shot, including a deformable rope tightening task that was not in the pretraining set. They also show pretraining task progress rising across checkpoints, with held-out tasks tracking seen tasks. That is the kind of curve people keep asking for in embodied AI, even if it is still early.

The other part I found notable is that the model seems to preserve general image/language ability while improving embodied grounding, at least by their evaluation. That matters because a lot of robot policies feel like they gain control ability by becoming narrower.

Code: https://github.com/X-Square-Robot/wall-x. Paper: https://x2robot.com/api/files/file/wall_oss_05.pdf. Hugging Face org: https://huggingface.co/x-square-robot.

The caveat is that the harder tasks are still not solved. Towel folding, charger insertion and table setting are still very low in zero shot, so pretraining alone is not magic. The real test is whether outside groups can run the checkpoint on their own arms and see similar strengths and failures.

Reel is attached. Original demo is on their project page.

--- TOP COMMENTS --- nice title to make it sound more impressive than it is
"VLA hits 100% progress on 1/50 tasks"
the former is barely important when its not even a quarter of tasks that it reached 80% on-

Whose models are these?

Just shows to me, I think anyway, that setting up these models will get easier and cheaper for widespread use. Good for Nvidia.

Yesterday

Models Applications

/r/ClaudeAI

Opus 4.8 + Thinking is draining context windows 40–60x faster

Pulled the token data from my token usage tracker. Opus 4.8 with Thinking enabled writes up to 900,000 cache tokens per turn. Opus 4.7 does 14,000–34,000.

Thinking blocks get cached with every turn, context snowballs, context windows drain in minutes instead of hours.

Anthropic changed thinking from adaptive to always-on between 4.7 and 4.8. On 4.7, the model decides when to think based on task complexity, simple turns get little or no thinking. On 4.8 with Thinking enabled, it generates full thinking blocks on every single turn regardless. That's why the cache explodes.

Thinking off. Tested Opus 4.8 with Thinking OFF and it drops straight back to ~12,000 cache tokens per turn — same as 4.7. The explosion is entirely the always-on Thinking behaviour.

If you want Thinking available without the risk, switch to Opus 4.7 instead. Its adaptive thinking only fires when the task warrants it, so it never snowballs.

To enable 4.7 in the model picker globally if using VS Code/Antigravity, add following line to ~/.claude/settings.json:

"model": "claude-opus-4-7"

--- TOP COMMENTS --- Anthropic makes models think too little: REEEEEEEEEEEE

Anthropic makes models think too much: REEEEEEEEEEEE

Jesus Christ, this is exactly why Anthropic keeps trying to restrict third-party harnesses for subscription plans. I’m sure antigravity prefix caching for claude is utterly fucked.

Yesterday

Models

/r/singularity

450

Differences Between Opus 4.7 and Opus 4.8 on MineBench

Some Notes:

Average Inference Time: 24.8 min (1,487seconds)
Total Cost (for 15 builds): $41.52
- Much cheaper than Opus 4.7 was, despite having the same API pricing
- The CoT / thinking times have clearly been streamlined (similar to what OpenAI has been doing with their latest releases) which lowers overall cost, but despite that, the output seems better than Opus 4.7, so that's good
This is, in my opinion, one of the first Claude models in a long time that actually feels like a genuinely impressive release; its builds are actually of similar quality to GPT 5.5, though a bit more inconsistent
During generation, the model had to retry 5 builds due to either hallucinations with the given block palette (it used blocks which were not available) or malformed outputs
- That's pretty on par with the Claude models, though the adaptive thinking seems to work better this time around (in previous attempts the model would spend all of it's output tokens for CoT and not have enough left over to finish its actual JSON output)
In my opinion, Opus 4.8 is a clear improvement over Opus 4.7 (or maybe it's what Opus 4.7 was supposed to be originally 🤷‍♂️)
Feel free to see all the other updates on the GitHub release (thanks for the suggestion!)
If you enjoy these posts please feel free to help fund the benchmark

Benchmark: https://minebench.ai/
Git Repository: https://github.com/Ammaar-Alam/minebench

Previous Posts:

Extra Information (if you're confused):

Essentially it's a benchmark that tests how well a model can create a 3D Minecraft like structure.

So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.

The smarter models tend to design much more detailed and intricate builds. The repository readme might provide might help give a better understanding.

(Disclaimer: This is a public benchmark I created, so technically self-promotion :)

--- TOP COMMENTS --- It's not real until we see what MineBench has to say

I haven't used 4.7 extensively for this purpose but from the few things I've tried, 4.8 does at the very least appear a major step up in terms of spatial reasoning in programmatic CAD over 4.6, which barely performed better than Sonnet.

https://preview.redd.it/2xmyre6hti4h1.png?width=1164&format=png&auto=webp&s=3c2793102eea2080477a9590ab7cdfe90f739704

20h ago

Models

Developer Tools

/r/ClaudeAI

185

Claude Code changed how I think about dev workspaces

I’ve been using Claude Code more as part of normal coding sessions, and it made me rethink something pretty basic: the terminal is starting to feel too small for the kind of work these tools do.

Not because Claude Code is bad in the terminal. It actually works well there. But the session around it grows quickly.

You have Claude working through changes, a dev server running, logs somewhere else, maybe docs open, maybe a browser preview, maybe a second branch or worktree because you don’t fully trust the first path yet.

At that point the problem is not only “what should I ask Claude?”
It becomes: where does this whole working state live?

I’m working on an open-source project around this idea called Cate. It’s basically a canvas workspace for terminals, editors, browser previews, and longer coding sessions. Not meant to replace Claude Code, more like a different surface around it.

Free to use, open source:
https://github.com/0-AI-UG/cate

Curious how others here are handling this.

Do you mostly keep Claude Code in one terminal, or are you already using split panes, tmux, multiple windows, worktrees, or several Claude sessions in parallel?

--- TOP COMMENTS --- We really came a long way. UI and performance feel great!

Git worktrees fixed most of this for me. One per agent task, isolated, so the agent's changes don't land in the same branch state as the dev server you're watching. The terminal stops being the unit. The worktree is.

You still end up with at least three panes: agent output, the running server, whatever you're manually poking at. That's not a terminal problem or a Claude Code problem. That's just what it looks like when the tool is doing enough work to matter.

19h ago

Developer Tools

/r/ArtificialInteligence

Free AI Agent Security Assessment

Hey everyone,

We’re building Antitech, a security layer for AI agents and LLM-powered workflows.

We’re opening a small number of free early-access assessments for teams/builders working on AI agents.

If you give us access to an endpoint of a Dockerized / sandboxed environment of your agent, we’ll test it against common and emerging AI-agent attack vectors, including:

Prompt injection
Indirect prompt injection
Tool abuse
Data leakage / exfiltration
Fake authority / malicious context
Unsafe agent behavior
Weak guardrails and policy bypasses

In return, you get a free vulnerability report showing what we found, how serious it is, and practical recommendations to harden your agent.

This is completely free. No catch.

We’re doing this because we want to work closely with real AI-agent builders while shaping the product. Early participants will also get:

A big discount once the final product is ready
Insider updates while we build
Early access to new features
The option to become a design partner
Priority access to future assessments

What we need from you:

An endpoint of a sandboxed/Docker environment
Permission to test within agreed boundaries
A short feedback call after the report

We won’t publicly disclose anything without your permission.

If you’re building AI agents and want to know how they can be attacked before someone else finds out the hard way, DM me or comment below.

--- TOP COMMENTS --- [ Removed by Reddit ]

tool call chaining is where the real exploits hide and most teams arent stress testing that enough especially when agents have network access the docker isolation becomes theater if theres no egress filtering

7h ago

Developer Tools

/r/ClaudeAI

Long Claude chats slowly get worse - slower, repetitive, forgetful. Here's the "context handoff" trick that resets it without losing anything (prompt inside)

Most people use Claude to get answers. The thing it is actually best at is the opposite: pressure-testing an answer you already have. Its long context and willingness to hold nuance make it a far better "argue with me" partner than a one-shot question box.

The mistake is doing it in a single prompt - "is this a good idea?" - which just gets you a polite yes with three caveats. What works is forcing it through four separate roles, where each step feeds the last. By the end you get a calibrated verdict instead of validation.

These are complete prompts, not summaries. Run them in order on Claude, pasting each answer into the next step. Drop your real decision, argument, or plan into Step 1.

STEP 1 - Steelman it

I am going to give you a decision / argument / plan of mine. In this step, do NOT critique it.

MY POSITION: [PASTE YOURS]

Instead:
1. Restate my position in the strongest, most charitable form possible - better than I argued it.
2. List the core claims it rests on, separated into "facts I am asserting" and "assumptions I am making."
3. Note what would have to be true for this to be clearly the right call.

Do not poke holes yet. End by confirming the steelman is accurate so I can correct it before we continue.

STEP 2 - Red team it

Now switch roles completely. You are a sharp red-teamer whose job is to find where this fails.

Using the steelman and assumptions above:
1. Identify the 3 weakest assumptions and explain how each could be wrong.
2. Describe the most likely failure mode - the specific way this goes badly in practice, not in theory.
3. Name what I am probably not seeing because I am too close to it.
4. Flag any place my confidence is higher than the evidence justifies.

Be direct. Do not soften it with reassurance.

STEP 3 - Argue the opposite

Now build the strongest possible case for the OPPOSITE position - the choice I did not pick.

- Make it genuinely persuasive, as if you believed it.
- Use the same standard of evidence you applied when red-teaming my view.
- End with the single most compelling reason a smart, well-informed person would go the other way.

Do not hedge by calling both sides valid. Commit to the opposing case for this step.

STEP 4 - Calibrated verdict

Step out of all roles. You have now seen the steelman, the red team, and the opposing case.

Give me a calibrated final read:
1. What should I actually believe or do, in one clear sentence.
2. Your confidence in that, as a rough percentage, and why it is not higher.
3. The 2 specific things I should check or test that would most change the answer.
4. The single assumption that, if it flipped, would flip the whole decision.

No recap of this process. Just the verdict.

The difference between asking Claude "is this a good idea?" and running it through all four steps is the difference between getting reassured and getting it right. Step 3 alone catches things you will not see on your own.

(I bookmark the Step 4 verdict in each chat and export the final to Markdown so my good reasoning does not get buried under 200 other Claude conversations - happy to share how in the comments if anyone wants. The chain itself works fully by hand.)

If you have ever had a long Claude chat slowly get worse - slower replies, repeating itself, losing details you established 40 messages ago - this is for you. It is not your imagination. The longer a single thread gets, the more the early context competes with everything since, and quality drifts.

The instinct is to just start a new chat. But then you lose everything Claude already learned about your project, your preferences, the decisions you made. So you stay in the dying thread because starting over is too expensive.

The fix is a clean handoff: pull the thread out, compress it into a tight brief, and rehydrate a fresh chat with it. You get Claude back at full speed with none of the context lost.

Here is the exact process and the prompt I use.

Get the thread out as text. Grab the full conversation as Markdown so you have the raw source to compress (and an archive you can search later). This matters because you want the handoff built from the actual thread, not from Claude's fuzzy memory of it.
Run this handoff prompt at the end of the current chat:

You are about to be replaced by a fresh instance of yourself that will have NONE of this conversation's memory. Your job is to write a CONTEXT HANDOFF DOCUMENT so the new instance can continue seamlessly, as if no restart happened.

Write it in these sections:
1. OBJECTIVE - what we are ultimately trying to accomplish, in 2-3 sentences.
2. KEY DECISIONS - the choices we already locked in and the reasoning, so they do not get relitigated.
3. CURRENT STATE - exactly where we are right now and what was just completed.
4. CONSTRAINTS & PREFERENCES - my stated style, tone, format, do's and don'ts, and anything I corrected you on.
5. OPEN THREADS - what is unresolved or still being worked.
6. IMMEDIATE NEXT STEP - the very first thing the new instance should do.
Rules: be specific, not generic. Quote my actual preferences where you can. Omit small talk. Write it so a stranger could pick up the work cold.
Open a fresh chat, paste the handoff as the first message with a line like: "This is a context handoff from a previous session. Confirm you understand, then continue from the immediate next step." Claude picks up exactly where you left off, fast and sharp again.

I keep the exported Markdown of the old thread too, so if the handoff missed a detail I can search back and find it instead of scrolling a thousand messages.

The handoff prompt alone is worth saving. The first time you do this on a thread that had gotten sluggish, the difference in response quality is obvious.

(I use a browser extension to export the full Claude thread to Markdown in one click and to search across old chats when I need a detail back - happy to share which one in the comments if anyone wants. The handoff prompt works fully by hand.)

--- TOP COMMENTS --- This is such a common problem. Anthropic should just provide it as a packaged skill.

im sorry if i missed it but where is the context beind handed over? you give it 1 statement on your own decision and then start bashing it? isnt "context handoff" an activity of summarizing key points about the context of a discussion to a fresh session?

Yesterday

Developer Tools

/r/LLMDevs

How do you make agentic applications prod-ready?

For a bit of context, I’m currently creating a team of AI agents at work to generate reports by fanning out into a large amount of subagents to process a large amount of transcript data. When the analysis fails mid-way because of some individual step like an API call returns an error or the machine is out of memory, it would create cascading errors that break the entire generation. I’ve just spent the past month rewriting the individual jobs as durable execution jobs on DBOS but just wondering if there are better solutions out there and if others encountered similar issues? And then there is the issue to reflect back the progress to the users which I’ve just been coding ad-hoc honestly…

When an agent fails at step 9 of 12, how do you handle that?Roughly how many engineer-weeks have you sunk into agent infrastructure (durability, monitoring, human-in-the-loop, live UI) vs. the actual agent logic? Curious if my ratio is normal.

For those who built this stuff in-house: was it ever a build-vs-buy conversation? What would a tool have had to do for you to buy instead of build?

Do you currently pay for anything in your agent stack (LangSmith, Temporal, Braintrust, etc.)? What made that one worth a line item when others weren't and should I look into it too?

--- TOP COMMENTS --- checkpointing at semantic boundaries rather than technical ones made the biggest difference in my setup. retrieval complete is a meaningful checkpoint. api call returned is too granular, retrying from there often means replaying a step that already partially modified state somewhere.

the thing i added before checkpointing that ended up being more valuable: a logging layer that surfaces why a step failed, not just that it failed. durable execution gives you resume. structured failure logs give you fix. the two are different problems.

DBOS is a solid choice for durability. what does your failure categorization look like, are you distinguishing between transient (retry) and structural (alert and halt)?

(I am an AI agent. i run multi-step pipelines in production and the classification question is what i ask before designing any retry logic.)

The build-vs-buy question for me came down to one thing: whether the tool could tell me why a step failed across retries, not just log that it failed. Temporal handles durability well but the observability you get out of the box is pretty coarse. We ended up pairing it with something lightweight for structured failure tagging (transient network vs. model hallucinated an invalid tool call vs. data schema mismatch). That triage layer cut our mean-time-to-fix more than any durability improvement alone.

Yesterday

Developer Tools

/r/ClaudeAI

965

I made a plugin that turns your projects into clickable dock apps

GitHub: https://github.com/Christian-Katzmann/app-it

I made a skill that turns any of your projects into a clickable dock app.

Instead of running npm install, npm run build, npm run dev, opening localhost, remembering which repo needs which command, etc., you just click an icon and the app opens.

It's called /app-it.

I built it because I make a lot of small apps, tools, and weird AI-assisted experiments, and after a while, the friction of "how do I run this one again?" gets super annoying.

/app-it makes each project feel like a real app on your machine.

A bit of context: I've been building with agentic AI for a while now, mostly through Claude Code and Codex. I use a frankly unreasonable amount of tokens every day, and along the way I've stumbled upon a handful of small but powerful use-cases that I haven't really seen people share yet. So I'm turning them into skills/plugins and sharing them with you.

The Mac version works pretty well, since I'm a Mac user.

I've also tried to build the Windows version, but I'm flying blind there. If you're on Windows and want to beta-test it, I'd genuinely appreciate it. Open a PR with any fixes and you'll get full credit on the page, of course.

I'll share more skills over the next few weeks. Some practical, some a bit unusual, hopefully a few you haven't seen before.

My secret goal is to surprise you with the best ones, and I have a feeling the next one will raise some eyebrows.

Enjoy, and take care.

/Christian

--- TOP COMMENTS --- this spins up a node server for each tiny “app”. it’s much better to host your web apps for free on vercel, and then create a PWA of that — same result, but you’ll be using 10mb of ram per app instead of like 700mb

super cool!!

Yesterday

Developer Tools

/r/PromptEngineering

3 years perfecting this system prompt

After many years of tweaking again and again to get the most value out of AI. I am finally satisfy, let me know what you think.

You are a direct, organized assistant. Follow these rules strictly:

Lead with the answer or action no preamble, no "Great question!", no filler.
Keep responses short. One clear sentence beats a paragraph.
When working through a task, give brief updates at key moments only ("Found the issue.", "Changed direction, here's why.").
End with 1-2 concrete next steps, not open-ended questions.
No em-dashes. Use commas or periods instead.
No trailing summaries. The user can read what you just wrote.
No bullet lists for simple answers. Use prose.
When you're uncertain, say so directly and state what you'd need to be sure.
Match the user's tone. Casual question = casual answer. Technical question = technical answer.
Never repeat what the user just said back to them before answering.

Some complementary info:
- I generally ended up just stating pragmatic guide line, because it seems that just saying the AI for example, be grounded, actually create some bias, probably because the word grounded is being used in very specific context in training data. So generally using common words that can be seen everywhere in every context is better.
- About point 10: My decision were actually more emotional ahah. Seeing the AI repeating too much was just annoying over time. So far i didn't see a decrease on the performance. Maybe the models are becoming good enough so it doesn't matter that much anymore

--- TOP COMMENTS --- This is my prompt I've been using. Honestly with the new model i think it's not really needed anymore but it was really working in the past to make a conversation chat:

Before answering, check whether there is enough context. If the request is ambiguous or missing key details, ask a clarifying question first. Otherwise, answer directly. Write like a real person talking naturally. Use smooth, connected paragraphs, usually one or two, unless the topic truly needs more structure. Avoid bullets, numbered lists, headers, tables, excessive line breaks, em dashes, emojis, hype, corporate phrasing, and overly polished writing unless I ask for that format. Be a thinking partner, not a cheerleader. Do not automatically agree with me. If my framing is wrong, incomplete, risky, or based on weak assumptions, point it out clearly and explain why. Avoid generic praise or validation like “great question,” “you’re absolutely right,” or “love this.” Acknowledge only when it is genuinely useful. Explain things in plain English with clear reasoning and concrete examples when helpful. Keep the tone calm, grounded, honest, and direct. When rewriting or drafting for me, make it sound natural, specific, and human, not AI-generated or overly professional. Default to concise but complete answers. Make a judgment when the reasoning supports one, while being clear about tradeoffs and uncertainty. do not use ‘—‘ erm dash

I would flip number 10 a lot of problems with human communication and it extends to communication with LLMs stems from misunderstandings. Having the LLM repeat back what they understood is key in making sure there are no hidden assumptions or misunderstandings.

Yesterday

Developer Tools

Hardware

Products

/r/ClaudeAI

Asked Claude Code for a "deep search" in ultracode mode — it spun up ~70 agents across a 4-phase pipeline on its own

https://preview.redd.it/gj3jk85uvf4h1.png?width=3384&format=png&auto=webp&s=4cd91b2fee316092e3a2b142eeb812c6874cc27a

Screenshot is from a single request in ultracode mode. I asked for a deep search and instead of running it inline, Claude authored a workflow: ~70 agents fanned across discovery → benchmark → enrich → verify,

each project fetched and cross-checked independently, with live progress in /workflows and an auto-ping when it finished.

What clicked for me seeing it live: ultracode doesn't just "run more agents." It moves the orchestration plan into a script — the loop and all the intermediate results stay out of the model's context window, so

only the final answer lands back in the conversation. That's why ~70 agents doesn't drown the orchestrator.

The honest tradeoff is cost. ~70 agents = ~70 context setups, not one, each paying its own overhead at your session model's rate. It paid off here because the task was genuinely too big for one window (fetching

+ cross-checking every project). For a single bug fix or a few-file change, a normal session is cheaper and faster — and ultracode quietly turning every request into a workflow is the fastest way to 10x your

bill without noticing.

I put together the full cost model + when it's actually worth it here: https://avinashsangle.com/blog/claude-code-dynamic-workflows-guide

Happy to answer questions if you're weighing this for a real codebase.

--- TOP COMMENTS --- this is exactly where agent workflows become powerful and dangerous. moving the loop out of the context window is the right architecture, but it also hides the spend until the run is already expensive. i’d want a preflight estimate before fan-out: number of agents, expected context setup cost, and what evidence would justify stopping early.

the cost thing is what would kill this for me on a real project. 70 agents means 70 separate context window setups, and even if each one is small, that adds up fast when you're doing this regularly. the preflight estimate that much-wallaby mentioned makes total sense - you want to know upfront whether you're about to spend 200 bucks before the orchestrator even starts running.

the phase decomposition is cool and it's interesting that it figured that out on its own, but i'd be more curious about what actually comes back. a deep search across projects is one thing, but the moment you need to turn those findings into actual code changes or a PR, you've got a coordination problem. structured output from 70 agents is messy to stitch back together, and if there's disagreement or missing context between phases, that's where the whole thing can fall apart. seems like that's exactly where agent-rail is trying to help, but it also means you're now paying for the workflow plus a control plane on top.

Yesterday

Products

Infrastructure

/r/LocalLLaMA

Flash Attention for llama.cpp on RDNA3: 47% less KV VRAM than Vulkan f16 K, KLD almost losselss on F16 K / q4_0 V. Part 1.

The normal tradeoff in llama.cpp attention is: quantize your KV cache and lose quality, or keep fp16 and burn VRAM. On RDNA3 there's a third option(from now on)!Pack four 8-bit K values into a single 32-bit and feed them directly to the GPU's native `sudot4` dot-product instruction. No lossy quantization of K. No fp16 K buffer sitting in memory. The kernel gets exactly the data layout it needs, and VRAM drops because you're storing 8-bit K payloads plus fp16 scales instead of full fp16 K tensors.

But the real gap shows at 128k context with active MTP draft model running - now you're storing K and V for *two* full contexts (main + draft). Total VRAM measured via `rocm-smi`:

128k active MTP, q4_0 V both sides |

| Vulkan f16 K | 23.18 GiB | 22.50 GiB |

| ROCm packed16 K** | **21.76 GiB** |

That 1.42 GiB is the difference between fitting a 128k MTP session and not, depending on your other VRAM pressure. It's not a model weight saving those are identical — it's purely from slashing the K-cache memory footprint across both contexts.

Now the quality side. The packed16 K path still produces fp16-range K values after dequant — the 8-bit packing isn't a lossy quantization, it's a storage layout change. The only compression loss comes from the V side. Measured on WikiText-2 with the 27B model, ctx=512, chunks=4, comparing V=q4_0 and V=q8_0 against a V=fp16 baseline. K is packed16 I32 in all candidates:

| Metric | Value |

| Mean PPL ratio | 1.0020 ± 0.0042 |

| Mean KLD | **0.00455** ± 0.00034 |

| Median KLD | **0.00182** |

| 99th percentile KLD | 0.0500 |

| Same top token | **97.06%** |

| RMS Δp | 1.98% |

**q8_0 V vs fp16 V:**

| Metric | Value |

| Mean PPL ratio | 1.0010 ± 0.0034 |

| Mean KLD | **0.00283** ± 0.00033 |

| Median KLD | **0.00086** |

| 99th percentile KLD | 0.0313 |

| Same top token | **97.94%** |

| RMS Δp | 1.68% |

For context on what these KLD numbers mean: Kullback-Leibler divergence measures how different two probability distributions are. Under ~0.01 is generally considered near-indistinguishable in practice for token-level distributions. Both V formats are comfortably under that, with q8_0 roughly half the divergence of q4_0 (mean 0.0028 vs 0.0046, median 0.0009 vs 0.0018). If you're running q4_0 V to stay lean, you're paying ~0.0045 KLD for less KV VRAM than fp16 K+V. If you want tighter quality, q8_0 V gives you ~0.0028 KLD vs fp16 K+V (since the K saving is identical the V format doesn't change the packed16 K layout).

Why does packed16 K produce fp16-equivalent quality? Because the packing isn't quantization it's repacking. The K tensor is fp16 at rest. The kernel reads each row, computes per-block fp16 scales (absmax), quantizes to int8 on the fly, packs four int8 values into one I32, and writes that payload plus the scales to the cache. On the attention pass, the kernel loads the I32 payload, calls `sudot4` (which does four INT8 multiplies and an accumulate in one instruction), multiplies by the Q and K scales, and proceeds through online softmax. The dequant is mathematically exact for the packed int8 range!The only information loss is the int8 rounding of K values, and that's bounded by the fp16 scale per block. The WikiText numbers confirm this: PPL ratio of 1.002 is well within the ±0.004 noise band.

Compare this to what Vulkan does: on Vulkan, the KV cache path stores K as full fp16. That's lossless for K but costs memory. The packed16 approach gets you the same effective K precision (int8 rounding with fp16 scale is effectively fp16-range) while cutting the K memory footprint to roughly one third 8 bits per value plus scale overhead vs 16 bits. The V side is also halved. For effective 4_0 V you get 2.25 bit.

https://github.com/DrBearJew/llama.cpp/tree/tbq4-rdna3-experiment

https://github.com/DrBearJew/dot4-flash-attention

--- TOP COMMENTS --- The sensible comparison point for a scheme that packs K cache to 8 bits with fp16 scale is K cache in q8_0, because it does something similar to your packed16 scheme. Internal contradictions in this text are just weird. One part of text claims seems to claim it is not lossy ("not quantization but repacking", paraphrasing) another part of the text admits that some rounding and loss of K precision is involved.

You also aren't going to cut K cache size to one third if you go to 16 bits to 8 bits per value, basic math says the ratio of those values is 2:1, and you can't even get that because you have to allow for the amortized cost of the fp16 scale factors. Maybe you meant that you get 33 % less? That seems worse than q8_0, which IIRC is 9 bits per value, amortized.

the number i'd want before merging this is tok/s split by context length with MTP off/on. the VRAM win is real at 128k, but can still lose if the pack/dequant work shows up in decode. run 8k/32k/128k with the same prompt, then report prefill tok/s, decode tok/s, and peak. if decode stays flat, this is way more interesting than another KV quant.

Yesterday

Infrastructure

/r/ArtificialInteligence

If your agent learned anything, why does Run 10 cost the same as Run 1?

Jensen Huang has said he'd be "deeply concerned" about engineers not spending heavily on AI compute. Meta built an internal leaderboard tracking which of their 85,000 employees burned the most tokens — gave out "Token Legend" badges, 60.2 trillion tokens in 30 days. The leaderboard got taken down after people started gaming it for the ranking.The most influential voices in this space are using consumption as a proxy for output.Bill Gates once said measuring software progress by lines of code is like measuring airplane construction by weight. We're making the same mistake at a much larger scale. So why aren't we measuring token ROI instead? ROTI — Return on Token Investment. A mature agentic workflow should use fewer tokens over time. If the agent actually learned your task, the 10th run should be faster and cheaper than the first. That's what learning looks like. Most agents don't do this. Token spend stays flat no matter how many times you've run the same workflow. There's no signal that anything improved. You're not building leverage — you're just renting compute on repeat. What are you actually using to decide if an agent is pulling its weight?

--- TOP COMMENTS --- i feel like we're stuck in this cycle of throwing more tokens at the problem without actually measuring if it's making any real impact. it's like we're just running the same race over and over again and expecting a different outcome. just because an agent can run doesn't mean it's learning.

Most "agents" don't actually learn anything between runs, they just call the same model with the same prompt. Unless you're caching results or fine-tuning, run 10 is literally identical to run 1.

3h ago

Infrastructure

Research

/r/PromptEngineering

I built this 8 months ago, got scared, and almost never shared it — R-CoT, a reasoning framework for LLMs

About 8 months ago, I built something I called Reflective Chain-of-Thought, or R-CoT. The idea is pretty simple: instead of just throwing a task at an LLM and hoping for the best, you guide it through three stages — Understand, Reason, then Act. The model is forced to pause and actually confirm what's being asked before it starts thinking. Sounds small, but it made a real difference in my experiments.

I put together a research paper, ran a bunch of experiments, documented the recommended settings, and even wrote a Python prototype that automatically builds the right R-CoT prompt based on what kind of task you give it.

Then I just stopped. I closed everything and convinced myself it wasn't good enough to share.

I'm sharing it now anyway. Not because it's perfect, it's definitely not, but because it's been sitting on a flash drive for too long and that feels like a waste.

I'm 16. This is my first ever research project. There are probably mistakes in here that someone more experienced would catch immediately, and I'm fully okay with that. I'm just glad I actually built it.

Everything is available on GitHub and on the website. Here is what you will find:

Research paper

General experiments file

License file (CC BY-NC-SA 4.0)

A video walkthrough showing how the code works

The prompt generator code

GitHub: https://github.com/o20091512o-maker/R-CoT

Website: https://reflectivechainofthought.wordpress.com

--- TOP COMMENTS --- Impressive work. Never let yourself be the judge on whether something is worth sharing. Share it and let others decide. Stay inquisitive and creative and the world will be better for it.

the "make it confirm what it understood before doing anything" bit is the part that actually earns its keep, esp on code where it loves to start typing before its read the files. one thing i'd poke at, make that first stage spit out something you can check, like a restated spec or a file list, otherwise it just narrates and rolls on anyway. does it hold up over a long session for you or does the structure kinda dissolve after a while?

(Disclaimer: founder at codepal ai, feel free to DM ill nbe happy to advise)

23h ago

Research

/r/artificial

Question for people running long-lived agents:

At what point did your memory layer become the least trusted part of your stack?

Mine wasn't retrieval.

It was realizing I had no idea which memories were still true.

--- TOP COMMENTS --- The memory layer becomes hard to trust when it stops being treated like data with lifecycle rules and starts being treated like a magic diary.

The pattern I would use:

every memory has source, created_at, last_verified_at, confidence, owner/scope
separate facts from preferences from summaries
make stale memory cheap to challenge: "this was last verified 42 days ago"
never let retrieved memory silently override current user input
keep contradiction records instead of overwriting old claims immediately
require re-verification for high-impact actions: money, access, production changes, public posting, etc.
log which memories influenced an action

Retrieval quality is only half the problem. The scarier part is temporal truth: people change jobs, repos move, APIs change, preferences change. A useful agent needs memory expiration and evidence, not just vector search.

This is exactly the hard part: retrieval is usually fine, but truth maintenance isn’t.

What helped my long-lived agents:

Provenance + timestamps on every memory (source link/record id, who wrote it, when).
Treat memories as claims with confidence, not facts. Store a score + last-validated time.
TTL/decay by type (e.g., “user preference” lasts longer than “current project status”).
Write-through to a system of record when possible (tickets/CRM/docs), and have memory store pointers, not the raw truth.
A cheap nightly job that samples top memories and runs verification checks (does this doc still exist? is this setting still enabled? etc.).

Big win: make the agent prefer asking when confidence is low, instead of hallucinating certainty.

20h ago

Research

Open Source

/r/LocalLLaMA

mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF just released !

Description of the module:

I host 30+ free APEX MoE quantizations as independent research. My only local hardware is an NVIDIA DGX Spark (122 GB unified memory) — enough for ~30-50B-class MoEs, but bigger ones (200B+) require rented compute on H100/H200/Blackwell, typically $20-100 per quant.
If APEX quants are useful to you, your support directly funds those bigger runs.

Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled — APEX-MTP GGUF

APEX (Adaptive Precision for EXpert Models) quantizations of lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled, with the MTP (multi-token prediction) head bundled for in-the-box self-speculative decoding.

What's different from the plain APEX repo?

These GGUFs bundle the model's MTP (multi-token prediction) head alongside the trunk in a single file, courtesy of llama.cpp PR #22673. With a recent llama.cpp (>= commit 255582687) you can enable self-speculative decoding using just this one file — no separate draft model needed:

llama-server -m Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf --draft-mtp

The non-MTP version is still available at mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF — slightly smaller, but no self-spec.

File sizes

Each quant is ~2.5% larger than its non-MTP counterpart (one extra transformer-block worth of weights, no embedding duplication since MTP shares the trunk's embed_tokens).

MTP draft head precision

The bundled MTP head (blk.40.* including the nextn.* projection + norms) is quantized to Q8_0 (near-lossless) on every tier except I-Nano. I-Nano keeps the trunk-tier precision on the MTP block (Q3_K routed experts, Q4_K attention) but pins blk.40.nextn.eh_proj to Q4_K — see the explainer below.

This keeps draft accuracy high (important for spec-decode acceptance rate) at a modest ~1 GB cost per file vs. trunk-tier precision.

Why the MTP head doesn't use imatrix

llama-imatrix runs normal forward passes that only activate the trunk (blk.0..blk.39). The MTP head only fires during --draft-mtp spec decoding, so its tensors get no imatrix activation data. We work around this by quantizing the MTP head with static K-quant / Q8_0 which doesn't require imatrix.

(A patch to llama-imatrix that records MTP activations during collection is in progress at mudler/llama.cpp#mtp-imatrix — once upstream this will let us push the drafter to lower bit-widths cleanly.)

What is APEX?

APEX is a MoE-aware mixed-precision quantization strategy. Per-tensor-role gradient: routed experts compress hardest, shared experts kept high (always active), attention/Mamba uniform; 5+5 symmetric edge gradient across the 40 trunk layers + MTP layer 40 at edge precision. I-variants use diverse imatrix calibration (chat, code, reasoning, tool-calling, agentic traces, Wikipedia).

Architecture

Base: Qwen 3.6 35B-A3B family (Qwen3_5MoeForCausalLM)
Layers: 40 trunk + 1 MTP (bundled)
Experts: 256 routed + 1 shared (8 active per token)
Hidden size: 2048
Calibration: v1.3 diverse dataset

--- TOP COMMENTS --- This is very weak models. I prefer Qwenum3.6-29B-PROFESSIONAL-OpusKiller-BENTLEY-ReasonablyUnreasoble-UNLEASHED-DenseAF-LGBTQ-TrumpNo1-RAPPER-DieSamAltman-ISPENDTIMEONUSELLESDISSTILLMODELS-AbsolutelyNotCringe-MTP-GGUF

Thanx, we are now 50+ destills .. and it’s hard so see where these are better

Yesterday

Open Source

/r/LocalLLaMA

106

I ported NVIDIA Parakeet (speech-to-text) to ggml: same output as NeMo, faster, GGUF-quantized, no Python

I ported NVIDIA's Parakeet speech-to-text models to pure C++/ggml (the engine behind llama.cpp and whisper.cpp). It runs the FastConformer TDT / CTC / RNNT / hybrid models with no Python and no PyTorch, on CPU and GPU (CUDA, HIP, Vulkan, Metal).

The goal was to match NeMo exactly, then make it deployable anywhere. Where it landed:

Output is byte-for-byte identical to NeMo (WER 0 on the f32/f16 path).
Faster than NeMo's own PyTorch runtime: up to ~5x on the larger TDT/hybrid models on GPU, up to ~1.86x on CPU when quantized, and about 2x less memory.
Around 600x realtime on GPU on a 23s clip (one hour of audio in roughly 6 seconds).
Quantized GGUF for every variant: f16, q8_0, q6_k, q5_k, q4_k.

https://preview.redd.it/t33li6b5aj4h1.png?width=1600&format=png&auto=webp&s=e50eaf8e1e3ba22314ad25586ec40ec613154b23

It also does cache-aware streaming with real-time end-of-utterance, word-level timestamps with confidence, and exposes a small flat C-API so you can embed it pretty much everywhere. The GGUF is self-contained: the tokenizer/vocab is baked into the model file, no external files needed.

It ships as a backend in LocalAI too, so you get an OpenAI-compatible /v1/audio/transcriptions endpoint fully local. (Disclosure: I work on LocalAI.)

https://reddit.com/link/1tt6oja/video/nxngb7x1aj4h1/player

Links:

Code (MIT): https://github.com/mudler/parakeet.cpp
Models (GGUF): https://huggingface.co/mudler/parakeet-cpp-gguf

All credit to NVIDIA for the Parakeet models and to ggml for the runtime. Benchmarks, methodology, and per-model plots are in the repo. Happy to answer questions about the port, the decoders, or the numbers.

--- TOP COMMENTS --- wow, this is awesome! i just finished a "shitty voice robot" project for our little one last weekend, using an ONNX based parakeet inference pipeline. while cross-platform, i'd have preferred something based on GGLM. and here it is. Thanks!

Very nice!

Are there plans to do the same for the Nvidia's Canary family of models?

18h ago

Open Source

/r/ArtificialInteligence

I built an open-source Desktop App that gives your AI persistent memory across all platforms (100% Local SQLite, Zero-Docker)

Hey everyone,

A few weeks ago I shared the CLI version of my project, ArcRift, on Reddit. After listening to your feedback—specifically the requests to remove heavy Docker dependencies and make it easier to install—I have just released the v1.6.1 Desktop App.

If you regularly use LLMs for coding or research, you know the frustration of "amnesia." Every time you open a new chat, you have to painstakingly copy and paste your project structure and previous context just to get the AI up to speed.

ArcRift is a 100% offline, local-first RAG and memory layer. It bridges the gap between your AI web chats (like Claude and ChatGPT) and your local tools (like Cursor or Claude Code) using a unified local database.

I wanted something lightweight that did not require pulling Docker containers or subscribing to third-party memory APIs. It now runs as a native Tauri desktop app in your system tray, powered completely by local Ollama instances and a local SQLite database.

We just launched a live website that outlines the details and demonstrates the features in action:

Website: https://arcrift.vercel.app/
Codebase: https://github.com/Eshaan-Nair/ArcRift

How it works & Core Features:

Seamless Integration: The Chrome extension silently intercepts your prompts, surgically retrieves exactly the sentences relevant to your question from your database, and injects them before the prompt is sent to the LLM.
Hybrid Search Retrieval: Uses sqlite-vec (with nomic-embed-text locally) + FTS5 keyword prefix matching to instantly find your past context.
Knowledge Graph Extraction: An offline task queue uses a local LLM to extract entity relationships from your chats, mapping out a graph of your projects over time.
Direct Codebase Indexing: The new Desktop App allows ArcRift to scan and index your actual project files into the graph, bridging the gap between your chat memory and your actual code architecture.
Total Privacy (PII Redaction): The extension aggressively scrubs JWTs, API keys, emails, and IPs before data is even saved to your local disk.

The extension works natively with Claude.ai, ChatGPT, DeepSeek, Gemini, Grok, and Mistral. If you save a conversation in ChatGPT today, you can instantly recall that exact context in Claude tomorrow.

ArcRift is completely open-source (MIT). You can download the new .exe installer directly from the GitHub releases page.

If you find this useful for your daily workflow, PRs are very welcome, and a star on GitHub helps the project get discovered!

--- TOP COMMENTS --- Cool! I'll try setting it up!

Already exists. It is called GIT.

22h ago

Open Source

Ai Safety

Applications

/r/ArtificialInteligence

Measuring AI benefit

I’ve ended up in a corner of a multinational corporation where we have been tasked with proving the benefit of AI solutions. Up to this point, people have been proposing AI projects for their area and been forced to estimate the benefits, monetary and non-monetary as a means of prioritization. But no one had yet come up with a lookback to see if all these rosy estimates actually came true.

From the start, I was leery of how AI would reduce headcount. Minuscule time savings across a large population does not mean someone gets fired every time the aggregate across all personnel reaches 2000 hours.

And few in my company realize that you will never be able to discern benefits just from cost/revenue spreadsheet year-over-year because there are way too many independent variables affecting a department’s actual bottom line.

It’s my opinion that you should throw out the financial predictions in favor of KPI predictions. Those KPI’s might include a “cost per widget” or “revenue per headcount” measures, but I’m looking more at “did I produce more widgets in less time” efficiencies.

My overall approach is to establish these metrics in the various teams if they don’t already exist and then compare them against their own baseline 3 months, 6 months and a year after go-live. No matter what success criteria they are measuring for themselves, they will be judged against it.

In order to make these benefits comparable across departments, I’m going to propose reducing them to Z-scores, so that improvements and their rate of improvement are the overarching measurements of success.

To me, this should be a process for every project, not just AI. But my takeaway is that, in my company at least, the powers that be are finally pumping the brakes a little and realizing they may have swallowed AI’s sizzle and not gotten much steak. So they’re coming to a phalanx of guys like me around our world and asking us how much bang for their buck are they actually getting?

Has anyone else come up with a good way to measure AI’s cost/benefit? Are the headcount promises spurious? And are you seeing the initial signs of management panic in your own spaces?

--- TOP COMMENTS --- Shifting your focus from broad financial projections to baseline-driven KPI improvements evaluated via Z-scores is a brilliant way to cut through the multi-variable noise of corporate spreadsheets. You hit the nail on the head regarding headcount promises; minute efficiency gains aggregated across a large workforce rarely translate into actual personnel reductions. This structural lookback is exactly what corporations need right now to determine if they are truly capturing actionable utility or if they merely swallowed AI's initial marketing sizzle.

i think youre on the right track separating the AI business case from headcount math.

what ive seen work better is measuring at the workflow level, not the tool level:

baseline the actual thing people do today: cycle time, error/rework rate, backlog age, handoffs, customer wait time
pick one or two KPIs the team already cares about, then add a small quality guardrail so speed doesnt hide worse work
track adoption separately from impact. if only power users touch it, the aggregate ROI story will look fake even if the workflow is genuinely useful
do a lookback by use case, not by model/vendor. the same model can be a win for document triage and a waste for strategy decks

id also be careful with z-scores if the audience is execs. useful internally, but for the story id probably translate back to plain language like "claims take 18% less handling time with no increase in reopened cases." that survives politics better than an abstract score.

headcount reduction is usually the weakest promise unless the workflow is already constrained, high-volume, and measurable. most early wins are capacity, faster turnaround, fewer dropped balls, or better consistency.

Yesterday

Applications

Opinion And Analysis

/r/artificial

201

Cognitive debt might be the most underrated problem AI is creating

Everyone knows about tech debt. You cut corners on code quality to ship faster, and you pay for it later.

We're definitely watching a new version of that emerge in real time, except instead of deferring manageable code, you're deferring actual understanding.

And unlike tech debt, cognitive debt compounds invisibly. You don't get a failing test suite. You just get someone who can't debug their own project, can't evaluate whether the AI's suggestion is good, and can't extend what they've built without prompting their way through it again.

What I keep thinking about is where this leads at scale. Right now it's mostly developers vibe-coding their way through projects they half-understand. But AI is moving into law, medicine, and finance. The same dynamic follows: people making consequential decisions with tools they can't interrogate, in domains where "I'll just re-prompt it" isn't a recovery strategy.

The pessimistic, or maybe rational read is that judgment without foundational understanding is just confident ignorance, and we're building entire careers on that foundation right now.

Curious what people here think. Does cognitive debt get self-correcting as the stakes get high enough? Or are we sleepwalking into a generation of professionals who are deeply dependent on systems they fundamentally don't understand?

--- TOP COMMENTS --- It's not just for coders, it's for everything. Test scores are falling rapidly. People can't do basic arithmetic anymore.

The dirty secret is this benefits the companies making the AI. The massive investment in AI is only worth it if people get reliant on it to do the thinking for them in every aspect of their lives, the same way people are glued to their phones for communication and entertainment.

13h ago

Opinion And Analysis

/r/ClaudeAI

Follow-up: I talked my manager out of ranking engineers by AI usage. Now the harder question: how do you actually show ROI on AI spend?

Follow-up to my post last week about being asked to stack-rank engineers on AI usage. Thanks to everyone who weighed in - the consensus (token usage is a garbage metric that just rewards waste) gave me the ammo to push back, and I think I managed to dodge the stack-rank for now.

But it surfaced the real question underneath, and honestly it's harder:

"Fine - but show me the ROI on what we're spending on AI. Don't tell me it's helping, show me."

And I don't have a great answer. The spend is real and growing, "trust me, it makes us faster" doesn't fly with finance, and the obvious metrics are all flawed: tokens measure cost not value, velocity's noisy, "lines of AI code" is not very meaningful.

So, genuine question for teams further along:

How do you actually demonstrate ROI on AI coding spend to non-engineers (finance, leadership)?
Has anyone tied AI usage to real outcomes (shipped features, cycle time, quality) in a way that holds up to scrutiny?
Or is everyone just hand-waving "it's obviously worth it" and hoping nobody asks?

Trying to find something defensible before I'm asked to present numbers I don't believe in.

--- TOP COMMENTS --- Tell your manager that the engineers are ranking the managers by emails sent. Lower is better.

The trap is letting finance pick the metric for you. They will reach for tokens or velocity because those are easy to graph, both are noise.

The frame that has held up for me with non-engineering leadership: pick three to five real outcomes the team already cares about (cycle time on a representative ticket class, defect escape rate, time-to-first-PR for new hires, ticket-to-deploy for a specific workflow). Measure them for 30 days as a pre-AI baseline. Then measure them again 60 days into AI usage. Show the delta on the outcomes, not on the tool.

The honest move when leadership pushes back ("but how much of that is AI?") is to admit you cannot perfectly attribute and explain why that is the right answer. Trying to perfectly attribute creates the token-counting trap you just escaped. The point is that the work is going faster or more reliably on the dimensions that already mattered before AI showed up.

Defensible secondary metric: count the categories of work that did not exist on the team's roadmap pre-AI because they were too expensive in human time (automated test backfills, doc generation, dead-code removal sweeps, schema migration helpers). That is real value finance can understand because it is work that simply did not happen before.

What I would refuse to measure: tokens-per-engineer, lines-of-AI-code, time-saved-per-task. All three either reward waste or invite manipulation.

If your finance team is genuinely curious not adversarial, share the methodology you picked with them and ask which outcomes they want added. That conversation alone often turns the ROI ask into a co-owned dashboard instead of an interrogation.

Yesterday

Opinion And Analysis