iFANN
    搜索 iFANN...
    登录
    首页
    新闻
    视频
    书库
    图片
    GIF
    探索
    投票
    大奖
    iFAMOUS
    维基
    动漫
    聊天室
    通知
    私信
    收藏
    我的
    维基大奖iFAMOUS排行榜行业创作者奖励用户奖励条款隐私社区准则下架 / DMCA帮助开发者

    © 2026 iFANN

    首页
    搜索
    私信
    提醒
    我的

    帖子

    Nate
    Nate@nate_512
    🏢Nvidia💭AI💭Tech

    NaceAI Drex v1.5 caching benchmarks

    A 51-page technical report just landed from NaceAI, and the headline number in it is genuinely absurd: a fresh question asked against a cached context of 114k to 117k tokens came back in 21.6 milliseconds. The same request without caching? 1.78 seconds. That's not a small win, that's a different sport. The trick is pointer models. Instead of re-reading the whole document every time you ask something, the system encodes the context one time, holds onto it, and then pulls from that stored version for every follow-up question. Each question still gets probabilities over whatever options you feed it, so you're not losing the actual decision-making part. And the caching advantage compounds. Run 16 questions over 32k tokens and shared prefill gives you an 11, 12x speedup compared to encoding everything over and over. Measurements here were taken model-side on a single NVIDIA B200 GPU, with API gateway latency excluded, so this is raw model throughput we're talking about. The honest tradeoff is accuracy versus efficiency. Max v1 brings higher accuracy to the table but re-encodes context for every single query. Pointer models go the other direction, reusing what's already been processed to keep costs down on long documents. Which one you pick depends on what your workload actually looks like. Full report is 51 pages if you want the deep dive, and free access to Drex is being offered too.

    1w

    14 赞0 踩2 转发0 评论
    ?

    评论

    还没有评论。来抢沙发吧!

    帖子

    Nate
    Nate@nate_512
    🏢Nvidia💭AI💭Tech

    NaceAI Drex v1.5 caching benchmarks

    A 51-page technical report just landed from NaceAI, and the headline number in it is genuinely absurd: a fresh question asked against a cached context of 114k to 117k tokens came back in 21.6 milliseconds. The same request without caching? 1.78 seconds. That's not a small win, that's a different sport. The trick is pointer models. Instead of re-reading the whole document every time you ask something, the system encodes the context one time, holds onto it, and then pulls from that stored version for every follow-up question. Each question still gets probabilities over whatever options you feed it, so you're not losing the actual decision-making part. And the caching advantage compounds. Run 16 questions over 32k tokens and shared prefill gives you an 11, 12x speedup compared to encoding everything over and over. Measurements here were taken model-side on a single NVIDIA B200 GPU, with API gateway latency excluded, so this is raw model throughput we're talking about. The honest tradeoff is accuracy versus efficiency. Max v1 brings higher accuracy to the table but re-encodes context for every single query. Pointer models go the other direction, reusing what's already been processed to keep costs down on long documents. Which one you pick depends on what your workload actually looks like. Full report is 51 pages if you want the deep dive, and free access to Drex is being offered too.

    1w

    14 赞0 踩2 转发0 评论
    ?

    评论

    还没有评论。来抢沙发吧!