Building frontier AI models is no longer just a research endeavor. It now requires a complex, industrial-scale R&D system. At leading AI companies, these systems rely on large teams of experts. Architecture, infrastructure, training, inference, deployment, and evaluation are only part of it. Behind every frontier AI model is a continuous cycle of experimentation, iteration, and refinement.

At NaiveAI, we broke with human-centered R&D from day one. We put AI models to work on AI R&D itself. They write code, run experiments, monitor progress, analyze results, and iterate. Human researchers set direction, define constraints and criteria, and make critical decisions. The human edge lies in experience, insight, and judgment.

NaiveAI built production-scale infrastructure for AI-centered R&D, giving AI models secure work environments and access to GPU compute. The system serves close to ten million sandboxes each week, with 100,000 active concurrently at peak. A unified control plane manages compute, environments, tools, permissions, and security at scale.

Naive-N0.5-Flash was built this way. AI explored and designed its hybrid attention architecture while optimizing its training, inference, and deployment systems. Human researchers provided guidance and made key decisions. Naive-N0.5-Flash is also trained for AI R&D, allowing it to participate directly in the R&D process and opening a path toward recursive self-improvement (RSI).

Model Overview

Naive-N0.5-Flash is a 309B MoE model with 15.5B active parameters, built for coding and AI R&D.

  • Native 1M context, without full attention. Naive-N0.5-Flash combines Sliding-Window Attention (SWA) and lightweight DeepSeek Sparse Attention (DSA) with GQA4 at a predominantly 5:1 SWA–DSA layout. The entire network remains local or sparse, with no full-attention layers.
  • AI-optimized inference up to 2,000 tokens/s. NaiveRT, our inference system for Naive-N0.5-Flash, was built and optimized through AI-centered R&D. It combines mega-kernel fusion, Programmatic Dependent Launch (PDL), and speculative decoding, delivering 50 tokens/s per user in Standard mode and up to 2,000 tokens/s in Ultrafast mode.
  • Open weights and API. Model weights and inference code are released under the MIT license. API access will also be provided, with pricing set at $0.10 / $0.40 / $0.01 per million tokens for input, output, and cache reads, respectively.

Evaluation Results

Evaluation setup. Unless otherwise noted, our evaluations of Naive-N0.5-Flash use Claude Code 2.1.207 with a 1M-token context window, temperature 1.0, and top-p 0.95. The harness exposes only basic file I/O and Bash tools.

Sources for reported benchmark scores

AI-Centered R&D in Practice

Naive-N0.5-Flash helps researchers with open-ended work across AI research and systems engineering.

Case 1

NaiveRT — AI-Optimized 2,122 Tokens/s Inference Runtime for RL

NaiveRT: AI-Optimized 2,122 Tokens/s Inference Runtime for RL

To fully exploit the lightweight architecture of Naive-N0.5-Flash in large-scale RL rollouts, we built NaiveRT, an inference runtime optimized for single-stream decode speed, with bitwise-deterministic sampling. With a fused DFlash draft model, NaiveRT reaches a peak single-stream decoding rate of 2,122 tokens/s on 8 GPUs. On the same system, it reduces the latency of a full speculative decoding round (draft, verify, sampling, and commit) from 12.3 ms under SGLang to 3.4 ms, a 72.4% reduction.

NaiveRT was built in six days by human researchers working with AI models, across 151 documented optimization trials.

Why single-stream speed matters for RL

Fast single-stream decoding is particularly valuable for RL workloads with contexts reaching 1M tokens. A rollout's context grows two ways:

  • Ingested tokens (observations, tool outputs, prior turns) arrive at 5,000–10,000 tokens/s.
  • Generated tokens arrive at 50–100 tokens/s with conventional decoding.

Generated tokens are the minority, roughly 20–40% at today's ratios, but they account for essentially all of the model-side wall-clock. A rollout nearing 1M tokens spends minutes ingesting and hours decoding.

These long rollouts become stragglers. In synchronous pipelines, a handful of slow rollouts leaves the entire batch waiting. In async pipelines with partial rollouts, a trajectory can span several policy versions, but only up to a maximum staleness of K steps. A trajectory that can't finish in time is discarded whole, along with every segment already generated.

Either way, the cost lands on the longest samples, which are often the hardest and most informative. Batching can't fix this: it amortizes weights across many requests, but it can't make any individual stream finish sooner. Per-stream decode speed is the only lever that shortens the rollout itself.

Determinism is the second requirement. Rollouts feed directly into gradients, and if the logprobs the sampler used differ from those the trainer computes, the policy is silently trained off-policy. NaiveRT is deterministic up to the random seed: the same sequence decoded twice with the same seed yields bitwise-identical tokens, logits, and KV cache. We treat this as a constraint on every optimization, not a debug mode.

The execution model

At the core of NaiveRT is aggressive mega-kernel fusion. In speculative decoding, draft, verify, sampling, and commit form a tightly connected execution loop. NaiveRT fuses and reorganizes these stages into larger GPU kernels. This reduces kernel-launch overhead, synchronization, and intermediate memory traffic, and improves GPU utilization even at small batch sizes.

Programmatic Dependent Launch (PDL) connects the remaining kernel boundaries, allowing downstream kernels to launch while upstream work is still finishing. The speculative decoding loop advances on-device: the tail kernel commits the token and directly launches the next round. Multiple rounds proceed without CPU intervention, leaving the host primarily responsible for streaming results.

The fusion is selective. Attention layers become single mega-kernels, as the next section shows. The MoE router, up/gate, and down projections remain separate kernels chained by PDL, for reasons we explain under A faster microbenchmark is not a faster model.

A DSA layer: from 29 kernels to one

In the SGLang configuration we profiled, a single DSA layer required 29 kernel executions per GPU per decoding step. They span Q/K/V projections, RoPE, KV-cache writes, the indexer GEMM, top-k selection, sparse gathering, attention, and output projection. That count doesn't include the input RMSNorm or the layer-boundary communication.

NaiveRT reorganizes this entire workload into a single cooperative mega-kernel running across 148 CTAs in one grid. The mega-kernel also takes in the preceding RMSNorm and the trailing TP8 communication. The four context-parallel kernels aren't fused one-for-one; NaiveRT's execution topology replaces them. Data that previously moved across kernel boundaries now stays within the fused execution path. The AI models worked through the data-visibility and synchronization dependencies required to keep each stage correct.

Optimizing inside the kernel

Fusion alone was not enough. The AI models continued optimizing the execution pipeline inside the kernels themselves.

One example came from QKV weight movement. The weights don't depend on the normalization result, but in the unfused implementation their transfer couldn't begin until the QKV projection kernel launched. After fusion, each CTA computes RMSNorm locally, eliminating a global synchronization. Meanwhile, the 124 QKV-projection CTAs each start a 128 KiB TMA transfer before computing RMSNorm, and wait at the barrier only when the weights are actually needed for projection.

Weight movement now overlaps normalization, and the same shared-memory region is reused by the index and attention stages. This single change reduced full-model latency by 21–23 μs per step.

Similar optimizations changed GPU execution without changing the model itself:

  • K/V staging. Selected K/V rows are staged in shared memory by split before computation, reducing the layer latency from 57.0 μs to 49.5 μs.
  • Tensor Core remapping. With eight query heads per GPU, a direct m16n8k16 mapping would pad the heads to sixteen rows. NaiveRT remaps the computation so the eight heads occupy the MMA's n8 dimension, eliminating the padded-head compute.

How it was built

NaiveRT was built through the same AI-centered R&D process used to develop Naive-N0.5-Flash. Human researchers set the technical direction, quality constraints, and acceptance criteria. AI models handled much of the implementation and optimization work:

  • profiling full-model performance
  • implementing candidate approaches
  • running numerical validation and multi-GPU tests
  • analyzing the results
  • determining which changes to keep, revise, or roll back

This human-guided, AI-driven process compressed the core optimization effort into six days, in three stages:

  • Whole-network engineering (43 trials, 28 adopted): building the fused full-model execution path and restructuring routing and data movement, on random weights.
  • Real-checkpoint execution (45 trials, 15 adopted): moving to the real checkpoint to tune Tensor Core and TP execution, remove redundant work, and overlap cross-GPU execution.
  • W8A8 kernel refinement (63 trials, 20 adopted): going deep inside critical kernels to refine load ordering, compute layouts, and kernel tails.
Latency of the target model's forward pass verifying an 8-token draft block, in milliseconds. The panels end near 3.1 ms rather than at the 3.4 ms full round, which also includes drafting, sampling, and commit. Each panel uses its own test conditions, so step heights aren't additive.

In total, the process covered 151 documented optimization trials:

  • 63 produced changes that were adopted.
  • 71 failed validation or were rolled back.
  • 17 explored alternative paths or prototypes.

Individual trials often included multiple implementation variants and repeated measurements. Altogether, that meant thousands of end-to-end runs covering compilation, performance profiling, numerical alignment, and correctness checks.

A faster microbenchmark is not a faster model

Throughout the process, the AI models judged candidate optimizations by their end-to-end impact on the real model path, not by isolated kernel benchmarks. The two often disagreed:

  • Short-context bypass. An optimization that skipped indexing and read K/V directly by position saved roughly 3 μs at 200 tokens. At 2,200 tokens it became roughly 3 μs slower, so it was rejected.
  • TMA prefetch. The prefetch described above showed the opposite behavior: slightly slower in an isolated warm-cache microbenchmark, yet 21–23 μs faster per step in the full model.
  • MoE fusion. A proposed fusion path went through seven rounds of implementation. Every version was numerically correct, and every version regressed end-to-end performance. PDL was already overlapping much of the original kernel-boundary cost, while fusion introduced additional synchronization. Human researchers ultimately stopped that direction, which is why the MoE kernels remain PDL-chained.

Correctness gated everything. Before any latency optimization could be merged, full-model logits and KV caches had to pass strict bitwise comparison, alongside end-to-end performance measurements.

Results

NaiveRT reaches a peak single-stream decoding rate of 2,122 tokens/s on 8 GPUs. This is measured over the best one-second window with thinking off, across 41 HTML/SVG generation requests at temperature 0.4 and top-p 0.95, excluding prefill. A full speculative round takes 3.4 ms, versus 12.3 ms for SGLang on the same system.

The result comes from the interaction of several pieces working across the full inference path:

  • light-weight sparse attention
  • fused kernels and fused DFlash drafting
  • full GPU-driven scheduling
  • communication overlap with lightweight primitive

Get NaiveRT

NaiveRT is open source, and everything needed to run it and reproduce the results in this post is available:

  • Source code. The NaiveRT GitHub repository contains the runtime, the fused kernels, and the benchmark scripts behind the figures in this post. The above content will be available by Oct, 12th.
  • Docker Compose. A reproducible build environment with the toolchain and dependencies NaiveRT is tested against.
  • Model weights. The NaiveRT Hugging Face repository hosts the Naive-N0.5-Flash W8A8 checkpoint and the matching DFlash draft model.
  • Updates. Follow @naiveailab on X for release notes, benchmarks, and what we're building next.
Case 2

AutoWM — AI-Built World Model with Top-Tier WorldArena Performance

AutoWM: AI-Built World Model with Top-Tier WorldArena Performance

What happens when Naive-N0.5-Flash is given an end-to-end research task outside NaiveAI’s existing areas of expertise?

A NaiveAI researcher tasked Naive-N0.5-Flash with building a world model. The researcher defined the objective, compute budget, and evaluation protocol. From there, Naive-N0.5-Flash worked through the research loop: designing approaches, training models, running evaluations, analyzing results, and deciding what to try next.

After 400 hours of cumulative research and 15 major experimental rounds, the resulting world model, AutoWM, reached a score of 77.43 under the public WorldArena-1 Track 1 evaluation protocol, above the highest published leaderboard score at the time, 73.64.

Researching Beyond a Fixed Recipe

The process began by reproducing the official FlowWAM recipe. Early experiments explored conventional directions such as classifier-free guidance and timestep sweeps, but produced little improvement.

Naive-N0.5-Flash then began changing the research setup itself. It rewrote captions, expanded the training set from 2.5K to 22.5K clips, filtered low-VLM-score samples, and explored different training data mixtures and frame-sampling strategies.

The results were not monotonic. Sixteen evenly spaced frames outperformed eight, while increasing to thirty-two yielded a smaller additional gain. Naive-N0.5-Flash also changed the rollout structure, dropping the second rollout and aligning conditioning within a single segment. Moving to a stronger base model with higher-quality training data improved performance further.

These were not steps in a predetermined recipe. Experimental results determined which directions continued and which became side branches.

Expanding the Search Space

Training and post-processing experiments were interleaved as the research progressed.

A key insight was that some WorldArena metrics can be computed without ground-truth reference videos. Naive-N0.5-Flash used these signals not only to evaluate generated videos, but also to guide output selection and post-processing.

It explored multi-timestep selection, choosing the best timestep for each video. This pushed AutoWM above the previous highest leaderboard score. It then expanded the search space to include individual frames, formulating frame selection as a knapsack problem solved with dynamic programming.

Further experiments explored Best-of-N video selection and choosing the best post-processing strategy for each video. An alternative DP configuration with N=32 underperformed the N=16 setting.

Later experiments explored flicker and jitter as part of post-processing aimed at increasing video dynamics. Together, these inference-time and post-processing optimizations substantially improved performance on WorldArena-1 Track 1.

From a New Domain to 77.43

By the end of the 400-hour research run, AutoWM reached 77.43 on WorldArena-1 Track 1, compared with the previous highest published result of 73.64.

World models were not an established research direction at NaiveAI. AutoWM shows that the same AI-centered R&D process can extend to a new domain when the objective, compute budget, and evaluation protocol are clearly defined.

In this case, Naive-N0.5-Flash participated in the research itself: proposing approaches, running experiments, abandoning directions that did not work, and deciding what to try next. This is the path we are pursuing toward recursive self-improvement (RSI): AI models taking on the research that produces the next generation of models, with each generation able to take on more of that research.

Technical Details

Model Architecture

Naive-N0.5-Flash builds on the open-weight MiMo-V2.5 base model, which has a simple architecture with strong capabilities in world knowledge and deep research. Most layers use Sliding-Window Attention (SWA), whose per-token decoding cost does not grow with context length, while a small number of global-attention layers preserve long-range information. At million-token context lengths, however, these global-attention layers account for much of the decoding overhead.

Naive-N0.5-Flash replaces the global-attention layers with DeepSeek Sparse Attention (DSA). A lightweight indexer scores the full history, while the backbone computes attention only over a selected subset of tokens. Although the indexer still scans the full history and the full KV cache is retained, sparse attention substantially reduces attention computation and memory access. Adapting the model to this new attention structure was one objective of continued pretraining.

NaiveAI researchers and AI models jointly explored the architecture. Researchers defined the objective and evaluation protocol: improve decoding efficiency at million-token context lengths while preserving model quality. AI models implemented candidate architectures and training approaches, ran ablations, and summarized the results. Among the candidates that met these objectives, researchers selected one of the simplest to implement.

Hybrid SWA–DSA Architecture

The network consists of eight six-layer modules. A standard module contains five SWA layers followed by one DSA layer, with the first layer of the first module also replaced by DSA. SWA uses a 128-token window, while DSA selects the top 2,048 tokens for backbone attention. Both attention types incorporate sink bias.

Unlike the original MLA-based DSA implementation, Naive-N0.5-Flash replaces MLA with grouped-query attention (GQA) using four KV groups. Through infrastructure–algorithm co-design, we developed a lightweight indexer with 16 query heads, reducing index-selection wall time by 44% relative to the original DSA implementation while preserving performance on agent tasks at 1M context.

Training Procedure

Following the architectural changes, Naive-N0.5-Flash completed 3.25T tokens of multi-stage training with a native 1M-token context window: 50B tokens of Indexer Warmup, 3T tokens of Sparse Attention Training, and 200B tokens of Learning Rate Decay. This process adapted the model to its new sparse attention architecture while substantially improving its AI R&D and coding capabilities.

  • Indexer Warmup — 50B tokens. Only the newly introduced DSA indexer is trained, while all other model parameters remain frozen. The layers being converted to DSA retain full attention for forward computation at 1M context, while the SWA layers remain unchanged. The full-attention distributions in those layers provide the supervision signal, with a KL-divergence loss aligning the indexer and backbone attention distributions and establishing the indexing behavior required for the subsequent transition to sparse attention.
  • Sparse Attention Training — 3T tokens. After warmup, the model switches to sparse attention and enters continued pretraining (CPT) with a language modeling (LM) loss at a fixed learning rate, focusing primarily on AI R&D and coding. The model remains at 1M context throughout. Continued pretraining adapts the model to the new information-selection and aggregation mechanisms while strengthening long-context modeling for tasks that require retaining task history, connecting dispersed information, and sustaining extended interactions.
  • Decay Stage — 200B tokens. The model then enters supervised fine-tuning (SFT), remaining at 1M context with the sparse-attention execution path and LM objective, while the learning rate is gradually reduced over the final 200B tokens.

Training System

Full-to-Sparse Transition

During indexer warmup, the layers being converted to DSA retained full attention, while the SWA layers remained unchanged. The indexer was aligned to the full-attention reference through a KL-divergence objective before the transition to sparse attention.

To validate the transition to sparse attention, AI models automatically analyzed top-k recall for the indexer’s top-2,048 selections against the full-attention reference. They identified and fixed numerical-stability issues in indexer top-k selection and independently verified the fixes. Based on these results, researchers determined the transition point and accuracy thresholds, and aligned validation standards across training and deployment.

Hybrid Sequence Parallelism

While analyzing attention computation at 1M context, AI models identified a key parallelism property of the hybrid architecture: DSA requires access to the full history, whereas SWA attends only to a 128-token window to the left. They therefore proposed and validated a hybrid sequence-parallel scheme using different execution paths for the two attention types.

DSA layers use Ulysses Sequence Parallelism, with all-to-all redistribution providing access to the full sequence. SWA layers use SWA Halo Sequence Parallel, where overlapping shards exchange only the required ghost region with their left-hand neighbor, with the overlap determined by sample boundaries. This preserves DSA’s global modeling capability while reducing SWA communication complexity from O(L) to O(w).

The same principle extends to inference: long-context prefill, row-sharded computation, and index retrieval use their respective context-parallel schemes rather than sharing a single cross-GPU communication pattern.

GPU Memory Optimization

AI models also analyzed and optimized GPU memory use. They extended the training framework’s activation-memory management to handle intermediate activations from sparse indexing and attention alignment. By analyzing recomputation paths, they refined offloading granularity down to individual operator outputs, with independently configurable offloading for each intermediate activation. Top-k indices are explicitly retained to prevent recomputation from changing the selected positions.

At 1M sequence lengths, correctness checks and performance profiling also uncovered precision issues in positional encoding and potential out-of-bounds sequence indexing.

Across numerical stability, hybrid sequence parallelism, tensor-layout transformations, long-sequence support, and memory optimization, AI models participated directly in the engineering loop of analysis, discovery, repair, and verification. Researchers defined the objectives, constraints, and decision boundaries, while AI models carried out automated analysis, problem discovery, and independent verification — a concrete example of using models to build models.

With a 1M-token context configuration, the resulting training system can process 1T training tokens in approximately four days on 512 GPUs.

License

Naive-N0.5-Flash model weights and inference code are released under the MIT License.

Citation

If you find Naive-N0.5-Flash useful in your research or work, please cite:

@misc{naiveai2026naiven05flash,
  title        = {Naive-N0.5-Flash: Building Frontier AI with AI},
  author   = {{NaiveAI Team}},
  year       = {2026},
}

Acknowledgments

Naive-N0.5-Flash builds on the work of the open-source community and gives back to it. We thank the Xiaomi MiMo team for making their MiMo-V2.5 base model publicly available, the DeepSeek team for their work on DeepSeek Sparse Attention (DSA), and the SGLang team and community for their open-source inference infrastructure.

Contact

For questions, feedback, or collaboration, please contact us at contact@naive.ai or follow us on X at @naiveailab. You can also find our open-source projects and model releases on GitHub and Hugging Face.