Predictive Sparse Intelligence

The project evolved through four generations:

State → Time → Compute → Memory / Capacity / Precision / Hardware

The central idea has remained the same:

Intelligence is not the ability to process everything.
It is the ability to predict what does not need to be processed.

TPH (Top-down Predictive Hierarchy) began with predictive coding.

Instead of forwarding all information upward, higher layers predict lower-layer states and only unexplained information — prediction error — is propagated.

The project later found that filtering error alone was not enough. The more important problem was what the model stored in its internal state.

This led to State Sparsity:

• Predict first
• Update from error
• Store only important information
• Keep higher-level representations sparse
• Treat prediction as an information bottleneck

Predict first. Propagate only the error.

The core principle was:

If something is already predictable, do not process it again.

TPH reduced unnecessary state updates, but its recurrent structure remained difficult to parallelize efficiently on GPUs.

MR-TPH (Multi-Resolution TPH) extended sparsity from state to time.

Instead of processing an entire sequence sequentially at full resolution:

• Fine-grained information is processed in parallel chunks
• Each chunk produces a summary and boundary state
• Only the lower-resolution summary sequence is processed sequentially
• Prediction error still controls state updates

This created a new form of sparsity:

Not every moment needs the same temporal resolution.

An MR-TPH implementation, Raptor, was evaluated on WikiText-2 under a specific experimental setup.

These results belong to that specific comparison and should not be interpreted as a general claim of superiority over Transformers.

MR-TPH separated fine-grained and coarse-grained time processing.

Spectrum-State extended the same philosophy to functions.

Instead of asking one mechanism to handle global context, sequential state, exact recall, and reasoning equally well, the architecture separated those roles.

Early Spectrum-State

The architecture evolved around several components:

A key development was GCS Sleep / Wake:

This extended the original idea from:

Sparse State → Sparse Compute

Spectrum-State v9

Spectrum-State v9 accepted that SSMs can lose exact long-range details when compressing history into fixed-size state.

Instead of removing attention entirely, it used attention selectively.

The reported configuration included:

• 85M parameters
• 12 layers
• 9 Mamba layers
• 3 selective attention layers
• 32k evaluation context

9 Mamba layers carry sequential state; 3 selective attention layers handle exact recall.

In a reported 32k Needle-in-a-Haystack test:

The test used N=30 with the needle approximately 29k tokens away.

Spectrum-State v14.5–v15

Later versions expanded selective computation further:

• Adaptive perception
• Sparse experts
• Compressed memory
• Dynamic reasoning depth
• Critic / PRM-style verification
• Fast and slow reasoning paths

The question changed from:

“When should the model update its state?”

to:

“When should the model think harder?”

Imugi extends the same research lineage from architecture into system-level resource allocation.

The goal is no longer only to react when a difficult input appears.

The system attempts to predict what resources will be needed next.

This includes:

• Multi-timescale recurrent processing
• Compressed associative memory
• Persistent semantic memory
• Generalist / Specialist routing
• Critic-based foresight
• Hybrid precision
• Specialist prefetching
• Hardware-aware execution
• Long-context state carry
• Distributed training

The key transition is:

Reactive Compute → Predictive Compute

Instead of only allocating more computation after difficulty appears, the system aims to anticipate future difficulty and prepare:

The broader conceptual evolution is:

Prediction → Error → Sparse State → Multi-Resolution → Global Context → Exact Recall → Adaptive Compute → Sparse Capacity → Persistent Memory → Foresight → Hardware-Aware Intelligence

Parts of the research program have been implemented and evaluated.

Spectrum-State v7

Reported training experiment:

• TinyStories
• 50,000 steps
• Loss: 0.54
• VRAM usage: 1.8 GB

Raptor / MR-TPH

Reported WikiText-2 comparison:

• Raptor Test PPL: 355.90
• Transformer Test PPL: 612.45

Spectrum-State v9

Reported long-context recall experiment:

• 85M parameters
• 32k context
• N=30
• ~29k token retrieval distance
• Spectrum-State: 100%
• Mamba-2: 83.3%

These results are specific to their respective experimental setups.

The earlier generations include implemented experiments and quantitative evaluations.

Imugi currently represents a broader architectural and systems specification, including:

• memory policy,
• specialist routing,
• foresight control,
• hybrid precision,
• training stability,
• distributed execution,
• and hardware orchestration.

Several components still require dedicated benchmarking, including:

• long-context memory retention,
• specialist routing efficiency,
• foresight accuracy,
• hardware prefetch gains,
• low-precision stability,
• and end-to-end Imugi performance.

Predictive Sparse Intelligence is a research program about building systems that predict not only what comes next, but what information, computation, memory, precision, and hardware resources will actually be necessary to handle it.