NeonBench: LLM Architecture Research
NeonBench is an experimental setup I built to test architectural variants on small language models (~5.0M non-embedding parameters) under fixed compute and parameter budgets. Key concepts under investigation:
1. Quasi-Encoder & Lookahead Attention
Splits standard causal attention into a primary causal stream and a bounded lookahead stream that can access local future tokens up to a specified window depth. This provides bidirectional contextual representations during training while preserving autoregressive generation.
Experiments evaluating different lookahead ratios (25%, 50%, 100%) show clear trade-offs between training throughput and downstream perplexity.
2. Learned Intent Streams
Maintains a parallel stream of residual vectors ('intent stream') alongside token representations across layers. Instead of recomputing full sequence context from scratch at each block, the intent stream aggregates multi-layer representations and modulates query projections in deeper attention layers.
This helps retain global context in deeper networks and improves multi-token associative recall without increasing the input sequence length.
3. Controlled Ablations & Parameter Parity
All model variants are evaluated under strict parameter-parity bounds (~5.0M params) by adjusting the feed-forward dimension (d_ff) when adding lookahead, intent, or convolutional mixing modules.
- Phase 6: Lookahead mask depth and visibility windows (50% vs 100%).
- Phase 7: Combining lookahead attention with learned intent streams.
- Phase 8 & 9 (Current): Isolated ablations (intent-only vs conv-mixing) to compare validation loss and throughput.