Short Convolution (ShortConv)
ShortConv is a small causal 1D convolution that mixes each channel across a few adjacent token positions. Kimi Linear and Inkling both use a kernel size of 4, so each output can combine the current token with the three preceding positions.
The convolution is depthwise. Each hidden channel learns its own temporal filter, while the surrounding linear projections, attention modules, and feed-forward layers handle cross-channel mixing.
Local receptive field
The current token plus the three preceding positions when the kernel size is 4
Decode state
A fixed-size rolling state whose size does not grow with the context length
Example architectures
How A Causal Depthwise ShortConv Works
For one channel c and a kernel size k, the convolution at position t is
y[t, c] = Σᵢ w[c, i] · x[t - i, c] for i = 0, ..., k - 1
The causal constraint only allows x[t] and earlier positions. At the start of a sequence, the missing earlier values are treated as zeros. With k = 4, the output uses x[t], x[t-1], x[t-2], and x[t-3].
Depthwise means that channel c has its own four weights and does not read another channel. This keeps the operation small. A standard dense projection can mix all channels before or after it.
A Minimal PyTorch Version
The following module implements the core operation. The optional activation and residual switches make it easy to reproduce the two model-specific variants discussed below.
import torch
import torch.nn as nn
import torch.nn.functional as F
class ShortConv(nn.Module):
def __init__(self, d_model, kernel_size=4, activation=None, residual=False):
super().__init__()
self.kernel_size = kernel_size
self.activation = activation
self.residual = residual
self.conv = nn.Conv1d(
in_channels=d_model,
out_channels=d_model,
kernel_size=kernel_size,
groups=d_model,
bias=False,
padding=kernel_size - 1,
)
def forward(self, x):
# x has shape (batch, tokens, channels)
y = self.conv(x.transpose(1, 2))
y = y[:, :, : x.shape[1]].transpose(1, 2)
if self.activation == "silu":
y = F.silu(y)
if self.residual:
y = y + x
return y
The truncation after Conv1d removes outputs that would depend on right-side padding. The remaining outputs are causal.
Why The Operation Is Cheap
For a sequence with T tokens, hidden width d, and kernel size k, a depthwise ShortConv takes roughly O(Tdk) work. Here k is only 4. The cost grows linearly with the sequence length.
Incremental decoding only needs a small rolling state for the recent activations. Its size is O(dk) and stays constant as the context becomes longer. Attention still handles broader retrieval in these models. ShortConv supplies a dedicated local mixing path.
Kimi Linear Filters Q, K, And V In KDA Layers
Kimi Linear has 27 decoder layers. Twenty use Kimi Delta Attention (KDA), and seven use MLA. Each KDA layer applies three depthwise ShortConv modules after the Q, K, and V projections.
All three use a kernel size of 4 followed by a SiLU activation. Their outputs become the Q, K, and V inputs to the recurrent KDA update. The seven MLA layers do not use these Q/K/V short convolutions.
This placement gives KDA an explicit four-token local mixing step before it updates its recurrent state. The recurrent state can then focus on carrying information across longer ranges.
Inkling Uses Four ShortConv Modules In Every Layer
Inkling applies ShortConv throughout all 66 decoder layers. Each layer contains four modules:
-
k_sconvafter the key projection -
v_sconvafter the value projection -
attn_sconvafter the attention output and before the main residual addition -
mlp_sconvafter the MLP or MoE output and before the main residual addition
Inkling also uses a kernel size of 4. Its implementation adds the module input back to each convolution output. This gives every ShortConv its own small residual path. The Hugging Face implementation keeps these four convolution modules in FP32.
The two projection convolutions provide local mixing inside attention. The two branch-output convolutions add local mixing after attention and after the feed-forward computation.
Same Kernel, Different Placement
Kimi Linear uses ShortConv as part of its linear-attention mechanism. Its three convolutions appear only in KDA layers, use SiLU, and prepare Q, K, and V for the recurrent update.
Inkling treats ShortConv as a more general local-processing component. Two modules modify K and V, while two more modify the attention and feed-forward branch outputs. The same four-token receptive field therefore influences both attention inputs and residual-stream updates.
Sources