What Is An RNN? | Sequence Memory In One Page

If you’ve worked with plain feed-forward networks, you know they treat each input as a standalone item. Sequence data doesn’t play by that rule. Words in a sentence depend on what came before. Sensor readings depend on prior readings. An RNN is built to keep a running state so the model can react to order, not just content.

What Is An RNN? And How It Differs From A Regular Network

An RNN is short for “recurrent neural network.” “Recurrent” means the network feeds part of its own output back into itself on the next step. That loop gives it a small, structured form of memory. Each new input updates that memory, then the updated memory helps produce the next output.

A dense network sees one vector and returns one vector. An RNN sees a stream of vectors and carries a hidden state from step to step. You can read the output at every step, or only at the end, depending on the task.

RNN Terms You’ll See And What They Mean

Term	Plain Meaning	What To Watch For
Sequence	An ordered list of inputs, like tokens or time ticks	Order changes meaning, so shuffling breaks the signal
Timestep	One position in the sequence	Long sequences raise compute cost and training strain
Input Vector	The numeric form of the item at a timestep	Scale and encoding choices can swing results
Hidden State	A compact summary carried to the next timestep	It can forget details on long gaps
Recurrent Weights	Weights that map the prior hidden state into the new one	They can cause vanishing or exploding gradients
Unrolling	Writing the loop as a long chain for training	More steps mean deeper graphs
Padding And Masking	Ways to batch variable-length sequences	Masking keeps padded tokens from skewing loss
Many-To-One	One output from a whole sequence	Use the last state or pooled states
Many-To-Many	One output per timestep	Alignment between inputs and labels must match

How An RNN Processes A Sequence Step By Step

An RNN runs the same small network cell again and again. At each timestep, the cell takes two inputs: the current input vector and the prior hidden state. It blends them with learned weights, passes the result through an activation, and emits a new hidden state. That new state becomes the “memory” for the next step.

Timestep Loop And Hidden State

It helps to treat the hidden state as a running note that keeps getting edited. It’s a compressed signal that the model learns to store because it helps the loss go down. If your data has long-range dependencies, the model must learn to keep the right bits alive over many steps.

Output Shapes You’ll Meet

RNNs come up in a few classic input-output patterns. In many-to-one, you feed the full sequence and read a single label at the end, like a sentiment tag. In one-to-many, you feed one item and generate a sequence, like a character string. In many-to-many, you output a label per timestep, like per-token tagging or per-tick prediction.

Why Order Changes Everything

Sequence models earn their keep when order carries signal. “dog bites man” and “man bites dog” share words, yet the order flips meaning. A bag-of-words view can’t catch that swap. RNNs can, since the hidden state evolves as tokens arrive.

Training An RNN Without Guesswork

Training looks like normal supervised learning on the surface: make a forward pass, compute loss, take gradients, update weights. The wrinkle is time. The RNN cell is reused across steps, so the same weights affect many timesteps. When you compute gradients, you’re pushing credit and blame backward through the unrolled chain.

Backpropagation Through Time In Plain Language

Backpropagation through time (BPTT) is just backprop on the unrolled graph. You run the sequence forward, store what you need, then run gradients backward from the loss to earlier steps. Many training loops use a truncated form, where you backprop through a fixed window, then detach the hidden state. That keeps memory use and gradient depth under control.

Vanishing And Exploding Gradients

RNNs can struggle on long sequences because gradients pass through the recurrent weights many times. If the repeated multiplications shrink values, gradients fade and early steps stop learning. If they grow, gradients can blow up and training can turn erratic. These issues are tied to the math of repeated matrix products.

Stabilizers That Work In Practice

• Gradient clipping: cap the gradient norm so one bad batch doesn’t wreck the update.
• Good initialization: start weights in a range that avoids wild activations.
• Shorter unroll windows: use truncated BPTT when full sequences are long.
• Batching with masks: keep padding from leaking into loss and gradients.
• Check shapes early: many bugs are silent shape swaps that train the wrong mapping.

If you want the exact parameter meanings for a vanilla RNN layer, the PyTorch nn.RNN docs spell out inputs, outputs, and shape rules.

LSTM And GRU: RNNs With Gates

Classic “simple” RNNs work, yet they can forget too fast when sequences get long. LSTM and GRU cells add gates that control what gets kept, what gets overwritten, and what gets read out. The goal is the same: keep gradients flowing so the model can learn longer dependencies without going unstable.

What Gates Actually Do

Think of a gate as a learned dial that scales a signal between 0 and 1. A forget gate can shrink part of the state. An input gate can decide how much new info gets written. Those dials are learned with the rest of the weights, so the cell can store patterns that help prediction across longer spans.

Picking Between Simple RNN, LSTM, And GRU

If your sequences are short and your task is light, a simple RNN can be fine and runs fast. If you see training stall on longer spans, try a GRU or LSTM. GRU cells are often a tidy middle ground: fewer parameters than an LSTM, yet better long-range handling than a simple RNN in many tasks.

Where RNNs Still Shine

Transformers get a lot of attention, yet RNNs still show up in real work. They can be a solid fit when sequences aren’t huge, latency matters, or you want a model that updates state as data arrives.

• Streaming sensor and IoT feeds where you want one prediction per tick
• Small to mid text tasks where you can’t afford heavy attention blocks
• Speech and audio pipelines where frames arrive in order
• Sequence classification when you value a compact model
• Auto-regressive generation at the character level for tiny vocabularies

When To Skip An RNN

If your sequences are long, attention models often train better and capture far-apart links with less strain. If you need parallel training on long inputs, RNNs can be slow because timesteps run in a loop. Also, if your data has weak order signal, a simpler model may win on speed and tuning time.

Model Choices For Common Sequence Tasks

Task Shape	Often A Good First Pick	Why It Fits
Short time series, many features	GRU	Handles order with a small state and modest cost
Long time series, long dependencies	Transformer encoder	Direct links across far steps via attention
Token tagging	Bi-directional LSTM	Uses left and right context for each token
Sequence to label	Simple RNN or GRU	Last state can be a clean summary when spans are short
Local pattern detection	1D CNN	Fast and strong on local motifs in signals
Real-time prediction	GRU with state carry-over	State can roll forward as new ticks arrive
Generation with small vocab	LSTM	Smoother long-span generation than a simple RNN

Getting Started In Code Without Getting Lost

You don’t need a big setup to try an RNN. Pick a library, build a small model, and verify shapes with a tiny batch. Most confusion comes from tensor layout: whether the time dimension comes first, and how hidden state is returned. Read the docs once, then print shapes in your first run.

A Minimal Shape Checklist

1. Decide your sequence length (T), batch size (B), and feature size (F).
2. Pick a layout: (T, B, F) or (B, T, F). Stick to it.
3. Confirm the hidden size (H). That’s the size of the running state.
4. Check the output: some layers return outputs for all timesteps plus the final state.
5. Mask padded steps so they don’t affect loss.

# Pseudocode shape flow
# x: (batch, time, features)
# rnn returns:
# y: (batch, time, hidden)
# h_last: (layers, batch, hidden)

If you’re using Keras, the Keras recurrent layer docs list SimpleRNN, LSTM, and GRU side by side, with the arguments that change behavior.

Practical Data Prep Notes For RNNs

RNNs are picky about how sequences are packaged. For time series, scale numeric features and keep the scaling consistent across train and test. For text, choose an encoding that matches your task, then watch sequence length. Long padded batches waste compute, so grouping sequences by length can help.

Also watch label alignment. If you predict the next step, shift labels by one. If you predict a whole-sequence label, be clear about which timestep’s state feeds the classifier head. These small choices decide whether training learns the mapping you meant.

Common Pitfalls And Quick Fixes

• Model learns nothing: start with a tiny dataset where you can overfit on purpose, then scale up.
• Loss drops, accuracy stalls: check class balance, label alignment, and masking.
• Training blows up: add gradient clipping and lower the learning rate.
• Runs slow: shorten sequence length, trim padding, or move to attention on long inputs.
• Hidden state misuse: reset state between unrelated sequences, carry state only when the stream is continuous.

Recap And Next Steps

If you came here asking what is an rnn?, the answer is a looped neural network cell that carries a hidden state across time. That state lets earlier inputs affect later outputs, which is what many sequence tasks need. Once you grasp the hidden state and unrolling, the rest is shape rules, loss setup, and a few training stabilizers.

When you test one, start small and log shapes. Then scale sequence length and model size in steps. If you hit long-span limits, switch to GRU or LSTM, or move to attention models for long inputs.

One last check: what is an rnn? It’s a tool for ordered data. Use it when order carries signal, skip it when order is weak, and you’ll save time.