15.27 min read

Circuit Model — Nodes, Wires, Functions

To formalize backpropagation, we model computation as a circuit. Data enters on the left as input wires $t_1, \ldots, t_n$ (collected as vector $\mathbf{t}$ ). Values flow left to right. The circuit has a single output wire, and the whole thing computes $f(\mathbf{t})$ .

A function node takes one or more input wires, applies some scalar function, and outputs a single wire. For example, $x = s_1 \cdot s_2$ , or $x = \sin(s_1)$ , or $x = s_1 + s_2 + s_3$ . Each node is named after its output wire (so the node and its output wire share the same name).

A splitter node copies one input wire into two identical output wires. This is needed whenever a value is used in multiple downstream computations. The splitter for input $s$ with outputs $x_1, x_2$ behaves as $\mathbf{x}(s) = [s,\, s]^t$ .

Notice that every function node outputs exactly one scalar. The circuit as a whole also outputs a single scalar $f(\mathbf{t})$ .

Formal View

Definition 15.2 — Function Node

A function node named

x

with input wires

s_1, \ldots, s_k

(vector

\mathbf{s}

) computes a scalar output

x = x(\mathbf{s})

. Both the node and the value on its output wire are called

x

Definition 15.3 — Splitter Node

A splitter node with input

s

produces two identical outputs

x_1 = x_2 = s

. In vector form:

\mathbf{x}(s) = [s,\, s]^t

. Splitters let one wire feed multiple downstream nodes.

Remark 15.2 — Notation Overloading

We use the symbol

f

for both the full circuit and for "cut circuit" variants.

f(\mathbf{t})

is the original output;

f(x, \mathbf{t})

is the circuit with the

x

-wire replaced by a free input. The argument list disambiguates.

Why This Matters

Any differentiable computation — loss functions, solvers, renderers — can be expressed as such a circuit, making backprop universally applicable.

Neural network layers (linear transform + activation function) as function nodes
Differentiable physics simulators expressed as large circuits
PyTorch and JAX internally represent computations as circuit DAGs
Compiler optimizations that exploit DAG structure

Learning Resources

What is backpropagation really doing?

3Blue1Brown

Builds up the circuit view of a neural network computation graph.

14 min

The spelled-out intro to neural networks and backpropagation: building micrograd

Andrej Karpathy

Implements function nodes as Python objects with stored values and backward methods.

144 min

Quiz

Question 1

What does a splitter node do?

Question 2

A function node can produce multiple output values.

Question 3

If $t_1$ feeds into two different downstream nodes, the circuit must contain:

Question 4

Node $x$ computes $x = s_1^2 + 3s_2$ . What type of node is this?

Common Mistakes

Forgetting splitter nodes when a value feeds multiple places — every fan-out requires one.
Thinking function nodes can return vectors — each outputs exactly one scalar in this model.
Confusing the circuit diagram (computation structure) with the graph of $f$ (input-output shape).