Linear Algebra
14.28 min read

Setup: Composition of Functions

To state the chain rule precisely, we need to set up the composition carefully. Suppose:

  • f:D⊆Rn→Rk\mathbf{f}: D \subseteq \mathbb{R}^n \to \mathbb{R}^k (an inner function from nn variables to kk variables)
  • g:E⊆Rk→Rm\mathbf{g}: E \subseteq \mathbb{R}^k \to \mathbb{R}^m (an outer function from kk variables to mm variables)
  • f(D)⊆E\mathbf{f}(D) \subseteq E (so the composition is defined)

The composition h=g∘f:D→Rm\mathbf{h} = \mathbf{g} \circ \mathbf{f}: D \to \mathbb{R}^m is defined by h(x)=g(f(x))\mathbf{h}(\mathbf{x}) = \mathbf{g}(\mathbf{f}(\mathbf{x})). The "intermediate variables" are the outputs of f\mathbf{f}, which serve as inputs to g\mathbf{g}.

We call the inputs x=(x1,…,xn)\mathbf{x} = (x_1, \ldots, x_n) the underlying variables and the intermediate outputs u=f(x)=(u1,…,uk)\mathbf{u} = \mathbf{f}(\mathbf{x}) = (u_1, \ldots, u_k) the intermediate variables. The chain rule expresses the Jacobian of h\mathbf{h} in terms of the Jacobians of f\mathbf{f} and g\mathbf{g}.

Formal View

Definition 14.1 — Composition Setup
Let f:D⊆Rn→Rk\mathbf{f}: D \subseteq \mathbb{R}^n \to \mathbb{R}^k and g:E⊆Rk→Rm\mathbf{g}: E \subseteq \mathbb{R}^k \to \mathbb{R}^m with f(D)⊆E\mathbf{f}(D) \subseteq E. The composition is h=g∘f:D→Rm\mathbf{h} = \mathbf{g}\circ\mathbf{f}: D \to \mathbb{R}^m, h(x)=g(f(x))\mathbf{h}(\mathbf{x}) = \mathbf{g}(\mathbf{f}(\mathbf{x})). The intermediate variables are u=f(x)\mathbf{u} = \mathbf{f}(\mathbf{x}).

Jacobian dimensions: JfJ\mathbf{f} is k×nk\times n, JgJ\mathbf{g} is m×km\times k, and Jh=Jg⋅JfJ\mathbf{h} = J\mathbf{g}\cdot J\mathbf{f} is m×nm\times n. Matrix sizes are compatible.

Why This Matters

Identifying the decomposition of a function into inner and outer parts is the first step in applying the chain rule.

  • Signal processing pipelines: each stage is an "outer function" applied to the "inner function" output
  • Neural network layer decomposition: each layer is composed with the previous
  • Coordinate transformations composed with physical models

Quiz

Question 1

If f:R3→R2\mathbf{f}: \mathbb{R}^3 \to \mathbb{R}^2 and g:R2→R4\mathbf{g}: \mathbb{R}^2 \to \mathbb{R}^4, the composition h=g∘f\mathbf{h} = \mathbf{g}\circ\mathbf{f} maps:

Question 2

For f:R3→R2\mathbf{f}: \mathbb{R}^3 \to \mathbb{R}^2 and g:R2→R4\mathbf{g}: \mathbb{R}^2 \to \mathbb{R}^4, what is the size of Jg⋅JfJ\mathbf{g} \cdot J\mathbf{f}?

Common Mistakes

  • Not checking that f(D)⊆E\mathbf{f}(D) \subseteq E before composing — the composition may not be well-defined.
  • Confusing which is the "inner" and which is the "outer" function.
  • Getting the Jacobian multiplication order backwards.