Error Analysis

To make the LLA derivation of the chain rule rigorous, we must show the composite error is $o(\|\mathbf{h}\|)$. Write: $\mathbf{f}(\mathbf{a}+\mathbf{h}) = \mathbf{f}(\mathbf{a}) + J\mathbf{f}(\mathbf{a})\mathbf{h} + \mathbf{e}_1(\mathbf{h})$ where $\|\mathbf{e}_1(\mathbf{h})\|/\|\mathbf{h}\| \to 0$. Let $\boldsymbol{\delta} = J\mathbf{f}(\mathbf{a})\mathbf{h} + \mathbf{e}_1(\mathbf{h})$, so $\mathbf{f}(\mathbf{a}+\mathbf{h}) = \mathbf{f}(\mathbf{a}) + \boldsymbol{\delta}$.

Then $\mathbf{g}(\mathbf{f}(\mathbf{a}+\mathbf{h})) = \mathbf{g}(\mathbf{f}(\mathbf{a})+\boldsymbol{\delta}) = \mathbf{g}(\mathbf{f}(\mathbf{a})) + J\mathbf{g}(\mathbf{f}(\mathbf{a}))\boldsymbol{\delta} + \mathbf{e}_2(\boldsymbol{\delta})$ where $\|\mathbf{e}_2(\boldsymbol{\delta})\|/\|\boldsymbol{\delta}\| \to 0$.

Substituting and rearranging, the composite error is $J\mathbf{g}(\mathbf{f}(\mathbf{a}))\mathbf{e}_1(\mathbf{h}) + \mathbf{e}_2(\boldsymbol{\delta})$. Since $\boldsymbol{\delta} = O(\|\mathbf{h}\|)$, both terms are $o(\|\mathbf{h}\|)$, completing the proof.