Deriva4ves of Pooling Pooling layer subsamples sta.s.cs to obtain summary sta.s.cs with any aggregate
func.on (or ﬁlter) g whose input is vector, and output is scalar. Subsampling is an
opera.on like convolu.on, however g is applied to disjoint (non- ‐overlapping) regions.
n Deﬁni.on: subsample (or downsample)
Let m be the size of pooling region, x be the input, and y be the output of the pooling layer.
subsample(f, g)[n] denotes the n- ‐th element of subsample(f, g).
yn = subsample x,g( ) n[ ]= g x n−1( )m+1:nm( ) y = subsample x,g( ) = yn[ ] g x( ) = xk k=1 m ∑ m , ∂g ∂x = 1 m mean pooling max x( ), ∂g ∂xi = 1 if xi = max x( ) 0 otherwise ⎧ ⎨ ⎩ max pooling x p = xk p k=1 m ∑ ⎛ ⎝⎜ ⎞ ⎠⎟ 1/p , ∂g ∂xi = xk p k=1 m ∑ ⎛ ⎝⎜ ⎞ ⎠⎟ 1/p−1 xi p−1 Lp pooling or any other differentiable Rm → R functions ⎧ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎪ ⎪ ⎪ ⎪ ⎪ x Pooling yn g m x(n- ‐1)m+1 …
Backpropaga4on in Pooling Layer Error signals for each example are computed by upsampling. Upsampling is an opera.on
which backpropagates (distributes) the error signals over the aggregate func.on g using
its deriva.ves g’n = ∂g/∂x(n- ‐1)m+1:nm. g’n can change depending on pooling region n.
p In max pooling, the unit which was the max at forward propaga.on receives all the error at backward
propaga.on and the unit is diﬀerent depending on the region n.
n Deﬁni.on: upsample
upsample(f, g)[n] denotes the n- ‐th element of upsample(f, g).