このページは http://www.slideshare.net/FlorianWilhelm2/explaining-the-idea-behind-automatic-relevance-determination-and-bayesian-interpolation-59498957 の内容を掲載しています。

掲載を希望されないスライド著者の方は、こちらよりご連絡下さい。

8ヶ月前 (2016/03/13)にアップロードin学び

Even in the era of Big Data there are many real-world problems where the number of input features...

Even in the era of Big Data there are many real-world problems where the number of input features has about the some order of magnitude than the number of samples. Often many of those input features are irrelevant and thus inferring the relevant ones is an important problem in order to prevent over-fitting. Automatic Relevance Determination solves this problem by applying Bayesian techniques.

- Dr. Florian Wilhelm

March 13th 2016

PyData Amsterdam

P 1 - What‘s the best model to

describe our data?

And what does „best“ actually mean?

2 - 3
- 4
- Simple model

Complex model

„Generality“

„Best Fit“

Occam‘s Razor:

„It is vain to do with more

what can be done with fewer“

5 - Simple Model

Complex model

ℋ1

ℋ2

Space of all possible datasets 𝐷

6 - Simple Model

Complex model

ℋ1

ℋ2

ℋ1 fits only a small subset of 𝐷 well

7 - Simple Model

Complex model

ℋ1

ℋ2

ℋ2 can fit large parts of 𝐷 well

8 - Prefer the model with high evidence for a given dataset

Source: D. J. C. MacKay. Bayesian Interpolation. 1992

9 - 1. Model fitting: Assume ℋ𝑖 is the right model and fit its parameters 𝒘 with Bayes:

𝑃 𝐷 𝒘, ℋ

𝑃 𝒘 𝐷, ℋ

𝑖 𝑃(𝒘|ℋ𝑖)

𝑖

=

𝑃(𝐷|ℋ𝑖)

“Business as usual”

2. Model comparison: Compare different models with the help of their evidence

𝑃 𝐷 ℋ𝑖 and model prior 𝑃 ℋ𝑖 :

𝑃 ℋ𝑖 𝐷 ∝ 𝑃 𝐷 ℋ𝑖 𝑃 ℋ𝑖

“Occam‘s razor at work“

10 - Marginalize & approximate:

𝑃 𝐷 ℋ𝑖 = 𝑃 𝐷 𝒘, ℋ𝑖 𝑃 𝒘 ℋ𝑖 𝑑𝒘

𝑃 𝐷 ℋ𝑖 ≅

𝑃 𝐷 𝒘𝑀𝑃, ℋ𝑖

𝑃 𝒘𝑀𝑃 ℋ𝑖 ∆𝒘

𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 ≅ 𝑏𝑒𝑠𝑡 𝑓𝑖𝑡 𝑙𝑖𝑘𝑒𝑙𝑖ℎ𝑜𝑜𝑑 × 𝑂𝑐𝑐𝑎𝑚 𝑓𝑎𝑐𝑡𝑜𝑟

Occam factor:

∆𝒘

∆𝟎𝒘

Source: D. J. C. MacKay. Bayesian Interpolation. 1992

11 - 12
- Given:

Dataset 𝐷 = 𝑥𝑛, 𝑡𝑛 with 𝑛 = 1 … 𝑁

Set of (non-linear) functions Φ = {𝜙ℎ: 𝑥 ⟼ 𝜙(𝑥)} with ℎ = 1 … 𝑀

Assumption:

𝑀

𝑦 𝒙; 𝒘 = 𝑤ℎ𝜙ℎ(𝒙) ,

ℎ=1

𝑡𝑛 = 𝑦 𝒙; 𝒘 + 𝜐𝑛,

where 𝜐𝑛 is an additive noise with 𝒩 0, 𝛼−1

Task: Find min‖Φ𝒘 − 𝒕‖2 (Ordinary Least Squares)

𝒘

13 - Problem:

Having too many features leads to overfitting!

Regularization

Assumption: „Weights are small“

𝑝 𝒘; 𝜆 ~𝒩(0, 𝜆−1𝕀)

Task: Given 𝛼, 𝜆 find

min 𝛼 Φ𝒘 − 𝒕 2 + 𝜆 𝒘 2

𝒘

14 - Consider each 𝛼𝑖, 𝜆𝑖 defining a model ℋ𝑖 𝛼, 𝜆 .

Yes! That means we can use

our Bayesian Interpolation to

find 𝒘, 𝜶, 𝝀 with the highest

evidence!

This is the idea behind BayesianRidge as found in sklearn.linear_model

15 - Consider that each weight has an individual variance, so that

𝑝 𝒘 𝝀 ~𝒩 0, Λ−1 ,

where Λ = diag(𝜆1, … , 𝜆𝐻), 𝜆ℎ ∈ ℝ+.

Now, our minimization problem is:

min 𝛼 Φ𝒘 − 𝒕 2 + 𝒘𝑡Λ𝒘

𝒘

Pruning: If precision 𝜆ℎ of feature ℎ is high, its weight 𝑤ℎ is very likely to

be close to zero and is therefore pruned.

This is called Sparse Bayesian Learning or Automatic Relevance

Determination. Found as ARDRegression under sklearn.linear_model.

16 - Crossvalidation can be used for the estimation of hyperparmeters but suffers from

the curse of dimensionality (inappropriate for low-statistics).

Source: Peter Ellerton, http://pactiss.org/2011/11/02/bayesian-inference-homo-bayesianis/

17 - • Random 100 × 100 design matrix Φ with 100 samples and 100

features

• Weights 𝑤𝑖, 𝑖 ∈ 𝐼 = 1, … , 100 , random subset J ⊂ 𝐼 with 𝐽 = 10, and

0,

𝑖 ∈ 𝐼\J

𝑤𝑖 = 𝒩(𝑤𝑖;0,

1 4),

𝑖 ∈ 𝐽

• Target 𝒕 = Φ𝒘 + 𝝂 with random noise 𝜈𝑖 ∼ 𝒩(0,

1 50)

Task: Reconstruct the weights, especially the 10 non-zero weights!

Source: http://scikit-learn.org/stable/auto_examples/linear_model/plot_ard.html#example-linear-model-plot-ard-py

18 - 19
- 20
- 21
- 22
- We have to determine the parameters 𝑤, 𝜆, 𝛼 for

𝑃 𝒘, 𝝀, 𝛼 𝒕 = 𝑃 𝒘 𝒕, 𝝀, 𝛼 𝑃 𝝀, 𝛼 𝒕

1) Model fitting:

For the first factor, we have 𝑃 𝒘 𝒕, 𝝀, 𝛼 ~𝒩(𝝁, Σ) with

Σ = Λ + 𝛼Φ𝑇Φ −1,

𝝁 = 𝛼ΣΦT𝐭.

23 - 2) Model comparison:

For the second factor, we have

𝑃 𝝀, 𝛼 𝒕 ∝ 𝑃 𝒕 𝝀, 𝛼 𝑃 𝝀 𝑃 𝛼 ,

where 𝑃 𝝀 and 𝑃(𝛼) are hyperpriors which we assume uniform.

Using marginalization, we have

𝑃 𝒕 𝝀, 𝛼 = 𝑃 𝒕 𝒘, 𝛼 𝑃 𝒘 𝝀 𝑑𝒘,

i.e. marginal likelihood or the “evidence for the hyperparameter“.

24 - Differentiation of the log marginal likelihood with respect to 𝜆𝑖 and 𝛼 as

well as setting these to zero, we get

𝛾

𝜆

𝑖

𝑖 =

,

𝜇2𝑖

𝑁 −

𝛼 =

𝑖 𝛾𝑖 ,

𝒕 − Φ𝝁 2

with 𝛾𝑖 = 1 − 𝜆𝑖Σ𝑖𝑖.

These formulae are used to find the maximum points 𝝀𝑀𝑃 and 𝛼𝑀𝑃.

25 - 1. Starting values 𝛼 = 𝜎−2(𝒕), 𝝀 = 𝟏

2. Calculate Σ = Λ + 𝛼Φ𝑇Φ −1 and 𝒘 = 𝝁 = 𝛼ΣΦT𝐭

𝑁−

3. Update

𝛾

𝜆

𝑖

𝑖 𝛾𝑖

𝑖 =

and 𝛼 =

where 𝛾

𝜇2

𝑖 = 1 − 𝜆𝑖Σ𝑖𝑖

𝑖

𝒕−Φ𝝁 2

4. Prune 𝜆𝑖 and 𝜙𝑖 if 𝜆𝑖 > 𝜆𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑

5. If not converged go to 2.

Sklearn implementation:

The parameters 𝛼1, 𝛼2 as well as 𝜆1, 𝜆2 are the hyperprior parameters

for 𝛼 and 𝝀 with

𝑃 𝛼 ∼ Γ 𝛼

−1

−1

1, 𝛼2

,

𝑃 𝜆𝑖 ∼ Γ 𝜆1, 𝜆2 .

𝛼

𝛼

𝐸 Γ 𝛼, 𝛽

=

and 𝑉 Γ 𝛼, 𝛽 =

.

𝛽

𝛽2

26 - Given a some new data 𝑥∗, a prediction for 𝑡∗ is made by

𝑃 𝑡∗ 𝒕, 𝝀𝑀𝑃, 𝛼𝑀𝑃

=

𝑃 𝑡∗ 𝒘, 𝛼𝑀𝑃 𝑃 𝒘 𝒕, 𝝀𝑀𝑃, 𝛼𝑀𝑃 𝑑𝒘

= 𝒩 𝝁𝑇𝜙 𝑥

−1

𝑡

∗ , 𝛼𝑀𝑃 + 𝜙 𝑥∗

Σ𝜙 𝑥∗ .

This is a good approximation of the predictive distribution

𝑃 𝑡∗ 𝒕 = 𝑃 𝑡∗ 𝒘, 𝝀, 𝛼 𝑃 𝒘, 𝝀, 𝛼 𝒕 𝑑𝒘 𝑑𝝀 𝑑α .

27 - 1. D. J. C. MacKay. Bayesian Interpolation. 1992

(… to understand the overall idea)

2. M. E. Tipping. Sparse Bayesian learning and the Relevance Vector

Machine. June, 2001

(… to understand the ARD algorithm)

3. T. Fletcher. Relevance Vector Machines Explained. October, 2010

(… to understand the ARD algorithm in detail)

4. D. Wipf. A New View of Automatic Relevance Determination. 2008

(… not as good as the ones above)

Graphs from slides 7 and 9 were taken from [1] and the awesome

tutorials of Scikit-Learn were consulted many times.

28 - 29