このページは http://www.slideshare.net/gpeyre/optimal-transport-in-imaging-sciences の内容を掲載しています。

掲載を希望されないスライド著者の方は、こちらよりご連絡下さい。

4年弱前 (2012/12/13)にアップロードin学び

Slides of the presentation at the conference "Optimal Transport to Orsay", Orsay, France, June 18...

Slides of the presentation at the conference "Optimal Transport to Orsay", Orsay, France, June 18-22, 2012.

- Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

Statistical Image Models

Colors di Source

str image

ibutio (X

n: ) each pixel

point in R3

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

Statistical Image Models

Colors di Source

str image

ibutio (X

n: ) each pixel

point in R3

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Source

Sliced W

image

asserstein

after

projectioncolor

of X to transf

style

er

image color statistics Y

Style image (Y )

J. Rabin

Wasserstein Regularization

Source image (X)

Sliced Wasserstein projection of X to style

image color statistics Y

Input

Source

image

image (X)

Modifi

Source

e

image d

after im

color age

transfer

Style image (Y )

J. Rabin

Wasserstein Regularization

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

Statistical Image Models

Colors di Source

str image

ibutio (X

n: ) each pixel

point in R3

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Source

Sliced W

image

asserstein

after

projectioncolor

of X to transf

style

er

image color statistics Y

Style image (Y )

J. Rabin

Wasserstein Regularization

Source image (X)

Sliced Wasserstein projection of X to style

image color statistics Y

Input

Source

image

image (X)

Modifi

Source

e

image d

after im

color age

transfer

Style image (Y ) Texture synthesis

Other applications:

J. Rabin

Wasserstein Regularization

Texture segmentation

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Overview

• Wasserstein Distance

• Sliced Wasserstein Distance

• Color Transfer

• Wasserstein Barycenter

• Gaussian Texture Models - Discrete Distributions

N 1

Discrete measure: µ =

pi Xi

X

p

i

Rd

i = 1

i=0

i - Discrete Distributions

N 1

Discrete measure: µ =

pi Xi

X

p

i

Rd

i = 1

i=0

i

Point cloud

Constant weights: pi = 1/N.

Xi

Quotient space:

RN d/ N - Discrete Distributions

N 1

Discrete measure: µ =

pi Xi

X

p

i

Rd

i = 1

i=0

i

Point cloud

Histogram

Constant weights: pi = 1/N. Fixed positions Xi (e.g. grid)

Xi

Quotient space:

Affine space:

RN d/ N

{(pi)i \

p

i

i = 1} - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

From Images to Statistics

Source image (X)

Discretized image f

RN d

fi

Rd = R3

N = #pixels, d = #colors.

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

From Images to Statistics

Source image (X)

Discretized image f

RN d

fi

Rd = R3

N = #pixels, d = #colors.

Source image after color transfer

Disclamers: images are not distributions.

Style image (Y )

Needs an estimator:

J. Rabin

Wasserstein Regularization

f

µf

Modify f by controlling µf. - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

From Images to Statistics

Source image (X)

Discretized image f

RN d

fi

Rd = R3

N = #pixels, d = #colors.

Source image after color transfer

Disclamers: images are not distributions.

Style image (Y )

Needs an estimator:

J. Rabin

Wasserstein Regularization

f

µf

Modify f by controlling µf.

Xi

Point cloud discretization: µf =

fi

i - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

From Images to Statistics

Source image (X)

Discretized image f

RN d

fi

Rd = R3

N = #pixels, d = #colors.

Source image after color transfer

Disclamers: images are not distributions.

Style image (Y )

Needs an estimator:

J. Rabin

Wasserstein Regularization

f

µf

Modify f by controlling µf.

Xi

Point cloud discretization: µf =

fi

i

Histogram discretization:

µf =

pi Xi

i

1

Parzen windows: pi =

(x

Z

i

fj)

f

j - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

From Images to Statistics

Source image (X)

Discretized image f

RN d

fi

Rd = R3

N = #pixels, d = #colors.

Source image after color transfer

Disclamers: images are not distributions.

Style image (Y )

Needs an estimator:

J. Rabin

Wasserstein Regularization

f

µf

Modify f by controlling µf.

Xi

Point cloud discretization: µf =

fi

i

Today’s focus

Histogram discretization:

µf =

pi Xi

i

1

Parzen windows: pi =

(x

Z

i

fj)

f

j - Optimal Transport Distances

Grayscale: 1-D

Vector X

µ

RN d

X =

Xi

(image, coefficients, . . . )

i

Colors: 3-D

B

G

R - Optimal Transport Distances

Grayscale: 1-D

Vector X

µ

RN d

X =

Xi

(image, coefficients, . . . )

i

Colors: 3-D

B

G

Optimal assignment:

⇥ ⇥ argmin

|Xi

Y (i)|p

R

N

i - Optimal Transport Distances

Grayscale: 1-D

Vector X

µ

RN d

X =

Xi

(image, coefficients, . . . )

i

Colors: 3-D

B

G

Optimal assignment:

⇥ ⇥ argmin

|Xi

Y (i)|p

R

N

i

Wasserstein distance: Wp(µX, µY )p =

|Xi

Y (i)|p

i

Metric on the space of distributions. - Optimal Transport Distances

Grayscale: 1-D

Vector X

µ

RN d

X =

Xi

(image, coefficients, . . . )

i

Colors: 3-D

B

G

Optimal assignment:

⇥ ⇥ argmin

|Xi

Y (i)|p

R

N

i

Wasserstein distance: Wp(µX, µY )p =

|Xi

Y (i)|p

i

Metric on the space of distributions.

Projection on statistical constraints:

C = {f \ µf = µY }

Proj (

C f ) = Y - Computing Transport Distances

Explicit solution for 1D distribution (e.g. grayscale images):

Xi

Yi

sorting the values, O(N log(N)) operations. - Computing Transport Distances

Explicit solution for 1D distribution (e.g. grayscale images):

Xi

Yi

sorting the values, O(N log(N)) operations.

Higher dimensions: combinatorial optimization methods

Hungarian algorithm, auctions algorithm, etc.

O(N 5/2 log(N )) operations.

intractable for imaging problems. - Computing Transport Distances

Explicit solution for 1D distribution (e.g. grayscale images):

Xi

Yi

sorting the values, O(N log(N)) operations.

Higher dimensions: combinatorial optimization methods

Hungarian algorithm, auctions algorithm, etc.

O(N 5/2 log(N )) operations.

intractable for imaging problems.

Arbitrary distributions:

µ =

pi X

⇥ =

q

i

i Yi

i

i

Wp(µ, )p solution of a linear program. - Overview

• Wasserstein Distance

• Sliced Wasserstein Distance

• Color Transfer

• Wasserstein Barycenter

• Gaussian Texture Models - Approximate Sliced Distance

Key idea: replace transport in Rd by series of 1D transport. Xi

[Rabin, Peyr´e, Delon & Bernot 2010]

Projected point cloud: X = { Xi, ⇥}i.

Xi, - Approximate Sliced Distance

Key idea: replace transport in Rd by series of 1D transport. Xi

[Rabin, Peyr´e, Delon & Bernot 2010]

Projected point cloud: X = { Xi, ⇥}i.

X

(

i,

Sliced Wasserstein distance:

p = 2)

SW (µX, µY )2 =

W (µX , µY )2d

| | =1 - Approximate Sliced Distance

Key idea: replace transport in Rd by series of 1D transport. Xi

[Rabin, Peyr´e, Delon & Bernot 2010]

Projected point cloud: X = { Xi, ⇥}i.

X

(

i,

Sliced Wasserstein distance:

p = 2)

SW (µX, µY )2 =

W (µX , µY )2d

| | =1

Theorem: E(X) = SW (µX, µY )2 is of class C1 and

E(X) =

Xi

Y⇥ (i),

d .

where

N are 1-D optimal assignents of X

and Y . - Approximate Sliced Distance

Key idea: replace transport in Rd by series of 1D transport. Xi

[Rabin, Peyr´e, Delon & Bernot 2010]

Projected point cloud: X = { Xi, ⇥}i.

X

(

i,

Sliced Wasserstein distance:

p = 2)

SW (µX, µY )2 =

W (µX , µY )2d

| | =1

Theorem: E(X) = SW (µX, µY )2 is of class C1 and

E(X) =

Xi

Y⇥ (i),

d .

where

N are 1-D optimal assignents of X

and Y .

Possible to use SW in variational imaging problems.

Fast numerical scheme : use a few random . - Sliced Assignment

Theorem: X is a local minima of E(X) = SW (µX, µY )2

N , X = Y - Sliced Assignment

Theorem: X is a local minima of E(X) = SW (µX, µY )2

N , X = Y

Stochastic gradient descent of E(X):

E (X) =

W (X , Y )2

Step 1: choose

at random.

Step 2: X( +1) = X( )

E (X( )) - Sliced Assignment

Theorem: X is a local minima of E(X) = SW (µX, µY )2

N , X = Y

Stochastic gradient descent of E(X):

E (X) =

W (X , Y )2

Step 1: choose

at random.

Step 2: X( +1) = X( )

E (X( ))

X( ) converges to C = {X \ µX = µY }. - Sliced Assignment

Theorem: X is a local minima of E(X) = SW (µX, µY )2

N , X = Y

Stochastic gradient descent of E(X):

E (X) =

W (X , Y )2

Step 1: choose

at random.

Step 2: X( +1) = X( )

E (X( ))

X( ) converges to C = {X \ µX = µY }.

Final assignment

Numerical observation: X( )

Proj (

C X(0)) - Overview

• Wasserstein Distance

• Sliced Wasserstein Distance

• Color Transfer

• Wasserstein Barycenter

• Gaussian Texture Models - Optimal transport framework Sliced Wasserstein projection Applications

Application to

Co

Color lTo

r r

ansf H

er istogram Equalization

1

Input color images: f

⇥

i

RN 3

Sliced W

.

i =

asserstein projection of X N

f

to style

i(x)

image color statistics Y

x

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Source image (X)

f1

f0

Sliced Wasserstein projection of X to style

image color statistics Y

f0

Source image after color transfer

Style

µ

image (Y )

Source

µ

image (X)

1

0

J. Rabin

Wasserstein Regularization

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to

Co

Color lTo

r r

ansf H

er istogram Equalization

1

Input color images: f

⇥

i

RN 3

Sliced W

.

i =

asserstein projection of X N

f

to style

i(x)

image color statistics Y

x

Optimal assignement:

min |f0 f1 ⇥ |

N

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Source image (X)

f1

f0

Sliced Wasserstein projection of X to style

image color statistics Y

f0

Source image after color transfer

Style

µ

image (Y )

Source

µ

image (X)

1

0

J. Rabin

Wasserstein Regularization

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to

Co

Color lTo

r r

ansf H

er istogram Equalization

1

Input color images: f

⇥

i

RN 3

Sliced W

.

i =

asserstein projection of X N

f

to style

i(x)

image color statistics Y

x

Optimal assignement:

min |f0 f1 ⇥ |

N

Transport:

T : f0(

Optimalx

tr )

ansport fr R3

amework Sliced f

W 1(

asserstein (i))

projection

R3

Applications

Application to Color Transfer

Source image (X)

f1

f0

Sliced Wasserstein projection of X to style

image color statistics Y

f0

Source image after color transfer

Style

µ

image (Y )

Source

µ

image (X)

1

0

TJ.Rabin WassersteinRegularization

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to

Co

Color lTo

r r

ansf H

er istogram Equalization

1

Input color images: f

⇥

i

RN 3

Sliced W

.

i =

asserstein projection of X N

f

to style

i(x)

image color statistics Y

x

Optimal assignement:

min |f0 f1 ⇥ |

N

Optimal transport framework Sliced Wasserstein projection Applications

Transport:

T : f

Application to Color Transfer

0(

Optimalx

tr )

ansport fr R3

amework Sliced f

W 1(

asserstein (i))

projection

R3

Applications

Application to Color Transfer

Equalization:

˜

f0 = T (f0)

˜

f0 = f1

Sliced Wasserstein projection of X to style

Source image (X)

image color statistics Y

f1

f0

T (f

Sliced Wasserstein

0)

projection of X to style

image color statistics Y

Source image (X)

T

f0

Source image after color transfer

Style

µ

image (Y )

Source

µ

image (X)

Source image after color transfer

1

0

µ

Style image (Y )

1

T

J. Rabin

Wasserstein Regularization

J. Rabin

Wasserstein Regularization

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Sliced Wasserstein Transfert

Solving min |f0 f1 ⇥ | is computationally untractable.

N

Approximate Wasserstein projection:

˜

f0 solves min E(f) = SW (µf , µf )

1

f

and is close to f0 - Sliced Wasserstein Transfert

Solving min |f0 f1 ⇥ | is computationally untractable.

N

Approximate Wasserstein projection:

˜

f0 solves min E(f) = SW (µf , µf )

1

f

and is close to f0

(Stochastic) gradient descent:

f (0) = f0

f ( +1) = f ( )

E(f ( ))

f ( )

˜

f0

At convergence: µ ˜ = µ

f0

f1 - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Color

Color T Trran

ansf sfert

er

Sliced Wasserstein projection of X to style

image color statistics Y

Sliced Wasserstein projection of X to style

image color statistics Y

Source image (X)

Input image

Source imagef(0

(X)

T

Source ransfe

image red

after image ˜

color tr f0

ansfer

Style image (Y )

Source image after color transfer

Target

Style image

image (Y f

Style image (Y ) 1

J. Rabin

Wasserstein Regularization

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Optimal transport framework Sliced Wasserstein projection Applications

Application

Optimal transport framework Sliced Wasserstein projection Applications

Optimal transport framework to Color

Sliced WassersteinTransf

projection

er

Application to Color Transfer

Applications

Application

Application to

to Color

Color Tr

Transf

ansfer

er

Color Exchange

Source

Source

Source image

image

image ((XX()X

) )

Source

Input image

image (X

f0 )

T

X

X

arget ⇥⇥ X

Y

Y

image ⇥ f1Y

X ⇥ Y

Style

Style image

image ((YY))

T Style

ransfeimage

red i (

m Y)

age ˜

f0

TransferYYe⇥d im

X age ˜

f1

Style image (Y )

⇥

XY

J. Rabin

Wasserstein Regularization

⇥

X

J. Rabin

Wasserstein Regularization

Y ⇥ X

J. Rabin

Wasserstein Regularization

J. Rabin

Wasserstein Regularization

• Wasserstein Distance

• Sliced Wasserstein Distance

• Color Transfer

• Wasserstein Barycenter

• Gaussian Texture Models- Wasserstein Barycenter

µ2

Barycenter of {(µ

:

i = 1

i, i)}L

i=1

)

L

i

,µ

µ

argmin

iW2(µi, µ)2

µ

(µ 2

2

µ

i=1

µ

W

1

W

2 (

W (µ1, µ )

2

µ3 ,µ ) µ3 - Wasserstein Barycenter

µ2

Barycenter of {(µ

:

i = 1

i, i)}L

i=1

)

L

i

,µ

µ

argmin

iW2(µi, µ)2

µ

(µ 2

2

µ

i=1

µ

W

1

W

2 (

If

, then

W (µ1, µ )

2

µ3

µi = X

µ =

i

X

,µ )

X =

iXi

µ3

i

Generalizes Euclidean barycenter. - Wasserstein Barycenter

µ2

Barycenter of {(µ

:

i = 1

i, i)}L

i=1

)

L

i

,µ

µ

argmin

iW2(µi, µ)2

µ

(µ 2

2

µ

i=1

µ

W

1

W

2 (

If

, then

W (µ1, µ )

2

µ3

µi = X

µ =

i

X

,µ )

X =

iXi

µ3

i

Generalizes Euclidean barycenter.

Theorem: [Agueh, Carlier, 2010]

if µ0 does not vanish on small sets,

µ exists and is unique. - Special Case: 2 Distributions

Case L = 2:

µt

argmin (1 t)W2(µ0, µ)2 + tW2(µ1, µ)2

µ

µ2

µ

µ

1

t

µt is the geodesic path. - Special Case: 2 Distributions

Case L = 2:

µt

argmin (1 t)W2(µ0, µ)2 + tW2(µ1, µ)2

µ

µ2

µ

µ

1

t

µt is the geodesic path.

Discrete point clouds:

µ =

N

,

k=1 Xi(k)

N

Assignment:

⇥ ⇥ argmin

|X1(k)

X2( (k))|2

N

k=1

µ =

N

,

k=1 X (k)

where X = (1 t)X1(k) + tX2( (k)) - Linear Programming Resolution

Discrete setting:

i = 1, . . . , L,

µi =

k Xi(k)

µ2

µ1

µ

µ3 - Linear Programming Resolution

Discrete setting:

i = 1, . . . , L,

µi =

k Xi(k)

L-way interaction cost:

C(k1, . . . , kL) =

i,j

i j | Xi(ki)

Xj(kj)|2

µ2

µ1

µ

µ3 - Linear Programming Resolution

Discrete setting:

i = 1, . . . , L,

µi =

k Xi(k)

L-way interaction cost:

= 1

C(k1, . . . , kL) =

i,j

i j | Xi(ki)

Xj(kj)|2

L-way coupling matrices:

P

PL

= 1

= 1

µ2

µ1

µ

µ3 - Linear Programming Resolution

Discrete setting:

i = 1, . . . , L,

µi =

k Xi(k)

L-way interaction cost:

= 1

C(k1, . . . , kL) =

i,j

i j | Xi(ki)

Xj(kj)|2

L-way coupling matrices:

P

PL

Linear program:

= 1

min

Pk

C(k

1,...,kL

1, . . . , kL)

= 1

P PL k1,...,kL

µ2

µ1

µ

µ3 - Linear Programming Resolution

Discrete setting:

i = 1, . . . , L,

µi =

k Xi(k)

L-way interaction cost:

= 1

C(k1, . . . , kL) =

i,j

i j | Xi(ki)

Xj(kj)|2

L-way coupling matrices:

P

PL

Linear program:

= 1

min

Pk

C(k

1,...,kL

1, . . . , kL)

= 1

P PL k1,...,kL

Barycenter:

µ2

µ =

P

µ1

k1,...,kL X (k1,...,kL)

k1,...,kL

µ

X (k

µ

1, . . . , kL) =

L

3

i=1

iXi(ki) - Sliced Wasserstein Barycenter

Sliced-barycenter:

µX that solves

min

i SW (µX , µX )2

X

i

i - Sliced Wasserstein Barycenter

Sliced-barycenter:

µX that solves

min

i SW (µX , µX )2

X

i

i

Gradient descent:

Ei(X) = SW (µX, µX )2

i

L

X( +1) = X( )

⇥

i

Ei(X( ))

i=1 - Sliced Wasserstein Barycenter

Sliced-barycenter:

µX that solves

min

i SW (µX , µX )2

X

i

i

Gradient descent:

Ei(X) = SW (µX, µX )2

i

L

X( +1) = X( )

⇥

i

Ei(X( ))

Advantages:

i=1

µX is a sum of N Diracs.

Smooth optimization problem.

Disadvantage:

Non-convex problem

local minima. - Sliced Wasserstein Barycenter

Sliced-barycenter:

µX that solves

min

i SW (µX , µX )2

X

i

i

Gradient descent:

Ei(X) = SW (µX, µX )2

i

L

X( +1) = X( )

⇥

µ1

i

Ei(X( ))

Advantages:

i=1

µX is a sum of N Diracs.

Smooth optimization problem.

Disadvantage:

Non-convex problem

µ

µ3

2

local minima. - W W Wasserstein

asserstein

asserstein Bar Bar

Bar

ycenter

ycenter

ycenter

Sliced

Sliced

Sliced W W Wasserstein

asserstein

asserstein Bar Bar

Bar

ycenter

ycenter

ycenter

Exper

Exper

Exper

imental

imental

imental

results

results

results

Applications

Applications

Applications

Conclusion

Conclusion

Conclusion

Color

Color

Color tr tr

tr ansf

ansf

ansfer er

er

Color

Color

Color

harmonization

harmonization

harmonization of of

ofse se

se veral

veral

veral

ima

ima

imaggg

es es

es

. . . Step

Step

Step 1: 1:

1: compute

compute

compute

Sliced-W

Sliced-W

Sliced-W

asserstein

asserstein

asserstein

Bar

Bar

Bar

ycenter

ycenter

ycenter of of

of color

color

color

statistics;

statistics;

statistics;

. . . Step

Step

Step 2: 2:

2:compute Ap

compute

compute

p

Sliced-Wlic

Sliced-W

Sliced-W

ation: C

asserstein

asserstein

asserstein projection

projection olor H

projection

of of

of each

each

each image a

image

image rmoniza

onto

onto

onto

the

the

the

tion

Bar

Bar

Bar

ycenter

ycenter

ycenter; ; ;

Raw image

sequence

RaRa

Ra

w w

w image

image

image

sequence

sequence

sequence

J. J. J. Rabin

Rabin

Rabin – – – GREYC

GREYC

GREYC, , ,

Univ Univ

Univ

ersityersity

ersity

of of of

CaenCaen

Caen

Appro

Appro

Appro

ximate

ximate

ximate W W Wasserstein

asserstein

asserstein Metr Metr

Metr

ic ic

for ic

for

T f

e or

Te Te

xture xture

xture

Synthesis

Synthesis

Synthesis and and

and

Mixing

Mixing

Mixing - W W Wasserstein

asserstein

asserstein Bar Bar

Bar

ycenter

ycenter

ycenter

Sliced

Sliced

Sliced W W Wasserstein

asserstein

asserstein Bar Bar

Bar

ycenter

ycenter

ycenter

Exper

Exper

Exper

imental

imental

imental

results

results

results

Applications

Applications

Applications

Conclusion

Conclusion

Conclusion

Color

Color

Color tr tr

tr ansf

ansf

ansfer er

er

Color

Color

Color

harmonization

harmonization

harmonization of of

ofse se

se veral

veral

veral

ima

ima

imaggg

es es

es

. . . Step

Step

Step 1: 1:

1: compute

compute

compute

Sliced-W

Sliced-W

Sliced-W

asserstein

asserstein

asserstein

Bar

Bar

Bar

ycenter

ycenter

ycenter of of

of color

color

color

statistics;

statistics;

statistics;

. . . Step

Step

Step 2: 2:

2:compute Ap

compute

compute

p

Sliced-Wlic

Sliced-W

Sliced-W

ation: C

asserstein

asserstein

asserstein projection

projection olor H

projection

of of

of each

each

each image a

image

image rmoniza

onto

onto

onto

the

the

the

tion

Bar

Bar

Bar

ycenter

ycenter

ycenter; ; ;

Raw image

sequence

Compute

Wasserstein

RaRa

Ra

w w

w image

image

image

sequence

sequence

sequence

barycenter

J. J. J. Rabin

Rabin

Rabin – – – GREYC

GREYC

GREYC, , ,

Univ Univ

Univ

ersityersity

ersity

of of of

CaenCaen

Caen

Appro

Appro

Appro

ximate

ximate

ximate W W Wasserstein

asserstein

asserstein Metr Metr

Metr

ic ic

for ic

for

T f

e or

Te Te

xture xture

xture

Synthesis

Synthesis

Synthesis and and

and

Mixing

Mixing

Mixing - W W Wasserstein

asserstein

asserstein Bar Bar

Bar

ycenter

ycenter

ycenter

Sliced

Sliced

Sliced W W Wasserstein

asserstein

asserstein Bar Bar

Bar

ycenter

ycenter

ycenter

Exper

Exper

Exper

imental

imental

imental

results

results

results

Applications

Applications

Applications

Conclusion

Conclusion

Conclusion

Color

Color

Color tr tr

tr ansf

ansf

ansfer er

er

Color

Color

Color

harmonization

harmonization

harmonization of of

ofse se

se veral

veral

veral

ima

ima

imaggg

es es

es

. . . Step

Step

Step 1: 1:

1: compute

compute

compute

Sliced-W

Sliced-W

Sliced-W

asserstein

asserstein

asserstein

Bar

Bar

Bar

ycenter

ycenter

ycenter of of

of color

color

color

statistics;

statistics;

statistics;

. . . Step

Step

Step 2: 2:

2:compute Ap

compute

compute

p

Sliced-Wlic

Sliced-W

Sliced-W

ation: C

asserstein

asserstein

asserstein projection

projection olor H

projection

of of

of each

each

each image a

image

image rmoniza

onto

onto

onto

the

the

the

tion

Bar

Bar

Bar

ycenter

ycenter

ycenter; ; ;

Raw image

W WWasserstein

asserstein

asserstein Bar Bar

Bar ycenter

ycenter

ycenter

Sliced

Sliced

Sliced W WWasserstein

asserstein

asserstein Bar Bar

Bar ycenter

ycenter

ycenter

Exper

Exper

Exper

imental

imental

imental

results

results

results

Applications

Applications

Applications

Conclusion

Conclusion

Conclusion

sequence

Color

Color

Color tr tr

tr ansf

ansf

ansferer

er

Color

Color

Color

harmonization

harmonization

harmonization of of

ofsese

se veral

veral

veral

ima

ima

imag g

geses

es

. .. Step

Step

Step 1: 1:

1: compute

compute

compute

Sliced-W

Sliced-W

Sliced-W asserstein

asserstein

asserstein

Bar

Bar

Bar

ycenter

ycenter

ycenter of of

of color

color

color

statistics;

statistics;

statistics;

. .. Step

Step

Step 2: 2:

2: compute

compute

compute

Sliced-W

Sliced-W

Sliced-W asserstein

asserstein

asserstein

projection

projection

projection of of

of each

each

each

image

image

image

onto

onto

onto

the

the

the

Compute

Bar

Bar

Bar

ycenter

ycenter

ycenter; ; ;

Wasserstein

RaRa

Ra

w w

w image

image

image

sequence

sequence

sequence

barycenter

J. J. J. Rabin

Rabin

Rabin – – – GREYC

GREYC

GREYC, , ,

Univ Univ

Univ

ersityersity

ersity

of of of

CaenCaen

Caen

Appro

Appro

Appro

ximate

ximate

ximate W W Wasserstein

asserstein

asserstein Metr Metr

Metr

ic ic

for ic

for

T f

e or

Te Te

xture xture

xture

Synthesis

Synthesis

Synthesis and and

and

Mixing

Mixing

Mixing

Project on

the barycenter

Sliced

Sliced

Sliced W W

W asser

asser

asser stein

stein

stein Pr Pr

Pr ojection

ojection

ojection onon

on the

the

the

bar

bar

bar

ycenter

ycenter

ycenter

J. J. J. Rabin

Rabin

Rabin – – – GREYC

GREYC

GREYC, , ,

Univ Univ

Univ ersity

ersity

ersity of of of Caen

Caen

Caen

Appro

Appro

Appro

ximate

ximate

ximate W WWasserstein

asserstein

asserstein Metr Metr

Metr

ic ic

foric

fT f

or or

eTeTexture

xture

xture

Synthesis

Synthesis

Synthesis and and

and

Mixing

Mixing

Mixing

• Wasserstein Distance

• Sliced Wasserstein Distance

• Color Transfer

• Wasserstein Barycenter

• Gaussian Texture Models- Texture Synthesis

“random”

Problem: given f0, generate f

perceptually “similar”

Exemplar f0 - Texture Synthesis

“random”

Problem: given f0, generate f

perceptually “similar”

analysis Probability

distribution

µ = µ(p)

Exemplar f0 - Texture Synthesis

“random”

Problem: given f0, generate f

perceptually “similar”

analysis Probability synthesis

distribution

µ = µ(p)

Exemplar f0

Outputs f

µ(p) - Texture Synthesis

“random”

Problem: given f0, generate f

perceptually “similar”

analysis Probability synthesis

distribution

µ = µ(p)

Exemplar f0

Outputs f

µ(p)

Gaussian models: µ = N (m, ), parameters p = (m, ). - Gaussian Texture Model

Input exemplar:

f0

RN d

N2

N

d = 1 (grayscale), d = 3 (color)

2

N3

N

Images N

1

1

Videos - Gaussian Texture Model

Input exemplar:

f0

RN d

N2

N

d = 1 (grayscale), d = 3 (color)

2

N3

Gaussian model:

X

µ = N (m, )

N

Images N

1

1

Videos

m

RN d,

RNd Nd - Gaussian Texture Model

Input exemplar:

f0

RN d

N2

N

d = 1 (grayscale), d = 3 (color)

2

N3

Gaussian model:

X

µ = N (m, )

N

Images N

1

1

Videos

m

RN d,

RNd Nd

Texture analysis: from f0 RN d, learn (m, ).

highly under-determined problem. - Gaussian Texture Model

Input exemplar:

f0

RN d

N2

N

d = 1 (grayscale), d = 3 (color)

2

N3

Gaussian model:

X

µ = N (m, )

N

Images N

1

1

Videos

m

RN d,

RNd Nd

Texture analysis: from f0 RN d, learn (m, ).

highly under-determined problem.

Texture synthesis:

given (m, ), draw a realization f = X( ).

Factorize

= AA (e.g. Cholesky).

Compute f = m + Aw where w drawn from N (0, Id). - Spot Noise Model [Galerne et al.]

Stationarity hypothesis: (periodic BC) X(· + ) X - Spot Noise Model [Galerne et al.]

Stationarity hypothesis: (periodic BC) X(· + ) X

Block-diagonal Fourier covariance:

y = f

computed as

ˆy( ) = ˆ( ) ˆ

f ( )

2ix

2ix

where ˆ

1⇥1

2⇥2

f ( ) =

f (x)e

+

N1

N2

x - Spot Noise Model [Galerne et al.]

Stationarity hypothesis: (periodic BC) X(· + ) X

Block-diagonal Fourier covariance:

y = f

computed as

ˆy( ) = ˆ( ) ˆ

f ( )

2ix

2ix

where ˆ

1⇥1

2⇥2

f ( ) =

f (x)e

+

N1

N2

x

Maximum likelihood estimate (MLE) of m from f0:

1

i,

mi =

f

N

0(x)

Rd

x - Spot Noise Model [Galerne et al.]

Stationarity hypothesis: (periodic BC) X(· + ) X

Block-diagonal Fourier covariance:

y = f

computed as

ˆy( ) = ˆ( ) ˆ

f ( )

2ix

2ix

where ˆ

1⇥1

2⇥2

f ( ) =

f (x)e

+

N1

N2

x

Maximum likelihood estimate (MLE) of m from f0:

1

i,

mi =

f

N

0(x)

Rd

x

1

MLE of :

i,j =

f

N

0(i + x) f0(j + x)

Rd d

x - Spot Noise Model [Galerne et al.]

Stationarity hypothesis: (periodic BC) X(· + ) X

Block-diagonal Fourier covariance:

y = f

computed as

ˆy( ) = ˆ( ) ˆ

f ( )

2ix

2ix

where ˆ

1⇥1

2⇥2

f ( ) =

f (x)e

+

N1

N2

x

Maximum likelihood estimate (MLE) of m from f0:

1

i,

mi =

f

N

0(x)

Rd

x

1

MLE of :

i,j =

f

N

0(i + x) f0(j + x)

Rd d

x

= 0,

ˆ( ) = ˆf0( ) ˆf0( )

Cd d

is a spot noise

= 0, ˆ( ) is rank-1. - Example of Synthesis

Synthesizing f = X( ), X

N (m, ):

= 0,

ˆ

f ( ) = ˆ

f0( ) ˆ

w( )

w

N (N 1, N 1/2IdN )

Cd

C

Convolve each channel with the same white noise.

Input f0 RN 3

Realizations f - Gaussian Optimal Transport

Input distributions (µ0, µ1) with µi = N (mi, i).

Ellipses: Ei = x Rd \ (mi x)

1(m

i

i

x)

c

E0

E1 - Gaussian Optimal Transport

Input distributions (µ0, µ1) with µi = N (mi, i).

Ellipses: Ei = x Rd \ (mi x)

1(m

i

i

x)

c

E0

T

E1

Theorem: If ker( 0) Im( 1) = {0},

T (x) = Sx + m1

m0

where

+

1/2

1/2

S = 1/2

1

0,1

1

0,1 = (

1/2

)1/2

1

0

1

W2(µ0, µ1)2 = tr ( 0 + 1

2 0,1) + |m0 m1|2, - Wasserstein Geodesics

Geodesics between (µ0, µ1): t [0, 1]

µt

Variational caracterization:

(W2 is a geodesic distance)

µt = argmin (1

t)W2(µ0, µ)2 + tW2(µ1, µ)2

µ - Wasserstein Geodesics

Geodesics between (µ0, µ1): t [0, 1]

µt

Variational caracterization:

(W2 is a geodesic distance)

µt = argmin (1

t)W2(µ0, µ)2 + tW2(µ1, µ)2

µ

µ

1

µ

µ

Optimal transport caracterization:

0

µt = ((1

t)Id + tT ) µ0 - Wasserstein Geodesics

Geodesics between (µ0, µ1): t [0, 1]

µt

Variational caracterization:

(W2 is a geodesic distance)

µt = argmin (1

t)W2(µ0, µ)2 + tW2(µ1, µ)2

µ

µ

1

µ

µ

Optimal transport caracterization:

0

µt = ((1

t)Id + tT ) µ0

µ0

Gaussian case: µt = Tt µ0 = N (mt, t)

mt = (1

t)m0 + tm1

t = [(1

t)Id + tT ] 0[(1

t)Id + tT ]

the set of Gaussians is geodesically convex.

µ1 - Geodesic of Spot Noises

Theorem: Let for i = 0, 1, µi = µ(f[i]) be spot noises,

i.e. ˆ

Then

i( ) = ˆ

f [i]( ) ˆ

f [i]( ) .

t

[0, 1], µt = µ(f[t])

f [t] = (1

t)f [0] + tg[1]

ˆ

ˆ

f [1]( ) ˆ

f [0]( )

g[1]( ) = ˆ

f [1]( ) | ˆf[1]( ) ˆf[0]( )| - Geodesic of Spot Noises

Theorem: Let for i = 0, 1, µi = µ(f[i]) be spot noises,

i.e. ˆ

Then

i( ) = ˆ

f [i]( ) ˆ

f [i]( ) .

t

[0, 1], µt = µ(f[t])

f [t] = (1

t)f [0] + tg[1]

ˆ

ˆ

f [1]( ) ˆ

f [0]( )

g[1]( ) = ˆ

f [1]( ) | ˆf[1]( ) ˆf[0]( )|

f [0]

0

1

t

f [1] - OT Barycenters

µ2

µ1

µ3 - OT Barycenters

µ2

µ1

µ3 - OT Barycenters

µ2

µ1

µ3 - 2-D Gaussian Barycenters

Euclidean

Optimal transport

Rao - Spot Noise Barycenters

ut

Inp - Spot Noise Barycenters

ut

Inp - 19/05/12

Static and dynamic texture mixing using optimal transport

19/05/12

Static and dynamic texture mixing using optimal transport

Abstract.- This paper tackles static and dynamic texture mixing by combining the statistical properties of an input set of images or videos. We focus on spot noise

textures that fol ow a stationary and Gaussian model which can be learned from the given exemplars. From here, we define, using optimal transport, the distance

between texture models, derive the geodesic path, and define the barycenter between several texture models. These derivations are useful because they al ow the

user to navigate inside the set of texture models, interpolating a new texture model at each element of the set. From these new interpolated models, new textures

can be synthesized of arbitrary size in space and time. Numerical results obtained from a library of exemplars show the ability of our method to generate new

complex realistic static and dynamic textures.

Abstract.- This p 19/05/12

aper tackles static and dynamic texture mixing b S

y ta

co ti

m c

bi a

ni n

ng d

t

h d

e y

s n

ta a

ti m

sti i

c c

al te

pr x

op tu

ert r

i e

es m

of ix

an i in

n g

p

utu

ss

eitn

og

f i o

m p

agti

e m

s oa

r l

vitr

d a

eon

s s

. p

W o

e r

f t

ocus on spot noise

Results following the geodesic path:

3

textures that fol ow a stationary and

Gaussian model which can be learned from the given exemplars. From here, we define, using optimal transport, the distance

between texture models, derive the g

eodesic path, and define the barycenter between several texture models. These derivations are useful because they al ow the

2

user to navigate inside the set of texture models, interpolating a new texture model at each element of the se4t. From these new interpolated models, new textures

can be synthesized of arbitrary size i M A T L A B C O D E

n space and time. Numerical results obtained from a library of 1

exemplars show the ability of our method to generate new

5

complex realistic static and dynamic textures.

6

0

7

19/05/12

Static and dynamic texture mixing using optimal transport

Results following the geodesic path:

3

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start

Abstract.- This paper tackles static and dynamic texture mixing by combining the statistical properties of an input set of images or videos. We focus on spot noise

te

i x

m taugre

es s

%th

2

[ a

%0 t]

%fo

%lo

%w

%a

%sta

% ti

%on

% ar

% y

[

%

1 a

] n

% d

% G

%au

% ss

% ia

% n

% m

%od

%

4

[ e

2 l

%] w

% hi

%ch

% c

%an

% b

% e

%lea

% rn

% e

%d

%fro

% m

% th

% e

% gi

%ve

% n

% ex

% e

%mp

% la

% rs

% . F

% r

%om

% h

% e

%re

%, w

% e

% de

% fi

%ne

% , u

% s

%ing

% o

% pt

% im

% al

% tr

%an

% sp

% o

%rt,

% th

% e

% di

%sta

% n

%ce

%%%%%%%%%%>

M A T L A B C O D E

between

f texture models, derifve the geodesic path

f , and define the barycenter between several texture models. These derivations are useful because they alow the

1

user to navigate inside the set of textu

5 re models, interpolating a new texture model at each element of the set. From these new interpolated models, new textures

can be synthesized of arbitrary size in spa

6 ce and time. Numerical results obtained from a library of exemplars show the ability of our method to generate new

complex realistic static and dynamic textures.

0

7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start

Abstract.- This paper tackles static a R

n e

d s

du

ylts

n

a fo

m lil

co

w

te in

xt g

u rth

e e

g

mi e

x o

i d

n e

g si

b c

y p

cat

o h:

mbining the statistical properties of 3

an input set of images or videos. We focus on spot noise

images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

textures th

[ a

0 t] fol ow a station [1]

[2]

between

f texture models, derf

f

user to navigate inside the s Dyn

ary and

Gaussian model which can be learned from the given exemplars. From here, we define, using optimal transport, the distance

2

ive the geodesic path, and define the barycenter between several texture models. These derivations are useful because they al ow the

M

et of texture am

A T L A B

models, int i

C

er c

O D E

polatin T

g a ex

new text t

ureur

model es

at each el M

emen i

4

t1 ox

f th i

e s n

et. Fg

rom these new interpolated models, new textures

can be synthesized of arbitrary size i

5

n space and time. Numerical results obtained from a library of exemplars show the ability of our method to generate new

6

complex realistic static and dynamic textures.

0

7

[See web page]

Results following the geodesic path: <%%%%%%%%%%%%%%%%%%

3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start

images%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

[0]

[1]

[2]

f

f

2

f

M A T L A B C O D E

4

1

0

1

5

2

3

4

5

6

7

6

ars

0

7

<%%%%%%%%%%%%%%%%% <

%%%

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %%%%%%%

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %%%%%%%

% %

% %

%S %

ta %

rt %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

images%%%%%%%%%%%%%% <

% %

% %

% %

% %

% %%%%%%

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %%%%%%%

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %%%%%%%

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %

% %%%%%%

> >

[0]

[1]

[0] [2]

[1]

[2]

f

f

0

1

empl

f f

f

f

2

3

4

5

6

7

Ex

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

[0]

[1]

[2]

f

f

0

f

1

2

3

4

5

6

7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

[0]

[1]

[2]

f

f

f

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

[0]

[1]

[2]

f

f

f

0

1

2

3

4

5

6

7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

0

1

2

3

4

5

6

7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

0

1

2

3

4

5

6

7

<%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%>

localhost/Users/gabrielpeyre/Documents/Dropbox/slides/2012-‐‑05-‐‑20-‐‑siims-‐‑textures/…/index.html

1/1

localhost/Users/gabrielpeyre/Documents/Dropbox/slides/2012-‐‑05-‐‑20-‐‑siims-‐‑textures/…/index.html

1/1

localhost/Users/gabrielpeyre/Documents/Dropbox/slides/2012-‐‑05-‐‑20-‐‑siims-‐‑textures/…/index.html

1/1

localhost/Users/gabrielpeyre/Documents/Dropbox/slides/2012-‐‑05-‐‑20-‐‑siims-‐‑textures/…/index.html

1/1 - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

Conclusion

Source image (X)

Image modeling with statistical constraints

Colorization, synthesis, mixing, . . .

Source image after color transfer

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

Conclusion

Source image (X)

Image modeling with statistical constraints

Colorization, synthesis, mixing, . . .

Wasserstein distance approach

Source image after color transfer

Fast sliced approximation.

Style image (Y )

J. Rabin

Wasserstein Regularization - Optimal transport framework Sliced Wasserstein projection Applications

Application to Color Transfer

Sliced Wasserstein projection of X to style

image color statistics Y

Conclusion

Source image (X)

Image modeling with statistical constraints

Colorization, synthesis, mixing, . . .

Wasserstein distance app

14

roac

Anon h

ymous

Source image after color transfer

Fast sliced approximation.

Style image (Y )

J. Rabin

Wasserstein Regularization

Extension to a wide range

of imaging problems.

Color transfert, segmentation, . . .

P (⇥⇥)

P (⇥ )

P (⇥ c )

P (⇥⇥)

Pin

(⇥ )

Pout

(⇥ c )

P (⇥⇥)

P (⇥ )

P (⇥ c )

P (⇥⇥)

P (⇥ )

P (⇥ c )

Fig. 4. Left: example of natural image segmentation. Below each image is displayed the 2-

D histogram of the image (in the space of the two dominant color eigenvectors as provided

by a PCA) for the whole domain

(which is P (⇥⇥)) and over the inside and outside the

segmented region.