このページは http://www.slideshare.net/EricMarsden1/modelling-correlations-using-python の内容を掲載しています。

掲載を希望されないスライド著者の方は、こちらよりご連絡下さい。

2年弱前 (2015/01/02)にアップロードinテクノロジー

Risk models often involve correlated random variables. Learn how to measure the degree of linear ...

Risk models often involve correlated random variables. Learn how to measure the degree of linear dependency between two random variables using Python and the SciPy library. Some warnings on dependency, causality and correlation in risk analysis.

Part of the free risk-engineering.org teaching materials.

- Modelling correlations

with Python and SciPy

Eric Marsden - Context

▷ Analysis of causal effects is an important activity in risk analysis

• Process safety engineer: “To what extent does increased process temperature and

pressure increase the level of corrosion of my equipment?”

• Medical researcher: “What is the mortality impact of smoking 2 packets of

cigarettes per day?”

• Safety regulator: “Do more frequent site inspections lead to a lower accident

rate?”

• Life insurer: “What is the conditional probability when one spouse dies, that the

other will die shortly afterwards?”

▷ The simplest statistical technique for analyzing causal effects is

correlation analysis

▷ Correlation analysis measures the extent to which two variables vary

together, including the strength and direction of their relationship

2 / 29 - Measuring linear correlation

▷ Linear correlation coefficient: a measure of the strength and direction

of a linear association between two random variables

• also called the Pearson product-moment correlation coefficient

cov(X ,Y )

𝔼[(X −𝜇

▷ 𝜌

X )(Y −𝜇Y )]

X ,Y =

=

𝜎X 𝜎Y

𝜎X 𝜎Y

• 𝔼 is the expectation operator

• cov means covariance

• 𝜇X is the expected value of random variable X

• 𝜎X is the standard deviation of X

▷ Python: scipy.stats.pearsonr(X, Y)

▷ Excel / Google Docs spreadsheet: function CORREL

3 / 29 - Measuring linear correlation

The linear correlation coefficient ρ quantifies the strengths and directions of

movements in two random variables:

▷ sign of ρ determines the relative directions that the variables move in

▷ value determines strength of the relative movements (ranging from -1

to +1)

▷ ρ = 0.5: one variable moves in the same direction by half the amount that

the other variable moves

▷ ρ = 0: variables are uncorrelated

• does not imply that they are independent!

4 / 29 - c o r r e l a t i o n ≠

d e p e n d e n c y

Image source: Wikipedia

Examples of correlations

5 / 29 - c o r r e l a t i o n ≠

d e p e n d e n c y

Image source: Wikipedia

Examples of correlations

5 / 29 - Examples of correlations

c o r r e l a t i o n ≠

d e p e n d e n c y

Image source: Wikipedia

5 / 29 - Online visualization: interpreting correlations

Try it out online: http://rpsychologist.com/d3/correlation/

6 / 29 - Not all relations are linear!

▷ Example: Yerkes–Dodson law

• empirical relationship between level of

arousal/stress and level of performance

▷ Performance initially increases with

stress/arousal

▷ Beyond a certain level of stress, performance

decreases

Source: https://en.wikipedia.org/wiki/Yerkes–Dodson_law

7 / 29 - Measuring correlation with NumPy

In [3]: import numpy

import matplotlib.pyplot as plt

import scipy.stats

In [4]: X = numpy.random.normal(10, 1, 100)

Y = X + numpy.random.normal(0, 0.3, 100)

plt.scatter(X, Y)

Out[4]: <matplotlib.collections.PathCollection at 0x7f7443e3c438>

E x e r c i s e : s h o w

t h a t w h e n t h e e r r o r

i n Y d e c r e a s e s , t h e c o r r e l a t i o n

c o e f f i c i e n t i n c r e a s e s

E x e r c i s e : p r o d u c e d a t a a n d a p l o t

w i t h a n e g a t i v e c o r r e l a t i o n

c o e f f i c i e n t

In [5]: scipy.stats.pearsonr(X, Y)

Out[5]: (0.9560266103379802, 5.2241043747083435e-54)

8 / 29 - Anscombe’s quartet

12

I

II

8

4

12

III

IV

8

E a c h d a t a s e t h a s t h e s a m e

4

c o r r e l a t i o n c o e f f i c i e n t !

0

10

20

0

10

20

Four datasets proposed by Francis Anscombe to illustrate

the importance of graphing data rather than relying

blindly on summary statistics

9 / 29 - Plotting relationships between variables with matplotlib

▷ Scatterplot: use function plt.scatter

14

▷ Continuous plot or X-Y: function plt.plot

12

10

8

6

import matplotlib.pyplot as plt

4

import numpy

2

0

X = numpy.random.uniform(0, 10, 100)

−2

Y = X + numpy.random.uniform(0, 2, 100)

−2

0

2

4

6

8

10

12

plt.scatter(X, Y, alpha=0.5)

plt.show()

10 / 29 - Correlation matrix

▷ A correlation matrix is used to investigate the dependence between

multiple variables at the same time

• output: a symmetric matrix where element mij is the correlation coefficient

between variables i and j

• note: diagonal elements are always 1

• can be visualized graphically using a correlogram

• allows you to see which variables in your data are informative

▷ In Python, can use:

• dataframe.corr() method from the Pandas library

• numpy.corrcoef(data) from the NumPy library

• visualize using imshow from Matplotlib or heatmap from the Seaborn library

11 / 29 - Correlation matrix: example

Vehicle_Reference

Casualty_Reference

0.8

Casualty_Class

Sex_of_Casualty

Analysis of the correlations between

Age_of_Casualty

0.4

Age_Band_of_Casualty

different variables affecting road

Casualty_Severity

0.0

casualties

Pedestrian_Location

Pedestrian_Movement

Car_Passenger

−0.4

Bus_or_Coach_Passenger

Pedestrian_Road_Maintenance_Worker

import pandas

Casualty_Type

−0.8

import matplotlib.pyplot as plt

Casualty_Home_Area_Type

import seaborn as sns

data = pandas.read_csv("casualties.csv")

cm = data.corr()

Casualty_Class

Car_Passenger

Casualty_Type

Sex_of_Casualty

sns.heatmap(cm, square=True)

Vehicle_Reference

Age_of_Casualty

Casualty_Severity

Casualty_Reference

Pedestrian_Location

plt.yticks(rotation=0)

Pedestrian_Movement

Age_Band_of_Casualty

plt.xticks(rotation=90)

Bus_or_Coach_Passenger

Casualty_Home_Area_Type

Pedestrian_Road_Maintenance_Worker

Data source: UK Department for Transport, https://data.gov.uk/dataset/road-accidents-safety-data

12 / 29 - Aside: polio caused by ice cream!

▷ Polio: an infectious disease causing paralysis, which primarily

affects young children

▷ Largely eliminated today, but was once a worldwide concern

▷ Late 1940s: public health experts in usa noticed that the

incidence of polio increased with the consumption of ice cream

▷ Some suspected that ice cream caused polio… sales plummeted

▷ Polio incidence increases in hot summer weather

▷ Correlation is not causation: there may be a hidden, underlying

variable

• but it sure is a hint! [Edward Tufte]

More info: Freakonomics, Steven Levitt and Stephen J. Dubner

13 / 29 - Aside: fire fighters and fire damage

▷ Statistical fact: the larger the number of fire-fighters attending

the scene, the worse the damage!

▷ More fire fighters are sent to larger fires

▷ Larger fires lead to more damage

▷ Lurking (underlying) variable = fire size

▷ An instance of “Simpson’s paradox”

14 / 29 - Aside: low birth weight babies of tobacco smoking mothers

▷ Statistical fact: low birth-weight children born to smoking mothers have

a lower infant mortality rate than the low birth weight children of

non-smokers

▷ In a given population, low birth weight babies have a significantly higher

mortality rate than others

▷ Babies of mothers who smoke are more likely to be of low birth weight

than babies of non-smoking mothers

▷ Babies underweight because of smoking still have a lower mortality rate

than children who have other, more severe, medical reasons why they are

born underweight

▷ Lurking variable between smoking, birth weight and infant mortality

Source: Wilcox, A. (2001). On the importance — and the unimportance — of birthweight, International Journal of Epidemiology.

30:1233–1241

15 / 29 - Aside: exposure to books leads to higher test scores

▷ In early 2004, the governor of the us state of Illinois R. Blagojevich

announced a plan to mail one book a month to every child in in the state

from the time they were born until they entered kindergarten. The plan

would cost 26 million usd a year.

▷ Data underlying the plan: children in households where there are more

books do better on tests in school

▷ Later studies showed that children from homes with many books did

better even if they never read…

▷ Lurking variable: homes where parents buy books have an environment

where learning is encouraged and rewarded

Source: http://freakonomics.com/2008/12/10/the-blagojevich-upside/

16 / 29 - Aside: chocolate consumption produces Nobel prizes

Source: Chocolate Consumption, Cognitive Function, and Nobel Laureates, N Engl J Med 2012, doi: 10.1056/NEJMon1211064

17 / 29 - Aside: cheese causes death by bedsheet strangulation

Note: real data!

Source: http://www.tylervigen.com/, with many more surprising correlations

18 / 29 - hidden

factor

However, correlation is not sufficient

to demonstrate causality. There might

be some genetic factor that causes

both lung cancer and desire for

nicotine.

I n l o g i c , t h i s i s c a l l e d t h e

p o s t h o c e r g o p r o p t e r h o c

f a l l a c y

Beware assumptions of causality

1964: the US Surgeon General issues a

report claiming that cigarette

smoking causes lung cancer, based

mostly on correlation data from

medical studies.

smoking

lung

cancer

19 / 29 - Beware assumptions of causality

1964: the US Surgeon General issues a

report claiming that cigarette

smoking causes lung cancer, based

hidden

factor

mostly on correlation data from

medical studies.

However, correlation is not sufficient

to demonstrate causality. There might

be some genetic factor that causes

smoking

lung

cancer

both lung cancer and desire for

nicotine.

I n l o g i c , t h i s i s c a l l e d t h e

p o s t h o c e r g o p r o p t e r h o c

f a l l a c y

19 / 29 - Beware assumptions of causality

▷ To demonstrate the causality, you need a randomized controlled

experiment

▷ Assume we have the power to force people to smoke or not smoke

• and ignore moral issues for now!

▷ Take a large group of people and divide them into two groups

• one group is obliged to smoke

• other group not allowed to smoke (the “control” group)

▷ Observe whether smoker group develops more lung cancer than the

control group

▷ We have eliminated any possible hidden factor causing both smoking and

lung cancer

▷ More information: read about design of experiments

20 / 29 - Constructing arguments of causality from observations

▷ Causality is an important — and complex — notion in risk analysis and

many areas of science, with two main approaches used

▷ Conservative approach used mostly in the physical sciences requires

• a plausible physical model for the phenomenon showing how A might lead

to B

• observations of correlation between A and B

▷ Relaxed approach used in the social sciences requires

• a randomized controlled experiment in which the choice of receiving the

treatment A is determined only by a random choice made by the experimenter

• observations of correlation between A and B

▷ Alternative relaxed approach: a quasi-experimental “natural experiment”

21 / 29 - Natural experiments and causal inference

▷ Natural experiment: an empirical study in which allocation between

experimental and control treatments are determined by factors outside

the control of investigators but which resemble random assignment

▷ Example: in testing whether military service subsequently affected job

evolution and earnings, economists examined difference between

American males drafted for the Vietnam war and those not drafted

• draft was assigned on the basis of date of birth, so “control” and “treatment”

groups likely to be similar statistically

• findings: earnings of veterans approx. 15% lower than those of non-veterans

22 / 29 - Natural experiments and causal inference

▷ Example: cholera outbreak in London in 1854 led to 616 deaths

▷ Medical doctor J. Snow discovered a strong association between

the use of the water from specific public water pumps and

deaths and illnesses due to cholera

• “bad” pumps supplied by a company that obtained water from the

Thames downstream of a raw sewage discharge

• “good” pumps obtained water from the Thames upstream from the

discharge point

▷ Cholera outbreak stopped when the “bad” pumps were shut

down

23 / 29 - Directionality of effect problem

aggressive behaviour

watching violent films

aggressive behaviour

watching violent films

Do aggressive children prefer violent TV programmes, or do violent

programmes promote violent behaviour?

25 / 29 - Further reading

You may also be interested in:

▷ slides on linear regression modelling using Python, the simplest

approach to modelling correlated data

▷ slides on copula and multivariate dependencies for risk models, a

more sophisticated modelling approach that is appropriate when

dependencies between your variables are not linear

Both are available from risk-engineering.org and from

slideshare.net/EricMarsden1.

26 / 29 - Image credits

▷ Eye (slide 21): Flood G. via https://flic.kr/p/aNpvLT, CC BY-NC-ND

licence

▷ Map of cholera outbreaks (slide 23) by John Snow (1854) from Wikipedia

Commons, public domain

For more free course materials on risk engineering,

visit https://risk-engineering.org/

27 / 29 - For more information

▷ SciPy lecture notes: https://scipy-lectures.github.io/

▷ Analysis of the “pay for performance” (correlation between a ceo’s pay

and their job performance, as measured by the stock market) principle,

http://freakonometrics.hypotheses.org/15999

▷ Python notebook on a more sophisticated Bayesian approach to

estimating correlation using PyMC,

nbviewer.ipython.org/github/psinger

For more free course materials on risk engineering,

visit https://risk-engineering.org/

28 / 29 - Feedback welcome!

This presentation is distributed under the terms of

the Creative Commons Attribution – Share Alike

licence

@LearnRiskEng

Was some of the content unclear? Which parts of the lecture

fb.me/RiskEngineering

were most useful to you? Your comments to

google.com/+RiskengineeringOrgCourseware

feedback@risk-engineering.org (email) or

@LearnRiskEng (Twitter) will help us to improve these course

materials. Thanks!

For more free course materials on risk engineering,

visit https://risk-engineering.org/

29 / 29