In the last chapter we saw how to describe the distribution of a single variable in a sample. However, in most cases, studies require to describe several variables that are often related. For instance, a nutritional study should consider all the variables that could be related to the weight, as height, age, gender, smoking, diet, physic exercise, etc.

To understand a phenomenon that involve several variables is not enough to study every variable by its own. We have to study all the variables together to describe how they interact and the type of relation among them.

Usually in a dependency study there is a dependent variable $Y$ that it is supposed to be influenced by a set of variables $X_1,\ldots,X_n$ known as independent variables. The simpler case is a simple dependency study when there is only one independent variable, that is the case covered in this chapter.

Joint distribution

Joint frequencies

To study the relation between two variables $X$ and $Y$, we have to study the joint distribution of the two-dimensional variable $(X,Y)$, whose values are pairs $(x_i,y_j)$ where the first element is a value of $X$ and the second a value of $Y$.

Definition - Joint sample frequencies. Given a sample of $n$ values and a two-dimensional variable $(X,Y)$, for every value of the variable $(x_i,y_j)$ is defined

  • Absolute frequency $n_{ij}$: Is the number of times that the pair $(x_i,y_j)$ appears in the sample.

  • Relative frequency $f_{ij}$: Is the proportion of times that the pair $(x_i,y_j)$ appears in the sample.

Watch out! For two-dimensional variables it make no sense cumulative frequencies.

Joint frequency distribution

The values of the two-dimensional variable with their frequencies is known as joint frequency distribution, and is represented in a joint frequency table.

Example (grouped data). The height (in cm) and weight (in kg) of a sample of 30 students is:

(179,85), (173,65), (181,71), (170,65), (158,51), (174,66),
(172,62), (166,60), (194,90), (185,75), (162,55), (187,78),
(198,109), (177,61), (178,70), (165,58), (154,50), (183,93),
(166,51), (171,65), (175,70), (182,60), (167,59), (169,62),
(172,70), (186,71), (172,54), (176,68),(168,67), (187,80).

The joint frequency table is

Scatter plot

The joint frequency distribution can be represented graphically with a scatter plot, where data is displayed as a collections of points on a $XY$ coordinate system.

Usually the independent variable is represented in the $X$ axis and the dependent variable in the $Y$ axis. For every data pair $(x_i,y_j)$ in the sample a dot is drawn on the plane with those coordinates.

Scatter plot

The result is a set of points that usually is known as a point cloud.

Example. The scatter plot below represent the distribution of heights and weights of the previous sample.

Scatter plot of heights and weights

Scatter plot interpretation

The shape of the point cloud in a scatter plot gives information about the type of relation between the variables.

Scatter plot of different types of relations

Marginal frequency distributions

The frequency distributions of each variable of the two-dimensional variable are known as marginal frequency distributions.

We can get the marginal frequency distributions from the joint frequency table by adding frequencies by rows and columns.

Example. The marginal frequency distributions for the previous sample of heights and weights are

and the corresponding statistics are

Covariance

To study the relation between two variables, we have to analyze the joint variation of them.

Devitations to means in an scatterplot

Dividing the point cloud of the scatter plot in 4 quadrants centered in the mean point $(\bar x, \bar y)$, the sign of deviations from the mean is:

Quadrant $(x_i-\bar x)$ $(y_j-\bar y)$ $(x_i-\bar x)(y_j-\bar y)$
1 $+$ $+$ $+$
2 $-$ $+$ $-$
3 $-$ $-$ $+$
4 $+$ $-$ $-$

Quadrants of a scatter plot

If there is an increasing linear relationship between the variables, most of the points will fall in quadrants 1 and 3, and the sum of the products of deviations from the mean will be positive.

Increasing linear scatter plot

If there is an decreasing linear relationship between the variables, most of the points will fall in quadrants 2 and 4, and the sum of the products of deviations from the mean will be negative.

Decreasing linear scatter plot

Using the products of deviations from the means we get the following statistic.

Definition - Sample covariance. The sample covariance of a two-dimensional variable $(X,Y)$ is the average of the products of deviations from the respective means.

It can also be calculated using the formula

The covariance measures the linear relation between two variables:

  • If $s_{xy}>0$ there exists an increasing linear relation.

  • If $s_{xy}<0$ there exists a decreasing linear relation.

  • If $s_{xy}=0$ there is no linear relation.

Example. Using the joint frequency table of the sample of heights and weights

we get that the covariance is equal to

This means that there is a increasing linear relation between the weight and the height.

Regression

In most cases the goal of a dependency study is not only to detect a relation between two variables, but also to express that relation with a mathematical function, in order to predict the dependent variable for every value of the independent one. The part of Statistics in charge of constructing such a function is called regression, and the function is kwnon as regression function or regression model.

Simple regression models

There are a lot of types of regression models. The most common models are shown in the table below.

Model Equation
Linear $y=a+bx$
Quadratic $y=a+bx+cx^2$
Cubic $y=a+bx+cx^2+dx^3$
Potential $y=a\cdot x^b$
Exponential $y=e^{a+bx}$
Logarithmic $y=a+b\log x$
Inverse $y=a+\frac{b}{x}$
Sigmoidal $y=e^{a+\frac{b}{x}}$

The model choice depends on the shape of the points cloud in the scatterplot.

Residuals or predictive errors

Once chosen the type of regression model, we have to determine which function of that family explains better the relation between the dependent and the independent variables, that is, the function that predicts better the dependent variable.

That function is the function that minimizes the distances from the observed values for $Y$ in the sample to the predicted values of the regression function. These distances are known as residuals or predictive errors.

Definition - Residuals or predictive errors. Given a regression model $y=f(x)$ for a two-dimensional variable $(X,Y)$, the residual or predictive error for every pair $(x_i,y_j)$ of the sample is the difference between the observed value of the dependent variable $y_j$ and the predicted value of the regression function for $x_i$,

Regression rediduals on Y

Least squares fitting

A way to get the regression function is the least squares method, that determines the function that minimizes the squared residuals.

For a linear model $f(x) = a + bx$, the sum depends on two parameters,the intercept $a$, and the slope $b$ of the straight line,

This reduces the problem to determine the values of $a$ and $b$ that minimize this sum.

To solve the minimization problem, we have to set to zero the partial derivatives with respect to $a$ and $b$.

And solving the equation system, we get

This values minimize the residuals on $Y$ and give us the optimal linear model.

Regression line

Definition - Regression line. Given a sample of a two-dimensional variable $(X,Y)$, the regression line of $Y$ on $X$ is

The regression line of $Y$ on $X$ is the straight line that minimizes the predictive errors on $Y$, therefore it is the linear regression model that gives better predictions of $Y$.

Example. Using the previous sample of heights ($X$) and weights ($Y$) with the following statistics

the regression line of weight on height is

And the regression line of height on weight is

Observe that the regression lines are different!

Regression lines of heights and wieghts

Relative position of the regression lines

Usually, the regression line of $Y$ on $X$ and the regression line of $X$ on $Y$ are not the same, but they always intersect in the mean point $(\bar x,\bar y)$.

If there is a perfect linear relation between the variables, then both regression lines are the same, as that line makes both $X$-residuals and $Y$-residuals zero.

Perfect linear regression

If there is no linear relation between the variables, then both regression lines are constant and equals to the respective means,

So, they intersect perpendicularly.

Non linear regression

Regression coefficient

The most important parameter of a regression line is the slope.

Definition - Regression coefficient $b_{yx}$. Given a sample of a two-dimensional variable $(X,Y)$, the regression coefficient of the regression line of $Y$ on $X$ is its slope,

The regression coefficient has always the same sign as the covariance.

It measures how the dependent variable changes in relation to the independent one according to the regression line. In particular, it gives the number of units that the dependent variable increases or decreases for every unit that the independent variable increases.

Example. In the sample of heights and weights, the regression line of weight on height was

Thus, the regression coefficient of weight on height is

That means that, according to the regression line of weight on height, the weight will increase $1.02$ Kg for every cm that the height increases.

Regression predictions

Usually the regression models are used to predict the dependent variable for some values of the independent variable.

Example. In the sample of heights and weights, to predict the weight of a person with a height of 180 cm, we have to use the regression line of weight on height,

But to predict the height of a person with a weight of 79 Kg, we have to use the regression line of height on weight,

However, how reliable are these predictions?

Correlation

Once we have a regression model, in order to see if it is a good predictive model we have to assess the goodness of fit of the model and the strength of the of relation set by it. The part of Statistics in charge of this is correlation.

The correlation study the residuals of a regression model: the smaller the residuals, the greater the goodness of fit, and the stronger the relation set by the model.

Residual variance

To measure the goodness of fit of a regression model is common to use the residual variance.

Definition - Sample residual variance $s_{ry}^2$. Given a regression model $y=f(x)$ of a two-dimensional variable $(X,Y)$, its sample residual variance is the average of the squared residuals,

The greater the residuals, the greater the residual variance and the smaller the goodness of fit.

When the linear relation is perfect, the residuals are zero and the residual variance is zero. Conversely, when there are no relation, the residuals coincide with deviations from the mean, and the residual variance is equal to the variance of the dependent variable.

Explained and non-explained variation

Variation decomposition by regression model

Coefficient of determination

From the residual variance is possible to define another correlation statistic easier to interpret.

Definition - Sample coefficient of determination $r^2$. Given a regression model $y=f(x)$ of a two-dimensional variable $(X,Y)$, its coefficient of determination is

As the residual variance ranges from 0 to $s_y^2$, we have

The greater $r^2$ is, the greater the goodness of fit of the regression model, and the more reliable will its predictions be. In particular,

  • If $r^2 =0$ then there is no relation as set by the regression model.

  • If $r^2=1$ then the relation set by the model is perfect.

When the regression model is linear, the residual variance is

and the coefficient of determination is

Example. In the sample of heights and weights, we had

Thus, the linear coefficient of determination is

This means that the linear model of weight on height explains the 65% of the variation of weight, and the linear model of height on weight also explains 65% of the variation of height.

Correlation coefficient

Definition - Sample correlation coefficient $r$. Given a sample of a two-dimensional variable $(X,Y)$, the sample correlation coefficient is the square root of the linear coefficient of determination, with the sign of the covariance,

As $r^2$ ranges from 0 to 1, $r$ ranges from -1 to 1,

The correlation coefficient measures not only the strength of the linear association but also its direction (increasing or decreasing):

  • If $r=0$ then there is no linear relation.

  • Si $r=1$ then there is a perfect increasing linear relation.

  • Si $r=-1$ then there is a perfect decreasing linear relation.

Example. In the sample of heights and weights, we had

Thus, the correlation coefficient is

This means that there is a rather strong linear, increasing, relation between height and weight.

Different linear correlations

The scatter plots below show linear regression models with differents correlations.

Linear regression models with different correlations

Reliability of regression predictions

The coefficient of determination explains the goodness of fit of a regression model, but there are other factors that influence the reliability of regression predictions:

  • The coefficient of determination: The greater $r^2$, the greater the goodness of fit and the more reliable the predictions are.

  • The variability of the population distribution: The greater the variation, the more difficult to predict and the less reliable the predictions are.

  • The sample size: The greater the sample size, the more information we have and the more reliable the predictions are.

In addition, we have to take into account that a regression model is only valid for the range of values observed in the sample. That means that, as we don’t have any information outside that range, we must not do predictions for values far from that range.

Non-linear regression

The fit of a non-linear regression can be also done by the least square fitting method.

However, in some cases the fitting of a non-linear model can be reduced to the fitting of a linear model applying a simple transformation to the variables of the model.

Transformations of non-linear regression models

  • Logarithmic: A logarithmic model $y = a+b \log x$ can be transformed in a linear model with the change $t=\log x$:

  • Exponential: An exponential model $y = e^{a+bx}$ can be transformed in a linear model with the change $z = \log y$:

  • Potential: A potential model $y = ax^b$ can be transformed in a linear model with the changes $t=\log x$ and $z=\log y$:

  • Inverse: An inverse model $y = a+b/x$ can be transformed in a linear model with the change $t=1/x$:

  • Sigmoidal: A sigmoidal model $y = e^{a+b/x}$ can be transformed in a linear model with the changes $t=1/x$ and $z=\log y$:

Exponential relation

Example. The number of bacteria in a culture evolves with time according to the table below.

The scatter plot of the sample is showed below.

Scatter plot of the evolution of a bacteria culture

Fitting a linear model we get

Linear regression of the bacteria evolution

Is a good model?

Although the linear model is not bad, according to the shape of the point cloud of the scatter plot, an exponential model looks more suitable.

To construct an exponential model $y = e^{a+bx}$ we can apply the transformation $z=\log y$, that is, applying a logarithmic transformation to the dependent variable.

Scatter plot of the evolution of the logarithm of bacteria culture

Now it only remains to compute the regression line of the logarithm of bacteria on hours,

and, undoing the change of variable,

Exponential regression of the bacteria evolution

Thus, the exponential model fits much better than the linear model.

Lack of fit does not mean independence

It is important to note that every regression model has its own coefficient of determination. Thus, a coefficient of determination near zero means that there is no relation as set by the model, but that does not mean that the variables are independent, because there could be a different type of relation

Linear regression on a cuadratic relation Cuadratic regression on a cuadratic relation

Outliers in regression

Outliers in regression studies are points that clearly do not follow the tendency of the rest of points, even if the values of the pair are not outliers for every variable separately.

Scatter plot with an outlier

Outliers influence in regression

Outliers in regression studies can provoke drastic changes in the regression models.

Linear regression with an outlier Linear regression without outliers