shahid lecture-8- mkag1273

MAL1303: STATISTICAL HYDROLOGY

Multiple Regression

Dr. Shamsuddin ShahidAssociate Professor

Department of Hydraulics and HydrologyFaculty of Civil Engineering

Room No.: M46-332; Phone: 07-5531624; Mobile: 0182051586

Email: [email protected]

11/23/2015 Shamsuddin Shahid, FKA, UTM

You created this PDF from an application that is not licensed to print to novaPDF printer (http://www.novapdf.com)

mailto:[email protected]

http://www.novapdf.com

Simple Linear Regression

Simple Linear Regression (SLR) is a statisticaltechnique that is used to determine thefunctional relationship between two variables.Regression gives an equation that best describesthe relationship between two variables.




Multiple Linear Regression (MLR)

Multiple linear regression is a statistical technique where adependent variable is predicted from a set of predictors

Multiple regression is a statistical technique that is used toidentify relationship between a dependent variable and acombination of independent variables.

The relationship is valid when few assumptions are fulfilled.Failing to satisfy the assumptions does not mean thatrelationship is not correct. It means that the relationship maynot be strong enough.




• The variables should be measure in interval/ratio scale.

• Dependent variable, Y must be normally distributed (noskewness or outliers)

• Predictors, X’s do not need to be normally distributed, butif they are it makes for a stronger interpretation.

• There should be linear relationship between Y and all X

• no outliers among Xs predicting Y

• Variance on Y is the same at all values of X(homoscedastic)

Linear Multiple Regression: Assumptions




Linear Multiple Regression: Outliers

• Outliers can distort the regression results in multiple regression aslike simple linear regression. When an outlier is included in theanalysis, it pulls the regression line towards itself. This can result in asolution that is more accurate for the outlier, but less accurate for allof the other cases in the data set.

• It is necessary to check for outliers in the dependent variable and inthe independent variables.

• Removing an outlier may improve the distribution of a variable.

• Transforming a variable may reduce the likelihood that the value for acase will be characterized as an outlier.




1. Decide dependent and independent variables.

2. Test for normality, linearity, homoscedasticity.

3. In necessary, remove the outliers.

4. If it does not satisfy the criteria for normality, transformationis required. Decide which transformations should be used.

5. Substitute transformations and run regression entering allindependent variables.

6. Do multiple regression analysis with variables specified in theproblem.

7. Test the significance of the regression equation.

Linear Multiple Regression: Steps




Simple Linear Regression

In Simple Linear Regression (SLR), the functional relationshipbetween two variables X and Y are determined.

Regression equation is the equation of a straight line that bestdescribes the relationship between two variables.

When the equation is used to calculate Y from observed X, itgives an error ε in the prediction. Therefore, the Y equals topredicted value plus error.




Multiple Linear Regression (MLR): Basics

A multiple linear regression model is called “linear” because onlylinear coefficients {β} are used. However, transforms of theregressor variables are permitted in an MLR model like SLR.

In Multiple Linear Regression (SLR), the functional relationship ofdependent variable Y with more than one independent variables aredetermined.





1 11 21

2 12 22 1

3 13 23 2

4 14 24

*

4 1 4 2 * 2 1

*

y x xy x x by x x by x x

x x x

data design matrix parameters




Multiple Linear Regression: Basics

Create the design Matrix

Calculate the parameters:

Where, XT is the transpose of Matrix XX-1 is the inverse of Matrix X




The Goodness of Fit of the Regression Model

One measure of how well a statistical model explains the observeddata is the coefficient of determination, that is, the square of thePearson correlation coefficient, r2, between y and x.

When x is replaced by ,it gives the correlation between actual and predicted value, R2

It can also be measure by,

y




Distinction between r and R are:

• r is a measure of association between two random variableswhereas R is a measure between a random variable y and itsprediction from a regression model.

• r lies in the interval - 1 r -1 while the multiple correlation Rcannot be negative; that is, it lies in the interval 0 R 1.

• R is always well defined, regardless of whether the independentvariable is assumed to be random or fixed. In contrast, calculatingthe correlation between a random variable, Y, and a fixed predictorvariable, X, that is, a variable that is not considered random, makesno sense.

The Goodness of Fit of the Regression Model




Multiple Linear Regression: Example

It is well known that groundwater recharge is directly related toRainfall and Soil Moisture Holding Capacity (SMHC). Instrumentaldata of groundwater recharge, Rainfall and SMHC at six sites hasbeen collected. Find a empirical equation that related groundwaterrecharge with Rainfall and SMHC




Multiple Linear Regression: Solution

Create the design matrix

Get solution by:11/23/2015 Shamsuddin Shahid, FKA, UTM



Multiple Linear Regression: Solution

Excel commands:

Matrix Inversion: MINV(array)

Matrix Multiplication: MMULT(array1, array2)

Matrix Transpose: Copy Matrix -> Past Special with tick ontranspose radio button.




Recharge = 1.38 + 0.12Rainfall – 0.01SMHC





Basic assumptions about the errors:

1. The mean of the errors is zero2. The errors are normally distributed.3. The variances of the errors for all observations are

constant4. The errors are independent of each other (uncorrelated)

Gross violations of these basic assumptions will yield apoor or biased model. However, if the variances of theerrors are unequal and can be estimated, weightedregression schemes can sometimes be used to obtain abetter model.

Multiple Linear Regression (MLR): Assumptions




is the Variance of residuals

Is the corresponding diagonal value of matrix (XTX)-1

Multiple Linear Regression: Confidence Interval


The parameter values have range. We can find the range of aparameter at a certain level of confidence by using followingformula:





Multiple Linear Regression: Confidence Interval

n = 6, p = 3

At α = 0.05,t(0.025, 3) = 4.18

s2 = 0.084

-0.35 ≤ β0 ≤ 3.11-0.10 ≤ β1 ≤ 0.35-0.16 ≤ β2 ≤ 0.14




• An estimator with lower variance is more efficient, in thesense that it is likely to be closer to the true value oversamples.

• The “best” estimator is the one with minimum variance of allestimators

Multiple Linear Regression: Efficient Estimator


-0.35 ≤ β0 ≤ 3.11-0.10 ≤ β1 ≤ 0.35-0.16 ≤ β2 ≤ 0.14




SST = SSE + SSR

Sum of Square Total (SST) = Total variability in the observed responsesSum of Square Error (SSE) = Total error by the model, or variability that is notexplained by the modelSum of Square Residual (SSR) = Systematic variability that is explained by theregression model.

Multiple Linear Regression: Strength




Mean variation in observations, MST = SST / n-1Mean Error, MSE = SSE / n-pMean regression, MSR = SSR / 1

Higher values of R2 indicate a better fit of the model to the sampleobservations.

Disadvantage of R2: Adding any regressor variable to an MLRmodel, even an irrelevant regressor, yields a smaller SSE andgreater R2. For this reason, R2 by itself is not a good measure ofthe quality of fit.






To overcome this deficiency in R2, an adjusted value can be used. The adjusted coefficient of multiple determination ( ) is defined as,

Because the number of model coefficients (p) is used incomputing, the value will not necessarily increase with theaddition of any regressor. Hence, is a more reliable indicatorof model quality.




SST = 1.27; SSR = 0.85; SSE = 0.42 MST = 0.26; MSR = 0.85; MSE = 0.14

= 0.67= 0.45

SST = SSE + SSR

Multiple Linear Regression: Strength (Example)

Mean variation in observations, MST = SST / n-1Mean Error, MSE = SSE / n-pMean regression, MSR = SSR / 1




F-test is used to assess the overall ability of a model.

When testing for the significance of the goodness of fit, our null hypothesis is that the explanatory variables jointly equal 0.

If our F-statistic is below the critical value we fail to reject the null and therefore we say the goodness of fit is not significant.

Multiple Linear Regression: F-statistics




The F-test is useful for testing a number of hypotheses and is oftenused to test for single, global and the joint significance of a group ofvariables.

Joint test often refer to ‘testing a restriction’.

This restriction is that a group of explanatory variables are jointlyequal to 0





The global F-test is used to assess the overall ability of a model toexplain at least some of the observed variability in the sampleresponses. The global F-test is performed in the following steps:

Null hypothesis: β1 = β2 = …. = βk = 0

The global F-statistics is calculated as,F0 = MSR/MSE

If F(calculated) > F (critical) (α, k, n-p),(where k = number of regressors; n = data points; p = parameters tobe estimated).

Reject the null hypothesis and conclude that at least one βj≠0 and atleast one model regressor explains some of the response variation.





Recharge = 1.38 + 0.12Rainfall –0.01SMHC


SST = 1.27 MST = 0.26SSR = 0.85 MSR = 0.85SSE = 0.42 MSE = 0.14

SST = SSE + SSR

F0 = MSR/MSE= 6.07

F (critical) (α, k, n-p)F (critical) (0.05, 2, 3)= 9.55

F(calculated) < F (critical) (α, k, n-

p)

Null hypothesis can notbe rejected.

No model regressorexplains some of theresponse variation.





Discharge = 21.97 – 0.19ET + 1.55BF + 0.94R -1.05GWR






Null hypothesis:β1 = β2 = β3 = β4 = 0

= 0.9865F0 = MSR/MSE

= 7.68F (critical) (α, k, n-p) =F (critical) (0.05, 4, 7) = 4.12

F(calculated) > F (critical) (α, k,

n-p)

Null hypothesisrejected.

Decision: At least one βj≠0 and at least one model regressor explains some of the response variation.





Discharge = 33.50 – 0.28ET + 1.53BF + 0.28R




Discharge = 33.50 – 0.28ET + 1.53BF + 0.28R


Null hypothesis:β1 = β2 = β3 = 0

F0 = MSR/MSE= 6.3

F (critical) (α, k, n-p) =F (critical) (0.05, 3, 8) = 4.07

F(calculated) > F (critical) (α, k,

n-p)

Null hypothesisrejected.

Decision: Groundwater recharge has no significant impact on Discharge.





Discharge = ? + ? ET + ? BF + ? GWR




To carry out this test you need to conduct two separate regression,one with all the explanatory variables in (unrestricted equation),the other with the variables whose joint significance is beingtested, removed.

Then collect the RSS from both equations.

Put the values in the formula

Find the critical value and compare with the test statistic. The nullhypothesis is that the variables jointly equal 0.

Multiple Linear Regression: Joint Significance




The test for joint significance has its own formula, which takes the following form:

RSSrestrictedRSSRSSedunrestrictRSS

equationedunrestrictinparametersknsrestrictioofnumberm

knRSSmRSSRSSF

R

u

u

uR

//






Obs. No. Y X1 X2 x3

1 5.1 2.3 2.5 4.2

2 6.2 1.9 2.8 3.3

3 4.8 2.0 3.1 4.0

. . . . .

. . . . .

. . . . .

60 5.9 2.4 3.8 4.6

3322110 xαxαxααy




If we have a model consists of three explanatory variables. We wish to test for the joint significance of 2 of the variables (x2 and x3), we need to run the following restricted and unrestricted models:

restrictedxααyedunrestrictxαxαxααy

t

t

110

3322110





Given the following model, we wish to test the joint significance of x2and x3. Having estimated them, we collect their respective RSSs (n=60).

51

750

110

3322110

.RSSrestrictedxββy

.RSSedunrestrictxαxαxααy

R

t

u

t





RSSrestrictedRSSRSSedunrestrictRSS

equationedunrestrictinparametersknsrestrictioofnumberm

knRSSmRSSRSSF

R

u

u

uR

//

28013403750

460750275051

..

/./..F


F (critical) (0.05, 2, 56) = 3.16




As the F statistic is greater than the critical value (28>3.15), wereject the null hypothesis and conclude that the variables x2 and x3are jointly significant and should remain in the model.

0:,0:,

32

320

AHHypothesiseAlternativHHypothesisNull





Choosing the Best MLR Model

• One of the major issues in multiple regression is the appropriateapproach to variable selection.

• To make a appropriate regression model, we need tosubsequently add or delete variables from model.

• The benefit of adding additional variables to a multipleregression model is to account for or explain more of thevariance of the response variable. The cost of adding additionalvariables is that the degrees of freedom decreases, making itmore difficult to find significance in hypothesis tests andincreasing the width of confidence intervals.

A good model will explain as much of the variance of y aspossible with a small number of explanatory variables.




The choice of whether to add a variable is based on a "cost-benefitanalysis", and variables enter the model only if they make asignificant improvement in the model.

There are at least two types of approaches for evaluating whethera new variable sufficiently improves the model. The first approachuses partial F-tests, and when automated is often called a"stepwise" procedure.

The second approach uses some overall measure of modelquality. The latter has many advantages.




