How to run regressions by industry and year in Stata

[byuus2011]

Hello,

I am new to Stata and want to run regressions and predict residuals by industry and year, e.g. 20 industries in 10 years so 20x10 regressions. I wrote the following code using loop:

levelsof sic, local(x)
foreach x of numlist 10/87 {
foreach z of numlist 2005/2010 {
reg cfo at s sch if sic==`x' & fyear==`z', noconstant
predict pcfo if e(sample) & sic==`x' & fyear==`z'
replace pcfo="`pcfo'"
predict rcfo if e(sample) & sic==`x' & fyear==`z', residuals
replace rcfo="`rcfo'"

}
}
end

But this code can generate predicted value for the first year only. Could anyone know what bug exists in my code.
Any help is appreciated. Thanks.

Mike

[bukharin]

I think there are a few problems with the code. The fundamental problem is that -predict- needs to create a new variable, so after you've run -predict- once it won't allow you to generate new predictions with the same variable name. One way around this is to create a variable for your predictions with missing values, use -predict- to place predictions in a temporary variable, then -replace- your variable with the values from the temporary variable (see code below).

A couple of other comments about the code:


levelsof sic, local(x)
foreach x of numlist 10/87 {

This -levelsof- command creates a macro called `x' containing all values of sic; but then the -foreach- command immediately overwrites `x' with the number 10. It doesn't look like you need the -levelsof- command at all.


predict pcfo if e(sample) & sic==`x' & fyear==`z'

You should just be able to say -if e(sample)- because the other conditions were already specified when you ran the regression. e(sample) is true if the observations were included in the regression.


replace pcfo="`pcfo'"

I'm not sure what you're trying to achieve here. In your code pcfo is a variable name, not a macro name. Using the single quotes `pcfo' is the way to reference a local macro, and putting it in double quotes as "`pcfo'" means you're getting the value of the local macro `pcfo' and converting it to a string - surely this is not what you're trying to do.


end

You don't need to use -end- here; you use it to end a program (see -help program-) or after manually entering data using the -input- command.

Suggested fix:
gen pcfo=. // empty variable for predictions
gen rcfo=. // empty variable for residuals
tempvar pcfo rcfo // temporary variables for each set of predictions
foreach x of numlist 10/87 {
foreach z of numlist 2005/2010 {
reg cfo at s sch if sic==`x' & fyear==`z', noconstant
predict `pcfo' // predictions are now in temporary variable
replace pcfo=`pcfo' if e(sample) // transfer predictions from temp variable
predict `rcfo' // residuals are now in temporary variable
replace rcfo=`rcfo' if e(sample), residuals // transfer residuals from temp variable
drop `pcfo' `rcfo' // drop temporary variables in preparation for next regression
}
}

[byuus2011]

bukharin,

Thank you very much for your quick reply and help!
I run the code you wrote. But it says no observation maybe because my sic code is continual. I have sic 10 12 13 .... 78 82 85 87. Then I adjust the code to:

gen pcfo=.
gen rcfo=.
tempvar pcfo rcfo
levelsof sic, local (x)
foreach x of local x {
foreach z of numlist 2005/2010 {
.....

Now the program can be run but generates all missing values for pcfo and rcfo.
Would you help please?

Thank you!

[bukharin]

Not sure why that's happened; sometimes a problem like that is due to a subtle syntax error. Can you please post a log of the code and output? Suggest changing the "reg cfo ..." to "quietly reg cfo..." so that the log isn't ridiculously long.

[byuus2011]

I doubt if the loop really works because no output shows up in log. So I remove -levelsof- and switch back to -foreach x of numlist 10/87-. Since my sic code looks like: 12-17, 20-40, 44-59, and 67-87, I run the program a couple of times by spliting -foreach- values to something like: -foreach x of numlist 12/17-, run once, then -foreach x of numlist 20/40-, run another time. It works but seems stupid. I know there might be an efficient way in loop to handle such discrete numbers in -foreach numlist values-

Thank you!

[bukharin]

levelsof sic, local (x)
foreach x of local x

Just looked at this again - you shouldn't have a space between local and (x), and you should use a new name (not x) - what you're effectively saying is "let's go through values of x and call the current value x" - there may be a problem with overwriting x.

In any case you can specify more complicated numlists than a/b, for example:
. foreach x of numlist 1/5 6/10 12 15/18 20 {
2. display `x'
3. }
1
2
3
4
5
6
7
8
9
10
12
15
16
17
18
20


See -help numlist-

[byuus2011]

Now the program works perfectly. Thank you so much!

[byuus2011]

Now I have another problem. The loop runs well till it encounters a year in which there are no obs or insufficient obs, then it stops. I don't know how to solve this problem. Any help is appreciated! Thanks.

[bukharin]

In that case you can use -capture- to prevent the loop from exiting with an error. -capture- "captures" the output, so for example if you have this error message:
no observations
r(2000);

... then -capture- prevents your loop from exiting but retains the error code (2000) in a system variable called _rc (rc=return code).

In Stata any command that completes successfully returns 0 in _rc, otherwise it returns a non-zero error code. This is useful to know because you can -capture- the code and then decide whether or not to run further code depending on whether _rc is 0 (command successfully ran) or non-zero (command didn't successfully run).

Silly example:

. sysuse auto, clear
(1978 Automobile Data)

. tab foreign, nolab

Car type | Freq. Percent Cum.
------------+-----------------------------------
0 | 52 70.27 70.27
1 | 22 29.73 100.00
------------+-----------------------------------
Total | 74 100.00

. regress price mpg

Source | SS df MS Number of obs = 74
-------------+------------------------------ F( 1, 72) = 20.26
Model | 139449474 1 139449474 Prob > F = 0.0000
Residual | 495615923 72 6883554.48 R-squared = 0.2196
-------------+------------------------------ Adj R-squared = 0.2087
Total | 635065396 73 8699525.97 Root MSE = 2623.7

------------------------------------------------------------------------------
price | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
mpg | -238.8943 53.07669 -4.50 0.000 -344.7008 -133.0879
_cons | 11253.06 1170.813 9.61 0.000 8919.088 13587.03
------------------------------------------------------------------------------

. di _rc
0

. regress price mpg if foreign==2
no observations
r(2000);

. capture regress price mpg if foreign==2

. di _rc
2000

So running with your example, I would implement something like this:
gen pcfo=. // empty variable for predictions
gen rcfo=. // empty variable for residuals
tempvar pcfo rcfo // temporary variables for each set of predictions
levelsof sic, local(levels)
foreach x of local levels {
foreach z of numlist 2005/2010 {
capture reg cfo at s sch if sic==`x' & fyear==`z', noconstant
if !_rc {
predict `pcfo' // predictions are now in temporary variable
replace pcfo=`pcfo' if e(sample) // transfer predictions from temp variable
predict `rcfo' // residuals are now in temporary variable
replace rcfo=`rcfo' if e(sample), residuals // transfer residuals from temp variable
drop `pcfo' `rcfo' // drop temporary variables in preparation for next regression
}
}
}

Note that you don't need to use -quietly- with -capture-, because -capture- suppresses all of the output from the command. If you still want to see the output you can use:
capture noisily command ...

[byuus2011]

Great! Thank you so much!