Programmatically download monthly economic indicators

This SAS code programmatically downloads monthly economic indicators from FRED, and merges multiple time series into a single data set for convenient analysis. FRED (Federal Reserve Economic Data) is an excellent resource from the Federal Reserve Bank of St. Louis: it’s easy to find, graph, download, and process data sets online. In our case, though, we focus on downloading for offline analysis.

St. Louis Fed has a Web Service API which requires a key, but this method downloads text files which bypasses the need for an API key.

Most of the data is selection of the Current Population Survey (Household Survey) which includes variations of the unemployment rate. Even though the unemployment data can be found on the web site for the U.S. Bureau of Labor Statistics, I didn’t see a good way to avoid manual downloading from the BLS web site, so I use FRED.

The SAS code should be easy to adapt for other economic time series.

/* initialize empty data set */
data fred_combined;
	format month monyy7.;
run;

/* this macro downloads a time series and merges it with the master set */
%macro add_fred(
	sm_url, /* URL of text data on FRED */
	sm_var, /* name of variable */
	sm_firstobs /* line of first data (if you are not sure and don't need the oldest data, ~25 is often safe) */
	);
filename fred url "&sm_url";

data fred_new;
  infile fred  firstobs=&sm_firstobs;
  format month monyy7.;
  input 
		@1 month yymmdd10. 
		@13 &sm_var;
run; 

filename fred; /* close file reference */


data fred_combined;
	merge 
		fred_combined
		fred_new;
	by month;
	if month; /* remove missing dates as found in the initialized (blank) data set */
run;
%mend;

/* Selection from Household Survey Data <http://research.stlouisfed.org/fred2/categories/12> ordered by popularity */
/* "The Current Population Survey (Household Survey) is a monthly survey of households conducted by the Bureau of Census 
for the Bureau of Labor Statistics. It provides a comprehensive body of data on the labor force, employment, unemployment, 
persons not in the labor force, hours of work, earnings, and other demographic and labor force characteristics. " */

/* Civilian Unemployment Rate, Seasonally Adjusted */
%add_fred(http://research.stlouisfed.org/fred2/data/UNRATE.txt, unrate, 22);

/* Civilian Unemployment Rate, Not Seasonally Adjusted */
%add_fred(http://research.stlouisfed.org/fred2/data/UNRATENSA.txt, unratensa, 13);

/* Civilian Employment-Population Ratio */
%add_fred(http://research.stlouisfed.org/fred2/data/EMRATIO.txt, emratio, 20);

/* Average (Mean) Duration of Unemployment */
%add_fred(http://research.stlouisfed.org/fred2/data/UEMPMEAN.txt, unempmean, 13);

/* Median Duration of Unemployment */
%add_fred(http://research.stlouisfed.org/fred2/data/UEMPMED.txt, uempmed, 20);

/* Total unemployed, plus all marginally attached workers plus total employed part time for economic reasons */
%add_fred(http://research.stlouisfed.org/fred2/data/U6RATE.txt, u6rate, 13);

/* Unemployment Rate - Bachelor's degree and higher, 25 years and over */
%add_fred(http://research.stlouisfed.org/fred2/data/LNS14027662.txt, un_bach_25, 13);

/* Unemployment Rate - White (LNS14000003) */
%add_fred(http://research.stlouisfed.org/fred2/data/LNS14000003.txt, un_white, 13);

/* Total unemployed, plus all marginally attached workers plus total employed part time for economic reasons */
%add_fred(http://research.stlouisfed.org/fred2/data/U6RATENSA.txt, u6ratensa, 13);

/* Special Unemployment Rate: Unemployed and Discouraged Workers [U4, seasonally adjusted] */
%add_fred(http://research.stlouisfed.org/fred2/data/U4RATE.txt, u4rate, 13);

/* Unemployment Rate - Job Losers [U2, seasonally adjusted] */
%add_fred(http://research.stlouisfed.org/fred2/data/U2RATE.txt, u2rate, 13);

/* Special Unemployment Rate: Unemployed and Discouraged Workers [U4, not seasonally adjusted] */
%add_fred(http://research.stlouisfed.org/fred2/data/U4RATENSA.txt, u4ratensa, 13);

/* Unemployment Rate - Some College or Associate Degree, 25 years and over */
%add_fred(http://research.stlouisfed.org/fred2/data/LNS14027689.txt, un_somecol_25, 13);

/* Persons unemployed 15 weeks or longer, as a percent of the civilian labor force [U1, not seasonally adjusted] */
%add_fred(http://research.stlouisfed.org/fred2/data/U1RATENSA.txt, u1ratensa, 13);

/* Unemployment Rate - Job Losers [U2, not seasonally adjusted] */
%add_fred(http://research.stlouisfed.org/fred2/data/U2RATENSA.txt, u2ratensa, 13);

/* Special Unemployment Rate: Unemployed and Marginally Attached Workers [U5, Not seasonally adjusted]*/
%add_fred(http://research.stlouisfed.org/fred2/data/U5RATENSA.txt, u5ratensa, 13);

/* Special Unemployment Rate: Unemployed and Marginally Attached Workers */
%add_fred(http://research.stlouisfed.org/fred2/data/U5RATE.txt, u5rate, 13);

/* Persons Unemployed 15 weeks or longer, as a percent of the civilian labor force */
%add_fred(http://research.stlouisfed.org/fred2/data/U1RATE.txt, u1rate, 13);

/***
 *** End of Household Survey Data 
 ***/

/* Consumer Price Index for All Urban Consumers: All Items */
%add_fred(http://research.stlouisfed.org/fred2/data/CPIAUCSL.txt, cpiaucsl, 15);

/* Payroll employment: All Employees: Total nonfarm */
%add_fred(http://research.stlouisfed.org/fred2/data/PAYEMS.txt, payems, 14);

/* University of Michigan: Consumer Sentiment */
/* source: Thomson Reuters/University of Michigan */
%add_fred(http://research.stlouisfed.org/fred2/data/UMCSENT.txt, umcsent, 16);

/* Kansas City Financial Stress Index */
/* source: Federal Reserve Bank of Kansas City */
%add_fred(http://research.stlouisfed.org/fred2/data/KCFSI.txt, kcfsi, 15);

/* S&P Case-Shiller 20-City Home Price Index */
/* Source(s): 	Standard and Poor's */
%add_fred(http://research.stlouisfed.org/fred2/data/SPCS20RSA.txt, spcs20rsa, 17);

/* S&P Case-Shiller 10-City Home Price Index */
%add_fred(http://research.stlouisfed.org/fred2/data/SPCS10RSA.txt, spcs10rsa, 17);

/* S&P Case-Shiller 20-City Home Price Index, NSA  */
%add_fred(http://research.stlouisfed.org/fred2/data/SPCS20RNSA.txt, spcs20rnsa, 17);

/* S&P Case-Shiller 10-City Home Price Index, NSA */
%add_fred(http://research.stlouisfed.org/fred2/data/SPCS10RNSA.txt, spcs10rnsa, 17);

/* Leading Index for the United States  */
/* Source(s): 	Federal Reserve Bank of Philadelphia */
%add_fred(http://research.stlouisfed.org/fred2/data/USSLIND.txt, usslind, 19);

/* Real Retail and Food Services Sales */
/* Federal Reserve Bank of St. Louis */
%add_fred(http://research.stlouisfed.org/fred2/data/RRSFS.txt, rrsfs, 16);

/* Retail Sales: Total (Excluding Food Services) */
/* Source(s): 	U.S. Department of Commerce: Census Bureau */
%add_fred(http://research.stlouisfed.org/fred2/data/RSXFS.txt, rsxfs, 13);


For the sake of time, at the end you probably want to cache the results locally to avoid re-downloading from the Internet. Also if you like, you may put descriptive labels on the variables.

Now if I could only find a way to programmatically download the Composite Leading Indicators (CLI) from the Organisation for Economic Co-operation and Development (OECD)—but for now I get it by email.

Advertisements

3 thoughts on “Programmatically download monthly economic indicators

  1. I just wrote a quick Python script to get the CLI indicators you mentioned using regular expressions. Feel free to tweak it to your likings:

    #!/usr/bin/python
    from urllib import urlopen
    import re

    regexs = [
    ‘(.*?)&lt;',
    '(.*?)’,
    ‘(.*?)’,
    ‘(.*?)&lt;']

    country = 0
    oddrow = 1
    evenrow = 2
    month = 3

    page = urlopen('http://stats.oecd.org/Index.aspx?DatasetCode=MEI_CLI&#039;).read()

    Data = []

    for regex in regexs:
    Data.append(re.findall(re.compile(regex), page))
    print Data[country]
    print Data[oddrow][0:47]
    print Data[evenrow][0:47]
    print Data[oddrow][47:94]
    print Data[evenrow][47:94]
    print Data[oddrow][94:141]
    print Data[evenrow][94:141]
    print Data[oddrow][141:188]
    print Data[evenrow][141:188]
    print Data[oddrow][188:235]
    print Data[evenrow][188:235]
    print Data[oddrow][235:282]
    print Data[evenrow][235:282]
    print Data[oddrow][282:329]
    print Data[evenrow][282:329]
    print Data[oddrow][329:376]
    print Data[evenrow][329:376]
    print Data[oddrow][376:423]
    print Data[evenrow][376:423]
    print Data[oddrow][423:470]
    print Data[evenrow][423:470]
    print Data[oddrow][470:517]
    print Data[evenrow][470:517]
    print Data[oddrow][517:564]
    print Data[evenrow][517:564]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s