Tag Archives: Information Technology

How Have We Improved the Consumer Price Index? Let Me Count the Ways

Soon after I became Commissioner, the top-notch BLS staff was briefing me on the many programs and details that make up BLS. I asked the staff working on the Consumer Price Index (CPI) if they could list for me some of the improvements that have occurred over the past few years. It has been nearly a quarter century since the Boskin Commission studied the CPI and recommended enhancements. I knew many of these enhancements had been implemented, along with other improvements. But I was shocked to see my staff come back with an 8-page, detailed listing of 77 substantive improvements that have been implemented since 1996.

You may have thought a price index that has been around since 1913 is happy to rest on its laurels. Well, you’d be wrong. There are improvements to the CPI going on all the time. As I reviewed the list, I noticed a number of improvements involve the introduction of new or changed goods and services, such as cell phones or streaming services. I also noticed improvements in how we handle product changes, as I wrote recently in a blog about quality adjustment. But these topics only scratch the surface.

I’m not going to describe every CPI enhancement that has taken place over the past 24 years; you can find much more detail on the CPI webpage. But I will whet your appetite by highlighting a few categories of improvements.

Keeping the CPI market basket up to date

The goal of the CPI is to track the change in the prices consumers pay for a representative market basket of goods and services. Let’s look at the what and where of that market basket.

  • What we collect — goods and services. We collect prices for a market basket of goods and services, designed to represent what consumers are buying. In January 2002, we switched from updating that market basket every 10 years to every 2 years, providing a better representation of current spending patterns.
  • What we collect — housing. We track the cost of housing by a separate sample of housing units. In 2010, we increased that sample to improve accuracy. In 2016 we began rotating that sample every 6 years. Previously, the housing sample was only rotated when new geographic areas were introduced after the U.S. census.
  • Where we collect — outlets. We collect prices from stores and businesses that are chosen based on where consumers shop and buy goods and services. In January 1998, we switched from updating this sample of “outlets” every 5 years to every 4 years. And in January 2020, we switched the source used to determine those outlets to the Consumer Expenditure Survey, which is also the source of our spending information for the market basket of goods and services people buy. Previously we used a separate survey of households to identify outlets.
  • Where we collect — geography. We collect prices for goods and services in selected geographic areas, designed to represent all urban areas of the United States. In January 2018, we updated the geographic areas, designed to represent current population trends. We last updated these areas in 1998.

Collecting CPI information

According to folklore, CPI data collection was accomplished by staff who dressed up in high fashion, the ladies in fancy hats and white gloves and the gentlemen in the finest haberdashery, who then went shopping to determine the latest prices. That’s not how it happens. CPI staff are not paid to “shop” to collect prices. We use trained experts who are skilled at gaining cooperation from many different types of businesses; ensuring they are obtaining price information for goods and services that are consistent from one month to the next or making appropriate adjustments; and gathering information from thousands of outlets about hundreds of thousands of goods and services over a short data-collection period. Let’s look at how the data-collection process has improved:

  • When we collect — In June 2005, the CPI switched from a collection period that spanned the first 15–18 days of the month to collection across the entire month. This provides more representative data, especially for items that frequently vary in price within the same month.
  • How we collect — In January 1998, the CPI began using computerized data-collection tools, which automate certain math functions and screen for errors or inconsistencies. We continue to upgrade our processes; CPI data-collection staff recently began using a new generation of tablet computers.
  • Alternative collection — Not all price information comes from traditional collection with stores. Some information comes from websites, corporate data files, third parties that combine data from different sources, and more. In fact, the CPI and other BLS programs are focused on identifying even more alternative collection methods in the coming years.

Calculating the CPI

Once we collect the prices on all these goods and services, we need to calculate an index. In simple terms, we find the difference between the price in month 1 and the price in month 2, and express that difference as a rate of change from month 1. We publish rates of change and also express current prices as an index, which is equal to 100 in a base period.

Many factors and decisions go into combining data for an item and then combining data for all items into the published CPI. We’ve improved those calculations in several ways over the past few years.

  • Geometric mean — In January 1999, the CPI switched the formula for calculating price changes at the component item level from an arithmetic mean to a geometric mean. This allows the overall index to capture substitutions consumers make across specific products within a component item category when the prices of those products change relative to one another. With the geometric mean formula, BLS does not assume consumers substitute hamburgers for steak, which are in different component categories. The formula only captures substitution within a component category, such as among types of steak.
  • More decimal places — In January 2007, the CPI began publishing index numbers to 3 decimal places, which improved consistency between published index numbers and rates of change.

New information available to the public

While the CPI has been around for over a century, we have added a number of new indexes over time, to provide a variety of inflation figures. Here are some of the newest products in the CPI family:

  • Research (Retroactive) series — The CPI Research Series incorporates many of the improvements that came out of the Boskin Commission. The series provides a pretty consistent way to measure price changes from 1978 up to the most recent full year.
  • Chained CPI series – The Chained CPI uses an alternative formula that applies spending data in consecutive months to reflect any substitution that consumers make across component item categories in response to changes in relative prices. For example, this index would capture consumer substitution of hamburger for steak. This measure is designed to come closer to a “cost-of-living” index than other BLS measures. The series was first produced in 2002.
  • Elderly research series – The CPI for the Elderly reweights the component CPI data based on the spending patterns of elderly households. This series, mandated by Congress, began in 1988. In 2008, we extended the series retrospectively back to 1982.

As I have mentioned in the past, we are always working to improve the CPI. We recently contracted with the Committee on National Statistics, part of the National Academy of Sciences, to provide guidance on a variety of issues. I’ll use this space to report on the Committee’s work, as well as other improvements underway in the CPI.

How We Collect Data When People Don’t Answer the Phone

I was asked recently how the U.S. Bureau of Labor Statistics can collect data these days when no one answers the telephone. A legitimate question and one we grapple with all the time. I had two answers – one related to data collection methods and one related to sources of data. I will elaborate here about both.

Beige wall phone with rotary dial

But first, do you remember the days before caller ID, when everyone answered the phone? If you were at home, the rotary phone, permanently attached to the kitchen wall, always rang during dinner.

If you were in the office, the phone probably had a row of clear plastic buttons at the bottom that would light up and flash. In either case, who was on the other end of the phone was a mystery until you answered. In those days, your friendly BLS caller could easily get through to you and ask for information.

Vintage office phone with rows of buttons

Fast forward to today’s world of smart phones and other mobile devices. Nobody talks on the phone anymore. Many phone calls are nuisances. A call from BLS might show up as Unknown Number, U.S. Government, or U.S. Department of Labor on your caller ID, or identified as potential spam. With the spread of “spoofing,” many people do not answer calls from numbers they don’t recognize. How do we get around these issues?

Data Collection

At BLS, we consider data collection as much an art as a science. Sure, our staff needs to be well-versed in the information they are collecting. But they also need to be salespersons, able to convince busy people to spend a few minutes answering key questions. Part of that art is making a connection. There are old-fashioned ways that still work, such as sending a letter or showing up at the door. And there are more modern techniques, such as email and text. We are nothing if not persistent.

Our data-collection techniques have been called “High Touch, High Tech.” We start by building a relationship—the High Touch step. BLS has a wide range of information that people and businesses can use to help make informed decisions. We can help you access that information, and we love to see survey respondents use BLS data they helped us produce. In return, we ask for some information from you. There’s where High Tech comes in. We continue to add flexibility to our data-collection toolkit. You can provide information in person, on paper, or on the phone. You also can email information or an encrypted file. Or you can access our online portal anytime and anywhere to provide information or upload a data file. We need your information, and we want to make providing that information as easy as possible.

For example, this chart shows the number of employer self-reports that we’ve received through our online portal over the past several years. Internet data collection has really taken off.

Editor’s note: Data for this chart are available in the table below.

Another data-collection strategy we use is asking businesses to allow us to get the information we need from their website. This might involve web scraping data or using an Application Programming Interface (API). We have had success showing businesses that we can get what we need from their website, often eliminating the need for them to compile data.

Alternative Data

Beyond these data-collection strategies, we are expanding efforts to get information from alternative sources, lessening our need to contact businesses and households. Some BLS programs, such as Local Area Unemployment Statistics, the Quarterly Census of Employment and Wages, and Productivity Studies, rely heavily on administrative data and information from other surveys. In these cases, there is little need to contact businesses or people directly.

Other BLS programs, such as the Consumer Price Index (CPI) and the Employment Cost Index (ECI), need to capture timely information. But there are alternatives that can complement direct data collection. The CPI, for example, has produced an experimental price index for new vehicles based on a file of vehicle purchase transactions provided by J.D. Power. Using information from sources like that may eventually lessen the need to have BLS employees contact vehicle dealerships. The ECI found that it was easier to capture employer premiums for unemployment insurance from state tax records than to ask employers.

Alternative data come in many forms, from government records, data aggregators, scanners, crowdsourcing, corporate data files, and many more. BLS is investing heavily in alternative data-collection techniques and alternative data sources. The High Touch and High Tech approach we use every day in our data-collection operations helps us to maximize data quality and minimize respondent burden and cost.

The telephone may go the way of the dinosaur, but that’s not stopping us from using every tool at our disposal to continue to produce gold standard data to inform your decisions.

Number of transactions with BLS internet data collection
YearNumber of transactions

































Ensuring Security and Fairness in the Release of Economic Statistics

The U.S. Bureau of Labor Statistics is the gold standard of accurate, objective, relevant, timely, and accessible statistical data, and I am committed to keeping it that way. As Commissioner, it is my obligation to do everything possible to protect the integrity of our data and to make sure everyone has equitable access to these data.

One step toward equitable access and data security is coming soon; on March 1, 2020, the U.S. Department of Labor (DOL) will eliminate all electronics from the lock-up facility where we allow members of the media to review economic releases and prepare news stories before the official release of the data. We are changing the procedures to better protect our statistical information from premature disclosure and to ensure fairness in providing our information to the public.

For many years the news media have helped BLS and the Employment and Training Administration (ETA) inform the public about our data. Since the mid-1980s, BLS and ETA have provided prerelease data access to news organizations under strict embargoes, known as “lock-ups.” We have provided this early access consistent with federal Statistical Policy Directives of the Office of Management and Budget. BLS uses the lock-up for several major releases each month, including the Employment Situation and Consumer Price Index. ETA uses the lock-up for the Unemployment Insurance Weekly Claims data. These economic data have significant commercial value and may affect the movement of commodity and financial markets upon release.

Because of technological advancements, the current lock-up procedure creates an unfair competitive advantage for lock-up participants who provide BLS data to trading companies. Today, the internet permits anyone in the world to obtain economic releases for themselves directly from the BLS or DOL websites. However, unlike media organizations with computer access in the current lock-up, others who use the data do not have up to 30 minutes before the official release to process the data. Their postings about the data may lag behind those released directly from the lock-up at official publication time, 8:30 a.m. Eastern. High-speed algorithmic trading technology now gives a notable competitive advantage to market participants who have even a few microseconds head start. To eliminate this advantage and further protect our data from inadvertent or purposeful prerelease, no computers or any other electronic devices will be allowed in the lock-up.

In recent years, BLS and ETA have devoted significant resources to introducing improved technologies that strengthen our infrastructure and ensure data are posted to the BLS or DOL websites immediately following the official release time.

We at BLS and ETA are committed to the principle of a level playing field—our data must be made available to all users at the same time. We are equally committed to protecting our data. We are now positioned to continue helping the media produce accurate stories about the data, while also ensuring that all parties, including the media, businesses, and the general public, will have equitable and timely access to our most sensitive data.

You can find more details about these changes in our notice to lock-up participants. We also have a set of questions and answers about the changes to the lock-up procedures.

Modernizing BLS News Releases for the Next Generation

At BLS we are always trying to refine our products to serve our customers better. Over the years, we have updated several of our publications to be more web-friendly and include more interactive features. One major exception has been news releases. In the past few years we have conducted a great deal of outreach and investigation with our news release readers to understand what would make our releases easier to digest and provide greater context to the data. The outcome of this research is the two news release prototypes we’re presenting.

On our beta site, you can find prototypes for the Consumer Price Index and The Employment Situation news releases. We incorporated interactive charts, downloadable excel tables, and a redesigned technical note (now called “About this release”).

We’d love to hear what you think! Please either drop a comment here, or on our beta site, so we can better refine these prototypes for future news releases.

New App for Career Information Now Available

Icon for CareerInfo app

BLS has partnered with the U.S. Department of Labor’s Office of the Chief Information Officer to develop the CareerInfo app that is now available from the Apple App Store and Google Play. CareerInfo presents information from the Occupational Outlook Handbook, the most popular BLS resource for career information.

The CareerInfo app helps you find data and information about employment, pay, job outlook, how to become one, and more for hundreds of detailed occupations. You can browse by occupational groups and titles or search by occupation or keywords. Within occupational groups, the app allows you to sort by occupation title, projected growth, and typical education or median pay.

Future updates will add features that will let you personalize the app by filtering searches and by “liking,” saving, viewing, and comparing favorites.

Check out the new CareerInfo app and explore the occupational information and data produced by BLS. You’ll be glad you did!