The Bigger Game – Big Data to improve the Census


James Petter, Vice President and Country Manager EMC UK and Ireland, discusses the potential for Big Data to make the Census more timely and cost effective.

Over the past 18 months, interest in big data analysis has grown exponentially due to its potential to unlock new business models and power economic growth.

This type of analysis has been whole heartedly embraced by the financial services, where, amongst other things, it is used to detect and prevent fraud. However, big data can also provide public sector organisations with opportunities to extract greater value from their existing data and improve operational efficiency.

One way this could be applied by government is in relation to the Census. Once every ten years since 1801, every household in England and Wales has been required to respond to a national census. The objective is to provide planners, policy makers and researchers with valuable population statistics in order to inform decision making about the funding of public services and the provision of infrastructure, like housing or transport links. However, today in the UK, the Census is carried out at great expense using antiquated paper-based processes similar to those used over two centuries ago-forms are sent to every household, filled in on a specific day of the year and returned to the authorities by post. The last Census in 2011 cost the government £480 million, and 35,000 temporary workers were needed to collect and process the data.

Beyond the high figures, more fundamental questions are being asked about the validity of the ‘pen and paper’ approach to the census. In an age of globalisation, mass travel, and huge technological change, figures collected by hand once a decade can quickly become outdated. As rapid changes shape our society, archaic data-collection techniques could mean crucial decisions are based on inaccurate information.

In light of these challenges, the Office for National Statistics (ONS), the organisation responsible for the Census, has been conducting a consultation on its future which is due to report later this year. One of the potential options the ONS has under consideration involves supplementing existing government data with compulsory annual surveys.

This is very similar to a recommendation made by leading think tank Policy Exchange, in its 2012 report “The Big Data Opportunity”, supported by EMC. The report explained that other existing government data sets, such as the council tax register, electoral roll, child benefit claims data, and state pension entitlement data, already provided partial coverage of the UK population. Collectively, these could be combined and de-duplicated to generate demographic information of comparable quality and superior timeliness to the Census. Policy Exchange also highlighted how a similar approach was being taken forward in the Netherlands. The Dutch conduct a “virtual census” in which administrative registers are linked at a pre-determined time to establish a record of the population.

Annual online surveys have the potential to offer the government a better understanding of the population than a ten-yearly census. Questions can be adjusted to respond to changing needs, while trends in migration or population growth can be identified much more quickly. Although these suggestions could create new hurdles in combining data and aligning insights, information exploration and analytics may offer the right solution, delivering more timely population estimates at lower cost.


This blog was originally posted on The Bigger Game Blog, which you can view here.   


Leave a Reply