Big Data in the Public Sector

The public sector is becoming increasingly aware of the potential value to be gained from big data, as governments generate and collect vast quantities of data through their everyday activities.

The benefits of big data in the public sector can be grouped into three major areas, based on a classification of the types of benefits: advanced analytics, through automated algorithms; improvements in effectiveness, providing greater internal transparency; improvements in efficiency, where better services can be provided based on the personalization of services; and learning from the performance of such services.

The chapter examined several drivers and constraints that have been identified, which can boost or stop the development of big data in the sector depending on how they are addressed. The findings, after analysing the requirements and the technologies currently available, show that there are open research questions to be addressed in order to develop such technologies so competitive and effective solutions can be built. The main developments are required in the fields of scalability of data analysis, pattern discovery, and real-time applications. Also required are improvements in provenance for the sharing and integration of data from the public sector. It is also extremely important to provide integrated security and privacy mechanisms in big data applications, as public sector collects vast amounts of sensitive data. Finally, respecting the privacy of citizens is a mandatory obligation in the European Union.

You have full access to this open access chapter, Download chapter PDF

Similar content being viewed by others

Big Data in HealthCare

The Principle of Purpose Limitation and Big Data

Strategies and Challenges in Big Data: A Short Review

Keywords

• Public Sector
• Sentiment Analysis
• Smart City
• Pattern Discovery
• Differential Privacy

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The public sector is becoming increasingly aware of the potential value to be gained from big data. Governments generate and collect vast quantities of data through their everyday activities, such as managing pensions and allowance payments, tax collection, national health systems, recording traffic data, and issuing official documents. This chapter takes into account current socio-economic and technological trends, including boosting productivity in an environment with significant budgetary constraints, the increasing demand for medical and social services, and standardization and interoperability as important requirements for public sector technologies and applications. Some examples of potential benefits are as follows:

• Open governmentand datasharing: The free flow of information from organizations to citizens promotes greater trust and transparency between citizens and government, in line with open data initiatives.
• Citizensentiment analysis: Information from both traditional and new social media (websites, blogs, twitter feeds, etc.) can help policy makers to prioritize services and be aware of citizens’ interests and opinions.
• Citizen segmentation and personalization while preservingprivacy: Tailoring government services to individuals can increase effectiveness, efficiency, and citizen satisfaction.
• Economic analysis: Correlation of multiple sources of data will help government economists with more accurate financial forecasts.
• Tax agencies: Automated algorithms to analyse large datasets and integration of structured and unstructured data from social media and other sources will help them validate information or flag potential frauds.
• Smart cityand Internet of things (IoT)applications: The public sector is increasingly characterized by applications that rely on sensor measurements of physical phenomena such as traffic volumes, environmental pollution, usage levels of waste containers, location of municipal vehicles, or detection of abnormal behaviour. The integrated analysis of these high volume and high velocity IoT data sources has the potential to significantly improve urban management and positively impact the safety and quality of life of its citizens.
• Cyber security: Collect, organize, and analyse vast amounts of data from government computer networks with sensitive data or critical services, to give cyber defenders greater ability to detect and counter malicious attacks.

1.1 Big Data for the Public Sector

As of today, there are no broad implementations of big data in the public sector. Compared to other sectors, the public sector has not been traditionally using data mining technologies intensively. However, there is a growing interest in the public sector on the potentials of big data for improvement in the current financial environment.

Some examples of the global growing awareness are the Joint Industry/Government Task Force to drive development of big data in Ireland, announced by the Irish Minister for Jobs, Enterprise and Innovation in June 2013 (Government of Ireland 2013), or the announcement made by the Obama administration (The White House 2012), on the “Big Data Research and Development Initiative” where six Federal departments and agencies announce more than $200 million in new commitments to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data.

1.2 Market Impact of Big Data

There is no direct market impact nor competition, as the public sector is not a productive sector, although its expenditure represented 49.3 % of GDP in 2012 of the EU28. The major part of the sector’s income is collected through taxes and social contributions. Hence, the impact of big data technologies is in terms of efficiency : the more efficient the public sector is, the better off are citizens, as less resources (taxes) are needed to provide the same level of service. Therefore, the more effective the public sector is, the more positive the impact on the economy, by transition for the rest of productive sectors, and more positive impact on society. Additionally, the quality of services provided, for example, education, health, social services, active policies, and security, can also be improved by making use of big data technologies.

2 Analysis of Industrial Needs in the Public Sector

The benefits of big data in the public sector can be grouped into three major areas, based on a classification of the types of benefits:

Big Data Analytics

This area covers applications that can only be performed through automated algorithms for advanced analytics to analyse large datasets for problem solving that can reveal data-driven insights. Such abilities can be used to detect and recognize patterns or to produce forecasts.

Applications in this area include fraud detection (McKinsey Global Institute 2011); supervision of private sector regulated activities; sentiment analysis of Internet content for the prioritization of public services (Oracle 2012); threat detection from external and internal data sources for the prevention of crime, intelligence, and security (Oracle 2012); and prediction for planning purposes of public services (Yiu 2012).

Improvements in Effectiveness

Covers the application of big data to provide greater internal transparency. Citizens and businesses can take better decisions and be more effective, and even create new products and services thanks to the information provided. Some examples of applications in this area include data availability across organizational silos (McKinsey Global Institute 2011); sharing information through public sector organizations [e.g. avoiding problems from the lack of a single identity database (e.g. in the UK) (Yiu 2012)]; open government and open data facilitating the free flow of information from public organizations to citizens and businesses, reusing data to provide new and innovative services to citizens (McKinsey Global Institute 2011; Ojo et al. 2015).

Improvements in Efficiency

This area covers the applications that provide better services and continuous improvement based on the personalization of services and learnings from the performance of such services. Some examples of applications in this area are personalization of public services to adapt to citizen needs and improving public services through internal analytics based on the analysis of performance indicators.

3 Potential Big Data Applications for the Public Sector

Four potential applications for the public sector were described and developed in Zillner et al. (2013, 2014) for demonstrating the use of big data technologies in the public sector (Table 11.1).

7.1 Pattern Discovery

• Data Analysis Technology: Semantic pattern technologies including stream pattern matching .
  • Research Question: Scalable complex pattern matching. Reaching trillions over datasets will take 5 years.
  • Research Question: Curation at scale depends on the interplay between automated curation platforms and collaborative approaches leveraging large pools of data curators. Commercial application results could be reached in 6–10 years.
  • Research Question: Standard Array Query Language. Currently there is a lack of standardized query languages but efforts such as ArrayQL are on their way. Currently there is no widespread adoption and existing DBs (SciDB, Rasdaman) are used in the scientific community. This may change in 3–5 years from now.

  7.2 Data Sharing /Data Integration

  • Data Acquisition Technology: To facilitate the integration as well as analysis.
    • Research Question: Data fragment selection, sampling and scalability. Solutions will be brought about by quantum computers (predicted to be available in 5–10 years, but 15–20 years seems more realistic.)
    • Research Question: Scalability, dealing with high speed of data and high variety. Dealing with trillions of nodes will take 3–5 years.
    • Research Question: Making semantic systems easy to use by non-semantic (logic) experts. It will take 5 years at least to have a comprehensive tooling support.
    • Research Question: What are standards for common data tracing formats? Provenance on certain storage types, e.g. graph databases, is still computationally expensive. The integration of provenance -awareness into existing tools can be achieved in the short term (2–3 years) once this reaches a critical market demand.

    7.3 Real-Time Insights

    • Data Analysis Technology: Linked data and machine learning technologies can support automated analysis, which is required for gaining real-time insights.
      • Research Question: High performance while coping with the 3 Vs (volume, variety and velocity). Real-time deep analytics is more than 5 years away.
      • Research Question: How can ad hoc and streaming queries on large datasets be executed with minimal latencies? This is an active research field and may reach further maturity in a few years’ time.

      7.4 Data Security and Privacy

      • Data Storage Technology: Encrypted storage and DBs; proxy re-encryption between domains; automatic privacy protection (e.g. differential privacy).
        • Research Question: Advances in “privacy by design” to link analytics needs with protective controls in processing and storage. A legal framework, e.g., the General Data Protection Regulation (GDPR), has to be harmonized among EU member states. Beyond legislation, data and social commons are required (Curry et al. 2014). This will require at least a further 3 years of research.

        7.5 Real-Time Data Transmission

        • Data Acquisition Technology: Kafka , Flume , Storm , etc., Curry et al. (2014).
          • Research Question: Distributed processing and cleaning. Current approaches should be able to let the user know the type of resources that they require to perform tasks specified by the user (e.g. process 10 GB/s). First approaches towards these ends are emerging and they should be available on the market within the next 5 years.
          • Research Question: How to improve random read/write performance of database technologies. The Lambda Architecture described by Marz and Warren reflects the current best practice standard for persisting high velocity data. Effectively it addresses the shortcoming of insufficient random/read write performances of existing DB technologies. Performance increases will be continuous and incremental and simplify overall development of big data technology stacks. Technologies could reach a level of maturity that leads to simplified architectural blueprints in 3–4 years.

          7.6 Natural Language Analytics

          • Data Analysis Technology: Information extraction, named entity recognition, machine learning, linked data. Entity linking and co-reference resolution.
            • Research Question: Increasing scalability and robustness. Robust scalable solutions are at least 3–5 years away.
            • Research Question: Curation at scale depends on the interplay between automated curation platforms and collaborative approaches leveraging large pools of data curators. Technically, this integration can be achieved in the short term (2–3 years).

            7.7 Predictive Analytics

            • Data Storage Technology: Analytical databases .
              • Research Question: How can databases efficiently support predictive analytics? From a storage point of view, analytical databases address the problem of better performance as the DB itself is able to execute analytical code. Currently there is a lack of standardized query languages but efforts such as ArrayQL are on their way. This may change in 3–5 years from now.

              7.8 Modelling and Simulation

              • Data Storage Technology: Best practices; batch and in-stream processing (Lambda architecture ), temporal databases.
                • Research Question: How can time-series data be managed in a general way for effective analysis? Spatiotemporal databases are an active research field and results may be beyond a 5-year time scale.
                • Research Question: Making models explicit and/or transparent. This is a research question with a long timeline (beyond 2020).

                8 Conclusion and Recommendations for the Public Sector

                The findings after analysing the requirements and the technologies currently available show that there are a number of open research questions to be addressed in order to develop the technologies such that competitive and effective solutions can be built. The main developments are required in the fields of scalability of data analysis, pattern discovery, and real-time applications. Also required are improvements in provenance for the sharing and integration of data from the public sector.

                It is also extremely important to provide integrated security and privacy mechanisms in big data applications, as the public sector collects vast amounts of sensitive data. In many countries legislation limits the use of the data only for purposes for which it was originally obtained. In any case, respecting the privacy of citizens is a mandatory obligation in the European Union.

                Other areas, especially interesting for the safety applications in public sector, are the analysis of natural language, which can be useful as a method to gather unstructured feedback from citizens, e.g. from social media and networks. The development of effective predictive analytics, as well as modelling and simulation tools for the analysis of historical data, are key challenges to be addressed by future research.

                References

                • 1105 Government Information Group. (n.d.). The chase for Big Data skills. Retrieved March 26, 2013, from GCN.com: http://gcn.com/microsites/2012/snapshot-managing-big-data/04--chasing-big-data-skill-sets.aspx
                • Correia, Z. P. (2004). Toward a stakeholder model for the co-production of the public-sector information system. Information Research, 10(3), paper 228. Retrieved February 27, 2013, from InformationR.net: http://InformationR.net/ir/10-3/paper228.html
                • Curry, E., Ngonga, A., Domingue, J., Freitas, A., Strohbach, M., Becker, T. et al. (2014). D2.2.2. Final version of the technical white paper. Public deliverable of the EU-Project BIG (318062; ICT-2011.4.4). Google Scholar
                • European Commission. (1998). COM(1998)585. Public sector information: A key resource for Europe. Green paper on public sector information in the information society. European Commission. Google Scholar
                • Government of Ireland. Department of Jobs, Enterprise and Innovation. (2013, June 24). Joint Industry/Government Task Force to drive development of Big Data in Ireland – Minister Bruton. Retrieved February 17, 2013, from Working for Jobs, Enterprise and Innovation: http://www.djei.ie/press/2013/20130624.htm
                • McKinsey Global Institute. (2011, June). Big Data: The next frontier for innovation, competition, and productivity. McKinsey & Company. Google Scholar
                • Ojo, A., Curry, E., & Sanaz-Ahmadi, F. (2015). A tale of open data innovations in five smart cities. 48th Annual Hawaii International Conference on System Sciences, IEEE (pp. 2326–2335) Google Scholar
                • Oracle. (2012). Big Data: A big deal for public sector organizations. Oracle. Google Scholar
                • PredPol. (n.d.). Retrieved September 08, 2013, from PredPol Web site: http://www.predpol.com/
                • The European Parliament and the Council of The European Union. (2003, November 17). Directive 2003/98/EC of the European Parliament and of the Council of 17 November 2003 on the re-use of public sector information. Official Journal L 345, 31/12/2003 P. 0090 - 0096. Brussels: The European Parliament and the Council of the European Union. Google Scholar
                • The White House. (2012, March 29). Big Data is a Big Deal. Retrieved January 18, 2013, from The White House: http://www.whitehouse.gov/blog/2012/03/29/big-data-big-deal
                • van Kasteren, T., Ulrich, B., Srinivasan, V., & Niessen, M. (2014). Analyzing tweets to aid situational awareness. 36th European Conference on Information Retrieval. Google Scholar
                • Wang, T., Rudin, C., Wagner, D., & Sevieri, R. (2013). Detecting patterns of crime with series finder. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases. Google Scholar
                • World Economic Forum. (2012). Big Data, Big Impact: New possibilities for international development. Geneva: The World Economic Forum. Google Scholar
                • Yiu, C. (2012). The Big Data Opportunity. Making government faster, smarter and more personal. London: Policy Exchange. Google Scholar
                • Zillner, S., Rusitschka, S., Munné, R., Lippell, H., Lobillo, F., Hussain, K. et al. (2013): D2.3.1. First draft of the sectorial requisites. Public Deliverable of the EU-Project BIG (318062; ICT-2011.4.4). Google Scholar
                • Zillner, S., Neururer, S., Munné, R., Prieto, E., Strohbach, M., van Kasteren, T. et al. (2014). D2.3.2. Final version of the sectorial requisites. Public Deliverable of the EU-Project BIG (318062; ICT-2011.4.4). Google Scholar

                Author information

                Authors and Affiliations

                1. Atos Spain, S.A., Av. Diagonal, 200, 08018, Barcelona, Spain Ricard Munné