University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Using LDA to Uncover the Underlying Structures and Relations in Smart City Data Streams

Puschmann, Daniel, Barnaghi, Payam and Tafazolli, Rahim (2017) Using LDA to Uncover the Underlying Structures and Relations in Smart City Data Streams IEEE Systems Journal, 12 (2). pp. 1755-1766.

Using LDA to Uncover the Underlying Structures and Relations in Smart City Data Streams.pdf - Accepted version Manuscript

Download (1MB) | Preview


Recent advancements in sensing, networking technologies and collecting real-world data on a large scale and from various environments have created an opportunity for new forms of real-world services and applications. This is known under the umbrella term of the Internet of Things (IoT). Physical sensor devices constantly produce very large amounts of data. Methods are needed which give the raw sensor measurements a meaningful interpretation for building automated decision support systems. To extract actionable information from real-world data, we propose a method that uncovers hidden structures and relations between multiple IoT data streams. Our novel solution uses Latent Dirichlet Allocation (LDA), a topic extraction method that is generally used in text analysis. We apply LDA on meaningful abstractions that describe the numerical data in human understandable terms. We use Symbolic Aggregate approXimation (SAX) to convert the raw data into string-based patterns and create higher level abstractions based on rules.

We finally investigate how heterogeneous sensory data from multiple sources can be processed and analysed to create near real-time intelligence and how our proposed method provides an efficient way to interpret patterns in the data streams. The proposed method uncovers the correlations and associations between different pattern in IoT data streams. The evaluation results show that the proposed solution is able to identify the correlation with high efficiency with an F-measure up to 90%.

Item Type: Article
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Date : 17 July 2017
DOI : 10.1109/JSYST.2017.2723818
Copyright Disclaimer : © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.
Uncontrolled Keywords : Data Streams; LDA; Data interpretation; Internet of Things; Data analytics
Depositing User : Clive Harris
Date Deposited : 14 Jun 2017 10:20
Last Modified : 16 Jan 2019 18:53

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800