Automatic collection and storage of smart city data with semantic data model discovery and sample data analysis

Abstract

Collecting and storing smart city data is a task that requires thorough data exploration, configuring and testing to be of value. Configuring a data collection pipeline for data from a new data provider needs to take into account what the various fields represent, what parts of the data is of interest, which data fields should be stored, and more. In some cases the data follows a predefined, and known schema, in other cases the data may be undocumented. This thesis presents a framework and a software for automating the process of collecting and storing smart city data, and other event based data sets. The problem, and solution is illustrated in this thesis by a use case where the task consist of storing public transportation data in a structured way in a storage system that can handle big data.