A smart sound fingerprinting system for monitoring elderly people living alone

Abstract

There is a sharp increase in the number of old people living alone throughout the world. More often than not, such people require continuous and immediate care and attention in their everyday lives, hence the need for round the clock monitoring, albeit in a respectful, dignified and non-intrusive way. For example, continuous care is required when they become frail and less active, and immediate attention is required when they fall or remain in the same position for a long time. To this extent, various monitoring technologies have been developed, yet there are major improvements still to be realised. Current technologies include indoor positioning systems (IPSs) and health monitoring systems. The former relies on defined configurations of various sensors to capture a person's position within a given space in real-time. The functionality of the sensors varies depending on receiving appropriate data using WiFi, radio frequency identification (RFIO), ultrawide band (UWB), dead reckoning (OR), infrared indoor (IR), Bluetooth (BLE), acoustic signal, visible light detection, and sound signal monitoring. The systems use various algorithms to capture proximity, location detection, time of arrival, time difference of arrival angle, and received signal strength data. Health monitoring technologies capture important health data using accelerometers and gyroscope sensors. In some studies, audio fingerprinting has been used to detect indoor environment sound variation and have largely been based on recognising TV sound and songs. This has been achieved using various staging methods, including pre-processing, framing, windowing, time/frequency domain feature extraction, and post-processing. Time/frequency domain feature extraction tools used include Fourier Transforms (FTs}, Modified Discrete Cosine Transform (MDCT}, Principal Component Analysis (PCA), Mel-Frequency Cepstrum Coefficients (MFCCs), Constant Q Transform (CQT}, Local Energy centroid (LEC), and Wavelet transform. Artificial intelligence (Al) and probabilistic algorithms have also been used in IPSs to classify and predict different activities, with interesting applications in healthcare monitoring. Several tools have been applied in IPSs and audio fingerprinting. They include Radial Basis Kernel (RBF), Support Vector Machine (SVM), Decision Trees (DTs), Hidden Markov Models (HMMs), Na'ive Bayes (NB), Gaussian Mixture Modelling (GMM), Clustering algorithms, Artificial Neural Networks (ANNs), and Deep Learning (DL). Despite all these attempts, there is still a major gap for a completely non-intrusive system capable of monitoring what an elderly person living alone is doing, where and for how long, and providing a quick traffic-like risk score prompting, therefore immediate action or otherwise.

In this thesis, a cost-effective and completely non-intrusive indoor positioning and activity-monitoring system for elderly people living alone has been developed, tested and validated in a typical residential living space. The proposed system works based on five phases: (1)Set-up phase that defines the typical activities of daily living (TADLs). (2)Configuration phase that optimises the implementation of the required sensors in exemplar flat No.1. (3)Learning phase whereby sounds and position data of the TADLs are collected and stored in a fingerprint reference data set. (4)Listening phase whereby real-time data is collected and compared against the reference data set to provide information as to what a person is doing, when, and for how long. (5)Alert phase whereby a health frailty score varying between O unwell to 10 healthy is generated in real-time. Two typical but different residential flats (referred to here are Flats No.1 and 2) are used in the study.

The system is implemented in the bathroom, living room, and bedroom of flat No.1, which includes various floor types (carpet, tiles, laminate) to distinguish between various sounds generated upon walking on such floors. The data captured during the Learning Phase yields the reference data set and includes position and sound fingerprints. The latter is generated from tests of recording a specific TADL, thus providing time and frequency-based extracted features, frequency peak magnitude (FPM), Zero Crossing Rate (ZCR), and Root Mean Square Error (RMSE). The former is generated from distance measurement. The sampling rate of the recorded sound is 44.1kHz. Fast Fourier Transform (FFT) is applied on 0.1 seconds intervals of the recorded sound with minimisation of the spectral leakage using the Hamming window. The frequency peaks are detected from the spectrogram matrices to get the most appropriate FPM between the reference and sample data. The position detection of the monitored person is based on the distance between that captured from the learning and listening phases of the system in real-time.

A typical furnished one-bedroom flat (flat No.2) is used to validate the system. The topologies and floorings of flats No.1 and No.2 are different. The validation is applied based on "happy" and "unusual" but typical behaviours. Happy ones include typical TADLs of a healthy elderly person living alone with a risk metric higher than 8. Unusual one's mimic acute or chronic activities (or lack thereof), for example, falling and remaining on the floor, or staying in bed for long periods, i.e., scenarios when an elderly person may be in a compromised situation which is detected by a sudden drop of the risk metric (lower than 4) in real-time.

Machine learning classification algorithms are used to identify the location, activity, and time interval in real-time, with a promising early performance of 94% in detecting the right activity and the right room at the right time.