EPSRC funded project: Towards flexible electronic hearing aid (HA)
implementations

In collaboration with the University of Manchester , we are
creating an
audio-visual (AV) HA
platform based upon flexible electronics, which are now being made as
“temporary
tattoos” for improved discreteness and social acceptability.
.

EPSRC funded project: On-chip big data processing

In collaboration with the University of Manchester, Alpha Data, and
ENU
, we are implementing deep cognitive neural network (DCNN) features
for
autonomous, privacy-preserving transfer learning (TL). Preliminary work has
demonstrated
such DCNN architectures are capable of highly energy-efficient, on-chip
implementations,
with fast decision-making, excellent generalization, and large gains
per-operation
scaling with deep structures for large scale processing. For very
large-scale
simulations, comprising 1M neurons and 2.5B synapses, have demonstrated up
to
300X
faster decision-making compared to DNNs.
.

EPSRC funded project: Privacy-preserving, multimodal (MM) lip-reading
(LR)

In collaboration with the University of Glasgow , we are
exploring
the
groundbreaking technology of ambient radio frequency (RF), for lip-reading.

EPSRC funded project: deep transfer learning (TL)

In collaboration with the University of Edinburgh and ENU ,
we
are
developing deep TL based generalized audio-visual (AV) speech enhancement
(SE)
algorithms. We are further building our innovative, context-aware DNN based
AV
mask
estimation and SE filtering models, including through top-down models of
speech,
inspired by human cognition and evolution.

Hearing Loss Testing:

In collaboration with Princeton University, New
Jersey and the National University of Computer and Emerging
Sciences,
we
are developing an automated cost-effective pre-screening test to
predict hearing loss at an early stage. The device
can potentially offer a second opinion to audiologists and can also be
utilized
in
developing countries or rural areas where there is a lack of well-educated
audiologists.

Dementia Sensitive Personalized Environment Planner App:

With the support of
Dementia Services Development Centre at the University of
Stirling,
we
are
empowering people with cognitive impairments (e.g. dementia, autism, major
depressive disorder) to proactively choose their personalized surrounding
environment using a 5G small cell technology driven proactive environment
planner
app.

Embedded Security for IoT:

Developing a pioneering technology that is capable of providing on- chip low
power
intrusion detection and encryption in embedded and multi-core
computing systems. These represent a cost-effective alternative, and a
comparatively superior approach to state-of-the-art ARM (Arm Cortex-A,
Cortex-
M23, and Cortex-M33) processors – TrustZone
http://arm.com/products/processors/technologies/trustzone and Intel’s work
(https://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-
io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices).

IoT sensors: In collaboration with the National University
of
Science
and Technology, we are developing a new IoT standard, DeepNode. DeepNode
stands
as a
major enabler for future smart cities, healthcare and industrial monitoring,
and
environmental/earth (remote) sensing. The DeepNodeWAN is capable of
processing
a large amount of sensitive data quickly with low power consumption and high
throughput, complying with the intelligent secure RRM and diverse
communication
requirements in massive real-time communication domains.

AV Ear Defenders – SE Application in Navy and Military:

Collaboration with the University of Texas at
Dallas to explore and exploit the potential of our develop AV
speech
enhancement
technology in the US Navy (for people controlling aircraft carriers deck
operations),
US military (for officers not wearing earplugs), air traffic control towers
(to
improve
communication and reduce the risk of accidents), and cargo trains (to
address
driver
distraction).

Disaster Management:

Collaboration with the Tianjin University of Technology to
explore the application of our disruptive multimodal speech processing
technology in
extremely noisy environments e.g. in situations where ear defenders are
worn,
such
as emergency and disaster response and battlefield environments.

Asthma:

Collaboration with the Edinburgh Medical School to
understand
the
role
of
exogenous sex steroid hormones in female patients with asthma. Specifically,
we are finding the correlation between the use of hormonal contraceptives
and
asthma exacerbations in reproductive age females.

Multiphase Flow Meter Calibration:

A novel deep learning driven time-series
predictive and optimization model for uncertainty growth prediction and
calibration
intervals optimization. The technology addresses the limitations of
state-of-the-art
mathematical/statistical uncertainty growth and calibration intervals
predictive
methods such as limited modelling assumptions, limited learning, lack of
ability
to
deal with non-linear complex behaviours, and poor scalability.
State-of-the-art
literature reveals that it is difficult to solve the calibration
optimization
equation
in
closed form.

Collision Free Wi-Fi routing:

A novel deep learning driven collision free Wi-Fi
routing algorithm to enable larger number of nodes in smart cities. Social
and
digital
infrastructure of a IoT-based smart city could be boosted by deploying
high-density
public WiFi. Indeed, WiFi is a key to smart cities. Existing Wi-Fi devices
operate
following the 802.11 standards with the aim to fairly use the channel that
the
devices
share. However, the throughput performance of the existing Wi-Fi networks
suffers
from high packet loss and supports very limited number of nodes with low
datarate.

Acute general hospital admission: Collaboration with the
Dementia
Services
Development Centre at the University of Stirling, we are helping
policymakers to
explore predictors of good/bad outcome following acute general hospital
admission
for people with cognitive impairment.

Interactive Prototype Demo
Next-generation Lip-Reading Hearing Aids: Exploiting the power of contextual
Big
Data
analytics

https://cogbid.github.io/cogavhearingdemo/

– CochleaNet: A Robust Language-independent Audio-Visual Model for
Speech Enhancement https://cochleanet.github.io/:

Noisy situations
cause huge problems for suffers of hearing loss as hearing aids often make
the signal more audible but do not always restore the intelligibility. In
noisy
settings, humans routinely exploit the audio-visual (AV) nature of the
speech
to selectively suppress the background noise and to focus on the target
speaker. In this paper, we present a causal, language, noise and speaker
independent AV deep neural network (DNN) architecture for speech
enhancement (SE). The model exploits the noisy acoustic cues and noise
robust visual cues to focus on the desired speaker and improve the speech
intelligibility. To evaluate the proposed SE framework a first of its kind
AV
binaural speech corpus, called ASPIRE, is recorded in real noisy
environments including cafeteria and restaurant. We demonstrate superior
performance of our approach in terms of objective measures and subjective
listening tests over the state-of-the-art SE approaches as well as recent
DNN
based SE models. In addition, our work challenges a popular belief that a
scarcity of multi-language large vocabulary AV corpus and wide variety of
noises is a major bottleneck to build a robust language, speaker and noise
independent SE systems. We show that a model trained on synthetic mixture
of Grid corpus (with 33 speakers and a small English vocabulary) and ChiME
3 Noises (consisting of only bus, pedestrian, cafeteria, and street noises)
generalise well not only on large vocabulary corpora but also on completely
unrelated languages (such as Mandarin), wide variety of speakers and noises.

– CHiME3 AV Corpus https://cogbid.github.io/chime3av/#about:

This new
publicly available dataset is based on the benchmark audio-visual GRID
corpus, which was originally developed by our project partners at Sheffield
for
speech perception and automatic speech recognition. The new dataset
contains a range of joint audiovisual vectors, in the form of 2D-DCT visual
features, and the equivalent audio log-filterbank vector. All visual vectors
were
extracted by tracking and cropping the lip region of a range of Grid videos
(1000 videos from five speakers, giving a total of 5000 videos), and then
transforming the region with 2D-DCT. The audio vector was extracted by
windowing the audio signal, and transforming each frame into a
log-filterbank
vector. The visual signal was then interpolated to match the audio, and a
number of large datasets were created, with the frames shuffled randomly to
prevent bias, and with different pairings, including multiple visual frames
to
estimate a single audio frame (from one visual to one audio pairings, to 28
visual to one audio pairings). This dataset will enable researchers to
evaluate
how well audio speech can be estimated using visual information only.
Specifically, the application of novel speech enhancement algorithms
(including those based on advanced machine learning), can be used to
evaluate the potential of exploiting visual cues for speech enhancement.

– ASPIRE dataset https://cogbid.github.io/ASPIRE/#about:

ASPIRE is a a first of its
kind, audiovisual speech corpus recorded in real noisy environment (such as
cafe, restaurants) which can be used to support reliable evaluation of
multi-

modal Speech Filtering technologies. This dataset follows the same sentence
format as the audiovisual Grid corpus.

– First audio-visual (AV)
speech in
real-noise challenge.

A detailed
description of the AV challenge, a novel real noisy AV corpus (ASPIRE),
benchmark speech enhancement task, and baseline performance results are
outlined in [Link]. The latter are based on training a deep neural
architecture
on a synthetic mixture of Grid corpus and ChiME3 noises (consisting of bus,
pedestrian, cafe, and street noises) and testing on the ASPIRE corpus.
Subjective evaluations of five different speech enhancement algorithms
(including SEAGN, spectrum subtraction (SS) , log-minimum mean-square
error (LMMSE), audio-only CochleaNet, and AV CochleaNet) are presented
as baseline results. The aim of the multi-modal challenge is to provide a
timely opportunity for comprehensive evaluation of novel AV speech
enhancement algorithms, using our new benchmark, real-noisy AV corpus
and specified performance metrics. This will promote AV speech processing
research globally, stimulate new ground-breaking multi-modal approaches,
and attract interest from companies, academics and researchers working in
AV speech technologies and applications. We encourage participants
(through a challenge website sign-up) from both the speech and hearing
research communities, to benefit from their complementary approaches to AV
speech in noise processing.