The SiVo-3 hearing aid provides voicing, fundamental frequency, voiceless excitation, and amplitude envelope information as a supplement to lip-reading for profoundly hearing impaired people. The extraction of voicing and voice fundamental frequency is performed by a real-time multi-layer perceptron that is trained to locate in noisy speech acoustic events associate with the instant of laryngeal closure. Training makes use of reference data from a laryngographic electrode recording. The performance of a 'C' language off-line prototype algorithm and a real-time version for the TI TMS320C50 processor has been compared to standard Cepstral and SIFT algorithms. In comparisons with quiet and noisy speech, the off-line 'C' language MLP showed superior detection of voicing in speech to the reference methods. The accuracy of the off-line MLP in fundamental frequency estimation was similar to that of SIFT. For speech in quiet, the Cepstral method was substantially more accurate than the other methods in fundamental frequency estimation. The Cepstral method's fundamental frequency estimation accuracy was more affected than the MLP and SIFT algorithms by the presence of noise so that at a +10 dB signal-to-noise ratio, the Cepstral method performed similarly to the others.
The real-time MLP was compared to the same two reference methods for speech in quiet, with similar results to that from the off-line algorithm. It is concluded that the MLP method provides acceptable performance given the requirements of this hearing aid application, where voicing detection is of primary importance and errors of fundamental frequency estimation are relatively innocuous given the limited frequency discrimination shown by profoundly impaired listeners.
The SiVo-3 hearing aid, a major component of the TIDE project OSCAR, features a speech-analytic mode designed to provide lipreading support for profoundly hearing impaired listeners. This speech-analytic mode provides simple acoustic signals that signal acoustic speech elements that are known to be valuable supplements to lipreading for this population (e.g. Faulkner et al., 1993). The selected elements are:
Periodic speech is represented by a sinusoid within the comfortable hearing range that follows the extracted fundamental frequency. Aperiodic speech is represented by a noise with a spectrum shaped to match the comfortable listening level between 125 and 2000 Hz. Speech amplitude envelope, after 2:1 logarithmic compression, is used to amplitude modulate both sinusoid and noise signals, with maximum and minimum levels limited by threshold and maximal comfortable levels.
The present report describes evaluations of both an off-line 'C' prototype and a real-time version of the algorithm employed by the SiVo-3 aid for the extraction of the voicing pattern and voice fundamental frequency. The real-time SiVo-3 algorithm is also compared with a previous algorithm of somewhat different structure that was used in the preceding SiVo-2 aid. One reason for the choice of a multi-layer perceptron (MLP) for the SiVo-3 is that the computations used by an MLP can be performed very efficiently by a fixed-point digital signal-processor chip. A second reason comes from the initial demonstration by Howard and Huckvale (1989) that an MLP can be rather effective for the extraction of voice fundamental frequency from noisy speech. The use of an MLP was expected to be a significant advance, particularly in respect of noise resistance, over previous real-time peak-picking methods (Howard, 1989) that had been used both in the original SiVo aid and also in the Nucleus cochlear implant speech processor.
1.1 Performance targets
In the audio-visual perception of consonants, the gross timing of the voicing pattern has been shown to be the principal source of auditory information (Faulkner, Rosen and Reeve). It is, therefore expected the accuracy of the voicing pattern cues from the SiVo-3 will be of major importance. An accurate representation of fundamental frequency (Fx) is highly desirable, since supra-segmental intonation information also contributes significantly to audio-visual perception of connected speech. Another important aspect of performance in this applications is that the extracted fundamental frequency information can represent the irregularity of vocal fold vibration in order to give the user feedback on their own voice control (Ball et al., 1990). For this purpose, a time-based analysis of fundamental frequency that operates cycle-by-cycle is highly desirable.
2. Algorithm structure
The approach presented here employs a multi-layer perceptron (MLP) neural network pattern classifier (Rumelhart & McClelland, 1988) which carries out a non-linear mapping of the input data set to the output unit of the network. The MLP classifiers discussed here depend on the use of supervised training by which the neural net is provided with labelled reference data that specifies the correct classification of the network output for each time-frame of the training data., whereby one frame per period (at the moment of larynx closure) is designated as voiced and all other frames as unvoiced. This allows the algorithm to give cycle-by-cycle estimates of Fx.
The algorithm consists of three parts: 1) the pre-processor, 2) the main extractor, and 3) a post-processor. The evaluations here cover two related MLP networks that differ in both the pre-processing and the network structure.
The input data to the MLP classifier is a number of waveform samples from the pre-processor. The analysis window needs to be long enough to allow the classifier to "see" one or more periods of speech. The two MLPs used here have analysis windows of 30.5 ms (MLPa) and 20.5ms (MLPb).
The MLP is trained with the objective of making its final output unit produce a high output at the time of vocal fold closure in the speech input, and a low output at other times. The algorithm is, therefore, estimating the time at which each vocal fold closure occurs, from which each successive fundamental period is then estimated.
2.1 MLPa Pre-processor
MLPa is the designation of the MLP used in the SiVo-3 MLP. The pre-processing stage principally performs data reduction of the speech signal. The speech input is low-pass filtered at 900 Hz, using a 4th order IIR low-pass filter. The signal is then down-sampled to a 2 kHz sample rate to match the cycle rate of the MLP. Each cycle of the MLP has as its input data the speech waveform samples from 61 successive 0.5 ms frames.
2.2 MLPb Pre-processor
The second MLP considered here, MLPb, is a predecessor to MLPa. It is similar to the algorithm described by Walliker and Howard (1990) and to that used in the earlier SiVo-2 device as evaluated in the TIDE project TP133/206 STRIDE. The input data for the MLPb algorithm is sampled both in time and frequency from the speech signal. The speech is first processed through 6 band-pass filters, and then the output of each filter is down-sampled to a 2 kHz sample rate. Forty-one successive frames from each of the six filters provide the input data to the MLP. MLPb differs from the MLP used in the SiVo-2 aid in the use of a post-processing buffer.
2.3 MLP structures
The main fundamental frequency extractor is a MLP pattern classifier containing an input layer, two hidden layers, and an output layer. Theoretically, a one-hidden-layer MLP network is capable of forming an arbitrarily close approximation to any non-linear decision boundary for any classification task (Makhoul et al., 1989). In practice a small two-hidden-layer network can be used where a one-hidden-layer network would require an unrealistically large number of nodes (Chester, 1990). The input layer's size is determined by the number of inputs. MLPa has 61 input units compared to the 246 input units of MLPb, and hence it is possible, with a similar level of computational effort, to use more units in the intermediate layers of MLPa. The MLPa algorithm has a 61-20-6-1 structure, while MLPb has a structure of 246-6-6-1 units. Both structures are fully connected.
2.4 Training reference data
The reference target data is derived from the laryngograph (Lx) signal (Fourcin & Abberton, 1971) which directly represents the events in the closed phase of vocal fold vibration. Other work has been reported using the raw Lx signal as the reference data to train an MLP classifier for larynx excitation (Denzler et al., 1993). Some disadvantages of using the unprocessed Lx signal are that 1) the DC offset of the Lx signal can be significant, and 2) the amplitude range of the Lx signal varies widely from speaker to speaker. In this study, the Lx signal was processed to make it more suitable for MLP training.
2.4.1 Lx processing for MLPb
The Lx-based training data for the earlier MLPb algorithm was a discontinuous unit-pulse train, with the value of one at each estimated time of vocal fold closure, and value zero elsewhere. The discontinuous nature of this training reference is likely to be a disadvantage, since there is no information in the reference signal to indicate frames that are close to, but not exactly synchronous with, a vocal fold closure. Hence, an MLP output pulse in the frame adjacent to the target frame cannot be treated differently from an output pulse many frames away from the target frame in respect of its contribution to the error during training. The reference data was generated by automatic labelling of the Lx signal to identify instants of larynx closure. As no perfect automatic labelling method was available, subsequent hand-correction was required.
2.4.2 Lx processing for MLPa
For MLPa, the reference signal was a continuous one. The Lx signal was filtered using a zero-phase 8th-order 20 Hz IIR high-pass filter to remove DC. It was then half-wave rectified and normalised cycle-by-cycle to the range between 0.0 - 1.0. By applying this processed Lx signal as the reference for the MLP classifier, we avoided hand-labelling which is time-consuming and can be inaccurate (Barnard, 1991). The continuously varying reference signal contains information that may be relevant to the training process at every sample value of each vocal fold cycle.
2.4.3 Recording of training data
The training data were recorded in a non-acoustically treated room at the Department of Phonetics and Linguistics, University College London, with a 50 cm speaker to microphone spacing. The data consisted of two signals, the speech pressure waveform and the laryngograph signal. Eight speakers (four male and four female) each read two passages "Arthur the Rat" and "The Rainbow Passage". A speech spectrum shaped noise was added to part of the speech waveform training data, at speech-to-noise ratios (S/N) of 20, 15 and 10 dB. S/N ratio was determined from the maximum speech rms level measured in a 500 ms window and the noise rms level. In order to provide training data that represented a range of speech input levels, parts of the noisy and quiet training data were attenuated by 0, 6 or 12 dB.
2.5 Training method
The network training used a standard back propagation method (Rumelhart et al., 1986). We know from the Generalised Delta Rule that the current weight change in the MLP network DWij(n) is proportional to the gradient descent in the error E of the input/output pattern,
µ - h ,
where h is a positive constant known
as the "learning rate". It is known that training can
be speeded up by increasing h. However,
oscillations eventually arise. The gradient descent method can
be enhanced by introducing a "momentum term" from the
previous weight change as:
µ - h
where the damping value acts to optimise the learning behaviour. It not only offsets oscillations but also increases the effective learning rate. However, if is too large, it will dominate the weight update and the updating direction can deviate far from the steepest gradient. Here an adaptive method for choosing h and was employed, in which the weights of the MLP are adapted by changing the learning parameter as a function of the direction of adaptation in the parameter space (Chan & Fallside, 1987). This method attempts to eradicate two specific weakness of the fixed coefficient learning algorithm: oscillations across the walls of the ravine of the error surface and the long search caused by the shallow gradient of the error surface in the weight space.
The standard gradient descent adaptation method modifies the MLP weights after all the training data samples have been presented to the network (batch learning). However, our experience has been that it is faster to update the weights after the presentation of a subset of training data (on-line learning).
The process of computing the gradient and adjusting the weights is repeated until a minimum is found. In practice, it is difficult to automate location of the error minimum. Here, the method of cross-validation was used to monitor generalisation performance during learning. The method of cross-validation works by splitting the data into two sets: a training set that is used to train the network, and a test set that is used to measure the generalisation performance of the classifier. During training, the performance of the classifier on the training data will improve, but its performance on the test data will only improve to a point, beyond which its performance on other data will start to deteriorate. This is the point where the network starts to over-learn, and it is here that the learning process should be terminated.
A number of training runs were made with different random initial coefficients for the MLP as a precaution against finding local minima in the error space. The run giving the lowest overall error from the training data was selected to determine the MLP coefficients that were ultimately used.
The post-processing stage performs error correction and smoothing of the raw estimates from the main extractor. The post-processor outputs a fundamental period marker for each estimated fundamental period. The history of several previous output periods needs to be stored in order to correct the errors and smooth the output from the main extractor. A 20.5 ms buffer was introduced which could hold 40 samples of previous MLP output and the present output. This is shown in Figure 1.
Figure 1:Buffer Post-Processor.
Each incoming MLP output is stored at the beginning of the buffer and all the other values in the buffer are shifted along by one frame. When the value at the end of buffer is both a local peak and exceeds a pre-determined primary threshold, the processor will look along the buffer for a second peak. If there is a second peak within the buffer, the time difference between the first and second peaks is calculated. If this fundamental period Tx is the first period after a silent or a voiceless speech segment, it is immediately accepted as a valid output. If it occurs within 30 ms of a previously detected period of voiced excitation, this Tx value is compared with the estimated previous fundamental period Te, where Te is the average of the previous two fundamental periods: (Tx-1+Tx-2)/2). If Tx is within 20% of Te, then Tx is accepted as a valid output. If Tx is not within 20% of Te, then the processor will look for a second peak that is within 20% of Te. If a peak has been found within this range that exceeds a lower secondary threshold, then it is used to calculate the present Tx. Otherwise, the previously detected second peak is used to calculate Tx.
Each Tx estimate is used to initiate the generation of a single cycle of a cosine signal. The sequence of cosine cycles forms a signal that is matched to the auditory abilities of profoundly hearing impaired listeners and represents both the fundamental frequency pattern and the on-off voicing pattern of the speech input. The MLP fundamental frequency extractor and the buffer processor introduce a delay of 35 ms (MLPa) or 30 ms (MLPb) between the input and output.
In the operation of the SiVo-3, each period estimate is used to control the generation of a single period of a cosine wave. This introduces a further delay of Tx(in the range 0.5 to 30 ms). Studies of aided lip-reading in normal listeners have shown that if the time delay of a fundamental frequency signal compared to the original speech is within 40 ms, there is no affect on performance, and that delays up to 80ms produce only slight and non-significant decreases in performance (McGrath & Summerfield, 1985).
3. Evaluation 1. UNIX 'C' implementation of MLPa:
performance in quiet and noise
This first study concerns the performance of an off-line 'C' language implementation of MLPa that was developed prior to the real-time version in TMS320C50 assembly language.
3.1 Comparison methods
In this section, the MLPa algorithm is compared to two standard methods. In each case, the implementations were those of the ILS signal processing software library (ILS, 1986).
The chosen methods were the "cepstrum" technique (Noll, 1967), and the "Simple Inverse Filter Tracking " or SIFT algorithm (Markel, 1972). While there are more recent algorithms in existence that may give superior performance (e.g. Secrest and Doddington, 1983) these standard Cepstral and SIFT methods have the merit be being widely used and documented. The ILS Cepstral analysis was applied with the default settings for a 10 kHz sample rate; the analysis window was 32 ms in duration. The default parameters for the ILS SIFT analysis were also used. Here the effective analysis window was 50 ms, with smoothing of the Fx estimates applied over four successive windows.
3.2 Test data
The test data, like the training data, contain both speech and Laryngograph waveforms. The laryngograph signal was used to derive the reference fundamental frequency for comparison with the estimates from each algorithm. Four speakers were recorded, two male and two female. They were different speakers to those recorded for the training data. Each read a standard speech passage, "The North Wind and the Sun".
The test data included noisy speech produced in the same way as the noisy training speech data, with S/N of 15 dB and 10 dB. The test speech data had a sampling rate of 10 kHz.
The fundamental frequency and fundamental period estimates from the algorithms were all expressed in terms of fundamental frequency. The Fx sample rate was 200 Hz for the MLPa output, and 100 Hz for the other methods. The Fx data from the MLP algorithm were resampled to 100 Hz before comparisons were made.
3.3 Voicing classification performance
One basic parameter of fundamental frequency extraction algorithms, especially in this SiVo-3 application, is the voiced-unvoiced decision. This decision has two error components, where voiced frames are wrongly classified as unvoiced, and where unvoiced frames are incorrectly classified as voiced. These two errors for the three algorithms are shown in Figure 2 through Figure 4.
|Figure 2Voicing classification errors in quiet||Figure 3 Voicing errors at 15 dB S/N|
Voicing errors at 10 dB S/N
The voicing errors in each case show a tendency for more false alarms as the signal-to-noise ratio deteriorates; this is probably an appropriate bias in a hearing aid to supplement lipreading, where the user may be able to ignore false alarms that occur during (visible) pauses in speech
An overall measure that combines both sources of error in voicing classification is the signal detectability index d' (Peterson et al., 1954). Voicing detection performance measured by d' is shown in Figure 5. This measure makes clear that the MLPa algorithm is the most effective for voicing classification, especially in quiet. The Sift algorithm is poorer as a result of the higher number of unvoiced frames that it classifies as voiced, while the Cepstral analysis is poorer than the MLP in missing a higher number of voiced frames.
Figure 5. Voicing detection accuracy measured by d'
3.4 Accuracy of fundamental frequency extraction
For all samples of the reference Fx pattern that were classified as voiced according to the laryngograph reference, Fx errors were computed for each algorithm. Errors of greater than one octave are designated gross errors. The proportion of frames where gross errors occurred is shown in Figure 6 through to Figure 8.
Figure 6: Proportion of frames of voiced speech showing Fx errors for quiet speech
Figure 7: Proportion of frames showing gross Fx errors: S/N = 15 dB
Figure 8: Proportion of frames showing gross Fx errors: S/N = 10 dB
All of the methods show relatively low rates of gross Fx error. The rate is markedly higher for MLPa than for either the Cepstral or the Sift method, but never exceeds 5%.
The distributions of Fx errors of one octave and less are shown in Figure 9 through Figure 11.
Figure 9: Distribution of Fx errors in quiet. Solid line - Cepstrum; dashed line - MLPa; dotted line - SIFT.
Figure 10. Distribution of Fx errors: 15 dB S/N
Figure 11: Distribution of Fx errors: 10 dB S/N
The distributions of errors of one octave and less show that the ILS Cepstral method is always more accurate than either the SIFT or the MLPa algorithms in the precision of Fx estimation. However, the error distribution from the Cepstral method is much more affected by the presence of noise than are the other methods, and at the poorest signal-to-noise ratio, the distribution is much more similar to that from the SIFT and MLPa algorithms. The SIFT and MLPa algorithms show very similar distributions of Fx error, except that the SIFT Fx estimates tend to be between one to two semitones too low. Such a constant error would not be important for an auditory supplement to lipreading.
The off-line prototype MLP has been shown to give superior performance in voicing classification than either reference method, both in quiet and in noise. The Fx estimation accuracy of the OSCAR MLP is less accurate than that of the Cepstral analysis both in terms of gross and less extreme Fx errors, but this difference is relatively small at poorer signal-to-noise ratios. The OSCAR MLP is more accurate than the Sift method in respect of less extreme Fx errors. The gross Fx error rate of the OSCAR MLP is larger than that of the two reference algorithms, but is still relatively low.
Since in this application it is the accuracy of voicing detection that is of primary significance over accuracy of Fx estimation, the results confirm that the MLP is likely to be appropriate for the SiVo application.
4. Evaluation 2. Real-time MLP running on SiVo-3
speech processor: performance in quiet
To evaluate the MLP algorithms running in real-time on the SiVo-3 device, simultaneous recordings were made (Alesis A-DAT digital tape recorder) of the following input and output and reference signals.
Recordings were made in a quiet room with a microphone distance of about 1 m. The microphone was that built into the case of the SiVo-3 unit. Recordings were taken from four female and four male talkers who had not been used for the MLP training data.
Evaluations are based on the voicing and fundamental frequency patterns sampled in 5 ms frames, compared to the voicing and fundamental frequency pattern derived from the Laryngograph waveform. Comparison data were obtained from the same speech samples using the Cepstral and SIFT fundamental frequency analysis algorithms provided in the ILS signal processing package. These were the same as the ILS reference methods used in the earlier evaluation of the Unix 'C' implementation of MLPa.
For each sample recording, the speech and laryngograph waveforms, and the outputs of the MLPa and MLPb algorithms after post-processing, were digitally acquired using a four-channel PCLx card and software from Laryngograph Ltd. The data were then copied to a Sun Unix computer system and processed using the Speech Filing System (SFS) software (Huckvale, 1996) to extract voicing and fundamental frequency patterns. Unvoiced frames were assigned a fundamental frequency of zero Hz.
These data were finally analysed using SPSS for Windows to make the following comparisons for each sample recording
4.1 Cross-correlations of estimated Fx with reference
Cross-correlation indicated that the time-alignment of the MLP output signals was correct - the delays involved in the MLP and post-processing buffer were 30 ms for MLPa and 35 Ms for MLPb. The ILS algorithms produced correctly time-aligned output without the need for correction.
4.2 Voiced/unvoiced classification
Voicing classification performance has been measured in two ways. Firstly, by the cross-correlation of the voiced/unvoiced classification frame-by-frame of the algorithm output with the reference from the Laryngograph. Secondly, by the signal detection statistic d'. Correlations and d' were compared between algorithms by pairwise t tests using a significance criterion of p=0.05.
The correlation comparisons are shown in Figure 12. Across the
eight talkers, the correlation with the reference voicing pattern
was not significantly different between MLPa and MLPb. The correlations
from MLPa were not significantly different to those from the Cepstral
analysis, but MLPb was significantly less accurate than the Cepstral
analysis. The correlations from the Sift analysis were significantly
lower than those from all of the other three algorithms.
Figure 12: Correlation of frame-by-frame voicing classification with Lx reference. The box and whisker plot shows the median value as a bar, the 75% range as the box, and the extreme values as the whiskers.
Figure 13: Voicing detection performance measured by d'
The d' measure (see Figure 13) showed no substantial differences between the MLP algorithms. Voicing detection by the Cepstral analysis was significantly higher than for the MLP methods, and voicing detection by Sift was significantly worse than any of the other methods.
4.3 Scatter-plots of estimated and reference Fx
Figure 14: Scatter-plots of reference Fx and estimated
Fx from the Cepstral and Sift algorithms. The panels show data
from the male and female speakers from each of the four algorithms.
The solid line is a linear regression fit for the data. The squared
correlation for each fit is shown to the right of each panel.
Figure 15: Scatter-plots of reference Fx and estimated Fx from MLPa and MLPb. The panels show data from the male and female speakers from each of the four algorithms. The solid line is a linear regression fit for the data. The squared correlation for each fit is shown to the right of each panel.
Scatter-plots of reference compared to estimated Fx are shown in Figure 14 and Figure 15. A perfect performance would result in a scatter plot where all points lay on a straight line of unit slope. None of the methods approach this level of accuracy. The Cepstral analysis provides the most accurate result, and the Sift method the poorest. MLPa is rather better than MLPb. Both MLP's show a tendency to overestimate higher fundamental frequencies for male speakers. The squared correlations shown in the figure represent a measure of goodness of Fx estimation that includes the effect of voicing error (with Fx arbitrarily valued at zero for unvoiced frames). This measure is higher generally for the female speakers than for male speakers. The algorithms rank as Cepstrum > MLPa > MLPb > Sift for this measure.
4.4 Distribution of Fx errors
Fx errors are shown here by histograms displaying the ratio between the estimated Fx value from each algorithm and Fx from the laryngograph reference. Frames of data where the Lx reference is unvoiced are not included, but frames where the algorithms fail to detect voicing are.
Figure 16: Distribution of Fx estimates. The four panels show data for male and female speakers from the ILS Cepstral and Sift algorithms. The data are for all frames where the Lx reference was voiced, and include frames that the algorithms classified as unvoiced for which the ratio is zero. Each bar represents a range of 6.25%. The algorithm is indicated on the abscissa, and the speaker sex above each panel. The mean and standard deviation of the displayed distribution is shown to the right of each panel, together with the total number of frames analysed.
Figure 17: Distribution of Fx estimates. The four panels show data for male and female speakers from the MLPa and MLPb algorithms. The data are for all frames where the Lx reference was voiced, and include frames that the algorithms classified as unvoiced for which the ratio is zero. Each bar represents a range of 6.25%. The algorithm is indicated on the abscissa, and the speaker sex above each panel. The mean and standard deviation of the displayed distribution is shown to the right of each panel, together with the total number of frames analysed.
The histograms in Figure 16 and Figure 17 reflect both the spread of estimated Fx about its true value, and the proportion of voiced frames classified as unvoiced. In precision of Fx estimation, the Cepstral method is clearly superior. The precision of Fx estimates from MLPa is reasonable, with some 70% of frames estimated with an error of 6.25% or less for male speech and around 50% with this degree of error for female speech. The broader Fx distribution from MLPa with female speech reflects the inherent limit of 0.5 ms to the precision of its Tx estimates. Female Fx for these speakers is typically between 200 and 300 Hz, but extends well beyond 400 Hz on occasion. A Tx precision of 5 ms translates to an Fx precision of 10% at 200 Hz and 20% at 400 Hz.
For male speech, MLPa is clearly superior to MLPb in regard to this measure.
4.5 Conclusions of Evaluation 2.
The MLPa algorithm is, on the evidence of the correlation of the estimated voicing pattern with the Lx reference, and accuracy of Fx estimation, superior to MLPb. This is most evident for male speech. In the accuracy of voicing detection, MLPa approaches the performance of the Cepstral reference, and both MLPs are superior to the Sift method.
4.6 Comparisons between Evaluations 1 and 2
Comparable measures are only available for voicing classification, since fundamental frequency estimation errors were processed in different ways in the two evaluations. It must be noted that the speech materials were also different between the two studies. The d' measure of voicing classification from MLPa is rather similar for speech in quiet in both evaluations; the overall d' in evaluation 1 was around 2.5, while for the real-time algorithm, d' was n the range 2.0 to 2.5. For the Cepstral method, d' for voicing classification was higher for the recordings of evaluation 2 (2.5 to 3.0) than from the recordings used in evaluation 1, where in quiet, d' was 1.8. This may reflect that the recordings used in evaluation 1 were inherently less quiet than those used in evaluation 2. It seems, therefore, that the different ranking of the Cepstral method and MLPa for voicing classification is likely to be due more to differences in the recorded materials and does not indicate a significant difference in performance between the prototype off-line and real-time versions of MLPa.
5. Overall conclusions
Both these evaluations, covering a UNIX and a real-time TMS320C50 implementation of the OSCAR MLP fundamental frequency extractor, indicate that its performance is likely to be appropriate for the SiVo-3 application. Voicing detection by the MLPa algorithm is similar to that of the Cepstral analysis, and much superior to that of the Sift algorithm. The results indicate that MLPa is, as intended, able to maintain much of its performance in levels of noise that are problematic for profoundly hearing impaired listeners. Evaluation 2 makes clear that MLPa is superior to the preceding MLPb on all criteria. Only the MLP algorithm is able to provide cycle-by-cycle F0 estimates, which gives it a further advantage over the other methods in this hearing aid application.
5.1 Significance of Fx and voicing errors
The significance of the errors of Fx estimation and voicing classification of these algorithms for a device that supplements lipreading can be assessed by reference to user-based assessments of the analysis method of the OSCAR field trials (van Son et al., 1997). Here tests in profoundly hearing impaired users of SiVo-3 compared an ideal extraction of the voicing pattern and Fx based on a recorded laryngograph signal to the use of MLPa to extract this information from speech in quiet. Speech perception using the analytic processing of MLPa did not differ significantly from a the condition in which the laryngograph signal was used to control the voicing and fundamental frequency information from the SiVo-3 aid. This suggests that profoundly impaired listeners are not highly sensitive to such errors. It has been suggested that high accuracy may be necessary in Fx based lipreading aids for optimal performance (Hnath-Chisolm and Boothroyd (1992), but these results were based on normally-hearing listeners who apparently are much more sensitive to such errors than are profoundly-impaired listeners.
5.2 User results for speech in noise
A second outcome of the OSCAR user trials has been the finding that speech perception in noise using the analytic processing of MLPa is little affected by the presence of noise at S/N up to 5 dB. This is consistent with the objective finding here that the voicing classification and Fx extraction performance of MLPa is little affected by noise.
Supported by the TIDE project OSCAR (TP1217) with funding from CEC DGXII (C5). We are grateful to John Walliker and Mark Huckvale for assistance in the development and training of the MLPs, and to both John Walliker and David Howells for their contribution to the real-time code of the SiVo-2 and SiVo-3. We are also grateful to Roel Smits for constructive comments on an earlier draft MS.
Ball, V., Faulkner, A., and Fourcin, A. J. (1990). "The effects of two different speech coding strategies on voice fundamental frequency control in deafened adults," Br. J. Audiol. 24, 393-409.
Barnard E., Cole R. A., Vea M. P., and Alleva F. A., "Pitch detection with a neural-net classifier", IEEE Trans. Acoustics, Speech, Signal Processing, Vol. 39, No. 2, pp 298-307, Feb., 1991.
Chan L. W., and Fallside F., "An adaptive training algorithm for back propagation networks", Computer Speech and Language, 205-218, 1987.
Chester D. L., "Why two hidden layers are better one", Proceedings of the International Joint Conference on Neural Networks, Vol. 1, pp 265-268, Erlbaum, 1990.
Denzler J., Kompe R., Kiebling A., Niemann H., and Noth E., "Going back to the source: inverse filtering of the speech signal with ANNs", Proceedings of Eurospeech, pp 111-114, Berlin, 1993.
Fourcin A. J., and Abberton E., "First applications of a new laryngograph", Medical and Biological Illustration, Vol. 21, pp 172-182, 1971.
Hess, W. (1983) "Pitch Determination of Speech Signals" Springer-Verlag, Berlin.
Hnath-Chisolm, T. and Boothroyd, A. (1992) "Speechreading enhancement by voice fundamental frequency: The effects of F0 contour distortions". J. Sp. Hear. Res., 35, pp 1160-1168.
Howard, D. M. (1989) "Peak-picking fundamental frequency estimation for hearing prostheses" J. Acoust. Soc. Am., 86, pp 902-910
Howard, I. S. and Huckvale, M. A. (1988) "Speech fundamental period extraction using a trainable pattern classifier" In. Proc. Speech '88: 7th FASE Symposium (Inst. Acoust. Edinburgh), pp129-136.
ILS (1986). Interactive Laboratory System, Signal Technology Inc.
Makhoul J, El-Jaroudi A, and Schwartz R, "Formation of disconnected decision regions with a single hidden layer", Proceedings of the International Joint Conference on Neural Networks, Vol. 1, pp 455-460, 1989.
Markel, J.D., "The SIFT algorithm for fundamental frequency estimation", IEEE Trans. AU-20, 367-377, 1972.
McGrath M and Summerfield Q (1985) Intermodal-timing relations and audio-visual speech recognition by normal-hearing adults", J. Acoust. Soc. Am., 77, 678-685.
Noll, A.M., "Cepstrum pitch determination" J. Acoust. Soc. Amer., 41, 293-309, 1967.
Peterson, W. W., Birdsall, T. G. and Fox, W. C. (1954). The theory of signal detectability. Inst. Radio Engineers Transactions, PGIT-4, 171-212
Rumelhart D. E., Hinton G., and Williams R., "Learning representations by back-propagating errors", Nature, Vol. 323, pp. 533-536, 1986.
Rumelhart D, McClelland J and the PDP Research Group (1988) Parallel Distributed Processing, The MIT Press, Cambridge, Massachusetts.
Secrest, B.G., and Doddington, G.R. (1983). "An integrated pitch tracking algorithm for speech systems," ICASSP 83, 1352-1355, Boston.
van Son, N., Beijk, C, and Faulkner, A. (1997), "Final report of Workpackage E: Field trials", in preparation. Ref. OSCAR-UCL-1997-WPE
Walliker, J. R. and Howard, I. S. (1990) The implementation of a real time speech fundamental period algorithm using multi-layer perceptrons. Speech Communication, 9, 63-71.
© Andrew Faulkner and Jianing Wei