CASE FOR SUPPORT

PART A: PREVIOUS RESEARCH TRACK RECORD

A.1 THE PROSYNTH1 PROJECT
EPSRC grants GR/L 52109 (UCL, £95,460), 53069 (Cambridge, £91,174), 51829 (York, £80,985), 1997-2000: An integrated prosodic approach to device-independent natural-sounding speech synthesis. PIs Jill House, Mark Huckvale (UCL), Sarah Hawkins (Cambridge), John Local, Richard Ogden (York). The present proposal builds on the collaboration successfully established under our current EPSRC grants between academic staff at University College London, the University of Cambridge and the University of York. These grants, covering the period October 1997 to March 2000, have made possible the employment of Researchers on each site for up to 24 months full-time. 
Our collaboration, between researchers with complementary skills, has achieved (i) the creation of a new open speech synthesis architecture based on XML and a declarative knowledge framework; (ii) a codification of knowledge for phonetic interpretation in the domains of duration, intonation and spectral colour for a limited number of sentence types and phonetic segments; (iii) demonstration of the value of the linguistic formalism and knowledge through perceptual tests; (iv) dissemination of the fruits of our research through articles and software. Our work has been reported at the 5th ICSLP in Sydney [1],[2]; at the 3rd ESCA/COCOSDA workshop on speech synthesis at Jenolan Caves [3]; at the XIVth ICPhS in San Francisco [4], [5], [6]; and at Eurospeech 99 in Budapest [7]. Aspects of the project have been presented at the international COST 258 workshops in Vigo (Nov 1998) and Lausanne (Apr 1999).  An invited paper [ ] is in press, and a joint position paper [8] submitted to Computer Speech & Language discusses the ProSynth philosophy and outlines research to date, including initial perceptual test results.
The development of common tools and access to a common database have been central to the research activity on all three sites. All sites have contributed to the enrichment and updating of the linguistic representation which drives ProSynth synthesis.  All have participated in the design of sometimes innovative perceptual test procedures. We see continued collaboration as not just mutually beneficial but essential if we are to build on current achievements, and crucial to the ProSynth principle, whereby linguistic knowledge at all levels of structure is integrated in phonetic interpretation. Individual sites have contributed as follows.  UCL: recording and automated annotation of a speech database; implementation of the linguistic representation and lexicon in XML; preliminary intonation modelling; development of a prototype text-to-speech (TTS) tool.  Cambridge: modelling and synthesis of systematically  varying acoustic fine detail at the segmental level; development of PROCSY, a new tool embodying a hybrid approach to fast copy-synthesis using HLsyn; [to produce stimuli for testing hypotheses about the perceptual salience of systematic acoustic variation]; coordination, running and analysis of perceptual tests. York: integration of linguistic levels; development of formal phonological representations for which phonetic interpretations can be stated; a linguistically informed  model of duration.

A.2 PARTNERS
A.2.1 Cambridge (Sarah Hawkins, Sebastian Heid)
Sarah Hawkins has published on acoustic phonetics, speech production and speech perception, is experienced in perceptual testing (natural and synthetic speech), and has contributed to the Infovox TTS system. She has worked on analysis, synthesis, and perception of temporal and spectral properties of phonetic contrasts, including prosodically‑induced variability [ ], [ ], [ ], [ ], [ ], [ ]. Recent work has focused on acoustic-phonetic properties that underlie the perceptual coherence of speech [ ], [ ], [*], [%]. Sebastian Heid has played a key role in ProSynth: conducting perceptual experiments; programming Hlsyn interface and rule system; acoustic analyses of systematic phonetic detail.  Funded Cambridge work has included: 
Infovox AB, Stockholm: Synthesis of British English segments in a multilingual synthesis system, 1990‑3;  part of Esprit SUNDIAL project. PI Sarah Hawkins.
Telia Promotor Infovox AB, Stockholm: Synthesis of British English, 1993‑5. PI Sarah Hawkins. Both grants funded RA, equipment and travel, and were conducted in collaboration with related work at UCL. Hawkins’ main contributions were to timing and segmental parameters, and intelligibility testing. She also supervised a broader range of work, including development of lexicon and grapheme-to-phoneme rules, and coordinated contracts between Infovox and Logica for the development of a (proprietary) syntactic parser.
Swiss National Fund for Scientific Research, 1999-2000 (£1,200). Towards a non-segmental computational model of spoken word recognition.  With Dr. N. Nguyen, U. Geneva. Acoustic-phonetic analyses, perceptual experiments, and preliminary modelling of dependencies between onset /l/ and coda voicing.
Hawkins also has longstanding active working relationships with Stevens and McGowan (amongst others) at Sensimetrics, and this facilitates the development of PROCSY. 
A.2.2 UCL (Jill House, Mark Huckvale, Jana Dankovicova)
Jill House has wide experience with modelling intonation and voice source for TTS synthesis, and served. on the Scientific Committee of the ESCA Speech Synthesis workshop, Mohonk, 1994. Recent work on F0 alignment [9], [10] has been directly relevant to ProSynth modelling. Mark Huckvale has extensive experience in speech synthesis, speech recognition, speech signal processing and software development.  He is vice-chairman of COST 258, primary author of the Speech Filing System software, and main 'voice' on a number of speech synthesis corpora. As RA on the ProSynth project, Jana Dankovicova made a significant contribution to intonation modelling. Her PhD research into linguistic factors contributing to articulation rate variation will feed into the research proposed here.  Relevant funded UCL projects have included: 
SRU F7T/50574/C, 1985-89: Improvements to speech synthesis-by-rule algorithms (P.I. Adrian Fourcin, Jill House RA).  Intonation modelling for the JSRU TTS system [11], [12], [13], [14].
SERC GR/F/30642, 1989-92 (£110,090): Natural voice source synthesis by rule (P.I. Jill House from 1990). Dynamic modelling of voice source parameters in formant synthesis;  perceptual relevance. [15], [16]. 
Infovox AB/Telia Promotor Infovox, Stockholm: consultancy, 1989-93 (Jill House): (i) Design and implementation within Infovox's British English TTS system of a revised prosodic model. (ii) Speech output component for British English using Infovox system, on Esprit SUNDIAL project. Interface with linguistic generator and context-sensitive intonation rules for dialogue (successfully evaluated) [17],[18], [19], [20], [21]. 
EPSRC GR/K75033, 1993-6 (£176,365): The development of an automatic parsing system using the ICE corpus as linguistic knowledge base. In collaboration with UCL Survey of English Usage, Huckvale contributed to development of a spoken corpus for prosody research (PROSICE), and supervised Fang=s development of a fast and robust grammatical parser [22], [23], subsequently adapted for automatic speech chunking in SpeechMaker [24]. 
EPSRC GR/L25639, 1996-9 (£170,111): Automatic cue-enhancement of natural speech for improved intelligibility. (PIs Huckvale and V. Hazan) [ no.], [etc. ]Y?
EPSRC GR/L81406, 1998-2001 (£167k ): Enhanced Language Modelling. (P.I. Huckvale) [ ], [ ]…?
A.2.3 York (John Local, Richard Ogden, Paul Carter)
John Local has a long history of research in spoken language, focusing on the analysis of phonetic detail, and Richard Ogden’s work has focused on declarative phonology. Both contributed to the development of YorkTalk until 1996. The success of this structure-based approach, with its emphasis on constraint satisfaction rather than arbitrary rules informed and motivated the previous ProSynth proposal. Work at York a demonstrated ability to combine phonological analysis with detailed phonetic interpretation, allowing for the generation of natural-sounding formant synthesis and improved timing of diphone synthesis. Paul Carter has played a vital role in ProSynth, elaborating the labelling system and implementing regression-based statistical models of duration using knowledge of high-level linguistic structure. 
York hosted the Sixth International Conference in Laboratory Phonology in July 1998, a recognition of our contribution to the field. Funded work has included: 
British Telecom: Collaborative work on non-segmental speech synthesis, 1988-94 (£233,000). Outcome: the YorkTalk model of speech synthesis. Some of the linguistic knowledge incorporated in YorkTalk was used by BT in the construction of the LAUREATE system, thus contributing to UK competitiveness.
British Telecom: Generating spoken language statistics from a non-segmental phonological model. Development of a test-signal for non-linear communication devices., 1994 (£30,000). P.I. John Local. [?]
British Council: British-German Academic Research Collaboration for work on the phonetics of rhythmic and prosodic systems and the conduct of conversation. 1996 (£3,000). John Local. [?]
ESRC Grant R000221880, 1996-7 (£28,920): A declarative account of deletion phenomena in English phonetics and phonology. P.I. Richard Ogden. Outcome: a corpus-based declarative, polysystemic analysis of apparent deletion phenomena in English function-words [O], of direct relevance to the present project. 
In addition,  John Local has collaborated with Ken Stevens and his team at Sensimetrics in the development of the Hlsyn quasi-articulatory synthesis system. 

A.3 EXTERNAL LINKS
Links to: BTL, Infovox, Sensimetrics, Keller, UEA, SOLE?

A.4 CONTRIBUTION
Through their collaborations with Industrial concerns such as BT Laboratories and Infovox, the partners have shown their ability to make a contribution to competitiveness.  Through their commitment to open source, many workers in the U.K. and the world now have access to free tools for speech research.

A.5 SITE FACILITIES
All sites enjoy state-of the-art research facilities such as unix machines running speech analysis software, including ESPS and waves+, a variety of speech synthesisers, software for statistical analysis and excellent recording facilities, with comprehensive technical support. Specialised perceptual testing facilities are available at Cambridge and UCL. All laboratories have excellent contacts with researchers in speech and language in other departments and units in their respective universities.

PART B PROPOSED RESEARCH AND ITS CONTEXT

B.1  BACKGROUND
Speech synthesised by rule has yet to make a significant impact on information systems, despite continued advances in textual analysis, pronunciation, signal generation, and voice quality. A recurrent complaint is that contemporary synthetic speech still sounds unnatural, as if it could not have been produced by a human speaker, and long sentences are often hard to understand. It suffers from unexpressive and often inappropriate prosody, and poorly modelled rhythm and coarticulation. Even when the grammatical analysis is right, one string of words can sound good, while another with the same grammar does not. These failings arise from the poverty of linguistic representation underlying the utterance to be produced, as well as a fundamental lack of attention to systematic fine detail in human production, much of which proves to be perceptually salient (see below). Segmental intelligibility data illustrate the scale of the problem. Heard in noise, most synthetic speech loses intelligibility much faster than natural speech: natural speech is about 15% less intelligible at 0 dB s/n ratio than in quiet, whereas for isolated words or syllables, Pratt (1986) reported that the intelligibility of typical synthetic speech drops by 35%-50%. We can expect similar results today.
ProSynth aims to rectify these flaws by taking an integrated prosodic (i.e. structure-based) approach to speech synthesis in a declarative computational framework. ESPRC-funded (part-time Oct. ‘97; full-time since April ‘98) and described by Ogden et al (attached), ProSynth’s core is a phonological model that allows structurally important distinctions to be made, even when their phonetic effects are subtle. The prosodic hierarchy is a grammatically-informed prosodic tree whose highest level is the intonational phrase (IP) and whose lowest elements are phonetic features grouped into syllabic constituents (onset, rhyme, [nucleus, coda]). Intermediate levels are syllable, foot and accent group. The tree builds on Nespor & Vogel (1986) and House & Hawkins (1995), while much of the phonological-phonetic model owes its antecedents to YorkTalk (Local, 1992; 1993; 1995a & b; Ogden, 1992; Local & Ogden, 1997) and computational phonology (e.g. Bird 1995).
In the current grant period, intonation modelling focused on simple patterns within IPs of differing internal structure in a discourse-final context. ProSynth’s phonological structure accounts for much of the systematic phonetic variation which affects f0 curves exemplifying specific pitch accents. For example, the "peak" in H*L pitch accents is often more of a plateau (with a duration) than a simple peak, and the properties of this plateau relate to the onset structure of the accented syllable; and alignment of turning-points in the f0 contour provides phonological justification for treating the foot as the primary domain of the contour rather than the accented syllable (HDH 99). Timing work compared YTs syllable-based compositional principles (e.g. constituent overlay, Ogden & Local 1996) with CART-based statistical analysis on segments (Riley 1992). Segments were distinguished by their position in the prosodic hierarchy. Currently, the segment-based and syllable-based approaches are being integrated. Work on spectral detail has demonstrated the perceptual salience of structure-dependent differences in excitation type at boundaries between sonorants and obstruents (H&H99) and ongoing work on resonance effects shows complex acoustic differences in formant frequencies due to /r/ vs. /l/ that spread (in decreasing strength) several syllables earlier than the /r/ or  /l/ itself, especially when it is in an unstressed syllable (cf. Tunley 1999 also). Factors that limit the spread are being examined. Preliminary tests (http://kiri.ling.cam.ac.uk/prosynth/percept.html; Ogden et al., attached) show that acoustic-phonetic distinctions that result from ProSynth modelling are perceptually salient, as one would expect from other research (e.g. Simpson, 1992; Hawkins & Slater, 1994; Manuel, 1995; Zsiga, 1995). For intonation, naturalness improved, and for duration and segmental quality, intelligibility in noise increased. Other tests involved asssessing speed of comprehension under conditions of high cognitive load while varying f0; these need more work.
The above work required significant infrastructure. We recorded, parsed and labelled structurally-relevant phrases, unavailable from any other source, coded them as XML files, and wrote software for their use. The software architecture is based on two key principles: an open framework where the internal representations and processing are easy to study, and the use of a declarative rule formalism for the codification of knowledge for phonetic interpretation. The first allows the system to be developed across sites and makes it accessible to other researchers. The second allows rule sets to be combined without clashes of rule ordering.  Most software is available as a Windows application for other researchers to experiment with. XML-file information is used for MBROLA synthesis. Software for perceptual tests includes PROCSY (Heid and Hawkins, in press, attached), a fast copy-synthesizer that exploits  Sensimetrics’ HLsyn quasi-articulatory formant synthesizer (Bickley et al, 1997). It allows manipulation of spectral detail as well as f0 and timing. Procsy is already partly rule-driven: vowels and approximants are copy-synthesized from the acoustic signal, while obstruents, nasals, and phonetic correlates of stress are synthesized by rule.
The focus in the current research on single intonational phrases and phonologically-determined distinctions between syllables (heavy-light, strong-weak) has produced linguistic structures that are already more subtle than in standard TTS systems, with demonstrable gains in intelligibility and/or naturalness. Other influences on intonational and segmental aspects of connected speech have different phonetic effects that we also expect to be fundamental to intelligibility, especially under adverse conditions. Amongst these are the phonetic consequences of foreground and background information, or how predictable and central to the main theme the information is. This can reflect marking knowledge that speaker and listener share (cf. Lindblom 1990). Future research will concentrate on two types of foregrounding and backgrounding.

B.2 PROGRAMME
B.2.1 Aims and Objectives
This proposal continues the main ProSynth aim of producing synthetic speech that is robust in adverse listening conditions. Current work will be extended to systematic variation in speech that is caused by differences in discourse function. The specific objectives are:
1. To codify and implement the phonetic and phonological knowledge which is required to generate synthetic speech that reproduces the natural expressiveness and variation of spoken discourse. This includes the phonetic consequences of variation due to shift in focus, foregrounding/backgrounding, word predictability and parenthetical expressions.
2. To integrate systematic temporal, intonational, and segmental details for the structures developed into a system-independent description.
3. To evaluate the model in both concatenative and formant synthesis using perceptual criteria of naturalness and robustness.

B.2.2 Overview
We shall model two types of foregrounding and backgrounding: variation in informational focus (deaccenting, including the “given-new” distinction) and parenthetical expressions. For example, I’m going to show you a lime green rug would be stressed on rug in a neutral context; on lime after an olive green rug, and I after a different speaker. Such distinctions have complex phonetic consequences, affecting pitch, timing and rhythm, and segmental articulatory precision (e.g. Nooteboom & Kruyt 1987, Fowler & Housum 1987, Eefting 1991, Hawkins & Warren 1994, Fowler, Levy & Brown 1997, Bard et al. 1995, 1999). The nasal of the function word I’m can assimilate with the following /g/, while in lime, a content word, it cannot. Patterns of devoicing and lenition also vary according to tempo, syllable position and sentential focus. XX When catapult is spoken fast, the acoustic properties of the first syllable change much less than those of the second syllable, which may be devoiced and raised to the extent that it is primarily an alveolar affricate. What happens spectrally to cat when spoken fast can probably be largely modelled as target undershoot; whereas entirely different targets must be specified for tap. XX Word frequency and predictability are known to have implications for phonetic detail, such that more frequent and more easily predicted words contain more reduced sounds (van Bergem 1993, Wright, to appear). I for example may be less diphthongal that lime. Function words and content words behave differently; these will be analysed and modelled separately. Changes in focus produce large changes in f0 and local durations that could affect overall articulation rate; changing the status of the sentence from main message to a parenthetical aside produces large changes in articulation rate. The proposed new work will adopt the museum guide scenario, to gain maximum benefit from SOLE’s research (www.cstr.ed.ac.uk/projects/sole.html), at present EPSRC-funded concurrently with ProSynth. 
We will investigate the temporal and spectral properties of function words in a variety of rhythmic contexts. We will investigate the representation of /l/ and /r/ resonance effects in higher-level prosodic structures. We will extend the current intonation model to include the systematic variations which occur in a set of defined connected speech contexts. We will investigate the AG and Foot as possible candidates for structuring variation in articulation rate, and the linguistic factors which determine it. We will extend and enrich our phonological structures so as to handle longer utterances, incorporate syntactic information, and model resonance effects more accurately.
B.2.3 Materials
The materials underlying our proposed research will be annotated spoken corpora: some adapted from existing sources and some recorded specially.
WP1.  The project database will consist of a series of texts constructed to allow for the integrated analysis, modelling and synthesis of grammatically and discourse conditioned systematic variation in prosodic, temporal, spectral and segmental features. Material for each of the datasets will be gathered from five speakers of the same variety (2 female, 3 male). One speaker (male) will be common to all the datasets for comparison and in order to facilitate the generation of speech signal that is coherent and sounds like it comes from one speaker. The database texts will use a museum context. This will allow the material to be framed in an appropriate discourse context, which will facilitate both production and testing. The texts will employ controlled variation in hypo- hyper-speech styles, information focus, rate of articulation frequency and predictability of content words and variation in prepositions, determiners and Pronoun+Auxiliary verb structures. 
WP2. The annotation scheme, based on XML, developed in ProSynth1 will be extended to incorporate additional mark up of discourse structure.  The parsing procedures which add the XML annotations will be extended to accept text marked up for rhetorical structure compatible with the formalism developed within the SOLE project.  The annotation scheme and parser may also need to be extended to incorporate further morphological and prosodic features.  The project database will be annotated by combined automatic and hand-labelling.
WP3.  As far as is useful to the project and as time allows, we will adapt existing databases into the common formalism.  Possibilities will depend on the availability of data, but we would be interested in the MARSEC () corpus, the PROSICE corpus (Fang & Huckvale), and …

B.2.4 Measurement and Analysis
Comparisons at all linguistic levels will focus on the phonetic consequences of varying in focus and rate in foregrounding vs. backgrounding information. Analyses will be divided in terms of long- vs. short-domain variables. Long-domain analyses focus on individual and collocated IPs; short-domain analyses focus on single words. Intonation and articulation rate are measured as long-domain variables; short-domain analyses distinguish function from content words (mainly restricted to nouns). Resonance effects arise in both long and short domains.
WP4: The database material will be analysed using the techniques that have proved successful in the current grant period. For temporal parameters: regression-tree analysis alongside other well-established linear statistical methods; formant-frequency and centre-of-gravity analyses for spectral differences; and the use of the general linear model to derive intonational and rate-of-articulation parameters. Function words: patterns of spectro-temporal variability will be established for determiners, pronoun + auxiliary structures and selected prepositions. Content words: patterns of spectro-temporal variability in mono- and polysyllabic nouns, including consonant lenitions and deletions, and vowel quality under influence of /r/ and /l/ when conditioning sound is variously in unstressed vs. stressed, and heavy vs. light syllables. Intonation: range and patterns of varibility for pitch accents over one or more IPs will be established. Articulation rate: Phonological words will be identified; variation rate in words, accent groups and feet will be analysed.
<<? is this redundant now ?>>
Function/content words
1.1	Design and record a database (5 speakers) containing a representative set of function words in a variety of metrical contexts. Measureable objective: an elaborated parametric labelling system; a labelled database containing function words in a systematically varied range of structures.
1.2	Conduct an analysis of the material in the database using and expanding techniques already established for the current database. These include regression tree analysis, alongside other well-established linear statistical techniques. Establish patterns of systematic variability.  Measureable objective: a set of phonetic descriptions of function words, including descriptions of ranges of variability.
1.3	Modify the parser so as to (1) identify function words (2) generate appropriate linguistic structures for them. Measureable objective: a parser capable of generating phonological structures which integrate syntactic and phonological constraints.
1.4	Establish statements of phonetic interpretation which model the appropriate temporal and spectral details.  Measureable objective: a device-independent set of phonetic interpretation statements which can be implemented in eg. HLsyn and/or MBROLA.
1.5	Perceptual tests. Measureable objective: improved intelligiblity under adverse listening conditions. Greater ‘naturalness’ of synthetic speech.
<<? section about intonation & workpackages >>
Intonation
Concentrating on discourse-final contexts has allowed us to identify the important phonetic parameters which define a given pitch accent, their relation to phonological structure, and preliminary information on f0 scaling of accent groups within an IP. We will now extend our intonation model to include the systematic variations which occur in a set of defined connected speech contexts.  Specifically, we will study parenthetical expressions and placement of linguistic focus, to identify the phonological (pitch accent selection) and phonetic (temporal alignment and f0 scaling) consequences of foreground and background status.
•	Within paragraph annotated for discourse structure (SOLE??) relate the structure to PA selection (and the phonetic details of alignment etc)
•	Look at sentences containing several IPs: scaling, alignment, PA selection etc in relation to focus structure.
•	Specific modelling of non-final IPs: different tonal configuration (PA) and scaling in relation to final IPs.  Will involve adding new level of structure: a super-domain such as the sentence within which IPs are organised.
•	For pairs of sentences matched for content and structure, but differing in focus: investigate effects on f0 scaling and temporal alignment of tonal turning-points
•	Microprosody???
Steps and measurable objectives:
2.1	Database.  Design and record database (3 speakers): text of connected speech (long paragraph, gallery guide scenario?) exemplifying variation in focus domains, parentheses B clear differences between given and new information B over representative  range of structures. (Controlled data)
2.2	Explore existing spoken database material (public domain) for comparable phenomena (Data for verification of observed patterns). Measurable objective: representation which relates prosodic hierarchy to the discourse phenomena under investigation.
2.3	Conduct analysis of d/base material using and expanding techniques already established for current d/base (general linear model)  Measurable objective: establish range and patterns of variation
2.4	Establish rules for phonetic interpretation which model systematic variation in f0 and articulation rate.  Measurable objective: device-independent rules for synthesis
2.5	Perceptual tests: adverse listening conditions.  Measurable objective: higher intelligibility/faster processing of correctly modelled intonation and artic rate in synthesis.
<<? section about articulation rate and work packages?>>
Articulation rate
It has been shown that articulation rate in connected speech follows a systematic pattern of a progressive slowing down throughout the intonation phrase (Dankovičova 1997,1999). The results reported relate to phonological words; we shall investigate also the AG and Foot as possible candidates for a temporal unit of structuring articulation rate variation (‘local rate variation’), and the linguistic factors which determine it, such as position within IP, word length and word class (Crystal & House, 1990; O’Shaughnessy, 1995).
4.1	Identification of phonological words in the databases
4.2	analysis of variation in rate across (i) phonological words; (ii) accent groups; (iii) feet (statistical analysis…)
4.3	design of model for articulation rate variation within IP (phonetic interpretation)
4.4	perceptual testing

B.2.5 Modelling and Integration
<<? parts of B.2.4 should be here and in B.2.5 ?>>

B.2.6 Implementation
To implement the new integrated model in synthesis will require design of the interface between this model and a range of synthesisers. Since one aim of the project is to design a device-independent system, the implementation will be sufficiently modular to allow both concatenative and formant synthesizers to be used as user-interfaces. The phonetic structure will be employed either to drive unit-selection for concatenative synthesis or to set parameters for formant synthesis. For example, we will develop PROCSY/HLsyn into a fully rule-driven synthesis system. This will involve rewriting PROCSY using Pro-XML to make it fully compatible with other ProSynth software. Also the Prosynth Windows application will be further developed to {bla bla bla – Mark??}.
The implementation will be essential for testing the integration of different aspects of knowledge in our model. The integrated model will be used to synthesise the original database text. A new text-database, similar in word-class content and metrical structure to the original database, will be designed, synthesised and used as independent test material. 
WP? Procsy/HLsyn will be developed as ProSynth’s formant synthesizer. Sensimetrics has agreed to release the HLsyn source code for rewriting onto a Unix platform, rather than the current PC. HLsyn is an appealing platform for formant synthesis by rule because it allows the complex acoustic consequences of vocal tract dynamics at segment boundaries to be easily synthesized, and because its parameters can straightforwardly be used in order to translate linguistic structure into acoustic specifications. Procsy will be rewritten from its current stand-alone state, based on Python, into PRO-XML for full compatibility with other ProSynth software. Some parameters are already assigned values by rule (see §2A). Values for parameters that are currently added by copy-synthesis (e.g. formant frequencies) will be calculated from target values, using principles of temporal overlaying developed in YorkTalk.

Resonance effects

3.2	Knowledge of the spread of /r/ and /l/ resonance effects gained form the current grant-period will be included: a research question is to what extent these can be represented in the higher-level prosodic structure vs. at featural or segmental levels. Both levels of representation may be necessary, depending on the balance between prosodic and segmental constraints on the spread of resonance effects. Durational values for content words will vary with changes in focus [xxor similar term] due to changes in the prosodic structure, and with whether the phrase has main or parenthetical status in the utterance. Local segmental and stress conditions determine whether durations can simply be shortened (nonlinearly, cf. Klatt 19xx) or whether entirely different spectral properties are needed. For example, when catapult is spoken fast, the acoustic properties of syllable 1 (cat) change much less than those of syllable 2 (tap), which may become devoiced and raised to the extent that it is primarily an alveolar affricate. What happens spectrally to cat when it is spoken fast can probably be largely modelled as target undershoot; whereas entirely different targets must be specified for tap. Such differences can be efectively produced by PROCSy/HLsyn; they would require selection of a different unit form the concatenative synthesis database.

B.2.7 Evaluation
We will continue our current practice of evaluating in 3 ways. (1) Close listening: as phoneticians, we are well placed to use this method, but it needs supplementing with objective methods. (2) Fit between measured and output parameter values: most useful for duration and in developing the prosodic tree, this method must be used with perceptual measures to indicate whether deviations from the measured value are important. (3) Perceptual tests with human subjects. This is our major method. ProSynth’s philosophy is that robust intelligibility will be achieved when we can model those aspects of naturalness that signal linguistic structure. This necessitates tapping comprehension (cf. Duffy and Pisoni 1992). Thus we are not seeking to improve general impressions of naturalness, which might include, e.g., a pleasant voice quality. Rather, the acoustic properties we aim to model and that set ProSynth apart from other systems are not always the features that are most immediately obvious to the casual listener. But they should make the speech better understood in adverse conditions, especially when sentences are long or syntactically complex. E.g. listeners often report no differences due to resonance-effect modelling, yet produce fewer id errors. Thus, though popular MOS tests (cf. http://fonpc1.hum.uva.nl:4711/ [ESCA Jenolan workshop]) provide useful information for systems in public use, and phoneme identification from nonsense syllables is likewise useful, neither is appropriate for ProSynth evaluation. We need the tight experimental design and statistical analyses that MOS tests allow (cf. Johnston 1997, Arden 1997), and at times a combination of subjective and objective assessment will be valuable (cf. Hirst et al. 1998), but our focus on (a) subtleties of linguistic structure and (b) robustness, means that to test our present work we can (a) rarely use text material developed by other groups, and (b) most of our tests must assess processing rather than opinion, usually in some type of adverse condition. Our tests must allow stringent comparisons between specific linguistic structures synthesized with and without regard to their structural nuances, and they must usually require word recognition.
Some tests will continue present methods of measuring intelligibility in randomly-varying noise: f0 and duration manipulations run on available concatenative systems; spectral manipulations using PROCSY. Fully factorial designs to compare contributions of f0, duration, and spectral changes will be run using PROCSY or its rule-driven development, PROSY?!. Naturalness tests will complement as necessary. A limited amount of prosodic evaluation (especially focus variation) will remove segmental information (Sonntag and Portele 1998). However, this method is inappropriate when prosody interacts with phonological structure, so we expect to use dual-task tests for most prosody evaluation. Since dual-task tests are costly and views about their value conflict (cf. Johnston 1997; Sonntag, Portele & Haas 1998), their worth will be assessed, with careful choice of materials and design, building on methods piloted in this grant period and in the cited literature; we expect to find them crucial for assessing intonation in particular. Current focus is on errors and reaction times to respond to the truth value of statements; we will also explore concurrent cognitive tasks such as simple arithmetic (cf. Sonntag, Portele & Haas 1998) and following instructions (including on-line e.g. by tracking an object on a screen) consistent with the view that tests should be relevant to the expected usage (cf. DISC:  http://www.elsnet.org/disc/, & Inger Karlsson, pers. comm.)
COLLEAGUES: two alternative workpackages are offered below, each for three years, the difference being whether the RA is full- or half-time. The full-time position should allow some general statistical support work from this RA for all sites during the second and third years, e.g. on general database analyses. I think some of you might find that helpful? The half-time post would be able to offer a lot less, but still some -- see WPs below. The amount of testing, and its comprehensiveness, would also be less ambitious for the half-time appointment. Before deciding which you prefer,  you might like to read the part on Justification near the end of this document. COMMENTS PLEASE.
Workpackages for evaluation: FUll-time RA
year 1
Search literature: 1 calendar month
Decide main type of dual-task method; program Tucker-Davis and computer, especially for dual-task designs (3 months)
Pilot dual-task and other comprehension designs (4 expts expected, includes analysis and trouble-shooting): 8 months
year 2
Test, analyse, and write reports for all sites at beginning of year 2: 5 months
Statistical analysis/support for all sites: 7 months
year 3
Test, analyse and write reports for all sites at beginning of year 3: 5 months
Statistical analysis/support for all sites:  3 months
Final perceptual tests and report-writing: 4 months
OR:  Workpackages for evaluation: half-time RA
year 1
Search literature: 2 calendar months
Decide main type of dual-task method; program Tucker-Davis and SGs, especially for dual-task designs (5 months)
Pilot dual-task and other comprehension designs (4 expts expected, includes analysis and trouble-shooting):  5 months
year 2
Partial perceptual tests; analyse and write report for all sites at beginning of year 2: 5 months
Continue to pilot dual-task and other comprehension designs: 7 months
year 3
Partial perceptual tests; analyse and write report for all sites at beginning of year 2: 5 months
Statistical analysis/support for all sites:  1 month
Final perceptual tests: 4 months

<<? this is just a summary - will be edited out ?>>
Work package summary
Total person hours available: 36 x 3 RAs = 108 person months;
6 PIs @ 5 hours per week = 1 month per year = 18 person months

1. Keep packages as simple and few as possible.
2. No distinction between sites for what’s done.
3. Each workpackage is adminisitered from one site.

1. Database collection: 	Calendar Months: 4	Person Months: 20
2. Database analysis: labelling: CM:			PM: 20
3. Implementation of model	CM:			PM: 50
4. Perceptual testing		CM:			PM: 30
5. Final write-up: 		CM:			PM: 6

B.3 TIMELINESS & NOVELTY
Despite engineering advances in text to speech synthesis (TTS) problems of intelligibility and naturalness persist, particularly when longer utterances are generated. For most of the past decade we have been involved in developing TTS systems (see section A2). Our combined experience leads us to propose an innovative model of computational phonology and phonetic interpretation that integrates timing, intonational information and systematic segmental variation into a coherent whole to produce synthetic speech that sounds natural and is highly intelligible in quiet and in noise. We believe the ProSynth approach to speech synthesis is fundamentally complementary to other work in the area in the U.K.  Our work has focussed on the linguistic and phonetic detail of the realisation of speech: whereas others aim to build statistical approximations that cover a whole corpus, we are the only group looking at the detailed phonetic consequences of linguistic contexts. Moreover, ours is the only work in the U.K. that systematically uses perceptual experiments to test whether the knowledge we have gained on detailed phonetic effects leads to improvements in cognitive processing of the synthetic output. The results of our work are open and available, in a portable knowledge representation, to the more systems-based projects such as Festival or SOLE.

B.4 RELEVANCE TO BENEFICIARIES
The proposed research will increase our expertise and experience in the increasingly important area of speech technology. It will be of benefit to companies and organisations with an interest in speech synthesis and speech technologies. Improvements in the naturalness and variety of synthetically generated speech will greatly increase the applicability of this technology. Commercial applications for high-quality, natural sounding speech synthesis include the next generation of dialogue systems, information enquiry services and machine translation systems. The work will also increase the research community's expertise and experience in the increasingly important domain of speech technology: developer of TTS systems will benefit from the databases, statistical results and prosodic knowledge obtained. Workers who evaluate speech synthesis systems will benefit from a new paradigm for the evaluation of prosody. The testing of the linguistic model will also contribute to the development of linguistic theory. Further beneficiaries could include users of speech protheses.

B.5 DISSEMINATION AND EXPLOITATION
We consider the dissemination of the fruits of our research activity to be very important, and accordingly we have dedicated work packages built into our research programme solely for the purposes of making our work available to others.
WP? Interfaces with Festival and Laureate.  Involves importing and exporting linguistic structures to be compatible with the higher and lower levels of these systems.
WP? Interfaces with text generated by SOLE project  Involves accepting XML marked up text generated by the ILEX text generation package as used in SOLE.
WP? Incorporation of HLSyn as alternative output channel.  Involves recompiling HLSyn for Unix and PC platforms, integrated with existing ProSynth software.
WP? Web-based demonstration system.  Involves providing web access to ProSynth software for on-line demonstrations.
WP? Software portability.  Involves configuration of software to maximise portability across computing platforms.
WP? Final Report
Interim and final results will be disseminated over the web, through conference presentations and their published proceedings (eg. International Conference on Speech and Language Processing; Eurospeech, International Congress of Phonetic Sciences; miscellaneous European Speech Communication Association and domestic workshops), and EPSRC reports. Papers will be submitted to refereed journals e.g. Computer Speech and Language, J. Phonetics, and J. Acoustical Society of America.

B.6 JUSTIFICATION OF RESOURCES
The RAs appointed to this project should be at post-doctoral level, since they all require a combination of advanced programming skills and linguistic knowledge. The York RA requires advanced skills in declarative and/or computational phonology, preferably combined with a training in phonetics.
Done properly, evaluation is time-consuming and challenging, still demanding significant pilot work especially to test intonation in task-appropriate ways. Funding in the current grant period allowed us to demonstrate gains in intelligibility and naturalness for factors currently modelled, and hence the worth of our approach, but we have only begun exploring the innovative designs required to evaluate intonation, and it has not allowed us to vary f0, duration, and spectral quality factorially. Intonation evaluation is especially challenging, and is expected to take at least one year before it is operational. To allow tests to start in year 2, a dedicated RA is required in year 1. This RA will probably be psychologist,  since experience in test design and statistics is essential. Dr. Heid can provide programming help.

Equipment. An Apple PowerBook is requested for portability of demonstrations, running software for acoustic analysis, writing up papers for dissemination. The York site uses predominantly Macintoshes and has support for them. 
Consumables. The DAT recorder and CD-ROM writer and CD-ROMs for data storage are needed for the database recording and analysis. A supported network connection is requested.
Travel. The conferences listed are directly relevant to the project. ICPhS (Barcelona) is the premier experimental phonetics forum; Eurospeech is an important venue for work in speech technology; ICSLP is a major international conference for IT, speech and language. BAAP provides the best national forum for phonetics research. Attendance at these meetings will benefit the project both because of the opportunity to publish and disseminate our results, and because of the valuable contacts with researchers in related fields.

Travel:
ICPhS, Barcelona 2003: air fare £300, conf. fees £200, accom. £220 x2 people = £1440
Eurospeech x 2 (1 person each time): Ditto. £1440.
ICSLP x 2 (1 person each time, assume intercontinental): air fare £500, conf. fees £200, accom. £220 = £1840
Domestic conferences/workshops 1/year @ £500/each = £1500
Travel for project meetings 4/year, £120 each, (i.e. 6 meetings, 4 requiring travel)
Overnight stays for annual jamboree, 2 per person at £60 each.

REferences
Arden, P. (1977) Subjective assessment methods for test-to-speech systems. In Evaluation in Speech and Language Technology (Salt CLub Workshop, University of Sheffield), 9-16. 
Bard, E.G., Sotillo, C., Anderson, A.H., Doherty-Sneddon, G. & Newlands, A. (1995) The control of intelligibility in running speech. In K. Elenius & P. Branderud (eds.) Proceedings of the XIIIth International Congress of Phonetic Sciences. 4, 188-191.
Bard, E.G. & Aylett, M.P (1999) The dissociation of deaccenting, givenness, and syntactic role in spontaneous speech. In Proceedings of the ICPhS, San Francisco  xx[please add correct ref from other refs] 1753-1756.
Bergem, Dick R van (1993). Acoustic vowel reduction as a function of sentence accent, word stress and word class. Speech Communication 12, 1-23.
Bickley, C.A., Stevens, K.N., & Williams, D.R. (1997) A framework for synthesis of segments based on pseudoarticulatory parameters. In J.P. van Santen, R.W. Sproat, J.P. Olive, and J. Hirschberg (eds.), Progress in Speech Synthesis, 211-220. New York: Springer.
Dankovicova. J. (1997) xx
Dankovicova. J. (1999) xx
Duffy, S.A. & Pisoni, D.B. (1992) Comprehension of synthetic speech produced by rule: A review and theoretical interpretation. Language and Speech 35, 351-389.
Eefting, W. (1991) The effect of “information value” and “accentuation” on the duration of Dutch words, syllables, and segments. Journal of the Acoustical Society of America 89, 412-424.
Fosler-Lussier, Eric and Nelson Morgan (1998). Effects of speaking rate and word frequency on conversational pronunciations. In ESCA Tutorial and Research Workshop on Modeling pronunciation variation for automatic speech recognition.
Fowler, C.A. & Housum, J. (1987) Talkers’ signaling of “new” and “old” words in speech and listeners’ perception and use of the distinction. Journal of Memory and Language 26, 489-504.
Fowler, C.A., Levy, E. & Brown, J. (1997) Reductions of spoken words in certain discourse contexts. Jounrnal of Memory and Language 37, 24-40.
Hawkins, S., and Warren, P. (1994) Phonetic influences on the intelligibility of conversational speech. Journal of Phonetics 22, 493-511.
Heid & Hawkins 1999 = ICPhS paper
Hirst, D.J., Rilliard, A. & Aubergé, V. (1998) Comparison of subjective and objective evaluation metric for prosody in text-to-speech synthesis. In Proceedings of the 3rd ESCA/COCOSDA Workshop on Speech Synthesis, Jenolan Caves, NSW, Australia. 1-4.
Huckvale, M.A. (1999), Representation and processing of linguistic structures for an all-prosodic synthesis system using XML, Proc. EuroSpeech-99, Budapest.
Johnston, R.D. (1997) Towards a reference system for speech technology. In Evaluation in Speech and Language Technology (Salt CLub Workshop, University of Sheffield), 90-97.
Keating, Patricia A., Dani Byrd, Edward Flemming, and Yuichi Todaka (1994). Phonetic Analyses of word and segment variation using the TIMIT corpus of American English. Speech Communication 14, 131-142.
Ladd, D.R. (1996) Intonational Phonology. Cambridge: Cambridge University Press.
Lindblom, B. (1990) Explaining phonetic variation: A sketch of the H&H theory. In W. Hardcastle and A. Marchal (eds.) Speech Production and Speech Modeling. Dordrecht: Kluwer, 403-439.
Nooteboom, S.G. & Kruyt, J.G. (1987) Accents, focus distribution, and the perceived distribution of given and new information: An experiment. Journal of the Acoustical Society of America 82, 1512-1524.
Ogden, R.A. & Local, J.K. (1996) YorkTalk Annual Report 1991. York Research Papers in Linguistics YRPL-96-01. ISSN 1350-2956.
Ogden, Richard (to appear). A declarative account of strong and weak auxiliaries in English. Phonology 16.
Riley, M.D. (1992) Tree-based modelling of segmental durations. In G. Bailly and C. Benoit (eds.) Talking Machines: Theories, Models and Designs.  Amsterdam: North Holland.
Sonntag, G.P., & Portele, T. (1998) PURR—A method for prosody evaluation and investigation.  Computer Speech and Language 12, 437-451.
Sonntag, G.P., Portele, T., & Haas, F. (1998) Comparing the comprehensibility of different synthetic voices in a dual task experiment. In Proceeedings of the Third International Workshop on Speech Synthesis. Jenolan Caves, NSW, Australia. 5-10.
Wright, Richard (to appear).