GALE Phase 4 Chinese BC Parallel Sentences
Authors: Zhiyi Song, Gary Krug, Stephanie Strassel

1.0 Introduction

This file contains documentation for GALE Phase 4 Chinese BC Parallel Sentences.
Along with other corpora, the parallel text in this release comprised
training data for Phase 4 of the DARPA GALE Program. This corpus
contains Chinese source sentences and corresponding English
translations for 63,829 tokens, selected from Broadcast Conversation (BC)
data collected and transcribed in GALE. The audio corresponding to the
source files in this release are distributed separately.

2.0 Package Structure

This package comprises two directories:

data/

The data directory is divided into four branches. These are
    "source_full", "source_selected", "translation_selected", and"index".
    These contain the full source tdf files, selected segments from source,
    selected translation segments files, and index files respectively.

      source_full/
         contains source full documents from which the sentences are
         selected in tdf format

      source_selected/
         contains source sentences selected for transaltion from each
         document in tdf format

      translation_selected/
         contains translated sentences correpsonding to source_selected/ in tdf format

      index/
         index files which indicate which sentences are selected from each
         full document with the first column listing file stems, second column 
	 listing segments in source_selected and third column listing 
	 corresponding segments in source_full

File names refer to the data source, source language and collection date,
e.g.

{SRC}_{PRG}_{LNG}_YYYYMMDD_hhmmss(.fileTypeExtension)

     where - {SRC} is the source ID (e.g., CNN, VOA, etc.)
           - {PRG} is the program ID (e.g., LARRYKING, etc.)
           - {LNG} is the three-letter language ID defined in the
             ISO639-3.  ARB is Modern Standard Arabic; CMN is Chinese,
             Mandarin; ENG is English.
           - YYYYMMDD is the data collection (broadcast) date.
           - hhmmss   is the start time of the program (hh is the hour
             in the 24-hour format)

File stems for source and translation file pairs are the same. Source full files
use the .tdf extension. Source selected files use the sel.tdf extension, while 
translation files use the sel.eng.tdf extension.

docs/

The docs directory contains documentation related to the release.

           docs/README.txt - this file
           docs/doc_list.txt - inventory of source and translation files
           with the token count for each file
           docs/file_list.txt - inventory of files in this release
           docs/GALE_Chinese_translation_guidelines_sentence_based_V2.pdf - translation
           guidelines
           docs/GALE_TranscriptionTranslationMarkup_V25.xls - explanation
           of special symbols
           docs/program_summary.txt -- audio programs in this release
           docs/TDF_format.txt -- TDF format description

3.0 Contents

This release includes 109 source-translation document pairs, comprising
63,829 tokens of Chinese source and their English translation. Data is
drawn from 17 distinct Chinese broadcast conversation (BC) sources.
BC programming is more interactive than traditional broadcast news
sources and may include talk shows, interviews, call-in programs and
roundtables.

The following table is a summary of data included in this corpus.

      source_lang   genre    files	sentences   source_tokens
      -----------------------------------------------------------
      Chinese         BC       109        1,504           63,829

Token counts are expressed in terms of characters and are taken from the source data.

The file called docs/file_list.txt contains a complete list of files in the
package. The file docs/doc_list.txt contains the inventory of translated docuemnts
with the source token count for each file.

3.1 TDF Format

Source data and translations are distributed in TDF format. TDF files are
tab-delimited text files containing one segment of text along with meta
information about that segment. Each field in the TDF file is described in
docs/TDF_format.txt.

A source TDF file and its translation are the same except that the
transcript in the source TDF is replaced by its English translation.

3.2 Encoding

All data are encoded in UTF8.

4.0 Translation Pipeline

Before audio files can be translated, they must be transcribed. The files
in this release were transcribed by LDC staff and/or transcription vendors
under contract to LDC. In addition to producing a verbatim transcript,
transcribers also indicate sentence boundaries. Sentence boundaries and
overall transcript quality were verified by LDC staff before sending files
out for translation.

After transcription and SU annotation, files were fed into the sentence-seletion scripts provided by IBM and SRI for selection.  The selected sentences are then reviewed by LDC annotators so that sentences that are not in the target language or dialect, or have formatting problems, or whose content is entirely unsuitable, such as commercial, religious, etc. are rejected. LDC also ran duplicate detection on all files prepared for translation and excluded any duplicate files from the translation pipeline.

After review, selected sentences from the same documents were grouped into selection file which was reformatted into a human-readable translation format and were assigned to translation vendors under contract to LDC. Source full documents with selected sentences highlighted were also provided to translators for context. Translators followed LDC's GALE Translation guidelines, which describe the makeup of the translation team, the source data format, the translation data format, best practices for translating certain linguistic features (such as names and speech disfluencies), and quality control procedures applied to completed translations. Transcribers and translators used special markup to indicate particular linguistic features, for instance unintelligible speech, partial words and typos in the transcript; these uses are described in the documentation accompanying this release.

After translations were completed, bilingual LDC staff performed
quality control by selecting a proportional sample from each delivery and
scrutinizing it for several kinds of mistakes, as described in the
translation guidelines. Low quality translations were returned to
the translators for revision.  After quality control is complete,
translation files were validated and reformatted into the release format.

5.0 Sanity Checks

LDC performed the following corpus-wide checks and corrected all errors
found:

      -- Number of source segments matches number of translation segments
         for all files (except full source text)
      -- Timestamps are identical between selected source and translation
      -- All non-blank source segments correspond to non-blank translation
         segments
      -- All translation files have a corresponding full source file
         selected
         source file, and index file
      -- All files contain only UTF-8 encoded characters, although they may
         contain non-ascii characters such as Western European characters
      -- Punctuation in translations is ASCII punctuation

6.0 Acknowledgement

This work was supported in part by the Defense Advanced Research Projects
Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this
publication does not necessarily reflect the position or the policy of the
Government, and no official endorsement should be inferred.

7.0 Content Copyright

Portions © 2008 Beijing TV, China Central TV, Hubei TV, © 2008, 2015 Trustees of the University of Pennsylvania

----
README Created 29 June, 2011 Gary Krug
       Updated 31 January, 2011 Zhiyi Song
       Updated 20 February, 2012 Stephanie Strassel
       Updated 20 February, 2012 Zhiyi Song
       Updated 1 June, 2012 Zhiyi Song
