To get you started with Cunei we will build a small French-English system trained on the freely-available Europarl corpus. More information on the Europarl corpus is available at http://statmt.org/europarl/. Cunei requires Java 6. In addition, your system must have Apache Ant installed to use the automated build scripts. Please make sure both of these components are installed before continuing. These are the only core requirements to run Cunei, but in order to build a new system we will use a few other tools as specified in later sections. The source code for Cunei is hosted in a Subversion repository. If you have Subversion, run the following command to check out the latest version of of Cunei into the directory cunei. Throughout this tutorial we will run commands and reference files relative to Cunei's base directory. Please change your working directory now. The Subversion repository only contains a copy of the source code. (If you are inclined to peek under the covers, feel free to browse the src directory.) Type the command below to run Apache Ant and compile the source code. Assuming all went well, a few seconds later you should be greeted with a BUILD SUCCESSFUL message. Download and extract the French-English Europarl with the commands below. This will create a new directory europarl containing the corpus. Next, use the following code to create a new Perl script europarl.pl. This script will massage the Europarl corpus into a suitable format for our use. While our example is using the French-English portion, the Europarl is available in many language pairs and this script can be used with any of them. Additionally, after completing this procedure once, you will see that the data format we use is very simple. Feel free to grab any other data, but be sure to set aside some out some of the text for development and testing. The following commands will execute the script and create the prepared Europarl corpus in data/corpora/fr-en/raw:

Distributed with the Cunei distribution is a configuration file in the systems/fr-en/default/ directory. Using the provided configuration, we first calculate the expected sentence ratios and then process the raw text into clean, tokenized text:

We will use the GIZA++ toolkit to induct word-alignments from the corpus. Download the latest version of GIZA++ from http://code.google.com/p/giza-pp/ and install it. If the GIZA++ tools are not in your $PATH then set the environment variable $GIZA_HOME to point to the location of your GIZA++ installation. To simply the process, a convenience script is provided to build the appropriate alignments. GIZA++ will take several hours to complete. When the script completes, data/corpora/fr-en/clean/europarl-v5-train.giza will be populated with several files. The files s.A3.final.gz and t.A3.final.gz are the Viterbi word alignments from GIZA++ for P(s|t) and P(t|s) respectively. We will use these files in the next step. In this step we will index the text files we processed earlier. In order to avoid re-processing these files, edit the configuration systems/fr-en/default/config and comment out the lines that begin with 'Processors' using the '#' character as shown below. Now run the following command to index the corpus. Expect this to take about an hour. Once indexing is complete, uncomment the 'Processors.Source.Input.Text' and 'Processors.Target.Input.Text' lines (as shown previously). This will ensure future text files are processed appropriately. The word-alignments from GIZA++ are not weighted. Run the following command to estimate lexicons for P(s|t) and P(t|s). These probabilities are incorporated into the alignments and are also used at run-time to calculate lexical features. We use SRILM to estimate n-gram probabilities. Download the latest version of SRILM from http://www.speech.sri.com/projects/srilm/download.html and install it. If the SRILM tools are not in your $PATH then set the environment variable $SRILM_HOME to point to the location of your SRILM installation. Usually it's best to build a language model from a larger monolingual corpus, but for this tutorial we'll just use the English half of our bilingual corpus to build a 4-gram model. The output of SRILM is then fed to Cunei which indexes and stores it to disk in a format that is faster to load. Once Cunei has indexed the language model, the SRILM file is no longer necessary and may be erased. If you successfully completed all the steps above you should have a working system that can produce translations. Let's give it a whirl!

To check that everything is working, generate the translation lattice with the command below. This will output the many possible translations found for each word or phrase in the test sentence.

Now try generating full-sentence translations with the command below. This will output the top four translations of the sentence. The default parameters are not adjusted for any particular language pair and, thus, are quite poor. Now it's time to remedy that. The following command will optimize Cunei's settings and significantly improve the quality of translations. The test document will be translated over and over, each time adjusting Cunei's parameters in order to produce translation that (as closely as possible) match the human translated document(s). This process will use a lot of memory and may take a few days to complete. Info messages will appear on the console, but the actual output will be logged in the file opt.log. When optimization is complete a new configuration file systems/fr-en/default/config.opt will be created for you with the optimized parameters. From now on, use this new configuration file for translation. Congratulations on completing the Cunei tutorial. Feedback is welcome. If something didn't quite work right or you have suggestions for improvement please send mail to cunei-support@lists.sourceforge.net.

Happy translating!