| <html> |
| |
| <head> |
| <meta http-equiv="Content-Language" content="en-us"> |
| <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> |
| <title>New Page 18</title> |
| </head> |
| |
| <body> |
| |
| <h1>UnicodeTools</h1> |
| <p>This file provides instructions for building and running the UnicodeTools, which<br> |
| can be used to:</p> |
| <ul> |
| <li>build the Derived Unicode files in the UCD (Unicode Character Database),</li> |
| <li>build the transformed UCA (Unicode Collation Algorithm) files needed by ICU.</li> |
| <li>run consistency checks on beta releases of the UCD and the UCA.</li> |
| <li>build 4 chart folders on the unicode site</li> |
| </ul> |
| <p><font color="#FF0000"><b>WARNING!!</b></font></p> |
| <ul> |
| <li>This is NOT production level code, and should never be used in programs.</li> |
| <li>The API is subject to change without notice, and will not be maintained.</li> |
| <li>The source is uncommented, and has many warts; since it is not production code, it has not |
| been worth the time to clean it up.</li> |
| <li>It will probably not work on Unix or Mac without changing the file separator.</li> |
| <li>Currently it uses hard-coded directory names.</li> |
| <li>The contents of multiple versions of the UCD must be copied to a local directory, as described |
| below.</li> |
| </ul> |
| <h2>Instructions:</h2> |
| <h3>0. You will need to get ICU4J on your system, using CVS.</h3> |
| <p>The rest of this will assume that you have set up CVS so that you load the ICU4J project into |
| C:\ICU4J<br> |
| <br> |
| You need both the main icu4j and a subproject called unicodetools. See: |
| <a href="http://ibm.com/software/globalization/icu/repository.jsp"> |
| http://ibm.com/software/globalization/icu/repository.jsp</a>. Inside unicodetools, look at com/ibm/text. The |
| main directories of interest are UCD, UCA and utility.</p> |
| <h4>0a. If you are using Eclipse for your IDE, look at the instructions on |
| <a href="http://icu.sourceforge.net/docs/eclipse_howto/eclipse_howto.html"> |
| http://oss.software.ibm.com/icu/docs/eclipse_howto/eclipse_howto.html</a> </h4> |
| <p>Set up Eclipse to build two projects: ICU4J and UnicodeTools:<br> |
| <br> |
| <b>Project Name: </b>ICU4J<br> |
| <b>Directory: </b>C:\ICU4J\icu4j<br> |
| <b>Default output folder = </b>ICU4J/classes<br> |
| <br> |
| <b>Project Name: </b>unicodetools<br> |
| <b>Create project from existing source: </b>C:\ICU4J\unicodetools<br> |
| <b>Default Output Folder: </b>unicodetools/classes<br> |
| <br> |
| After Eclipse is set up with these, exclude certain files from unicodetools:<br> |
| <br> |
| Right-Click UnicodeTools > Properties > Java Build Path > Exclusions<br> |
| com/ibm/rbm/<br> |
| com/ibm/text/utility/UnicodeMapInt.java<br> |
| com/ibm/text/utility/TestUtility.java<br> |
| com/ibm/text/UCD/GenerateThaiBreaks-old.java/<br> |
| com/ibm/text/UCD/ProcessUnihan.java/<br> |
| com/ibm/text/UCA/WriteHTMLCollation.java/<br> |
| <br> |
| UnicodeTools must also include the ICU4J project, with<br> |
| <br> |
| Right-Click UnicodeTools > Properties > Java Build Path > Projects</p> |
| <h3>1. In UCD, you must edit UCD_Types.java at the top, to set the directories for the build:</h3> |
| <p>public static final String DATA_DIR = "C:\\DATA\\";<br> |
| public static final String UCD_DIR = BASE_DIR + "UCD\\";<br> |
| public static final String BIN_DIR = DATA_DIR + "BIN\\";<br> |
| public static final String GEN_DIR = DATA_DIR + "GEN\\";<br> |
| <br> |
| Make sure that each of these directories exist. Also make sure that the following<br> |
| exist:<br> |
| <br> |
| <GEN_DIR>/DerivedData<br> |
| <GEN_DIR>/DerivedData/ExtractedProperties<br> |
| <UCD_DIR>/EXTRAS-Update</p> |
| <h3>2. Download all of the UnicodeData files for each version into UCD_DIR.</h3> |
| <p>The folder names must be of the form: "3.2.0-Update", so rename the folders on the<br> |
| Unicode site to this format. I<span style="background-color: #FFFF00">f the |
| folder contains ucd, then make the contents of that directory be the contents of |
| the x.x.x-Update directory. That is, each directory will directly contain files |
| like PropList....txt</span></p> |
| <h4>2a Ensure Complete Release</h4> |
| <p>If you are downloading any "incomplete" release (one that does not contain a complete set of data |
| files for that release, you need to also download the previous complete release). Most of the N.M-Update |
| directoriess are complete, *except*:</p> |
| <p>4.0-Update, which does not contain a copy of Unihan.txt and some other files<br> |
| 3.1-Update, which does not contain a copy of BidiMirroring.txt</p> |
| <p>Also, make the following changes to UnicodeData for 1.1.5:</p> |
| <p><b>Delete</b></p> |
| <pre>3400;HANGUL SYLLABLE KIYEOK A;Lo;0;L;1100 1161;;;;N;;;;; |
| ... |
| 4DFF;HANGUL SYLLABLE MIEUM WEO RIEUL-THIEUTH;Lo;0;L;1106 116F 11B4;;;;N;;;;; |
| 4E00;<cjk IDEOGRAPH REPRESENTATIVE>;Lo;0;L;;;;;N;;;;;</pre> |
| <p><b>Add:</b></p> |
| <pre>4E00;<cjk Ideograph, First>;Lo;0;L;;;;;N;;;;; |
| 9FA5;<cjk Ideograph, Last>;Lo;0;L;;;;;N;;;;; |
| E000;<private Use, First>;Co;0;L;;;;;N;;;;; |
| F8FF;<private Use, Last>;Co;0;L;;;;;N;;;;;</pre> |
| <p><b>And from a late version of Unicode, add:</b></p> |
| <pre>F900;CJK COMPATIBILITY IDEOGRAPH-F900;Lo;0;L;8C48;;;;N;;;;; |
| ... |
| FA2D;CJK COMPATIBILITY IDEOGRAPH-FA2D;Lo;0;L;9DB4;;;;N;;;;;</pre> |
| <h4>2b. UCA data</h4> |
| <p>If you are building any of the UCA tools, you need to get a copy of the UCA data file<br> |
| from http://www.unicode.org/reports/tr10/#AllKeys. The default location for this is:<br> |
| <br> |
| BASE_DIR + "Collation\allkeys" + VERSION + ".txt".<br> |
| <br> |
| If you have it in a different location, change that value for KEYS in UCA.java, and <br> |
| the value for BASE_DIR</p> |
| <h4>2c. Here is an example of the default directory structure with files. All of |
| the yellow ones should exist</h4> |
| <pre>C://DATA/ |
| |
| BIN/ |
| |
| <span style="background-color: #FFFF00"> Collation/ |
| allkeys-3.1.1.txt |
| </span> |
| GEN/ |
| DerivedData/ |
| <span style="background-color: #FFFF00"> </span><span style="background-color: #FFFF00">UCD/ |
| 3.0.0-Update/ |
| Unihan-3.2.0.txt |
| ... |
| 3.0.1-Update/ |
| ... |
| 3.1.0-Update/ |
| ... |
| 3.1.1-Update/ |
| ... |
| 3.2.0-Update/ |
| ... |
| 4.0.0-Update/ |
| ArabicShaping-4.0.0d14b.txt |
| BidiMirroring-4.0.0d1b.txt |
| ... |
| EXTRAS-Update/</span></pre> |
| <h3>3. Versions</h3> |
| <p>All of the following have "version X" in the options you give to Java (either on the |
| command line, or in the Eclipse 'run' options. If you want a specific version like 3.1.0, then you |
| would write "version 3.1.1". If you want the latest version (4.1.0), you can omit the "version X".</p> |
| <h3>4. Building Files</h3> |
| <ol> |
| <li><b>Setup</b><ol> |
| <li>In Eclipse, open the Package Explorer (Use Window>Show View if you |
| don't see it)</li> |
| <li>Open UnicodeTools<ul> |
| <li>com.ibm.text.UCD<ul> |
| <li>MakeUnicodeFiles.<span style="background-color: #FFFF00">txt</span><p>This file drives the production of |
| the derived Unicode files. The first three lines contain |
| parameters that you may want to modify at some times:</p> |
| <pre>Generate: <b>.*script.*</b> <i>// this is a regular expression. Use .* for all files</i> |
| DeltaVersion: <b>10</b> <i> // This gets appended to the file name. Pick 1+ the highest value in Public</i> |
| CopyrightYear: <b>2006</b> <i> // Pick the current year</i></pre> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Open in Package Explorer |
| <ul> |
| <li>com.ibm.text.UCD<ul> |
| <li>Main</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Run>Run As...<ol> |
| <li>Choose Java Application<ul> |
| <li>it will fail, don't worry; you need to set some parameters</li> |
| </ul> |
| </li> |
| </ol> |
| </li> |
| <li>Run>Run...<ul> |
| <li>Select the Arguments tab, and fill in the following<ul> |
| <li>Program arguments:<pre>build 5.0 MakeUnicodeFiles</pre> |
| </li> |
| <li>VM arguments: |
| <pre>-Xms512m -Xmx512m</pre> |
| </li> |
| </ul> |
| </li> |
| <li>Close and Save</li> |
| </ul> |
| </li> |
| </ol> |
| </li> |
| <li><b>Run</b><ol> |
| <li>You'll see it build the 5.0 files, with something like the following |
| results:<pre>Writing UCD_Data5.0.0 |
| Data Size: 109,802 |
| Wrote Data 109802</pre> |
| </li> |
| <li>For each version, the tools build a set of binary data in BIN that |
| contain the information for that release. This is done automatically, or |
| you can manually do it with the Program Arguments<pre>version X build</pre> |
| <p>This builds an compressed format of all the UCD data (except blocks |
| and Unihan) into the BIN directory. Don't worry about the voluminous |
| console messages, unless one says "FAIL".</p> |
| <p><font color="#FF0000"><i>You have to manually do this if you change |
| any of the data files in that version!</i></font></p> |
| <p>Note: if for any reason you modify the binary format of the BIN files, you also have to bump the |
| value in that file:</p> |
| <pre>static final byte BINARY_FORMAT = 8; // bumped if binary format of UCD changes</pre> |
| </li> |
| </ol> |
| </li> |
| <li>Results in <a href="file:///C:/DATA/GEN/DerivedData"> |
| C:\DATA\GEN\DerivedData</a><ol> |
| <li>The files will be in this directory.</li> |
| <li>There are also DIFF folders, that contain BAT files that you can run |
| on Windows with CompareIt. (You can modify the code to build BATs with |
| another Diff program if you want).<ol> |
| <li>For any file with a significant difference, it will build two |
| BAT files, such as the first two below.<pre>Diff_PropList-5.0.0d10.txt.bat |
| OLDER-Diff_PropList-5.0.0d10.txt.bat |
| |
| UNCHANGED-Diff_PropertyValueAliases-5.0.0d10.txt.bat</pre> |
| </li> |
| </ol> |
| </li> |
| <li>Any files without significant changes will have "UNCHANGED" as a |
| prefix: ignore them. The OLDER prefix is the comparison to the |
| last version of Unicode.</li> |
| <li>On Windows you can run these BATs to compare files:</li> |
| </ol> |
| </li> |
| </ol> |
| <h3>5. Invariant Checking</h3> |
| <ol> |
| <li>Setup<ol> |
| <li>Open in Package Explorer<ul> |
| <li>com.ibm.text.UCD<ul> |
| <li>TestUnicodeInvariants.java</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Run>Run As... Java Application<br> |
| Will create the following file of results:<pre><a href="file:///C:/DATA/GEN/UnicodeInvariantResults.txt/">C:\DATA\GEN\UnicodeInvariantResults.txt\</a></pre> |
| </li> |
| <li>Open that file and search for "**** START Error Info ****" Each such |
| point provides a dump of comparison information.</li> |
| </ol> |
| </li> |
| </ol> |
| <h3>6. Options</h3> |
| <ol> |
| <li>If you want to see files that are opened while processing, do the |
| following:<ol> |
| <li>Run>Run</li> |
| <li>Select the Arguments tab, and add the following<ol> |
| <li>VM arguments: |
| <pre>-DSHOW_FILES</pre> |
| </li> |
| </ol> |
| </li> |
| </ol> |
| </li> |
| </ol> |
| <h3>5. UCA</h3> |
| <ol> |
| <li> |
| <h3>You will use com.ibm.text.UCA.Main as your main class, creating along |
| the same lines as above.</h3></li> |
| <li> |
| <h4>To build all the UCA files used by ICU, use the Program arguments:</h4> |
| <pre>Main ICU</pre> |
| </li> |
| <li> |
| <h4>To build all the charts, use the UCA project, with options: </h4> |
| <pre>normalizationChart caseChart scriptChart indexChart</pre> |
| </li> |
| </ol> |
| |
| </body> |
| |
| </html> |
