| <html> |
| |
| <head> |
| <title>ICU - Formats and API for Binary Data Files</title> |
| </head> |
| |
| <body> |
| |
| <h1>ICU - Formats and API for Binary Data Files</h1> |
| |
| <h2>Finding ICU data</h2> |
| |
| <p>ICU data, when stored in files, is loaded from the file system |
| directory that is returned by <code>u_getDataDirectory()</code>. |
| That directory is determined sequentially by |
| <ul> |
| <li><code>getenv("ICU_DATA")</code> - |
| the contents of the ICU_DATA environment variable</li> |
| <li>on Windows, by the value named <code>"Path"</code> of the registry key |
| <code>HKEY_LOCAL_MACHINE "SOFTWARE\\ICU\\Unicode\\Data"</code></li> |
| <li>relative to the path where <code>icuuc.dll</code> or <code>libicu-uc.so</code> or similar |
| is loaded from: if it is loaded from <code>/some/path/lib/libicu-uc.so</code>, then |
| the path will be <code>/some/path/lib/../share/icu/1.3.1/</code> |
| where <code>"1.3.1"</code> is an example for the version of the ICU library that |
| is trying to locate the data directory</li> |
| <li>relative to the path where <code>icuuc.dll</code> or <code>libicu-uc.so</code> or similar |
| is found by searching the <code>PATH</code> or <code>LIBPATH</code> |
| as appropriate; the relative path is determined as above</li> |
| <li>hardcoded to <code>(system drive)/share/icu/1.3.1/</code>, |
| where <code>(system drive)</code> is empty or a path to the system drive, like |
| <code>"D:\"</code> on Windows or OS/2</li> |
| </ul></p> |
| |
| <p>When ICU data is loaded using the <code>udata</code> API functions, then |
| there is a defined sequence of file locations and entry point names that are |
| used to locate the data. See the description in <code>icu/source/common/udata.h</code> for |
| details. Note that the exact data finding depends on the implementation |
| of this API and may differ by platform and by build configuration. |
| See also <code>icu/source/common/udata.c</code> for implementation details.</p> |
| |
| |
| <h2>Binary Data File Formats</h2> |
| |
| <p>Data files for ICU and for applications loading their data with ICU, |
| should have a memory-mappable format. This means that the data should be |
| layed out in the file in an immediately useful way, so that the code that uses |
| the data does not need to parse it or copy it to allocated memory and |
| build additional structures (like Hashtables). |
| Here are some points to consider:</p> |
| |
| <ul> |
| <li>The data memory starts at an offset within the data file |
| that is divisible by (at least) <code>sizeof(double)</code> |
| (the largest scalar data type) |
| if you use <code>unewdata.h/.c</code> |
| to write the data. |
| To be exact, <code>unewdata</code> writes the data 16-aligned, |
| and it is 16-aligned in memory-mapped files. However, the build |
| process forced us to insert a <code>double</code> before the |
| binary data to get any alignment, thus only 8-aligning |
| (<code>sizeof(double)==8</code> on most machines) the data.</li> |
| <li>Write explicitly sized values: explicitly 32 bits with an |
| <code>int32_t</code>, not using an ambiguous <code>int</code>.</li> |
| <li>Align all values according to their data type size: |
| Align 16-bit integers on even offsets, 32-bit integers on |
| offsets divisible by 4, etc.</li> |
| <li>Align structures according to their largest field.</li> |
| <li>When writing structures directly, avoid implicit |
| field padding/alignment: if a field may not be aligned |
| within the structure according to its size, then |
| insert additional (reserved) fields to explicitly |
| size-align that field.</li> |
| <li>Avoid floating point values if possible. Their size and structure |
| may differ among platforms.</li> |
| <li>Avoid boolean (<code>bool_t</code>, <code>bool</code>) values |
| and use explictly sized integer values instead |
| because the size of the boolean type may vary.</li> |
| <li>Write offsets to sub-structures at the beginning of the data |
| so that those sub-structures can be accessed directly without |
| parsing the data that precedes them.</li> |
| <li>If data needs to be read linearly, then precede it with its length |
| rather than terminating it with a sentinel value.</li> |
| <li>When writing <code>char[]</code> strings, write only "invariant" |
| characters - avoid anything that is not common among all ASCII- |
| or EBCDIC-based encodings. This avoids incompatibilities and |
| real, heavyweight codepage conversions. |
| Even on the same platform, the default encoding may not always |
| be the same one, and every "non-invariant" character |
| may change.<br> |
| (The term "invariant characters" is from |
| <a href="http://www.unicode.org/unicode/reports/tr16/"> |
| Unicode Technical Report 16 (UTF-EBCDIC)</a>.)</li> |
| </ul> |
| |
| |
| <h2>Platform-dependency of Binary Data Files</h2> |
| |
| <p>Data files with formats as described above should be portable among |
| machines with the same set of relevant properties:</p> |
| |
| <ul> |
| <li>Byte ordering: If the data contains values other than byte arrays.<br> |
| Example: <code>uint16_t</code>, <code>int32_t</code>.</li> |
| <li>Character set family: Some data files contain <code>char[]</code>. |
| Such strings should contain only "invariant characters", but |
| are even so only portable among machines with the same character set |
| family, i.e., they must share for example the ASCII or EBCDIC |
| graphic characters.</li> |
| <li>Unicode Character size: Some data files contain <code>UChar[]</code>. |
| In principle, Unicode characters are stored using UTF-8, UTF-16, or UTF-32. |
| Thus, Unicode strings are directly compatible if the code unit size is the same. |
| ICU uses only UTF-16 at this point.</li> |
| </ul> |
| |
| <p>All of these properties can be verified by checking the |
| <code>UDataInfo</code> structure of the data, which is done |
| best in a <code>UDataMemoryIsAcceptable()</code> function passed into |
| the <code>udata_openChoice()</code> API function.</p> |
| |
| <p>If a data file is loaded on a machine with different relevant properties |
| than the machine where the data file was generated, then the using |
| code could adapt by detecting the differences and reformatting the |
| data on the fly or in a copy in memory. |
| This would improve portability of the data files but significantly |
| decrease performance.</p> |
| |
| <p>"Relevant" properties are those that affect the portability of the |
| data in the particular file.</p> |
| |
| <p>For example, a flat (memory-mapped) binary data file |
| that contains 16-bit and 32-bit integers and is |
| created for a typical, big-endian Unix machine, can be used |
| on an OS/390 system or any other big-endian machine.<br> |
| If the file also contains <code>char[]</code> strings, |
| then it can be easily shared among all big-endian <em>and</em> |
| ASCII-based machines, but not with (e.g.) an OS/390.<br> |
| OS/390 and OS/400 systems, however, could easily share such |
| a data file <em>created</em> on either of <em>these</em> systems.</p> |
| |
| <p>To make sure that the relevant platform properties of |
| the data file and the loading machine match, the |
| <code>udata_openChoice()</code> API function should be used with a |
| <code>UDataMemoryIsAcceptable()</code> function that checks for |
| these properties.</p> |
| |
| <p>Some data file loading mechanisms prevent using data files generated on |
| a different platform to begin with, especially data files packaged as DLLs |
| (shared libraries).</p> |
| |
| |
| <h2>Writing a binary data file</h2> |
| |
| <p>This is a raw draft.</p> |
| |
| <p>... Use <code>icu/source/tools/toolutil/unewdata.h|.c</code> to write data files, |
| can include a copyright statement or other comment...See <code>icu/source/tools/gennames</code>...</p> |
| |
| </body> |
| |
| </html> |
