rust-jmdict/data/README.md

data/

We cannot put the JMdict into Git as one single file because its size is over 100 MiB, and GitHub does not like files that big. I don't want to use LFS because it's still a text file and thus delta-compresses really well if you let Git do its job. Therefore we split the file into chunks of roughly 1000 entries each.

Since we're pre-processing anyway, we're also converting from XML to JSON. The original XML file uses a lot of memory when parsed as a whole, and parsing in pieces is finicky because we need to carry over the DTD into each chunk, if only for the entity definitions. The JSON files in this directory, on the other hand, do not have any magical entities and thus trivially parse as individual entries. It also turns out that parsing JSON is much quicker than parsing XML, which makes a significant impact on the build time of the whole crate.

Import workflow

To update the JMdict copy in this directory, run make import JMDICT_PATH=/path/to/JMdict. Check the git diff afterwards; it should usually only show changes for a few places where upstream edited the respective JMdict entries.

Export workflow

We cannot bundle the data files with the crates when publishing because crates.io imposes a 10 MiB limit on crates. The data files are therefore stored in a compressed bundle by make export. The output file appears in this directory as entrypack-YYYY-MM-DD.json.gz, with the date being extracted from JMdict's own modification timestamp in entries-999.json.

This file can then be copied to its web server location, currently residing on http://dl.xyrillian.de/jmdict/ under the control of @majewsky. Finally, update the constants at the top of jmdict-traverse/src/file.rs to refer to the new file.