How IPinfo Made Its MMDB Files 31% Smaller Without Dropping Records
IPinfo’s MMDB files shrank by 31%, and existing readers still open them with no decompression step. What did it remove, and can you do the same to your own database?
- IPinfo removed duplicate search-tree structure while preserving lookup results.
- The smaller files use the existing MMDB format and reader libraries.
- Custom database builders can use mmdbshrink, but the optimization process needs substantially more memory than the input file size.
Less duplicated structure, the same lookup answers
IPinfo’s smaller databases retain the records returned for IP lookups. Its engineering explanation describes an optimization of the structure used to reach those records, with no decompression step added.
For applications distributing an IP database, that makes the saving useful beyond a compressed download: the file itself occupies less space while existing readers remain usable. Preserving a location record is different from improving its geographical accuracy; the optimization changes storage, not the underlying information.
One subtree can serve several parents
The MMDB specification separates the search tree, data and metadata. A lookup follows the bits of an IP address through nodes containing two references, eventually reaching a data record or an empty result.
If different address ranges lead through identical structures to identical records, those subtrees can be shared. IPinfo processes children first and reuses a node when its ordered pair of child references matches an existing node. The technique is called hash-consing.
Matching a country name alone is insufficient: the downstream structure and targets must match. Sharing node references fits the existing format, which is why this does not require a new decoder.
An 80% result is not a promise for every file
The tool’s September 28 measurements show how widely the benefit varies across IPinfo datasets.
| Database | Before | After | Reduction |
|---|---|---|---|
| ipinfo_lite | 38 MB | 25 MB | 34% |
| ipinfo_location | 687 MB | 316 MB | 54% |
| ipinfo_privacy | 789 MB | 155 MB | 80% |
The 31% headline refers to IPinfo’s combined catalogue, not each database. Repetition determines the opportunity. An already optimized file will not shrink further on another run.
Current downloads need no extra pass
IPinfo says its downloadable MMDBs already contain the optimization. Builders of their own databases can use the standalone tool, installed separately, to create a new file while retaining the input:
mmdbshrink custom.mmdbThe default output is custom.shrunk.mmdb. Validation compares metadata, designated address boundaries and one million random addresses per address family. That default check is distinct from a full traversal of every prefix.
Plan the build machine’s capacity before running it: the README estimates peak memory at roughly six to twelve times the input size. This is the optimizer’s working memory, not a prediction of the finished database reader’s footprint.
For automation, the exit-code distinction matters. Standalone mmdbshrink returns a nonzero status for validation failure. The integrated mmdbctl shrink prints errors but exits with zero, so an ordinary shell success check is insufficient for that command.
Keep the original and compare the validated output with the lookups your application depends on before replacing it. The useful result is a smaller deployable database that still returns the expected records.