Dictionary Compressing Indexes
Did anyone ever explore "dictionary compressing" indexes ?
Could significantly reduce DB size in some cases (at least for SQLite)
i.e: having two tables for indexes, computed index keys then ad hoc table to join with docs
It's a classical space/time tradeoff, not helpful when index-keys are highly unique, but for lower cardinality scenarios (enums, ...) it should be both faster & smaller
I ran some simulations and it reduced size by a factor of 2 for low-medium and up to +50% throughput (for SQLite on synthetic workloads)
Obviously it all depends on the effective
Could significantly reduce DB size in some cases (at least for SQLite)
i.e: having two tables for indexes, computed index keys then ad hoc table to join with docs
It's a classical space/time tradeoff, not helpful when index-keys are highly unique, but for lower cardinality scenarios (enums, ...) it should be both faster & smaller
I ran some simulations and it reduced size by a factor of 2 for low-medium and up to +50% throughput (for SQLite on synthetic workloads)
Obviously it all depends on the effective
distinct(keys) / rowsdistinct(keys) / rows ratio (some other DB engines might also internally dedupe themselves)
