I know kdb is a in-memory database. my dataset is over two terabytes in size. However my RAM is limited to 32GB. How will this work?

Like any computer process, the maximum amount of data loaded into memory is limited by the real physical memory addressable by your process. On a 32-bit server, this can be little as 2GB. On a 64-bit server, 64TB is possible as of this writing, and that limit will continue to increase in the future.

However, with memory mapped tables, the size of your table can theoretically be as large as your disk (assuming it does not exceed the kdb limitations). As long as the operations you perform on your database don’t require more than your RAM can handle at once, you’ll be fine. That’s how kdb has been handling multi-terabyte databases for years.


This work is licensed under a Creative Commons License.
The views and opinions expressed herein are those of the authors and do not necessarily reflect those of any other person or legal entity.
Kdb+ is the registered trademark of Kx Systems, Inc.