I have a very big text file. Can I read a text file in chunks?

By defining a function to handle one chunk

process_chunk: {[list_of_lines]
// Split and parse each line in list_of_lines and
// then - most likely - create some sort of output.
// After all, the idea here is to avoid holding
// all of the data in memory.
}

and passing your chunk handler along with the file handle to .Q.fs:

bytes_read: .Q.fs[process_chunk; `:big_file_name]

For details on how to parse lines of text inside your chunk handler, see this related faq.


This work is licensed under a Creative Commons License.
The views and opinions expressed herein are those of the authors and do not necessarily reflect those of any other person or legal entity.
Kdb+ is the registered trademark of Kx Systems, Inc.