I have a very big text file. Can I read a text file in chunks?

By defining a function to handle one chunk

process_chunk: {[list_of_lines]
// Split and parse each line in list_of_lines and
// then - most likely - create some sort of output.
// After all, the idea here is to avoid holding
// all of the data in memory.
}

and passing your chunk handler along with the file handle to .Q.fs:

bytes_read: .Q.fs[process_chunk; `:big_file_name]

For details on how to parse lines of text inside your chunk handler, see this related faq.

Share:

Leave a Reply