« How do I remove a key from a dictionary? | Main | How do I select all columns in q? »

I have a very big text file. Can I read a text file in chunks?

By defining a function to handle one chunk

process_chunk: {[list_of_lines]
    // Split and parse each line in list_of_lines and
    // then - most likely - create some sort of output.
    // After all, the idea here is to avoid holding
    // all of the data in memory.
    }

and passing your chunk handler along with the file handle to .Q.fs:

bytes_read: .Q.fs[process_chunk; `:big_file_name]

For details on how to parse lines of text inside your chunk handler, see this related faq.

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments

There are no comments for this journal entry. To create a new comment, use the form below.

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
All HTML will be escaped. Textile formatting is allowed.