Short answer: 2>&1 (i.e., redirect STDERR to STDOUT), echo $?, and add parentheses.

Note: This applies to bash, and has not been tested in csh, ksh, etc.


We can invoke arbitrary external programs from q using the system command. For example:

q)system "echo Hello, world!"
"Hello, world!"
q)

External programs often signal errors by returning a non-zero exit code, which is stored in the shell variable ?:

$ ls nonexistent_file
ls: nonexistent_file: No such file or directory
$ echo $?
1
$ foo
-bash: foo: command not found
$ echo $?
127

When an external program called from system in q runs into a situation like this, q detects the non-zero exit code from the child process and raises a signal, ‘os:

q)system "ls nonexistent_file"
ls: nonexistent_file: No such file or directory
'os
q)

When trying out ideas at the console as in the above example, the full details of the error are available on the screen. Getting those error details programmatically is a bit trickier, however. Consider the following function, in which we use protected execution to invoke an error handler when the ‘os signal is raised:

use_protected_execution: {[]
: @[system;
"ls nonexistent_file";
{[error_info] -1 "error is ", error_info;}];
}

The error handler in use_protected_execution knows only that an operating system error occurred. None of the error details are available within the code:

q)use_protected_execution[]
ls: nonexistent_file: No such file or directory  // invisible
error is os
q)

We can improve the situation somewhat by appending “; echo $?” to the command:

q)result: system "ls nonexistent_file; echo $?"
q)result
,"1"    // actually a list of strings
q)

This enables us to distinguish error codes as follows:

distinguish_error_codes: {[]
result: system "ls nonexistent_file; echo $?";
exit_code: "I" $ last result;
$[0   = exit_code;
; // everything's alright
1   = exit_code;
; // minor problem
2   = exit_code;
; // major problem
127 = exit_code;
; // command not found
/ else
// unhandled exit code
]}

What we really want, though, is to capture the output sent by the failed command to STDERR. You might expect the usual shell redirection syntax to do the job:

q)result: system "ls nonexistent_file 2>&1; echo $?"
ls: nonexistent_file: No such file or directory
q)result
,"1"
q)

No such luck. The trick is to put parentheses around the whole thing:

q)result: system "(ls nonexistent_file 2>&1; echo $?)"
q)result
"ls: nonexistent_file: No such file or directory"
,"1"
q)

We can wrap all this up in a handy function that, instead of raising os, raises the descriptive message output to STDERR by the child process:

call_external: {[command]
result: system "(", command, " 2>&1; echo $?)";
exit_code: "I" $ last result;
if [0 < exit_code;
' raze -1 _ result];
-1 _ result}

Short answer: .Q.view partition_list

By default, all partitions of a given historical kdb database are available. The .Q.view function enables a given kdb process instance to selectively expose an arbitrary set of partitions into view.

The following example restricts all operations of the database instance to the most recent partition date (typically the previous business day):

q)yesterday: last date
q).Q.view yesterday
q)count date
1
q)yesterday ~ value exec from select distinct date from trade
1b
q)

Here is a more realistic example of a date partitioned historical database where viewable partitions are limited to those year-to-date:

q)start_date: "D" $ string[`year $ .z.D], ".01.01"
q)end_date: last date
q).Q.view date where date within (start_date; end_date)
q)count select from trade where not date within (start_date;
end_date)
0q)

See also: .Q.pn

By defining a function to handle one chunk

process_chunk: {[list_of_lines]
// Split and parse each line in list_of_lines and
// then - most likely - create some sort of output.
// After all, the idea here is to avoid holding
// all of the data in memory.
}

and passing your chunk handler along with the file handle to .Q.fs:

bytes_read: .Q.fs[process_chunk; `:big_file_name]

For details on how to parse lines of text inside your chunk handler, see this related faq.