How does each-both (aka multivalent each) work?

Short answer: It takes a function of N operands and creates a new function that applies the original to each set of corresponding elements from N same-length vectors.

We can categorize the ways in which q handles vectors into the following three groups:

1) Treat the vector as a single entity.

2) Perform an operation on each element of a vector.

3) Perform an operation on each set of corresponding elements from multiple vectors of the same length.

We use ‘ (each-both) to ask q to apply a function in the third manner.

Let’s review a few examples to distinguish the three forms of vector handling described above. Consider the , (join) operator, which takes two arguments and concatenates them. When used by itself, , (join) treats a vector as a single entity:

q)1 , 2
1 2
q)1 2 3 , 10
1 2 3 10
q)1 , 10 20 30
1 10 20 30
q)1 2 , 10 20 30
1 2 10 20 30

Notice that the lengths of the two vectors to be joined doesn’t matter; in fact, they don’t have to be vectors.

Next, as an example of applying the same operation to each element of a vector, we can use \: (each-left) to append the letter “e” to each element in a vector of strings:

q)("car"; "far"; "mar") ,\: "e"

Lastly, we’ll use ‘ (each-both) to join corresponding elements from two vectors of the same length (like zip from Python or Haskell):

q)1 2 3 ,' 10 20 30
1 10
2 20
3 30

As an aside, the use of ‘ (each-both) is not needed for many built-in functions in q, because those functions (sometimes referred to as atomic functions in kx documentation) automatically assume that this is the desired behavior when presented with same-length vectors as arguments. The + operator is a good example:

q)1 2 3 + 10 20 30
11 22 33

Even a user defined function, if that function’s body is exclusively made up of applications of atomic functions, does not require the use of ‘ (each-both) to display this behavior:

q)atomic: {[x; y] (x * x) + y * y}
q)atomic'[1 2 3; 10 20 30]
101 404 909
q)atomic[1 2 3; 10 20 30]  // ' not required
101 404 909
q){[x; y] atomic[x; y]}[1 2 3; 10 20 30]
101 404 909

The last example shows that such user defined atomic functions are truly atomic in the eyes of q.

The following code simulates the behavior of ‘ (each-both) – on functions of two arguments – by creating a new function (that acts on vectors) from the function passed as an argument:

However, ‘ (each-both) doesn’t stop at two lists, which is why it is sometimes called multivalent each. Suppose we need to generate a bunch of html hyperlinks for a web page we are creating on-the-fly (perhaps in a custom http POST handler). Each link needs to have its own styling, so we want to add a distinct class attribute to each anchor tag.

The .h namespace includes numerous functions for generating HTML. The one we need is the .h.htac (html tag with attributes and closing tag) function:


q).h.htac[`a; `class`href ! ("myclass"; ""); "kdbfaq"]
"<a class=\"myclass\" href=\"\">kdbfaq</a>"

Let’s wrap .h.htac (html tag with attributes and closing tag) up in a function to create a styled link:


We’ll store the information we need to generate the links in a table:


name        url                    class
"kx"        ""        "c1"
"Wikipedia" "" "c2"
"kdbfaq"    ""    "c3"

Now we can use ‘ (multivalent each) to generate all the links easily:

q)styled_link'[links `name; links `url; links `class]
"<a class=\"c1\" href=\"\">kx</a>"
"<a class=\"c2\" href=\"\">Wikipedia</a>"
"<a class=\"c3\" href=\"\">kdbfaq</a>"

Or, alternately, we can combine ‘ (multivalent each) with . (apply) for a shorter, more general approach to applying a function to each row in a table:

q)styled_link .' flip value flip links
"<a class=\"c1\" href=\"\">kx</a>"
"<a class=\"c2\" href=\"\">Wikipedia</a>"
"<a class=\"c3\" href=\"\">kdbfaq</a>"

On more wrinkle: sometimes ‘ (each) is used where \: (each-left) or /: (each-right) would be more explicit. For instance, our earlier example in which we appended the letter “e” to each element of a list of strings could have been written as follows:

q)("car"; "far"; "mar") ,' "e"

We find the original, more explicit form, quicker and easier to read.

How can I capture STDOUT, STDERR and the exit status of a system command invoked from q?

Short answer: 2>&1 (i.e., redirect STDERR to STDOUT), echo $?, and add parentheses.

Note: This applies to bash, and has not been tested in csh, ksh, etc.

We can invoke arbitrary external programs from q using the system command. For example:

q)system "echo Hello, world!"
"Hello, world!"

External programs often signal errors by returning a non-zero exit code, which is stored in the shell variable ?:

$ ls nonexistent_file
ls: nonexistent_file: No such file or directory
$ echo $?
$ foo
-bash: foo: command not found
$ echo $?

When an external program called from system in q runs into a situation like this, q detects the non-zero exit code from the child process and raises a signal, ‘os:

q)system "ls nonexistent_file"
ls: nonexistent_file: No such file or directory

When trying out ideas at the console as in the above example, the full details of the error are available on the screen. Getting those error details programmatically is a bit trickier, however. Consider the following function, in which we use protected execution to invoke an error handler when the ‘os signal is raised:

use_protected_execution: {[]
: @[system;
"ls nonexistent_file";
{[error_info] -1 "error is ", error_info;}];

The error handler in use_protected_execution knows only that an operating system error occurred. None of the error details are available within the code:

ls: nonexistent_file: No such file or directory  // invisible
error is os

We can improve the situation somewhat by appending “; echo $?” to the command:

q)result: system "ls nonexistent_file; echo $?"
,"1"    // actually a list of strings

This enables us to distinguish error codes as follows:

distinguish_error_codes: {[]
result: system "ls nonexistent_file; echo $?";
exit_code: "I" $ last result;
$[0   = exit_code;
; // everything's alright
1   = exit_code;
; // minor problem
2   = exit_code;
; // major problem
127 = exit_code;
; // command not found
/ else
// unhandled exit code

What we really want, though, is to capture the output sent by the failed command to STDERR. You might expect the usual shell redirection syntax to do the job:

q)result: system "ls nonexistent_file 2>&1; echo $?"
ls: nonexistent_file: No such file or directory

No such luck. The trick is to put parentheses around the whole thing:

q)result: system "(ls nonexistent_file 2>&1; echo $?)"
"ls: nonexistent_file: No such file or directory"

We can wrap all this up in a handy function that, instead of raising os, raises the descriptive message output to STDERR by the child process:

call_external: {[command]
result: system "(", command, " 2>&1; echo $?)";
exit_code: "I" $ last result;
if [0 < exit_code;
' raze -1 _ result];
-1 _ result}

Why can’t I index some elements of a dictionary?

Short answer: Check that the keys of your dictionary are uniformly atoms or lists, not both.

To experiment, let’s construct two dictionaries, one with atomic keys and one with list keys.

q)dict1: `keya`keyb`keyc ! 10 20 30
q)dict1                // dictionary with atomic keys
keya| 10
keyb| 20
keyc| 30
q)dict2: (`keya1`keya2; `keyb1`keyb2; `keyc1`keyc2) ! 1 2 3
q)dict2                // dictionary with list keys
keya1 keya2| 1
keyb1 keyb2| 2
keyc1 keyc2| 3
q)dict1[`keya]         // index by atom key
q)dict2[`keya1`keya2]  // index by list key

Note how dict1 contains a keys made of exclusively of atoms, and dict2 contains keys that are lists.

Now let’s consider a dictionary whose keys are both atoms and lists:

q)dict3: (`keya1; `keyb1`keyb2; `keyc1`keyc2) ! 1 2 3
q)key dict3
`keya1                // The first key is an atomic symbol,
`keyb1`keyb2          // while the rest of the keys are
`keyc1`keyc2          // lists of symbols.
q)dict3[`keya1]       // The first key indexes fine,
q)dict3[`keyb1`keyb2] // but the remaining keys are broken;
0N 0N
q)dict3[`keyc1`keyc2] // they are treated as multiple keys.
0N 0N

Even though the keys are present they cannot be found by a dictionary lookup. Reverse lookup does work:

`keya1      | 1    // ok
`keyb1`keyb2| 2    // broken
`keyc1`keyc2| 3    // broken
q)dict3 ? 2
q)dict3 ? 3

Analogous behavior can be observed when searching a list that contains both atoms and lists:

q)list: key dict3
q)list ? `keya1
0q)list ? `keyb1`keyb2
3 3

Always remember that a dictionary lookup is essentially the following:

q)lookup: {[dict; thekey] value[dict] @ key[dict] ? thekey}

Sometimes types in q behave differently depending on the data contained within. Take a look at this related faq to read more.

This work is licensed under a Creative Commons License.
The views and opinions expressed herein are those of the authors and do not necessarily reflect those of any other person or legal entity.
Kdb+ is the registered trademark of Kx Systems, Inc.