What fby does is aggregate values from one list based on groups defined in another, parallel, list. For example, suppose we have one list of cities and another list with a few temperature samples for each. We can use fby to calculate the minimum temperature sample for each city, and then replicate those values at each position for each corresponding city:

q)city:`NY`NY`LA`SF`LA`SF`NY
q)temp:32 31 75 69 70 68 12
q)(min;temp) fby city
12 12 70 68 70 68 12
q)

Thus, NY's minimum temperature, 12, appears at every index in fby's output where `NY appears in city.

At the core of fby (like by) is group, which organizes the distinct values in a list into a dictionary mapping those values to their indices:

q)city:`NY`NY`LA`SF`LA`SF`NY
q)group city
NY| 0 1 6
LA| 2 4
SF| 3 5
q)

We can group the temperatures for each city together by indexing temp with the value of the grouped city dictionary:

q)grouped: value group city
q)temp[grouped]
32 31 12
75 70
69 68
q)

Note that the result of indexing temp with grouped is a nested list with the same shape as grouped. This is a general principle: the result of an indexing operation has the shape of the index.

Now we can apply an aggregation function to each of the temperature groups:

q)min each temp[grouped]
12 70 68
q)

We're almost there. The real trick of fby is placing each aggregation result into a new list so that each element has the correct value per the grouping list. We can use @ (functional amend) to get the job done (see also the functional apply/amend faq):

q)@[temp; grouped; :; min each temp[grouped] ]
12 12 70 68 70 68 12
q)

The real fby is just slightly more complicated to ensure that the first argument to @ has the correct type.

A table is a reference to a (column) dictionary.

The internal representation of a table is nearly identical to that of a dictionary. We can use the flip command to create a table from a dictionary:

q)t: ([] a: 1 2 3; b: 4 5 6; c: 7 8 9)
q)d: `a`b`c ! (1 2 3; 4 5 6; 7 8 9)
q)t ~ flip d
1b
q)

In fact, when a dictionary is flip'ed, the underlying core data structure remains untouched. The table itself is a simple, small object that refers to the original dictionary. Using .Q.w, we can measure how much more memory a table takes than the corresponding dictionary:

q).Q.w[]`used         // memory usage baseline
112992j
q)x:`a`b`c!3 3#til 9  // create a small dictionary
q).Q.w[]`used
113424j
q)x:flip x
q).Q.w[]`used         // memory usage delta is
113456j               // just 32 more bytes
q)

No matter how large the underlying dictionary is, creating a table is fast and still takes only 32 bytes:

q)x:`a`b`c!3 100000#til 10
q).Q.w[]`used
1686192j
q)x:flip x
q).Q.w[]`used
1686224j          // 32 = 1686224 - 1686192
q)

Now, Let's examine how a keyed table is related to a dictionary. We start by creating a simple keyed table:

q)t: ([] a: 1 2 3; b: 4 5 6; c: 7 8 9)
q)keyedTable: `a`b xkey t
q)keyedTable
a b| c
---| -
1 4| 7
2 5| 8
3 6| 9
q)keys keyedTable
`a`b
q)

Since keyedTable is a table, one might expect it to have the same type as t but instead, q presents the following surprise:

q)type t              
98h                  // type table as expected
q)type keyedTable
99h                  // *NOT* 98h
q)

Type 99h is the type number for dictionaries. If keyedTable really is a dictionary, we should be able to extract its key and value:

q)key keyedTable
a b
---
1 4
2 5
3 6
q)value keyedTable
c
-
7
8
9
q)

Indeed, keyedTable is a dictionary - one that holds unkeyed tables for both its key and its value:

q)type key keyedTable
98h
q)type value keyedTable
98h
q)

This suggests that we can create a keyed table by using the ! (dict) operator with two unkeyed tables:

q)(key keyedTable)!(value keyedTable)
a b| c
---| -
1 4| 7
2 5| 8
3 6| 9
q)

Lastly, joining the two flipped tables brings us back to the original dict.

q)(flip key keyedTable),(flip value keyedTable)
a| 1 2 3
b| 4 5 6
c| 7 8 9
q)

For more information, see Creating dictionaries and tables from C.

q)phandle: hopen `:serverhost:5012
q)phandle (function; arg1; arg2 ..)
..
q)hclose phandle
q)

To invoke a function in another q process (i.e., a function that already exists in that q process), you need the following:

1. a process handle to the other q process,
2. the name of the pre-existing function in the other process, and
3. the parameters you want to pass.

Then all you need to do is to apply the process handle to a list whose first element is the function name (as a symbol) and whose remaining elements are the arguments.

For example, let's start one q process, which will be our server, listening on TCP port 5012. In our server, we'll define a function square (we'll make the background color different for the server to make it easier to distinguish from the client):

$ q -p 5012             // server
..
q)\p                    // display the listening port
5012
q)square: {x * x}
q)square 3
9
q)

Next, we'll start up another q process (the client), create a process handle connected to the first q process (the server), and request the first process to compute square 5:

$ q                     // client
..
q)phandle: hopen `:localhost:5012
q)phandle (`square; 5)  // remote execution
25
q)

To call a function with more parameters, simply add them to the end of the list. We'll demonstrate by defining a 2-argument function on the server that calculates the length of a right triangle's hypotenuse:

q)hypotenuse: {sqrt sum square x,y}
q)hypotenuse[3;4]
5f
q)

q)phandle (`hypotenuse; 5; 12)
13f
q)

What if the function you're calling doesn't take any parameters? For example, we'll define a function in the server called serverTime that returns the local time according to the server:

q)serverTime: {[] .z.T}
q)serverTime[]
11:51:34.762
q)

You can't pass zero parameters over IPC:

q)phandle enlist `serverTime  // a list with just
'length                       // the function name
q)                            // is not allowed

However, you can pass whatever you like, and it won't matter:

q)phandle `serverTime`ignored
11:52:28.537
q)phandle (`serverTime; ())    // () is often used
11:52:40.923
q)phandle (`serverTime; ::)    // so is ::
11:52:47.338
q)

See :: (generic null).

So far, all of our examples involved a client invoking a predefined function on the server. You can also pass a function defined on the client to be executed on the server. To see this, let's define a global variable on the server:

q)SERVERGLOBAL: 47
q)

Now, on the client, we'll define a function called getSERVERGLOBAL to retrieve the value of SERVERGLOBAL on the server. Instead of passing the name of the function (i.e., `getSERVERGLOBAL), we pass the function's value:

q)getSERVERGLOBAL: {[] SERVERGLOBAL}
q)getSERVERGLOBAL[]
'SERVERGLOBAL                   // not defined here
q)phandle (getSERVERGLOBAL; ::) // note the missing `
47
q)

Notice that this operation does not cause getSERVERGLOBAL to become defined on the server:

q)getSERVERGLOBAL
'getSERVERGLOBAL
q)

This technique can be used to run built-in functions and anonymous functions (aka lambdas) on the server as well:

q)phandle (+; 2; 4)
6
q)phandle ({til SERVERGLOBAL - x}; 42)
0 1 2 3 4
q)

There is one more way to convey code to the server to run: you can pass the code in a string.

q)phandle "SERVERGLOBAL + 4"
51
q)

We prefer passing a function over a string, because - especially as the expression to be passed gets more complex - it's easier to read.

q)container: 1 2 3
q)100 * container
100 200 300
q)

Sometimes we are interested in only a subset of elements from a container:

q)container: til 10
q)container
0 1 2 3 4 5 6 7 8 9
q)container[where 0 = container mod 2]
0 2 4 6 8
q)                                                                                                         

Sometimes, however, you need to update particular elements of a structure while leaving the remaining elements unchanged. That's what functional apply and amend are for; they transform specific elements of a container without touching the others. The variations are distinguished by 3 choices:

1. @ or .

Which operator is used, @ or ., determines the interpretation of the indices used to select the elements to transform.

2. container or name

The first argument is either the value of a container or the name of a global variable referring to a container. In the former case, a new object is returned; in the latter, the global variable is modified and its name is returned.

3. monadic or dyadic function

If the transformation requires additional information beyond that contained in each element itself, that is accomplished by using a dyadic function and supplying the additional information in a fourth argument to the operator.

For a detailed discussion, please see the following faqs:

Functional @ (apply)
Functional . (apply)

Lastly, note that there is another pair of overloads for @ and . - each with three arguments - called protected execution, which are invoked when the first argument to @ or . is a function or projection; protected execution is discussed in another faq.

1. @[container; indices; function]
2. @[container; indices; function; second_args]

(Note that there is another overload for @ with three arguments, called protected execution, which is invoked when the first argument to @ is a function or projection; protected execution is discussed in another faq.)

In the 3-argument case, q applies function to those elements of container specified by indices, leaving the rest of container alone. In other words, the behavior of 3-argument @ (apply) resembles that of the following function:

Although this model breaks down when the first argument is a global variable name, it is very helpful in understanding what is going on even in that case. Let's look at a couple of simple examples to clarify this:

q)list: 1 + til 8
q)list
1 2 3 4 5 6 7 8
q)@[list; 0; neg]         / negate element 0
-1 2 3 4 5 6 7 8
q)list                    / list remains unmodified
1 2 3 4 5 6 7 8
q)@[list; 1 3 4; {x * x}] / square elements 1, 3, and 4
1 4 3 16 25 6 7 8
q)

The diagram below illustrates the last example above:

Because the new container has the same type and shape as the original, function must return the same type as its argument, unless the container is a mixed list (i.e., type 0):

q)@[list; 4 5 6 7; {x div 2}] / list is an int list
1 2 3 4 2 3 3 4
q)@[list; 4 5 6 7; {x % 2}]   / but % returns float
'type
q)mixed_list: 1 2, `3`4
q)type mixed_list
0h
q)@[mixed_list; 0 3; string] / anything goes with type 0
,"1"
2
`3
,"4"
q)

Next, we'll consider the second case listed at the top of this faq. When function is dyadic and a fourth argument is supplied, @ (apply) behaves like the following function:

The basic idea is still the same, i.e., to transform selected elements while leaving the rest of the container intact. The difference is that, instead of function being modified by each, it is modified by '(each-both) (also see this faq on each-both and multivalent each), so that the selected elements of container are paired up with the corresponding elements from second_args:

q)@[list; 1 3 5 7; +; 10 20 30 40]
1 12 3 24 5 36 7 48
q)@[list; (1 3 5 7; 0 2 4 6); *; 2 -2]
-2 4 -6 8 -10 12 -14 16
q)

One function often used with 4-argument @ (apply), is : (amend). We can replace selected elements with a certain value (or values) as follows:

q)list: 1 + til 8
q)@[list; 1 3 5 7; :; 47]
1 47 3 47 5 47 7 47
q)@[list; 1 3 5 7; :; 10 20 30 40]
1 10 3 20 5 30 7 40
q)

It might appear that the above example exposes a flaw in our model, apply_dyadic, of @ (apply). However, recall that, when modified by an adverb, : (amend) does not modify its first argument:

q):'[list 1 2 3; 20 30 40]
20 30 40                     / :' returns its 2nd argument
q)list                       / without modifying its 1st
1 2 3 4 5 6 7 8
q)

Where our models do break down is when, as alluded to earlier, the first argument is the name of global variable referring to a container, rather than the value of a container. In that case, the mechanics of the operation are the same, but original container is modified, and the return value is the name:

q)list: 1 + til 8
q)@[`list; 0; neg]
`list
q)list
-1 2 3 4 5 6 7 8
q)

This behavior is handy for writing functions of your own that work in both scenarios - either creating a new value or modifying one in place - like the following:

We can also use @ to apply a function to selected entries in a dictionary:

q)dictionary: `a`b`c`d ! `1`2, 3 4
q)dictionary
a| `1
b| `2
c| 3
d| 4
q)@[dictionary; `a`b; {"I"$ string x}]
a| 1
b| 2
c| 3
d| 4
q)

Tables can be transformed as well. Consider the following table:

q)t: ([] col1: `foo`bar`baz; col2: 5 10 15f)
q)t
col1 col2
---------
foo  5   
bar  10  
baz  15  
q)

Recall that each row of a table is a dictionary:

q)t 0
col1| `foo
col2| 5f
q)

Thus, we can use @ (apply) to accomplish an update in the following manner:

q)update col2 - 4 from t where i > 0
col1 col2
---------
foo  5   
bar  6  
baz  11  
q)@[t; 1 2; {[row] row[`col2] -: 4; row}]
col1 col2
---------
foo  5   
bar  6  
baz  11  
q)

We could also obtain the previous result using nested calls to @ (apply):

q)@[t; 1 2; @[; `col2; -[; 4]]]
col1 col2 
---------
foo  5 
bar  6
baz  11
q)

Since we can index tables using column names, we can double all of the entries in col2 as follows:

q)@[t; `col2; 2*]
col1 col2
---------
foo  10  
bar  20  
baz  30  
q)

1. .[container; indices; function]
2. .[container; indices; function; second_args]

(Note that there is another overload for . with three arguments, called protected execution, which is invoked when the first argument to . is a function or projection; protected execution is discussed in another faq.)

In the 3-argument case, q applies function to those elements of container specified by indices, leaving the rest of container alone.

The behavior of . (apply) is very similar to that of @ (apply) (which is described in another faq). The only difference between @ (apply) and . (apply) is that the indices, in the case of . (apply), are applied at depth along the dimensions of the container. The following code behaves like . (apply) for a two-dimensional container:

To explore the different treatment of the indices argument between @ (apply) and . (apply), we'll consider the simplest, multi-dimensional container: a two-dimensional list, aka a matrix.

q)matrix: 0N 4 # til 16
q)matrix
0  1  2  3 
4  5  6  7 
8  9  10 11
12 13 14 15
q)

We can use simple indexing of matrix with both @ (index) and . (index) to illustrate the difference:

q)@[matrix; 0]           // fetch row 0
0 1 2 3
q)@[matrix; 0 0]         // fetch matrix[0;0] fails.  why?
0 1 2 3                  // @ cannot index more
0 1 2 3                  // than one dimension
q)
q).[matrix; 0 0]         // dot indexes "at depth"
0

Now let's apply a function to selected elements from matrix using . (apply):

q).[matrix; (0; ::); 100+]
100 101 102 103
4   5   6   7  
8   9   10  11 
12  13  14  15 
q).[matrix; 0 0; 100+]
100 1  2  3 
4   5  6  7 
8   9  10 11
12  13 14 15
q).[matrix; (::; 0); 100+]
100 1  2  3 
104 5  6  7 
108 9  10 11
112 13 14 15
q)

Notice that we used :: in order to elide a dimension from the indices.

We can also use . (apply) with dictionaries of lists:

q)dict: `a`b ! (1 2 3; 4 5 6)
q).[dict; (`a; 0); 50*]
a| 50 2 3
b| 4  5 6
q).[dict; (::; 1); 50*]
a| 1 100 3
b| 4 250 6
q)

The general case of applying a function at depth via functional . (apply) is not implemented for tables:

q)t: ([] a: `foo`bar; b: 5 10f)
q).[t; (0; `b); %[; 5]]
'nyi
q).[t; (`b; 0); %[; 5]]
'nyi
q)

The . form of apply has another trick up its sleeve: the empty list index. When the second parameter to . (apply) is (), the entire container is passed to function in a single invocation, as the following example demonstrates:

q).[matrix; (); 0N!]
(0 1 2 3;4 25 36 7;8 81 100 11;12 13 14 15)
0  1  2   3 
4  25 36  7 
8  81 100 11
12 13 14  15
q)

When using . (apply) in this manner, we can return anything from function; the type and shape do not matter:

q).[matrix; (); {"hello"}]
"hello"
q)

Returning to the second case, when function is dyadic, . (apply) takes a fourth argument (named second_args in our example), and each indexed element of container is paired with the corresponding element of second_args. This means that second_args must conform to indices (or be an atom). The following function expresses this behavior:

(Also see this faq on each-both and multivalent each.)

We can demonstrate how this works by altering the center square of matrix:

q).[matrix; (1 2; 1 2); *; (10 20; 100 200)]
0  1   2    3 
4  50  120  7 
8  900 2000 11
12 13  14   15
q)

We can categorize the ways in which q handles vectors into the following three groups:

1) Treat the vector as a single entity.
2) Perform an operation on each element of a vector.
3) Perform an operation on each set of corresponding elements from multiple vectors of the same length.

We use ' (each-both) to ask q to apply a function in the third manner.

Let's review a few examples to distinguish the three forms of vector handling described above. Consider the , (join) operator, which takes two arguments and concatenates them. When used by itself, , (join) treats a vector as a single entity:

q)1 , 2
1 2
q)1 2 3 , 10
1 2 3 10
q)1 , 10 20 30
1 10 20 30
q)1 2 , 10 20 30
1 2 10 20 30
q)

Notice that the lengths of the two vectors to be joined doesn't matter; in fact, they don't have to be vectors.

Next, as an example of applying the same operation to each element of a vector, we can use \: (each-left) to append the letter "e" to each element in a vector of strings:

q)("car"; "far"; "mar") ,\: "e"
"care"
"fare"
"mare"
q)

Lastly, we'll use ' (each-both) to join corresponding elements from two vectors of the same length (like zip from Python or Haskell):

q)1 2 3 ,' 10 20 30
1 10
2 20
3 30
q)

As an aside, the use of ' (each-both) is not needed for many built-in functions in q, because those functions (sometimes referred to as atomic functions in kx documentation) automatically assume that this is the desired behavior when presented with same-length vectors as arguments. The + operator is a good example:

q)1 2 3 + 10 20 30
11 22 33
q)

Even a user defined function, if that function's body is exclusively made up of applications of atomic functions, does not require the use of ' (each-both) to display this behavior:

q)atomic: {[x; y] (x * x) + y * y}
q)atomic'[1 2 3; 10 20 30]
101 404 909
q)atomic[1 2 3; 10 20 30]  // ' not required
101 404 909
q){[x; y] atomic[x; y]}[1 2 3; 10 20 30]
101 404 909
q)

The last example shows that such user defined atomic functions are truly atomic in the eyes of q.

The following code simulates the behavior of ' (each-both) - on functions of two arguments - by creating a new function (that acts on vectors) from the function passed as an argument:

However, ' (each-both) doesn't stop at two lists, which is why it is sometimes called multivalent each. Suppose we need to generate a bunch of html hyperlinks for a web page we are creating on-the-fly (perhaps in a custom http POST handler). Each link needs to have its own styling, so we want to add a distinct class attribute to each anchor tag.

The .h namespace includes numerous functions for generating HTML. The one we need is the .h.htac (html tag with attributes and closing tag) function:

q).h.htac[`a; `class`href ! ("myclass"; "kdbfaq.com"); "kdbfaq"]
"<a class=\"myclass\" href=\"kdbfaq.com\">kdbfaq</a>"
q)

Let's wrap .h.htac (html tag with attributes and closing tag) up in a function to create a styled link:

We'll store the information we need to generate the links in a table:

q)links
name        url                    class
----------------------------------------
"kx"        "http://kx.com"        "c1" 
"Wikipedia" "http://wikipedia.com" "c2" 
"kdbfaq"    "http://kdbfaq.com"    "c3"
q)

Now we can use ' (multivalent each) to generate all the links easily:

q)styled_link'[links `name; links `url; links `class]
"<a class=\"c1\" href=\"http://kx.com\">kx</a>"
"<a class=\"c2\" href=\"http://wikipedia.com\">Wikipedia</a>"
"<a class=\"c3\" href=\"http://kdbfaq.com\">kdbfaq</a>"
q)

Or, alternately, we can combine ' (multivalent each) with . (apply) for a shorter, more general approach to applying a function to each row in a table:

q)styled_link .' flip value flip links
"<a class=\"c1\" href=\"http://kx.com\">kx</a>"
"<a class=\"c2\" href=\"http://wikipedia.com\">Wikipedia</a>"
"<a class=\"c3\" href=\"http://kdbfaq.com\">kdbfaq</a>"
q)           

On more wrinkle: sometimes ' (each) is used where \: (each-left) or /: (each-right) would be more explicit. For instance, our earlier example in which we appended the letter "e" to each element of a list of strings could have been written as follows:

q)("car"; "far"; "mar") ,' "e"
"care"
"fare"
"mare"
q)

We find the original, more explicit form, quicker and easier to read.

Note: This applies to bash, and has not been tested in csh, ksh, etc.

We can invoke arbitrary external programs from q using the system command. For example:

q)system "echo Hello, world!"
"Hello, world!"
q)

External programs often signal errors by returning a non-zero exit code, which is stored in the shell variable ?:

$ ls nonexistent_file
ls: nonexistent_file: No such file or directory
$ echo $?
1
$ foo
-bash: foo: command not found
$ echo $?
127

When an external program called from system in q runs into a situation like this, q detects the non-zero exit code from the child process and raises a signal, 'os:

q)system "ls nonexistent_file"
ls: nonexistent_file: No such file or directory
'os
q)

When trying out ideas at the console as in the above example, the full details of the error are available on the screen. Getting those error details programmatically is a bit trickier, however. Consider the following function, in which we use protected execution to invoke an error handler when the 'os signal is raised:

use_protected_execution: {[]
    : @[system;
        "ls nonexistent_file";
        {[error_info] -1 "error is ", error_info;}];
    }

The error handler in use_protected_execution knows only that an operating system error occurred. None of the error details are available within the code:

q)use_protected_execution[]
ls: nonexistent_file: No such file or directory  // invisible
error is os
q)

We can improve the situation somewhat by appending "; echo $?" to the command:

q)result: system "ls nonexistent_file; echo $?"
q)result
,"1"    // actually a list of strings
q)

This enables us to distinguish error codes as follows:

distinguish_error_codes: {[]
    result: system "ls nonexistent_file; echo $?";
    exit_code: "I" $ last result;
    $[0   = exit_code;
        ; // everything's alright
      1   = exit_code;
        ; // minor problem
      2   = exit_code;
        ; // major problem
      127 = exit_code;
        ; // command not found
    / else
        // unhandled exit code
      ]}

What we really want, though, is to capture the output sent by the failed command to STDERR. You might expect the usual shell redirection syntax to do the job:

q)result: system "ls nonexistent_file 2>&1; echo $?"
ls: nonexistent_file: No such file or directory
q)result
,"1"
q)

No such luck. The trick is to put parentheses around the whole thing:

q)result: system "(ls nonexistent_file 2>&1; echo $?)"
q)result
"ls: nonexistent_file: No such file or directory"
,"1"
q)

We can wrap all this up in a handy function that, instead of raising os, raises the descriptive message output to STDERR by the child process:

call_external: {[command]
    result: system "(", command, " 2>&1; echo $?)";
    exit_code: "I" $ last result;
    if [0 < exit_code;
        ' raze -1 _ result];
    -1 _ result}

To experiment, let's construct two dictionaries, one with atomic keys and one with list keys.

q)dict1: `keya`keyb`keyc ! 10 20 30
q)dict1                // dictionary with atomic keys
keya| 10
keyb| 20
keyc| 30
q)dict2: (`keya1`keya2; `keyb1`keyb2; `keyc1`keyc2) ! 1 2 3
q)dict2                // dictionary with list keys
keya1 keya2| 1
keyb1 keyb2| 2
keyc1 keyc2| 3
q)dict1[`keya]         // index by atom key
10
q)dict2[`keya1`keya2]  // index by list key
1
q)

Note how dict1 contains a keys made of exclusively of atoms, and dict2 contains keys that are lists.

Now let's consider a dictionary whose keys are both atoms and lists:

q)dict3: (`keya1; `keyb1`keyb2; `keyc1`keyc2) ! 1 2 3
q)key dict3
`keya1                // The first key is an atomic symbol,
`keyb1`keyb2          // while the rest of the keys are
`keyc1`keyc2          // lists of symbols.
q)dict3[`keya1]       // The first key indexes fine,
1
q)dict3[`keyb1`keyb2] // but the remaining keys are broken;
0N 0N
q)dict3[`keyc1`keyc2] // they are treated as multiple keys.
0N 0N
q)

Even though the keys are present they cannot be found by a dictionary lookup. Reverse lookup does work:

q)dict3
`keya1      | 1    // ok
`keyb1`keyb2| 2    // broken
`keyc1`keyc2| 3    // broken
q)dict3 ? 2
`keyb1`keyb2
q)dict3 ? 3
`keyc1`keyc2
q)

Analogous behavior can be observed when searching a list that contains both atoms and lists:

q)list: key dict3
q)list ? `keya1
0
q)list ? `keyb1`keyb2
3 3
q)

Always remember that a dictionary lookup is essentially the following:

q)lookup: {[dict; thekey] value[dict] @ key[dict] ? thekey}
q)

Sometimes types in q behave differently depending on the data contained within. Take a look at this related faq to read more.

Let's test with the following two dictionaries, sfdict and fsdict:

q)sfdict: `symbol`float ! (`abc; 123.4)  // type sym is first
q)fsdict: `float`symbol ! (123.4; `abc)  // type float is first
q)sfdict
symbol| `abc
float | 123.4
q)fsdict
float | 123.4
symbol| `abc
q)

Note that while both dictionaries contain the same key-value pairs, because dictionaries are an ordered list of key value pairs, the above two dictionaries do not match.

q)sfdict = fsdict
symbol| 1
float | 1
q)sfdict ~ fsdict
0b
q)

A dictionary search returns a null value when the key sought is not found:

q)null sfdict[`badkey]
1b
q)null fsdict[`badkey]
1b
q)

Getting a null back from a failed lookup is expected. On the other hand, many are surprised to find that the two nulls returned for the above two lookups differ:

q)sfdict[`badkey] ~ fsdict[`badkey]
0b
q)sfdict[`badkey]
`                    // null symbol
q)fsdict[`badkey]
0n                   // null float
q)

The types of the returned nulls are different according to the types of the first values in sfdict and fsdict. This is the same behavior observed when indexing a mixed list with an out-of-range index:

q)(`a; `b; 1f; 2f)[5]
`
q)(1f; 2f; `a; `b)[5]
0n
q)

There's a similar "first element wins" behavior with the atomicity of dictionary keys. see this faq on dictionary indexing to learn more.

Normally, the : (amend) function is used to assign a value to a name:

q)foo: 47
q)foo
47
q)

Like other built-in functions that take two arguments, : (amend) can be called using either infix notation (as above) or function call notation:

q):[foo; 747]
q)foo
747
q)

Note that : (amend) displays its special semantics (which it shares with assignments in all strict languages) of not evaluating its first argument when that argument is simply a name, regardless of whether : (amend) is invoked infix or functionally.

q)delete bar from `.    / make sure bar is not defined
q):[bar; 42]
q)bar
42
q)

On the other hand, : (amend) does evaluate its first argument when that argument is an expression, but : (amend) still produces l-values when it does so:

q)list: 0 1 2 3
q)list[0]: 47
q)list
47 1 2 3
q):[list 1 2 3; 10 20 30]
q)list
47 10 20 30
q)

Based on the previous examples, you might be tempted create a projection from : (amend) and a left-hand side, but it turns out you can't:

q):[foo]
'foo
q)

It is possible to close the first argument to : (amend), but the resulting projection is no longer an assignment; it is an identity function:

q)foo: 42
q)(:[foo])[47]
47
q)foo
42
q)

In fact, the value attached to the first argument of : (amend) is irrelevant:

q)(:[`xyzzy])[47]
47
q)

Closing the second argument will cause : (amend) to return the same value always:

q)f: :[; 3]     / bind the 2nd argument
q)f 18          / no matter what we pass
3
q)f "hello"     / f will always return 3
3
q)

The other case when : (amend) does not perform assignment is when it is modified by an adverb:

q)list1: 0 1 2 3
q)list2: 4 5 6 7
q)(list1; list2) :\: 10 20 30 40   / we get the rhs
10 20 30 40                        / once for each list
10 20 30 40
q)list1                            / neither list
0 1 2 3
q)list2                            / has been modified
4 5 6 7
q)list2 :/: 0 1 2 3
0 1 2 3
q)list1[0 1 2 3] :' 10 20 30 40
10 20 30 40
q)

q)table: ([] x: 10 ? "abc"; y: 10 ? til 10)
q)table
x y
---
a 7
a 7
b 1
b 9
a 1
a 0
c 8
b 8
c 3
c 1
q)

To sort by column x, instead of "ORDER BY x", we write `x xasc table:

q)`x xasc table // sort by column x
x y
---
a 7
a 7
a 1
a 0
b 1
b 9
b 8
c 8
c 3
c 1
q)

The application of xdesc is similar.

In addition, to sort by multiple columns, instead of "ORDER BY y, x" or "ORDER BY y, x DESC", we pass the list of column names as the left argument to xasc or xdesc, respectively. For example,

q)`y`x xdesc select from table where y > 5
x y
---
b 9
c 8
b 8
a 7
a 7
q)

Don't confuse xasc and xdesc with asc and desc, which operate on a vector instead of a table. Read more about sorting vectors in this related faq.

q)(`hh`mm`ss $ .z.T), `int $ .z.T mod 1000
16 49 2 233
q)

You can extract the hours, minutes, and seconds from a time by passing `hh, `mm, and `ss, respectively, as left arguments to $ (cast):

q)now: .z.T
q)now
16:49:02.233
q)`hh $ now
16
q)`mm $ now
49
q)`ss $ now
2
q)`hh`mm`ss $ now
16 49 2
q)

Getting the milliseconds from a time is slightly less obvious. Times (type -19) are represented internally by q as 32-bit integers; typically the value counts the number of milliseconds since midnight, but it can also represent a span of time. We can cast freely back and forth between the two types and the values are preserved:

q)`time $ 0
00:00:00.000
q)`int $ 00:00:00.000
0
q)`time $ 24 * 60 * 60 * 1000
24:00:00.000
q)`int $ 24:00:00.000
86400000
q)`int $ now
60542233
q)`time $ 60542233
16:49:02.233
q)

Not only is the internal representation of time simply an integer, we can mix integers and times in integer arithmetic operations, and the result is always a time:

q)01:00:00.000 + 00:01:00.000
01:01:00.000
q)01:00:00.000 + 60000
01:01:00.000
q)01:00:00.000 * 4
04:00:00.000
q)now - 01:30:20.123
15:18:42.110
q)

By using div and mod, then, we have an alternative means to calculate the components of a time:

q)now div 3600000 // milliseconds per hour
00:00:00.016
q)now mod 1000    // just the milliseconds, please
00:00:00.233
q)

Although extracting milliseconds from a time while keeping the time type (as in the second example above) is sometimes useful, we normally want to get back these components of a time as integers, so let's cast it:

q)`int $ now mod 1000
233
q)

By the way, there is a shortcut for getting hours, minutes, and seconds from global variables that hold times: dot notation.

q)x: .z.T
q)x.hh, x.mm, x.ss
16 49 2
q)

However, we rarely use global variables to hold time values.

In general you can parse strings to some particular type by passing the uppercase form of the corresponding type character as the left argument to $ (cast).

q can parse dates in YYYYMMDD, YYYY-MM-DD, or YYYY.MM.DD format:

q)"D"$ ("20070809"; "2007-08-09"; "2007.08.09")
2007.08.09 2007.08.09 2007.08.09
q)

Times must have the form HH:MM:SS.mmm, but you can shorten them:

q)"T"$ ("07:08:09.010"; "07:08:09"; "07:08"; "07")
07:08:09.010 07:08:09.000 07:08:00.000 07:00:00.000
q)

In addition, "T"$ will tolerate a missing zero for hours (and hours only) less than 10:

q)"T"$ "7:08:09"
07:08:09.000
q)

"Z"$ expects (as in XML) that dates and times are joined with a T:

q)"Z"$ "2007-08-09T07:08:09.101"
2007.08.09T07:08:09.101
q)

"Z"$ usually treats the text analogously to "D"$ and "T"$; that is, the date must be complete, but the time may have its finer points elided:

q)"Z"$ "2007"
0Nz
q)"Z"$ "20070809"
2007.08.09T00:00:00.000
q)

Note, however, the "Z"$ - unlike "T"$ - requires that hours less than 10 be prepended with zero:

q)"Z"$ "20070809T7:08:09"
0Nz
q)

You can also parse months with "M"$, minutes with "U"$, and seconds with "V"$:

q)"M"$ ("2007-08"; "2007.08"; "200708")
2007.08 2007.08 2007.08m
q)"U"$ "07:08:09.010"
07:08
q)"V"$ "07:08:09.010"
07:08:09
q)

Make sure you don't pass any extra text to "M"$, though:

q)"M"$ "2007.08.01"
2007.01m
q)

There are two more type characters for parsing temporal values. "N"$ and "P"$ are for parsing nanosecond-precise time spans and datetimes, respectively:

q)"N"$ "12D07:08:09.123456789"
12D07:08:09.123456789
q)"P"$ "2007-08-09T07:08:09.123456789"
2007.08.09D07:08:09.123456789
q)

You may have discovered that "N"$ can parse floating point numbers, as well, but beware - depending on the range of the input, you might not get what you expect:

q)"N"$ "0.123456789"
0D00:00:00.123456789
q)"N"$ "1.123456789"
0D01:00:00.123456789
q)"N"$ "12.123456789"
0D12:00:00.123456789
q)"N"$ "24.123456789"
1D00:00:00.123456789
q)"N"$ "48.123456789"
2D00:00:00.123456789
q)"N"$ "4800.123456789"
2D00:00:00.123456789

Stick with strings that represent durations in the way q does, and you'll steer clear of surprises.

Lastly, if you find yourself with a text full of times expressed as seconds (and, optionally, fractions thereof) since the Unix epoch, "P"$, and "Z"$ can parse them, too:

q)unix_time: first system "date +%s"
q)unix_time
"1311635286"
q)"Z"$ unix_time
2011.07.25T23:08:06.000
q)"P"$ unix_time , ".123456789"
2011.07.25D23:08:06.123456789

count list
or
count each column_name
to get the length of a list-valued field in a query.

The length of any list can be found using the count function; the only complication occurs with certain applications of count in queries. We'll illustrate with a table of simulated trade data:

q)trade
time         sym price size
---------------------------
09:30:17.623 glo 26.79 3200
09:30:26.602 ojb 28.17 1600
09:30:33.657 fjc 91.31 5400
09:30:40.884 knb 59.7  2900
09:31:19.256 apl 1.26  7900
..
q)

The following query uses count to return the number of rows that satisfy the where clause:

q)exec count i from trade where sym = `aif
6
q)

This is usually what you want.

However, suppose we created another table (perhaps to improve the performance of certain queries) that grouped prices for each symbol:

q)grouped: select price by sym from trade
q)grouped
sym| price                                                   ..
---| --------------------------------------------------------..
acl| 18.1 85.5 86.31 45.65 10.91 31.25 5.49 51.66 1.02 31.82 ..
aif| 44.02 47.74 11.83 14.28 85.11 99.09                     ..
alk| 51.12 16.7 78.71 96.56 26.55 32.09 33.66 30.01 83.24 30...
amg| 59.25 56.18 92.63 46.57 57.25 14.58 69.21 88.25 94.1 28...
aog| 92.59 51.52 96.95 83.2 6.21 59.77 44.19 94.19 3.13 41.94..
..
q)

In table grouped, the price column is of type list-of-float - note the uppercase type letter F:

q)meta grouped
c    | t f a
-----| -----
sym  | s    
price| F    

It appears that we can retrieve the prices for a given symbol easily enough:

q)exec price from grouped where sym = `aif
44.02 47.74 11.83 14.28 85.11 99.09
q)

Suppose, however, that we want to know how many trades occurred for a particular symbol:

q)exec count price from grouped where sym = `aif
1    
q)

What went wrong? A where function yields a list of row indexes that meet the constraints, and then each projection (i.e., the column names between exec and from — in this case, price) yields a list of corresponding field values. Since the number of rows that met our sole constraint is 1, the result of the projection is an untyped list with a single element:

q)type exec price from grouped where sym = `aif
0h
q)count exec price from grouped where sym = `aif
1
q)

Projection results are the arguments to aggregation functions in queries. In other words, the untyped list in our example is the same as the one passed to count. Since the projection's single element contains the list of prices we want to count, the way out is to combine count with first:

q)exec count first price from grouped where sum = `aif
6
q)

Applying the same logic when counting the trades for every symbol, we need to use count each:

q)select count price by sym from grouped
sym| price
---| -----
acl| 1    
aif| 1    
alk| 1    
amg| 1    
aog| 1    
..
q)select count each price by sym from grouped
sym| price
---| -----
acl| 14   
aif| 6    
alk| 10   
amg| 12   
aog| 13   
..
q)

Constraints in where clauses behave the same as well. Consider the following select statement:

q)t
x y z
-------------
a 1 "xyzzy"
b 2 "grue"
c 3 "frobozz"
q)select from t where 5 < count z
x y z
-----
q)

Newcomers to q often expect the above query to return the rows from t whose z column has more than 5 characters (i.e., c 3 frobozz). Rather than counting the contents of each z field, however, count is actually counting the list t `z:

q)count t `z
3
q)t `z
"xyzzy"
"foobar"
"frobozz"
q)

This insight suggests the solution:

q)count each t `z
5 6 7
q)select from t where 5 < count each z
x y z
-------------
c 3 "frobozz"
q)

select sum `long $ size from large_table

In many programming languages, including q, anytime you add integers (unless you somehow know for sure that the sum will fit in - roughly - 31 bits) you risk integer overflow. If you're lucky, the error will be obvious (e.g., you'll get a negative number when you expected a positive one). Casting the arguments to sum to long will give you (almost) 63 bits of breathing room.

If your sum won't fit in 63 bits, you'll need to explore other options:

• Switch to floats. Although overflow is still possible, it's rare. However, you lose some precision.
• Use a bignum library such as gmp.
• Use a language (e.g., haskell or clojure) that has built-in support for arbitrary-precision arithmetic.

Lastly, keep in mind that literal values that cannot fit into an int must be suffixed with j - even if the context suggests that a long is expected:

q)select from meta large_table where c = `id
c | t f a
--| -----
id| j
q)count select from large_table where id = 84066472837652480
'84066472837652480
q)count select from large_table where id = 84066472837652480j
1
q)

Although expressions in q are usually evaluated from right to left, there is one exception: the constraints in a where clause are evaluated from left to right. Consider the following example:

q)t: ([] x: `a`b`c; y: 1 2 3)
q)select from t where x < `c, y > 1
x y
---
b 2
q)

The comma separating x < `c from y > 1 is not the join operator; instead, it delimits the constraints of the where clause. If you need to use the join operator in a where clause, use parentheses:

q)select from t where x in (`a, `b), y > 1
x y
---
b 2
q)

Each constraint is evaluated in the usual order, i.e., right-to-left:

q)select from t where x < first -1 # `p`c, y > 1
x y
---
b 2
q)

Because constraints are evaluated from left to right, putting the most restrictive constraint first will speed up queries on large tables. On a date partitioned database, for example, the placement of a constraint on the virtual date column will make a marked performance difference:

select from large_table where date = 2011.02.25, name = `JOE

is significantly faster than

select from large_table where name = `JOE, date = 2011.02.25

The latter query will attempt to load the entire contents of table for rows matching name `JOE prior to narrowing the result set down to a single date. kdb does not provide behind-the-scenes optimization in cases like these.

testfunc: {[]
x: `aaa;
y: `bbb;
z:: `ccc;
-1 "finished!";
}

Let's demonstrate the placement of a breakpoint prior to the assignment of global variable z. Although there is no explicit support for breakpoints in q, insertion of non-compliant code, such as breakhere; shown below, does the job (don't forget the trailing semicolon):

testfunc: {[]
x: `aaa;
y: `bbb;
breakhere;
z:: `ccc;
-1 "finished!"
}

We are tricking q into throwing a signal 'breakhere.

q)testfunc[]
{[]
x: `aaa;
y: `bbb;
breakhere;
z:: `ccc;
-1 "finished!"
}
'breakhere
q))

At this point, we can examine the local stack.

q))x
`aaa
q))y
`bbb
q))z
()
q))

kdb has suspended execution, leaving the remaining two lines of function testfunc unexecuted. : (colon) resumes execution.

q)):
finished!
-1
q)x
'x
q)y
'y
q)z
`ccc
q)

See: global amend (::)

By default, all partitions of a given historical kdb database are available. The .Q.view function enables a given kdb process instance to selectively expose an arbitrary set of partitions into view.

The following example restricts all operations of the database instance to the most recent partition date (typically the previous business day):

q)yesterday: last date
q).Q.view yesterday
q)count date
1
q)yesterday ~ value exec from select distinct date from trade
1b
q)

Here is a more realistic example of a date partitioned historical database where viewable partitions are limited to those year-to-date:

q)start_date: "D" $ string[`year $ .z.D], ".01.01"
q)end_date: last date
q).Q.view date where date within (start_date; end_date)
q)count select from trade where not date within (start_date;
end_date)
0
q)

See also: .Q.pn

$ unzip verylargefile.zip /destination_dir/
I/O error: File too large

Make sure your file system can handle LFS.

$ 7za e -o /destination_dir/ verylargefile.zip

See also: 7-zip usage examples.