How does fby work?

According to, fby is short for filter-by, and it is commonly used as the q equivalent to SQL’s HAVING clause (though, like where, fby is a q function, and its use is not limited to where clauses).

What fby does is aggregate values from one list based on groups defined in another, parallel, list. For example, suppose we have one list of cities and another list with a few temperature samples for each. We can use fby to calculate the minimum temperature sample for each city, and then replicate those values at each position for each corresponding city:

q)temp:32 31 75 69 70 68 12
q)(min;temp) fby city
12 12 70 68 70 68 12

Thus, NY’s minimum temperature, 12, appears at every index in fby‘s output where `NY appears in city.

At the core of fby (like by) is group, which organizes the distinct values in a list into a dictionary mapping those values to their indices:

q)group city
NY| 0 1 6
LA| 2 4
SF| 3 5

We can group the temperatures for each city together by indexing temp with the value of the grouped city dictionary:

q)grouped: value group city
32 31 12
75 70
69 68

Note that the result of indexing temp with grouped is a nested list with the same shape as grouped. This is a general principle: the result of an indexing operation has the shape of the index.

Now we can apply an aggregation function to each of the temperature groups:

q)min each temp[grouped]
12 70 68

We’re almost there. The real trick of fby is placing each aggregation result into a new list so that each element has the correct value per the grouping list. We can use @ (functional amend) to get the job done (see also the functional apply/amend faq):

q)@[temp; grouped; :; min each temp[grouped] ]
12 12 70 68 70 68 12

The real fby is just slightly more complicated to ensure that the first argument to @ has the correct type.

What does it mean for a table to be a flipped dictionary?

Short answer:

A table is a reference to a (column) dictionary.

The internal representation of a table is nearly identical to that of a dictionary. We can use the flip command to create a table from a dictionary:

q)t: ([] a: 1 2 3; b: 4 5 6; c: 7 8 9)
q)d: `a`b`c ! (1 2 3; 4 5 6; 7 8 9)
q)t ~ flip d

In fact, when a dictionary is flip‘ed, the underlying core data structure remains untouched. The table itself is a simple, small object that refers to the original dictionary. Using .Q.w, we can measure how much more memory a table takes than the corresponding dictionary:

q).Q.w[]`used         // memory usage baseline
q)x:`a`b`c!3 3#til 9  // create a small dictionary
q)x:flip x
q).Q.w[]`used         // memory usage delta is
113456j               // just 32 more bytes

No matter how large the underlying dictionary is, creating a table is fast and still takes only 32 bytes:

q)x:`a`b`c!3 100000#til 10
q)x:flip x
1686224j          // 32 = 1686224 - 1686192

Now, Let’s examine how a keyed table is related to a dictionary. We start by creating a simple keyed table:

q)t: ([] a: 1 2 3; b: 4 5 6; c: 7 8 9)
q)keyedTable: `a`b xkey t
a b| c
---| -
1 4| 7
2 5| 8
3 6| 9
q)keys keyedTable

Since keyedTable is a table, one might expect it to have the same type as t but instead, q presents the following surprise:

q)type t
98h                  // type table as expected
q)type keyedTable
99h                  // *NOT* 98h

Type 99h is the type number for dictionaries. If keyedTable really is a dictionary, we should be able to extract its key and value:

q)key keyedTable
a b
1 4
2 5
3 6
q)value keyedTable

Indeed, keyedTable is a dictionary – one that holds unkeyed tables for both its key and its value:

q)type key keyedTable
q)type value keyedTable

This suggests that we can create a keyed table by using the ! (dict) operator with two unkeyed tables:

q)(key keyedTable)!(value keyedTable)
a b| c
---| -
1 4| 7
2 5| 8
3 6| 9

Lastly, joining the two flipped tables brings us back to the original dict.

q)(flip key keyedTable),(flip value keyedTable)
a| 1 2 3
b| 4 5 6
c| 7 8 9

For more information, see Creating dictionaries and tables from C.

How do I execute a q function call with parameters over IPC?

Short answer:

q)phandle: hopen `:serverhost:5012
q)phandle (function; arg1; arg2 ..)
q)hclose phandle

To invoke a function in another q process (i.e., a function that already exists in that q process), you need the following:

  1. a process handle to the other q process,
  2. the name of the pre-existing function in the other process, and
  3. the parameters you want to pass.

Then all you need to do is to apply the process handle to a list whose first element is the function name (as a symbol) and whose remaining elements are the arguments.

For example, let’s start one q process, which will be our server, listening on TCP port 5012. In our server, we’ll define a function square (we’ll make the background color different for the server to make it easier to distinguish from the client):

$ q -p 5012             // server
q)\p                    // display the listening port
q)square: {x * x}
q)square 3

Next, we’ll start up another q process (the client), create a process handle connected to the first q process (the server), and request the first process to compute square 5:

$ q                     // client
q)phandle: hopen `:localhost:5012
q)phandle (`square; 5)  // remote execution

To call a function with more parameters, simply add them to the end of the list. We’ll demonstrate by defining a 2-argument function on the server that calculates the length of a right triangle’s hypotenuse:

q)hypotenuse: {sqrt sum square x,y}
q)phandle (`hypotenuse; 5; 12)

What if the function you’re calling doesn’t take any parameters? For example, we’ll define a function in the server called serverTime that returns the local time according to the server:

q)serverTime: {[] .z.T}

You can’t pass zero parameters over IPC:

q)phandle enlist `serverTime  // a list with just
'length                       // the function name
q)                            // is not allowed

However, you can pass whatever you like, and it won’t matter:

q)phandle `serverTime`ignored
q)phandle (`serverTime; ())    // () is often used
q)phandle (`serverTime; ::)    // so is ::

See :: (generic null).

So far, all of our examples involved a client invoking a predefined function on the server. You can also pass a function defined on the client to be executed on the server. To see this, let’s define a global variable on the server:


Now, on the client, we’ll define a function called getSERVERGLOBAL to retrieve the value of SERVERGLOBAL on the server. Instead of passing the name of the function (i.e., `getSERVERGLOBAL), we pass the function’s value:

'SERVERGLOBAL                   // not defined here
q)phandle (getSERVERGLOBAL; ::) // note the missing `

Notice that this operation does not cause getSERVERGLOBAL to become defined on the server:


This technique can be used to run built-in functions and anonymous functions (aka lambdas) on the server as well:

q)phandle (+; 2; 4)
q)phandle ({til SERVERGLOBAL - x}; 42)
0 1 2 3 4

There is one more way to convey code to the server to run: you can pass the code in a string.

q)phandle "SERVERGLOBAL + 4"

We prefer passing a function over a string, because – especially as the expression to be passed gets more complex – it’s easier to read.

This work is licensed under a Creative Commons License.
The views and opinions expressed herein are those of the authors and do not necessarily reflect those of any other person or legal entity.
Kdb+ is the registered trademark of Kx Systems, Inc.