The kcats Programming Language

See Releases. Binaries currently do not have installers, you will need to place the executable in a convenient place and chmod it (on mac/linux).

You can skip this section if you're using pre-built binaries described above.

It can be checked out from git or viewed here: Production Source

See emacs-ide.org in the source tree for more info.

Now that we know the very basics, we can explore and learn as we go. Kcats lets you treat the standard library (the dictionary) as data, and you can process it with… itself. Documentation is in there. You just need to know how to ask for it. So here's how you do it, and you'll understand how exactly it works later.

In all the examples in this document, you can run them on the command line, by running kcats -p, followed by the program in single quotes, like this:

kats -p 'words [first] map [] sort'

In case you want to view it in its entirety, the standard library is part of the source, it lives in the lexicon file.

This program retrieves the dictionary of the current environment, and prints just the name of each, sorted in alphabetical order.

words [first] map [] sort

[* + - / < <= = > >= abs addmethod advance and animate any? assemble assert assign
 association association?
 attend autoformat bail bits both both? branch break breakpoint butlast bytes? cache
 catcher ceiling clone clonedeep clonedown close collect combinations
 compare confine contains? count cram cut database days dec decache decide decodejson
 decorate decorated definition dictionary dictmerge dip dipdeep dipdown
 dive divedeep divedown draft drop dropdeep dropdown dropper dump each emit empty
 empty? encode encodejson encodenumber encodestring ends? entry environment
 environment? error? eval-step evaluate even? evert every? execute exp fail file-in
 file-out filter finished? first flatten flip float floor fold
 format frequencies future generate generator get group handle handoff hashbytes heatmap
 hours if inc indexed indexer indexof inject inscribe inspect
 integers interpose intersection into join joiner keep label last let liberate list?
 log lookup loop map max method? milliseconds min
 minutes mod module negative? not number number? odd? or over pack pad pair pair?
 pairwise partition persist pipe-in pipe-out pipe?
 pop positive? prepend prime primrec print put quot radix range read receiver recover
 recur rem repeat resolve rest restore retry
 reverse round second seconds select sender serversocket set set? shield shielddeep
 shielddown sink siphon skipper sleep slice slurp snapshot socket
 something? sort sort-indexed spawn spit split sprint sqrt stage standard starts?
 step stepper string string? swap swapdown take taker timer
 times timestamps toe tos tracer triplet tunnel type unassign under until unwrap update
 updates use using value walk when while
 within? word word? words wrap xor yes zero? zip]

Even though the rest of this document will explain a lot of these words and how they work, the above program does the following:

• words: retrieves the dictionary words, and puts a copy of them on the stack
• [first] map: for each item in the dictionary words (which is a key/value pair, where the key is the word and the value is the definition) take the first, which is the word.
• [] sort: sort takes a program to transform each item in the list it's sorting, to use for comparison. We want to use the word itself for comparison, so we don't need to transform it at all, hence the empty program.

The specification of a word's input and output types is stored in the dictionary too. It's in the property called spec. Let's say you're interested in the word swap.

words [swap spec] lookup

[[[item a]
  [item b]]
 [[item b]
  [item a]]]

What this program does is fetches the dictionary words, then looks up the swap definition, then within that definition, looks up the spec property.

In the result, what we have here is two lists - the spec of the input, and the spec of the output.

The input spec is [[item a] [item b]]. The output spec is [[item b] [item a]]. What it's telling you is that it requires two items on the stack, any two, we'll call them a (on top) and b beneath. There may be more items below that but they won't be touched. When swap is finished, a and b will have their places swapped so that b is on top. And in fact that's what we get:

"b" "a" swap

"b" "a"

Remember the top of the stack is printed first, and so b is now on top.

words [swap examples] lookup

[[[1 2 3 swap] [1 3 2] "Swap top two items"]]

This is a list of examples, and each example is a triple:

• A program that calls the given word
• A program that doesn't call the word that gives the same result ³
• A description of what the example is demonstrating

Use the same technique to explore other words. You can simply replace the word in the code snippets above with some other word. Here's how you find the examples for =, which tests for equality of two items - just replaced swap with =.

words [= examples] lookup

[[[1 2 =] [[]] "Different Numbers are not equal"]
 [[1 1 =] [yes] "Same numbers are equal"]
 [[1 1 =] [yes] "Same value integer and float are equal"]
 [[[1] [] =] [[]] "Number and Nothing are unequal"]
 [[[1 [[]]] [1 [[]]] =] [yes] "Same nested list with numbers are equal"]
 [[[1 ["foo"]] [1 ["foo"]] =] [yes] "Same nested list with string are equal"]
 [["hi" "hi" =] [yes] "Same strings are equal"]
 [["hi" "there" =] [[]] "Different strings are unequal"]
 [[\h \h =] [yes] "Same characters are equal"]
 [[\h \i =] [[]] "Different characters are unequal"]
 [["hi" encode "hi" encode =] [yes] "Same bytes are equal"]
 [["hi" encode "there" encode =] [[]] "Different bytes are unequal"]
 [[[] yes =] [[]] "Different booleans unequal"]
 [[[1 ["foo"]] [1 ["bar"]] =] [[]] "Nested lists with different strings are unequal"]
 [[[] [] =] [yes] "'Nothing' is equal to itself"]
 [[[] [] association =] [yes] "List/Association empty container types are equal"]
 [[[] [] set =] [yes] "List/Set empty container types are equal"]
 [[[[a b]] [[a b]] association =] [[]] "Nonempty List/Association types are unequal"]
 [[[1 2 3] set [3 1 2] set =] [yes] "Sets constructed from different lists are equal"]]

There are words that operate on multiple types, and it's helpful to talk about what those types have in common. Specs use these traits to describe groups of types that a word will accept or produce.

Data types are automatically converted when needed.

For example, if you have a list of pairs and you use the word lookup, it assumes your intention is to use the list as an associative data type, so it will be automatically converted, and remain converted after lookup completes.

You can often tell by the spec when the return type is a promoted type:

words [assign spec] lookup

[[[item value]
  [list keys]
  sized]
 [association]]

Here you can see that the spec for assign takes a sized and returns an association. This allows you to do things like this:

[[name "Susie"] [age 25]] [sport] "bowling" assign

[[age 25]
 [name "Susie"]
 [sport "bowling"]]

The initial value of [[name "Susie"] [age 25]] is not an associative, it's just a list. You could explicitly convert it using the word association but assign will do it for you, because it is a function that operates on an associative type.

Note that the conversion can fail, because converting to associative requires that you have a list of pairs. If you don't, that's an error:

["foo" "bar"] [age] 25 assign

[[actual "foo"]
 [asked [pair]]
 [handled []]
 [reason "type mismatch"]
 [type error]
 [unwound [assign]]]
25 [age] ["foo" "bar"]

The most common promotion is from list to associative but there are others.

if takes 3 programs from the stack:

• condition a program whose result decides which branch to take
• the yes branch
• the no branch

An important detail: after condition runs, its stack effects are erased. The yes or no branch runs on whatever was underneath the 3 programs at the start.

1 2 3 [odd?] ["it's odd"] ["it's even"] if

"it's odd" 3 2 1

Notice how the 3 is still there. The word odd? normally consumes its argument.

3 odd?

yes

Here's a more extreme example:

1 2 3 [drop odd?] ["it's odd"] ["it's even"] if

"it's even" 3 2 1

See how we drop the 3 and test odd? against 2 instead? Normally we'd have consumed both the 3 and the 2 but the conditional is not allowed to have any stack effect. See 5.11.

Loops take a program to run, and a boolean (See 5.4.1.2) condition whether to run the program. If the condition is false, the program never runs. If it's true, the program runs, and loop expects another boolean condition to be on top to see whether to run the program again.

Note that the item on top only determines whether the program runs again, it's dropped and not accessible to the program. If the program needs it, be sure to clone it. Usually it doesn't need that item for anything except deciding whether to continue the loop, which is why it's dropped automatically.

Here's an example:

1 yes [2 * clone 100 <] loop

128

Notice that loop receives the program and yes the first time. The program never sees yes, only the 1 underneath it - it multiplies it by 2, then clones it and checks if it's less than 100. If so, it drops that boolean value and continues and multiplies the number beneath by 2 again, and so on, until the number is greater than or equal to 100. Finally that false value is dropped and the loop is done, leaving just the final number 128.

Kcats also has while, which is a bit higher level than loop. Instead of expecting a boolean value on top each time through, you provide a condition program similar to what if requires. while runs the condition program, if it leaves a affirmative value, the loop continues. Like loop, while's body program does not have access to the affirmative value.

1 [100 <] [2 *] while

128

Like if, the condition program cannot permanently affect the stack. So we don't need clone like we did with loop. After we compare the number to 100, it's restored so the body program can see it on top.

It's just like while but with the condition's logic reversed, so that it stops when the condition is true.

1 [100 >=] [2 *] until

128

Unlike while (which runs the body 0 or more times), until will always run it at least once.

1 [yes] [2 *] until

2

One unusual and important feature of kcats is that you can program the runtime. What does that mean? It means you can control exactly how a program is executed.

Why would you want to do that? Well, one common use case is that you want to debug the program: you'd like to manually control execution while you examine the state of the program, to figure out where it's going wrong.

Another use case is security: you want to execute a program but not allow it to access things that programs normally can access (like the filesystem or network). You also don't want the program to be able to permanently redefine what words mean.

So how exactly do we alter how programs are executed? A kcats program keep track of 3 things as it is running, its state: the stack (the data the program is working with), the program (the remaining instructions left to be executed), and the dictionary (the words that have meaning in a program). We call the whole state an environment, and we can create and work with environments in kcats just like any other data, including executing the progams within them.

First let's look at how to create an environment:

[[program [1 2 +]]] environment

[[dictionary [[modules []]
              [words 249_entries]]]
 [program [1 2 +]]
 [stack []]]

Notice how `environment` takes an association and fills out the `dictionary` and `stack`. Why does it say `249<sub>entries</sub>`? Normally dictionaries would be printed out in their entirety, just like any other data, but because they're large (there are hundreds of words) by default the runtime prints it as the count of the words.

So this is an environment. What words are useful here? Well, first of all we can treat it as an association, which it is. Here we replace the last item in the program so we're subtracting instead of adding.

[[program [1 2 +]]] environment
[program] [pop drop [-] join] update

[[dictionary [[modules []]
              [words 249_entries]]]
 [program [1 2 -]]
 [stack []]]

Ok, well, this is not that exciting, if environment is just another kind of association right? Point taken, but now let's use the word `eval-step`:

[[program [1 2 +]]] environment
eval-step

[[dictionary [[modules []]
              [words 249_entries]]]
 [program [2 +]]
 [stack [1]]]

That evaluates the environment one step! We can do two more steps to finish the program:

[[program [1 2 +]]] environment
[eval-step] 3 times

[[dictionary [[modules []]
              [words 249_entries]]]
 [program []]
 [stack [3]]]

One hard rule of kcats: "an environment cannot alter its own dictionary". That means, a program that's running can't do anything to change the meanings of words that come later in the program. This is in order for kcats to be able to provide a relatively trustworthy execution environment - you can load a library and be sure there's nothing it can do to alter the meaning of your program. Monkeypatching is not allowed.

So the question then is, if we can't alter the current dictionary, what can we do? You can run your program inside another environment with a different dictionary.

The word dictionary retrieves the current dictionary and places it on the stack. Then we can treat it like any other data and alter it. Then we can create a new environment and evaluate that environment, and retrieve its stack.

Doing this with low level constructs looks like this:

;; Fetch the current dictionary
dictionary
;; Create a new set of words
[[square [clone *]]] draft
;; apply the change to the dictionary
shielddown
;; The program to run with the new dictionary
[9 square]
;; Create an environment and evaluate it
[program dictionary] label environment
evaluate
;; make the inner stack the new stack
[stack] lookup restore

81

Obviously this is quite cumbersome, so there are higher level words to do all of this for you, like `let`.

The most straightforward and common change you can make to the dictionary, is to add a word that wasn't in there before, and use it in some limited scope after which it is no longer accessible.

The word let is handy for small bits of code where you don't want to repeat yourself:

[[square [clone *]]]
[9 square 8 square +]
let execute

145

Let's break this down. The word let takes two arguments, a list of new words paired with their definitions, and a program to run that uses those words. We define a new word square to mean [clone *], and then we create a program that runs [9 square 8 square +] inside an environment with the new word square defined. Then finally execute that program to get 145.

The list of new words can even refer to another word from the same list:

[[square [clone *]]
 [fourth [square square]]]
[3 fourth]
let execute

81

Words you're defining can refer to themselves - recursive functions are great!

[[factorial [[swap positive?]
             [[*] shielddown [dec] dip factorial]
             when]]]
[9 1 factorial dropdown]
let execute

362880

Creating new words is relatively safe - presumably no one is using those words, so giving them meaning doesn't cause any confusion. It's when you start changing an existing meaning that things get hairy.

As we have seen, words perform actions, and those actions are specified by other words. So a word foo can use the word bar as part of its execution. So let's say I change the meaning of the word bar. Does that mean I changed the meaning of foo as well (because bar is part of the meaning of foo)?

The answer is it depends.

In kcats, by default, the answer is no. When you are changing the meaning, it's for you own immediate use of the word. Let's go over some examples.

Let's say we want to alter the meaning of a rather important word that's used all over the place in the standard library: swap. "When I say swap I want to just insert the number 5."

[[swap [5]]]
["a" "b" "c" swap]
let execute

5 "c" "b" "a"

Ok, straightforward enough, right? But what happens if we call another word that uses swap, like times? First let's look at the definition of times, to see that it really does call swap:

[times] definition

[swap [dec] swap put [dip] join [0 >]
 swap while drop]

It calls swap quite a bit! Now let's alter the meaning of swap and call times.

[[swap [5]]]
[["hi"] 3 times swap]
let execute

5 "hi" "hi" "hi"

Notice that times still works as expected! Even though it calls swap internally, it didn't insert any 5's. Only our own swap did that.

You can even refer to the old behavior of a word when defining new behavior:

[[swap [5 swap]]]
[["hi"] 3 times swap]
let 

"hi" 5 "hi" "hi"

Here we redefine swap to mean "insert 5, and then do whatever swap did before". This works even though swap is an axiom word:

[swap] definition

builtin

There are two other words related to let that are more flexible for times when you need to do a bit more complex alterations of existing meaning. One of the primary use cases is adding a method. You've got a word that behaves differently depending on its argument and you want to add a new behavior.

[[hash [[type [foo] unwrap =]
        [drop "foo" hash]
        addmethod]]]
[[[foo myfoo]] association hash] revise
"foo" hash =

yes

What have we done here? We're taking the word hash, which is just a decide:

[hash] definition

[[[[bytes?] [hashbytes]]
  [[yes] [encode hash]]]
 decide]

See how it behaves differently for a byte array by calling hashbytes and by default it calls encode and then tries hash again? So now we've added a new logic branch there:

[[hash [[type [foo] unwrap =]
        [drop "foo" hash]
        addmethod]]]
[[hash] definition] revise

[[[[type [foo] unwrap =] [drop "foo" hash]]
  [[bytes?] [hashbytes]]
  [[yes] [encode hash]]]
 decide]

When the type is 'foo', we use the hash of the string "foo".

Now, there are times when you do want to alter the meaning of a word globally. It's called Monkey Patching, and it's not something you should do unless you have no other choice. It can make a program's behavior very hard to analyze or debug, and is very easy to misuse, and is referred to as a "foot gun", as in, a gun with which it is very easy to shoot yourself in the foot.

Kcats does support both temporarily and permanently monkey patching. See the words define for permanent and lingo for temporary.

One use case for permanent changes might be to add methods to a multimethod. If done carefully, it can add new behavior without interfering with existing behavior.

dictionary
[encode definition] [[type [foo] unwrap =]
                     [drop "foo" encode]
                     addmethod] 
update
define
[[foo myfoo]] association hash
"foo" hash =

yes

Here we change how a foo is encoded, and specify it's always encoded as the word "foo" converted to bytes. Then we test that equality holds.

Sometimes in programming, having the concept of an indefinite sequence is handy. You have part of your program producing data, and another consuming it, but the producer doesn't know how much the consumer will actually need. A producer might calculate a huge number of items at great expense, only for the consumer to only need a tiny fraction of them. Generators allow the consumer to tell the producer when to produce, but the producer still retains all the logic of how that's done.

In kcats there's no special sauce for generators, we can implement them as a pattern with just the standard words we've already seen.

Let's say you want to create the fibonacci sequence. Let's see how we can code that without worrying about how many items in the sequence we'll eventually need.

A generator consists of two things: state, and a program. Each time we want to generate an item, we run the program. The program should produce a new item and update the state. We just put however many state items we need on the stack, and then a program that can work with those items.

1 0 [[+] shielddown swap clone]

So here we start with 1 0. That's the starting state. Normally we'd start fibonacci with 1 1 but this isn't the actual first two numbers in the sequence, it's starting values we use to calculate them. Then we have a program that takes two numbers as input and leaves one new number. Let's just execute that program and see the result:

1 0 [[+] shielddown swap clone] execute

1 1 1

We can see the 0 is now 1 and there's an extra 1 on the stack. Remember the generator must do two things, produce a new item and update the state. It updated the state from 0 1 to 1 1, and produced the first item, 1.

This gets us one number, but not the whole fibonacci sequence. Let's look at the word generate. All it does is run the program, pulls the generated item to the top of the stack, and puts a new copy of the program in place so that when we want the next item, we can call generate again:

1 0 [[+] shielddown swap clone] generate

1 [[+] shielddown swap clone] 1 1

Notice here that the only difference from before is that the program is sandwiched between the fibonacci number we produced, and the state.

Let's keep going and call generate again! But wait, before we do that we need to do something with item we just produced, to get it out of the way. For now we'll just drop it. We've seen it and we want to see what's next.

1 0 [[+] shielddown swap clone] generate ;; what we had before
drop ;; throw away the first item
generate ;; the 2nd item

1 [[+] shielddown swap clone] 1 2

Ok, so the 2nd item is 1 and we can see the state is updated - instead of 1 1 we have 1 2.

One more time:

1 0 [[+] shielddown swap clone]
[generate drop] 2 times ;; generate and drop the first two items
generate ;; the 3rd item

2 [[+] shielddown swap clone] 2 3

Ok we can see that we can get items one at a time by calling generate, but this is not very useful. What we really want is to get the first 20 numbers in the fibonacci sequence, and collect them into a list. We can do exactly that:

1 0 [[+] shielddown swap clone] ;; our original generator
20 taker ;; another generator that stops generating after 20 items
collect ;; collects all the generated items into a list

[1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765]
[[positive?] [dec [generate] dive] [[]] if] 0 [[+] shielddown swap clone] 6765 10946

There's the fibonacci sequence! Hey wait, what's all that stuff after it? We just want fibonacci! That's there in case you wanted to keep generating more items. If you want to just get the result and throw away the generators, you can do that with shield, which erases all stack effects except whatever was on top. So we'll just shield the entire thing:

[1 0 [[+] shielddown swap clone]
 20 taker
 collect]
shield

[1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765]

So what is happening here? We're stacking up generators. Starting with the last, we have collect which will repeatedly call generate on the generator below it. It keeps going and collecting the generated items in a list, until the generator below returns nothing. Then it stops and returns what it collected.

Then below collect we have a generator 20 taker - what that does is keeps its own state of how many items we want it to take. It counts down as it generates items below it, passing them up to collect and when it hits zero, it returns nothing (even if the generator below it would have produced something, taker won't even ask). That will signal collect to stop.

We have other handy generators we can stack up. Let's say for whatever reason we want to know what are the first 20 odd fibonacci numbers? Well, we have keep:

[1 0 [[+] shielddown swap clone] ;; our original generator
[odd?] keep ;; a generator that keeps calling the one
            ;; below it until it gets something that
            ;; passes the predicate we specified
20 taker ;; another generator that calls generate 20 times
collect] ;; collects all the generated items into a list
shield

[1 1 3 5 13 21 55 89 233 377 987 1597 4181 6765 17711 28657 75025 121393 317811 514229]

There it is, the first 20 odd fibonacci numbers!

Let's say instead we wanted to know the prime factors that make up each of the first 20 fibonacci numbers. We can do that with each:

[1 0 [[+] shielddown swap clone] ;; our original generator
 ;; a program to give the prime factors of a given number
 [[] swap 2 ;;  current-divisor input result
  [[sqrt] dip >=]
  [[mod zero?] 
   [clone ;; c-d c-d i r
    sink ;; c-d i c-d r
    [put] dipdown ;; c-d i new-r
    / 2] ;; dividend new-r
   [inc] ;; c-d++ i r
   if]
  while
  drop put]
 each

 20 taker ;; another generator that calls generate 20 times
 collect] ;; collects all the generated items into a list
shield

[[1] [1] [2] [3] [5] [2 2 2] [13] [3 7] [2 17] [5 11]
 [89] [2 2 2 2 3 3] [233] [13 29]
 [2 5 61] [3 7 47] [1597] [2 2 2 17 19] [37 113]
 [3 5 11 41]]

There we have it. We can see that [2 2 2] is what makes up 8, etc.

Other included generators are:

dropper

Inverse of taker - drops the first n items of the sequence and returns the rest.

joiner

Joins items together

integers

all the numbers starting with 0

reduce will consume what a generator produces. You provide a program that takes 2 arguments, and reduce will generate all the items, and pass to your program: the result so far and the next item generated, and repeat that until there are no items left:

[integers
 1 dropper ;; drop 0 so we start with 1
 10 taker
 [3 *] each
 [+] reduce]
shield

135

Let's say you go to the trouble of making a beautiful stack of transformations and you want to re-use it, but you don't have a generator, you have a list! Our transformation stack needs a generator! How are we supposed to use it? Never fear, there is a simple way to adapt transformations to work on anything that works with the word take. You can use the word liberate to convert a list to a generator. (It's just an alias for [take] which is even shorter than liberate so feel free to just use [take]).

Do you see why [take] converts a list to a generator? Remember, generators are a state and a program. If we already have a list or pipe, we can just treat that as the state. And [take] as the program does exactly what we want, removes an item from the list and returns it, leaving the state with one fewer item.

One pitfall with generators is that sometimes you want to transform generated items and use some item from the stack to help do it. The problem with this is that generators can be arbitrarily deep and you won't know exactly how deep that item is.

The solution is to capture the items you want.

Let's look at a simple example. Let's say we want to generate every multiple of n (where n is some number on the stack). We can already generate every integer, we just need to multiply each one by n. A naive solution would be to just use each, but it doesn't work:

3 integers
[*] each
1 dropper ;; drop 0
10 taker collect

[[reason "type mismatch"]
 [unwound [* [[0 [inc clone]
               0 3]] unwrap evert first dropdown [[generate [[*] bail]
                                                   shielddown]] unwrap swap drop [1] unwrap dec [positive?] shield [[generate drop]
                                                                                                                    dip dec [positive?] shield]
           loop [generate swap]
           dip float
           [[[[positive?] [[generate drop]
                           dip dec]
              while [generate swap]
              dip float]
             bail]] unwrap swap [9] unwrap swap [[[positive?] [dec [generate] dive] [[]] if]]
           unwrap swap [] swap clone [put [generate] dip swap clone] loop drop]]
 [actual [inc clone]]
 [asked [number]]
 [type error]
 [handled yes]]
0 [inc clone]
0 3

This doesn't work because n and the last integer we generated aren't next to each other on the stack, there's a bunch of generator machinery in between. We could try to guess exactly how deep the machinery is, but then our generators aren't composable anymore - we couldn't move that call to each somewhere else in the generator stack, without having to change the program. What we really should do is create our program for each first, before we start stacking up generators, and bind n:

3 
[*] bind
[integers] dip ;; insert the integers generator below the each program
each
1 dropper
10 taker collect

[3 6 9 12 15 18 21 24 27 30]
[[positive?] [dec [generate] dive] [[]] if] 0 [[[positive?] [[generate drop]
                                                             dip dec]
                                                while [generate swap]
                                                dip float]
                                               bail]
0 [generate [[3 *]
             bail]
   shielddown]
[inc clone]
10

What exactly is this doing? We're taking values from the current stack, and prepending them to a program, so that later when the program executes, it'll find that value on the top of the stack. Put another way, we're binding the value of the first argument to the program now, rather than letting it take a value from the top of the stack later.

In kcats, both coordination and input/output are done with pipes. See the definition for pipe.

Let's take a common example of coordination. Your program has to do several very long and intensive calculations but doesn't want to make the user wait to do other things. The way that's done in kcats is by creating multiple environments, and have them communicate with each other using pipes. You can send any item through a pipe that you could put onto the stack, including other pipes. You can clone a pipe to give access to it to more than one environment.

There are two main operations a pipe supports: put and take. You either put an item in, or take an item out. Either one of those operations may block, if the pipe is either full (when putting) or empty (when taking). Your environment would have to wait for some other environment to take something out so there's space to put, or put something in so that there's something to take out.

All pipes share the put and take operations but they can differ in other ways.

Note that put and take can also be used on plain lists. put adds to the end, and take removes the first item. Neither will ever block when used on a list. Another slight difference is what happens when you've reached the end of the content (either the list is empty or the pipe has, for example, hit the end of file condition): a take from an empty list will just return nothing, but a take from a pipe that is at EOF will result in an error.

Let's look at how we do I/O using files as an example - let's say we want to write the word foo to a file called bar:

[[file "bar"]] pipe-in ;; create the pipe to the given file "foo"
"foo" encode ;; we have to convert string to bytes first, using the word
            ;; =encode=.
put ;; finally, put the bytes into the pipe, and they are written to
    ;; the file

[[to [[file "bar"]]]
 [type tunnel]
 [values [[type bytes]]]]

Note the representation of the pipe shows where it leads (the to field), and what types of items it can carry (the values field).

Neither put nor take consume the pipe from the stack, for convenience, as most of the time you'll want to use it again.

Let's look at reading from a file:

[[file "bar"]] pipe-out
take string

"Hello World!" [[type tunnel]
                [values [[type bytes]]]
                [to [[file "bar"]]]]

Note that the amount of bytes you'll get from a file on each take, is limited. You will only get the entire contents if the file is small. We'll want to repeatedly take until there's nothing left, and put all the taken parts together.

Here's how we do it:

• turn the pipe that provides chunks of a file into a generator, with [take].
• Assemble the chunks with reduce. It requires a program to say how to combine the chunks. We want to join them, so the program is [join].

We can also use the word file-out as a shortcut to get a pipe given a file's name.

"bar" file-out [take] join reduce string

"Hello World!" [take] [[type tunnel]
                       [values [[type bytes]]]
                       [to [[file "bar"]]]]

Finally there's a convenient alias for [take] [join] reduce string, it's called slurp:

dictionary [slurp] lookup

[[definition [[take] [join] reduce string [drop drop]
              dip]]
 [spec [[pipe] [item]]]]

It actually drops the generator for you as well, since we know it's already been fully read from. So you can do this:

"bar" file-out slurp

"Hello World!"

It's tempting to want the flexibility of EAV (where there's basically just one big db table with 3 columns and every attribute is a row).

However this may be a little hasty. Perhaps what we're really after here is custom tables - the idea being that each user's db schema might be different depending on what data is important to them.

We've basically got a database schema consensus problem. Maybe Alice has a table CATS with columns SIZE COLOR AGE and Bob has a table CATS with columns HEIGHT COAT-COLOR AGE. How do they share data? The two tables are not really compatible without a specialized conversion tool and even then some data would be missing. So Alice and Bob ideally should agree on what a CATS schema is, otherwise they can't really share CAT facts. The advantage of EAV might be that even if they had different schemas they could stlil perhaps meaningfully talk about AGE and possibly even COLOR (with a bit of intervention, or even another fact that equates COLOR and COAT-COLOR in CATS).

The drawback of EAV is of course that it would perform rather terribly as the database grows. I can't say for sure how many facts could potentially be stored here, but here are some constraints:

• Assume individual data only (no facebooks that store millions of people's data)
• Assume popularity of the app (users may try to cram every fact they "know" into this db)

• Assume there's some kind of garbage collection - Alice may collect weather observations or predictions constantly but doesn't need to keep old data. Maybe facts have a TTL? Not sure how that could be determined automatically.

  It's hard to estimate how large the db might get, but I suspect a lower bound of supporting 1M words is safe. As for upper bound, it's more difficult to say, but I would think the hardware limits of mobile devices would come into play. As of 2023 I think a db size on the order of 10gb would be approaching the device's capability limits, so maybe 100M words or so. I think it would be difficult to get an EAV database to perform well at that size, especially on mobile. Note datomic can handle that size so it's theoretically within reach.

  It may be possible to pick a standard db now (sqlite maybe) and not worry too much about performance. As long as the facts are portable to another db (which shouldn't be that hard), the issue can be revisited when it becomes an issue.

  Even using sqlite though, just building proper queries may be difficult. It may be possible to skirt that problem too and just do a minimal query to get a dataset that fits easily in memory and then post-process the rest. Let's say the query is "List all predictors (people who made predictions) and their accuracy", you could get all the unique predictor ids in a query, then one by one get all their predictions, then get all the relevant observations and compare them. Slow but not the type of query that will be done often, and possibly indexable.