These style of languages often come up here. Could a proponent give some examples of practical applications written in these array languages? I feel like that's missing in these discussions. People say "finance", but let's say I have a finance team that have never heard of it - why might they be interested?
I have no doubt any experienced C programmer could beat a k solution in run-time performance, but the k programmer will always be done faster.
> proponent give some examples of practical applications written in these array languages?
Databases are the most common: Building an API library to quickly answer questions about data is probably about the easiest thing to do in an array language, but isn't almost anything a database with a definition like that? E-commerce and storefronts, CRMs, flight trackers, and so on. In this way, I often suggest to people that the closest analogue is maybe something like SQL stored-procedures, but with a less-shit language. But this isn't exactly right.
Games are an interesting problem that aren't databases (for example), and a lot of games have vector-problems that have nice solutions in k. So let's play hangman.
If you don't know this game, we will only be playing with lowercase letters:
A:`c$"a"+!26; // A gets the characters cast [from] "a" + til 26
First, let's agree on a dictionary to use:
W:0:"/usr/share/dict/words"
and I will only choose a word with at least 7 letters in it and make it lowercase
so you have a chance:
w:_*1?W@&7<#'W; // w gets lowercase [the] first [of] 1 random-take [from the] Words [that/at] where 7< count each Words
Now when you make a guess:
G:();g:{G::G,x}
I'll now tell you what you know, but you're only allowed N guesses.
N:5
k:{@[w;&|/w=/:A^N#G;:;"."]}; // k gets a lambda, [that] amends w where the max-over [when the] word equal [to] each of-the Alphabet without [the first] N [characters] taken from the Guess
That's it. You keep guessing until you guess the word, or you run out of guesses.
g"e";k[]
"...e..e..'."
g"s";k[]
"..se..e..'s"
It's a silly game. I wrote comments above (with // characters so they should be easy to spot), and I now want you to take a look at them.
The comment is what I "read" when I am reading the k stanza. The square-brackets cover words (prepositions mostly) that help make it sound more like English, but look at the other words too: x=/:y is x equals each-of y. That's all it is. I am reading it and writing it that quickly. Yes it takes longer to write and read the English, and of course, any non-array language will be a lot more code to read and write as well.
That's what I think the value is in an array language: It's just so clear once you know how to read and write it, that you are simply faster. And being clear means the computer does what you expect (usually on the first try!) and so the programmer gets on to the next problem sooner.
I've written ad servers and presentation tools in array languages; I don't think there's anything I can't write (after all, almost everything is a database or a game if you squint hard enough), but there's a lot I wouldn't: I did an MP4 decoder in JavaScript once (to autoplay ads on iphones back when they blocked video playback in iframes) but I would never say JavaScript is the right language to write MP4 decoders. Array languages are tools in your toolbox -- like JavaScript or any other -- and you should try to put as many tools in your toolbox so that you have the best chance of using the right tool for the job.
> let's say I have a finance team that have never heard of it - why might they be interested?
In my experience it's very good at quickly developing real-time analytics applications with only a small set of developers. A couple of q developers can develop, maintain and operate the server side of 5 or 6 separate applications without breaking a sweat. Changes come in at a high speed too.
It's a highly interactive language. A bit like a lisp, you start up a q process, open a port and then you iterate and update your application live without needing to restart. Typically on our projects we've had a well iterated program running in QA for a day or 2 before opening a PR (which becomes more of a formality for getting the solution to the problem into prod at that stage).
The q language itself is quite wordy. Check the reference page: https://code.kx.com/q/ref/ Many programs written in q consist mainly of the key words with the special operators interspersed. Also see some example libraries: https://github.com/finos/kdb
It's been a fairly stable language to work with, having few breaking changes between successive versions. q code written 8/9/10 years ago on older versions will most likely still run the same today. We have source code on one project at work which hasn't had a code change in 6 years now (despite moving through different versions 2.8->3.0->3.3->3.5->4.0) and it runs daily without a hiccup.
Mostly it's a joy working with it because I feel like I get to tell the computer what I want it to do, without also having to tell it how to do it.
We stick with q as much as possible. We've found it's better for our front end developers who only touch it a bit here and there to make edits on the APIs they use.
That said we do make heavy use of the functional form of querying tables (https://code.kx.com/q/basics/funsql/#select) in the more complex parts of our systems. The toolbars in the GUIs allow users to select a wide range of possible ways to request data. It's a lot easier to dynamically generate the functional format of the different parts of a select query than it is to try and play with manipulating lots of stored procs. Anyone with experience in a LISP would be familiar with this way of manipulating code as data in this manner, since it's a similar concept.
Some would say this is k, not q and the lines are probably a bit blurred here. Either way there's no question that doing so has made things more productive.
As an aside, there's no performance difference between using k and q. Most of the q keywords are implemented in k. Programs written in k do look quite different to those written in q, even if you stick to the mantra of one thought per expression, one expression per line.
Myself, I used to only like q since it was easier, especially when I started. My only programming background was C# and Python; q made it easy to get into the different paradigm of the APL family. Now, after years working with it, I'd prefer to use k where-ever possible. The brevity is nice. When you write something non-trivial you see a lot of the same patterns of characters on the screen. And it just feels better to write one character for | instead of typing out reverse.
Personally I'd prefer to go with Arthur Whitney's advice from an interview he did years ago (that I can't find a link to unfortunately). He basically said work in k until you need to do big database work with kdb+, then pull in q. Convincing the team and management to let me do that is probably a lost cause though.
The main product is is “kdb+”, a simple, fast, proprietary, embedded, time series database. Especially good for tick data.
I work in finance, and literally all of my employers used it. Not much secret sauce, just simple algorithms implemented very well. No real open source competitor. Well, maybe DuckDB if they support as-of joins.
An issue in writing a thoroughly helpful response, is that successful APL and K implementations are usually applied in a largely corporate and proprietary context.
I lack a list of applications to link which you could look at.
However I'd still say APL/J/K applications require vastly less resource, and initial investment than most tools whilst providing on-par or greater computational power.
I can offer you the contrary opinion: why I would not use these kind of languages.
A couple of years ago I worked on a non-trivial APL application with one of my university professors and another student. We were trying to build a CPU simulator flexible enough to handle stuff ranging from PDP-11 up to Intel x86. The goal was to run some analysis on memory accesses performed by the x86 architecture. Quite an interesting project in which I worked on for around two year.
The first implementation was done in APL using a book which I don't remember as reference. We had a couple of meetings where we learned APL and the general idea behind the design. Pretty soon we started to deal with a lot of issues like:
- We only found two implementations for the APL interpreter: GNU and Dyalog. GNU is free but pretty much abandoned. Support for Windows was (is?) nonexistent. Dyalogs version is proprietary so we couldn't use that (even when a "student" version was available).
- You had to use a special keyboard layout since APL uses non ascii characters, which was only available for Linux. Every time you wanted to write some operator I had to look it up in a paper sheet. Eventually you start memorizing, but it was a pain in the earlier days.
- There is literally no community support. You can't just 'StackOverflow' some APL stuff.
- Dynamic scoping + dynamic typing make working on a large codebase pretty much impossible. You don't know when things are defined, and you don't know what they are.
- The language is extremely terse. I remember that a single line of code was able to fetch, decode and execute instructions. You're incentivized to write this kind of code, and even if you're able to understand all operators you still need to figure out the semantics of the code. For reference, I work with Haskell everyday and it looks like Java compared to APL.
- The code tends to be very hacky. You (ab)use the fact that most things are arrays and solve all problems using some form of indexing/map/reduce. You tend to forget about proper conditionals or custom data types. Domain modeling is pretty much non-existent.
- There is no built in support for test, and we could not figure out how to use a third party library for that (is it even possible to use third party code?). Our tests consisted in a lot of code that checked some conditionals, and then we printed "Pass" or "Fail" on each. Very primitive stuff.
- No debugging at all. You run the code and pray for the best.
A year after we started my classmate decided to drop the project since he felt he couldn't keep up with the complexity: each line of code was non-trivial and really hard to understand.
Eventually we had to rewrite the whole project because GNUs interpreter didn't support big integers, and trying to circumvent that resulted in very poor performance. The new version was written in Julia (https://github.com/emlautarom1/Julia_Simulator), so we were able to reuse a lot of "array code". The project got cancelled in the middle of the rewrite and we kind of forgot about it.
This might have been true a couple of years ago but it is totally untrue now.
I'm not sure why you couldn't use the student version of Dyalog? Sounds like it would have been fine. There are also many more FOSS implementations of array languages now, such as ngn/k and April. https://github.com/phantomics/april
'only available for Linux' - not true https://github.com/abrudz/Kbd/ and others (also different input modes like `w for ⍵)
- My professor was completely against the idea of using proprietary software. Also, we were supposed to use APL, not other languages.
- I see that support has improved. I didn't find any keyboard support for Windows at the time. Still, dealing with a different keyboard layout was a total pain.
- I wouldn't call the APL community "big". The're are around 2K users combining all of those platforms. Now include the language barrier (I'm from Argentina) and go back a couple of years and you can see the issue.
- The APL version we were using had dynamic scoping. TIL that there are multiple variations of the same language with totally different semantics!
- Yes, that level of terseness involves hacky code which is hard to understand, review and mantain. This is 100% my opinion.
Overall I wouldn't recommend APL or derivatives at all for any serious project.
Or derivatives?? Generalizing from GNU APL to APL as a whole is a stretch in my opinion, but telling people not to use K on the basis of your APL experiences is something else entirely. They are very different languages. For example K uses only ASCII and has never supported dynamic scoping, only (somewhat restricted) lexical scoping.
'The APL version we were using had dynamic scoping. TIL that there are multiple variations of the same language with totally different semantics!' - your own code uses localisation within tradfns! Dfns just do that by default.
