Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Statistical Analysis with Lisp-Stat (lisp-stat.dev)
101 points by ngcc_hk on March 24, 2022 | hide | past | favorite | 44 comments


One thing to keep in mind is that common lisp is actually a super new and exciting language... All the implementations used to be proprietary for a long time (but there's been widespread acknowledgement of the merits of the language from the very beginning).

Now that we've got super awesome FOSS implementations the party has gotten started and immediately there are so many crazy cool things coming out. I think common lisp is basically a better rust for a lot of the things rust is being used for.. Although of course rust is way better for many things also. It's not really an apples-to-apples comparison but the point is that common lisp is quite fast and good for application development (which rust is too but often with more complexity than necessary).

Another thing to remember is this rule of thumb where if you want to make a neutral prior for how long a technology will be around you should basically just look at how long it has already existed. Something exists for a week, probably it will be around for another week. Something exists for a decade, probably it will be around for another decade. Of course this is not a very detailed assessment but SICL and CIEL are respectively efforts at minimizing the bootstrap and modernizing the final product. There will be more of these efforts and eventually lisp will become a totally different language (as we approach the energy minimum for the vocabulary)... Essentially what I am trying to say is that right now; lisp code is the code that I have the most confidence in w.r.t. longevity.


SBCL was adopted in production in Lisp shops like ITA circa 2005 or so? So even if you consider production-grade, open source Common Lisp only, it's still older than Go or Rust. Common Lisp is not "super new" by any means, but it is still exciting. It's like the redwoods of California: ancient, but evergreen.


And SBCL is fork of long mainstay of open source Common Lisp, CMUCL (pretty sure ITA moved from CMUCL to SBCL)


I don't have exact dates, but I definitely used cmucl on linux/x86 before 2000 (though not much before; 1999 maybe?).


IMO Common Lisp is extremely unlike Rust, which is basically (C++)++, and I would not use them for the same things


CL is quite unlike Rust, but I've actually found lots of similarities between CL and C++: religiously stressing backwards compatibility and the ability to run very old software, providing the ability to hook and customize as many things as possible while providing default implementations, being unopiniated with regards to programming paradigms...


Symbolics and Xerox PARC beg to differ, and their products failed to establish themselves more due to their mismanagement than technical.

Rust still needs to grow up quite a bit before it can assert to be (C++)++ specially in anything that involves a GPU.


> I think common lisp is basically a better rust for a lot of the things rust is being used for..

More like a superior python with an awkward syntax and some crazy legacy (loop macro being the worst offender perhaps).

Just the other day, my ipython interpreter killed itself after having an uptime of 8 hours. No reason, just got tired of living. In such occasions, I hope I would've spent more time assimilating my brain to some common lisp craziness so I wouldn't have to deal with shit like that.


LOOP gets weird flack from some purist mindset. There's no issue with it in practice.


purist? no, it's a very practical mindset. if you are into lisp, you should already appreciate the power of parenthesis. i literally have to spend time parsing the barren (longish) loop macros to understand wth is going on and what keyword is suppossed to stick to where. in other, totally unrelated news, forget about customizability and embrace the ever-crappy indentation (especially with aggressive-indent) and the awful non-sexp-based editing. what looks like purism is in fact guided by simple pain avoidance.

no, fren, iterate all the way. it is regular common lisp code and doesn't make you context switch due to some _obviously_ misguided DSL design.

i don't understand why some lispers are so adamantly supportive of LOOP. what's more, some of them _despise_ iterate (because reasons). i just don't get it.

PS As a bonus, iterate has an actually usable manual [1]. For LOOP, i have to go to gigamonkeys and whatnot.

[1] https://iterate.common-lisp.dev/doc/index.html#Top


To each their own. I'm a huge fan of LOOP. Is very easy to read, feeling much more natural to how I think about those sections of code.


Yes, I was a fan of LOOP too, until I found iterate.

I mean, we are discussing the two things comparatively here, aren't we? So, how is loop better than iterate? Iterate has nearly the same semanitcs as LOOP. It's basically LOOP on s-exprs, with some handy idioms sprinkled in. But often times the match is 1-1. So, how is the absence of sexprs making anything easiers to read or write? It makes it harder for me in both cases.


LOOP is better than ITERATE because I don't have to load a library to use it, and I don't have to tell my colleagues learning Lisp to "please ignore the built in iteration construct and instead use the one that has more parentheses from this library."

Also note that LOOP is an S-expression. Just because there are more sub-lists doesn't make it more or less of an S-expression.


> Also note that LOOP is an S-expression.

Well, it's body doesn't look like s-expressions to me, that's the point. And there's 0 reasons why they shouldn't be. But, conversely, there are real benefits to them being sexps. Coherency, for one.

It's not the first time I hear this defense: "it's part of CL standard therefore it's lispy", "it's just a DSL". But these are qualities, aren't they? Some things will look lispier that other things. Some DSLs are better designed.

Look at the DO keyword. Why exactly do I need it? Why do I need :when or :if when I have cl:when and cl:if?

> don't have to load a library to use it, and I don't have to tell my colleagues

That's a valid point, this is the price you would have to pay (and don't forget importing the symbols!). But you just might have to gain more in the long term, if you switch. I for one, don't mind, but It Depends TM.

If I had a large project nearing its completion that uses LOOP, i wouldn't switch to iterate, sure. But on a new project? I might as well convince everybody else just to try it out for the hell of it.


I'll try iterate. For me it is just that loop is already very easy to read. Moving all things to s exprs doesn't seem like it would be that big of a deal.

And I should say that the construct that I find easiest is the "then" clause.

    (Loop repeat 8
          For a = 0 then b
          For b = 1 then c
          For c = 1 then (+ a b)
          Collect a)
That just reads very easily to me.

(Typed on phone...)


First of all, if you are going to try it out, I recommend using this macro:

   (defmacro itr (&rest rest)
     `(iterate:iterate ,@rest))
because it keeps all the clauses on the same indentation level.

Now, to the translation: iterate has previous clause. https://iterate.common-lisp.dev/doc/Previous-Values-of-Drive... which may help translate the loop form you provided into this:

   (itr (repeat 8)
        (for a previous b initially 0)
        (for b previous c initially 1)
        (for c = (+ a b))
        (collect a))
However, if you wanted that exact syntax, you could try:

   (defmacro-clause (FOR var = init-value THEN form)
     `(progn (with ,var = ,init-value)
             (after-each (setf ,var ,form))))
to get a direct translation:

   (itr (repeat 8)
        (for a = 0 then b)
        (for b = 1 then c)
        (for c = 1 then (+ a b))
        (collect a))
> Moving all things to s exprs doesn't seem like it would be that big of a deal.

One of the biggest things you will "miss" from LOOP is the do keyword. The control structures is just regular lisp code. Also, I found generators to be pretty handy (not often, but I am glad I had them when it was called for).

I mean, it's not exactly groundbreaking, but it does make a difference especially that a looping construct is probably the heaviest-used macro. You can certainly get away with LOOP, but I see iterate as more or less how LOOP should've been done.

I also recommend reading this short discussion in the manual https://iterate.common-lisp.dev/doc/Don_0027t-Loop-Iterate.h...


Previous doesn't have the same flow, so does feel more cumbersome than I'd like. (That is, I specifically like the left to right flow of "this then that" for variables.)

I do have a softspot for how this is embracing the parens. :D

This is funny in that it seems to dodge the biggest complaint of loop, that it has different syntax for lists and vectors/hashtables.


It has in-sequence, however it uses elt I think. For efficiency reasons, there's a separation. Otherwise how should it work? In any case, you could probably write a macro that does the right thing based on type.

> Previous doesn't have the same flow,

I have provided a clause macro that uses the exact syntax you showed, you don't have to use previous.


Apologies, I was agreeing with your macro on why I like "then" over "previous". I did not intend mine as a correction or challenge.

My point on "in-sequence" and such was more that the main complaint I ever hear regarding LOOP is that:

    (loop for c across "hello"
          collect c)
is not the same as

    (loop for c in '(#\h #\e #\l #\l #\o)
          collect c)
Specifically, "in" versus "across" seems to upset a lot of folks. I don't recall ever hearing too many complaints about the syntax not being more nested.

Again, though, looks neat and I'm excited to dive more into it.


Okay, gotcha.

Yeah, I can see how having distinct keywords may be seen as inconvenient. I think if there were a general container-keyword that wouldn't introduce any cost based on a type declaration, that would be cool, but it's not something I have encountered a real need for.


Yeah, the other, possibly worse, problem is that experienced common lisp programmers think that LOOP is ok ;)


Why does this crazy macro exist for one part of the language but not others? It feels obviously out of place and gives you the feeling that you're using a legacy language with historical oddities rather than a coherently designed system.


Me thinks you just haven't looked at more parts. FORMAT, in particular, is just as dense.


Is there an alternative to FORMAT?


I mean. Not really, though I have seen folks make a herculean effort with print and friends.


> an awkward syntax

And yet C likes get praise with their {[(;*&^| stuff...


- "if you want to make a neutral prior for how long a technology will be around you should basically just look at how long it has already existed"

AKA Lindy's effect.

https://en.wikipedia.org/wiki/Lindy_effect


Static for 25+ years, but "super new"?? Of implementations, CMUCL even predates Common Lisp, but I don't know when it implemented CL. GNU Common Lisp was available GPL'd around the time of the standard and, I think Clisp.


While FOSS implementations are great, their tooling is still missing the whole stack experience as offered by Allegro and Lispworks.


Not sure why OP didn't use TFA's actual title, but oh well.

Lisp-Stat is maybe not as full-featured as its analogues in the python ecosystem, but it's still quite good. I recently developed an algorithmic trading bot in Common Lisp using statistical arbitrage methods and I used lisp-stat quite a bit in the development process, and while I enjoyed vega-lite (the plotting library that lisp-stat uses) I will say that I despite JSON as a format and found exporting to gnuplot scripts more flexible and useful.


Deliberate. I think the original title is good for the writer. But for those are statisticians, I am not sure you should go there. R, pathos, SAS and SPSS (plus one I am not sure but is around) would be more common. Hence, the change to the emphasis. You know Common Lisp or Lisp, a good tool back to life.


Not to be confused with XLispStat[1]

1: https://en.wikipedia.org/wiki/XLispStat


This was going to be my question. "Didn't statisticians abandon this for R like 25 years ago?"


Yes they did to my regrets. I did admit whilst xlisp-stat is much cleaner implementation for stat. and is my entry to Common Lisp (via simulation of social scientist 1st edition). But lisp is in the way, sort of, for many.

It is nice to have a common lisp version to use in statistical area, for those who does not hinder by it.


There is some migration aid for this, as it is really common lisp based statistical package like xlisp.

https://lisp-stat.dev/docs/resources/xlisp/


Ah I did get confused. I have Luke's book from 1990 still!


Wow, I didn't know this is still alive. I used this project for some strange reason good 15 years ago. The most funky feature it had (still has?) is data visualization in 6 dimensions (yes, six), it was done as a kind of rotating plot that is switching between variables to show three each time. Very cool, although rather useless for average human.


People who promote the virtues of lisp online usually talk about the powerful repl.

Here they advertise jupyter notebooks.

Does that give that lisp repl experience I’ve heard so much about or is it just like my familiar r and Python repl experience?


For that you need to dive into Lisp commercial offerings.

Jupiter notebooks are the poor man's experience of what using a Symbolics or Interlisp-D workstation used to be like.


The REPL experience in Lisp stems from how Lisp works; notebooks on top of Python don't remove Python's fundamental limitations. You can put a better interface on top of Python and it will still have Python's limitations, such as the necessity to create new instances if you redefine a class as opposed to old instances adapting to redefined structure and behavior.


Lisp is excellent even if you write every line of code in a file, and compile every time you test, using a fresh image (using the REPL only for some exploration of the code committed in files, and debugging).


I do not like emacs and is a vim guy. With that https://news.ycombinator.com/item?id=30812912 for using Slimv and vim (the source talked about slime as well, but I am not sure I like that). I can do the expression by expression and buffer by buffer kind of building an image, not just file by file.

In this regards, the notebook approach is very different. I am still learning from this site about this under common lisp. Based on python notebook (and R), I think it helps after you build your system you can use this to help others to understand what you have build.


“Statistical Analysis with Lisp-Stat”


It is more Common Lisp with an extension into my area, even though my first degree and jobs are all stat. Thanks for the suggestion. Just not how I thinking of it. In fact if you just want to do stat, I recommend spss, R, python … not really sure you will start with this and learn the lisp underlying. Clips-stat guy moved to develop R for a reason.

Still I think it is lovely they restart this path. Love Common Lisp.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: