Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

My current thinking is that it would make sense to implement the standard DataFrame abstraction (with all the generic functionality it offers) with a couple kinds of distributed data frame. One approach is to make a distributed stack of local data frames; another is to make a single data frame where each column is a distributed vector. Not sure which is better. Fortunately, Julia makes it really easy to experiment with these kinds of things. Of course, as soon as you have a system like that you need a scheduler for work at which point you want something like Dask or Dagger (https://github.com/JuliaParallel/Dagger.jl).


We are so strong with the more general Tables.jl abstraction. Tables.jl took us a step away from having lots of types of `AbstractDataFrame`. Subtyping `AbstractDataFrame` is hard, I have to deal with two packages that do it and it is a big and not entirely documented interface.

More natural is extending Tables.jl (like DataFrames and JuliaDB does). and we continue to build more tools that are table agnostic, and have APIs that DataFrames and a distributed table package special case when they can do it more efficienctly




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: