I wish this website spent its banner space on "what it is" rather than "star us on GitHub!". I doubly wish this because it's pretty confusing what JuliaDB is! From the name, I expect it to be a database, but the website immediately compares it to Pandas. I think of Pandas as a library for in-memory manipulation and analysis of tabular data, not a database / persistence engine. So is JuliaDB not a database at all, but in fact an alternative to Pandas / DataFrames.jl?
In fact, I've been using Julia for work and following the ecosystem since version 0.4 (we're at 1.5 now), and I'm still not sure what JuliaDB is. No doubt this is mostly due to me not having reason to look very deeply (and/or not being very perceptive), but certainly doesn't feel like the marketing copy is giving me any help...
I think it is actually a database, so it seems they haven't committed the cardinal sin of putting "DB" in the name of something that's not a database.
> JuliaDB is a pure Julia analytical database. It makes loading large datasets and playing with them easy and fast. JuliaDB needs to support a number of features: releational database operations, quickly parsing text files, parallel computing, data storage and compression.
> This talk is a bottom-up look at the construction of JuliaDB. We will talk about the scope and implementation of underlying building block packages, namely IndexedTables, TextParse, Dagger, OnlineStats and PooledArrays.
> they haven't committed the cardinal sin of putting "DB" in the name of something that's not a database
It doesn't seem to store data in any way - so it is definitely a data processing engine, but a database without an "INSERT" command feels a little off.
From that point of view, this looks a lot like what original MapReduce did - the data lives outside it & is referenced as urls, but the engine itself does processing out-of-core and in-memory for very large datasets.
"It doesn't seem to store data" is a complaint that doesn't really make sense to lodge against a library rather than a standalone program. If your program is using a database library, it's your job to write the line of code that tells the library to load data from a particular file into memory. The library cannot persist as a running process of its own across multiple executions of your program that is using the library. This is as true of SQLite as it is of JuliaDB.
I agree that it's a bit odd to not have a direct analog of SQL's INSERT, but you can definitely add rows to an existing table by making a new one and doing a merge operation.
Side note: this is what we tell every startup about talking to HN: Lead with a clear statement of what your company does. If you don't, the discussion will consist of "I can't tell what your company does". Same for open source projects of course.
Arrow is a file format, albeit one designed to allow easy interoperation of different analytical frameworks, but it doesn't provide any computation functionality itself.
JuliaDB is more like Dask, or DataFrames.jl - it provides that computation functionality.
In fact, I've been using Julia for work and following the ecosystem since version 0.4 (we're at 1.5 now), and I'm still not sure what JuliaDB is. No doubt this is mostly due to me not having reason to look very deeply (and/or not being very perceptive), but certainly doesn't feel like the marketing copy is giving me any help...