they have 200 employees on linkedin, many of whom are not engineers. How they can carry two such major and complicated projects(browser and search) with such headcount?
Can't say for search (it seems like massive work indeed - or maybe you can actually build a decent and comprehensive search engine with few people but with a massive amount of money), but for the browser they really provide a browser UI (and I'd guess most of it is actually built by Google too). It requires work, but it's not massive like a browser.
There are many browsers out there, maintained by a few devs, sometimes in their free time.
Konqueror, Gnome Web, qutebrowser, WebPositive...
Whatever the SerinityOS is doing, reimplementing a browser engine from scratch for their browser Ladybird [1], is vastly more impressive.
search.marginalia.nu consisted of 1 Swedish engineer and 1 server last I heard and still managed to outclass DDG in a number of query types relevant to me.
In fact, back when I used DDG I think I fell back to marginalia more often than I fell back to Google, partly because of my dislike for Google and partly because Google doesn't respect my queries - which of course is a contributing reason for my dislike for them.
Let's say 100 of the Brave employees are engineers and 50 of them work on the Chromium skin, that still leaves 50 to work on search and related efforts. If 5 of them are as good as the marginalia guy and they are allowed to work with equally clear direction, lack of interruptions and more funding, I think that could almost explain a working search engine.
Remember: In the first 20 years of Google existence (or in any 20 years of the semiconductor age until recently) Moores law had over 13 cycles. A lot of what used to be hard problems before isn't anynore.
> search.marginalia.nu consisted of 1 Swedish engineer and 1 server last I heard and still managed to outclass DDG in a number of query types relevant to me.
Still just 1 dinky lil' consumer hardware server in my living room. I think what is limiting the project the most is the hardware. Like I can definitely squeeze more out of it, but I could probably do 100 times more if I had any sort of operational budget.
But at least the development is funded for the moment. We'll see where I am when the NLnet money runs out...
Would having a few more people working on it make stuff move faster? Undeniably. But I don't think lack of manpower is an obstacle at the moment in the same lack of hardware. It's reasonable to expect there's an upper limit to how big a project you can manage as a single developer, but I don't think I'm there yet.
Right now I've got about two years runway, which isn't a lot but it also isn't nothing. Considering the project had barely started two years ago, and considering the opportunities it's already created, hopefully more options will make themselves available along the way somehow.
hopefully not the kind where you lose 90% of your forces, many to desertion and others to fighting with ethnic warlords, before finally defeating the encirclement campaign
Interesting question. If I wanted to say grow the index 10-fold and aim for a billion documents indexed (~2.5bn crawled), I'd need:
* More crawlers. This is mostly a network limitation. Disk requirement would be about 50 Tb in total of literally any hard drive, split along the crawlers.
* A dedicated number crunching machine for index construction and processing. Basically needs to be all cores and all RAM.
* A dedicated SQL server. Doesn't have to be super big or fast, but right now it's contending with the index for RAM.
* Either a bigger index server, or a few machines for a sharded index. I'd basically need 40 Tb worth of Enterpise SSD and 1 Tb total of RAM.
The problem space is very nice in that almost everything is either embarrassingly parallel along some axis or another or O(n)-O(n log n) so adding more hardware pays off well.
This is clearly too much hardware to run out of my apartment, even if the fuses held up the noise and heat would be intolerable, so likely some form of colocation hosting as well.
I dunno what it would land on. Probably a one-off cost of about $50k-$100k, depending the degree of redundancy. There's definitely stuff that could be done for less as well. Like right now the entire operation runs out of a single machine, and it's causing me to have to bring the service down for days at a time when I'm switching to a new index.
> Remember: In the first 20 years of Google existence (or in any 20 years of the semiconductor age until recently) Moores law had over 13 cycles. A lot of what used to be hard problems before isn't anynore.
The web also grow tremendously, and probably user expectations too (in the beginning of the century we were told in school not to speak to search engine like we would to a human, but with keywords and operators). We also do and search for many more kinds of things
If anything my expectations sunk massively between 2009 and the introduction of Kagi.
To be blunt: for me, mainstream search engines Google and Bing very much feel like the things they replaced, Altavista and Yahoo, just with some fancy bolt ons like maps etc.
I understand some people like to be able to write sentences to their search engines, but as long as the results have worse quality than they had 15 - 20 years ago, that "understanding" is just another fancy bolt on feature.
The only things that exist today that could threaten Google quality is Kagi which has gone all in on quality and ChatGPT (and other similar solutions) which finally have produced a working "answer machine" instead of breaking perfect or at least working search engines.
