Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Duck Duck Go Searches Are Now Externally Anonymous (gabrielweinberg.com)
132 points by phsr on May 21, 2010 | hide | past | favorite | 48 comments


Thank you, Gabriel.

The way you listen to your users and implement fixes and suggestions quickly is probably almost as important to your success as good search results right now, at least to me. Makes me want to use your site more.


You're very welcome. And please do--I need as much help as I can get :) http://duckduckgo.com/spread.html


I'll copy here what I wrote on GW's blog:

Thank you for taking the privacy of your users seriously! Most of us aren't spies or Chinese dissidents, but the principle of privacy is very important to me, and I'm glad to see that it is to you too.


I love the fast fixes and changes you're giving. As much as I like Google I just can't see them reacting this fast.


Google has billions of dollars riding on their search, I'd bet they would react pretty freaking fast if a game changing bug or user concern appeared :)


I have every confidence that Google would give any concerns about search every bit of the attentive customer service they are famous for showering their paying customers with.


Are they famous for that? I have never heard stories of how they deal with paying customers, but then, I don't think I know any paying customers of Google. No snark intended, I'm really curious to hear.


That was sarcasm. I have had no direct experience, but from what I have read, you are generally ignored, and if they dont like you , your results ranking can be trashed.


Oh crud, I didn't realize anyone could read that literally. My bad: you have to know the context.

Hi, I'm a longtime AdWords customer (I spend in the mid four figures per year). It is wonderful... except that customer service is provided through Google. Or, frequently in my experience, not provided through Google.

When the algorithms are working well, AdWords is amazing. Quite literally life changing for me. But for AdWords, I'd still be at the day job.

Computer algorithms do not always work the way we'd hope they would. In many multi-billion dollar businesses run by grownups, for example banks, there are humans in the loop for when computers make mistakes, for example marking as fraudulent my $550 monthly AdWords bill. There are a variety of means by which I can contact a human at the bank, explain my problem, and get effective help at resolving it. My bank makes less than $200 off of me a year.

Google hates dealing with customers because it doesn't scale nearly so well as throwing another 10,000 cheap boxes at the problem. So they do everything possible they can to avoid having to talk to you: Matt Cutts calls these "scalable communication approaches", but when my business got essentially mothballed during busy season they felt a lot more like "You're making me click through five different screens to find the contact link and then limiting my message to 250 characters, come on guys, why don't you just write Eff Off And Die so I won't waste any time hoping for a resolution."

On submitting that message to support, I got back an autogenerated reply which was utterly non-responsive. From Google's perspective, I think we were done. Further support requests were did not succeed in reaching anyone who knew what was happening.

The problem was eventually resolved a week later -- after I had complained about it on my blog. I have no knowledge of whether complaining was causationally related to resolution but on previous incidents with Amy Hoy's Google Checkout account and at least three others I can think of off the top of my head, causing a PR issue was the only way to get attention to utterly routine issues.

In addition to these issues, the few beleaguered humans working in the borg tasked with dealing with customers are unaware of Google's own policies, ignorant of most of the information of key importance to their customers (or they hide their knowledge very well), and discouragingly forgetful of promises such as "I will escalate this and someone will get back to you."

One of my other minor frustrations: Google will hold your ad for review if it is possibly objectionable. For example, involving gambling products. I do not actually sell a gambling product, but I've had ads held for weeks or months waiting for review because I tend to use the word "bingo" a lot.

The last time I complained about this on HN, a helpful chap from a foreign Google office emailed me to give me some suggestions, saying that essentially I was likely to fall through the algorithmic crack every single time and that in addition to putting my ad through on the web interface I should immediately open a ticket asking them to review it. This answer makes him the most helpful Google employee I have ever even heard about.

I actually tweeted early today that I edited an ad yesterday and was pleasantly surprised that it was actually published. It has been several months since I've worked with AdWords actively because it is so frustrating.

Full disclosure: I am an AdWords case study.


Wow, thanks for the detailed account of your support experience with AdWords. Makes you realize that if it is a monopoly, you can pretty much treat customers as you wish.


FYI, https://ddgw.s3.amazonaws.com/external.png is still being loaded direct by s212.css, rather than being proxied via your server. The search term isn't included in the referer though.


Good catch :). Will fix!


I use a Firefox addon called "TamperData". It's really good for stuff like this. I just go to a web page, open TamperData, hit <ctrl>refresh and it will give me a list of all the requests that are made. I can then go and take a look at the full set of HTTP Request and Response headers for each individual request.

It lets you intercept and modify request headers on the fly too. Very useful tool.


TamperData is ridiculously useful. I suggest everyone check it out.


Thx for the reference. It should be fixed now.


People here are probably not going to like this but I will not trust Duck Duck Go until they release their whole source code and allow me to run my own instance. For now, there's still a middle man called Duck Duck Go that I have to blindly trust. I salute his efforts to make his page privacy friendly, but as long as there is a middle man and no one to check his source code to check if he's not sending anything about us, there can't possibly and technically be any guaranteed privacy. Any one who has taken basic cryptography course would agree with that.


Presumably you've sworn off Google, Yahoo, and Microsoft as well, and accomplish Internet search by using a self-hosted copy of ARCHIE.


Would be satisfied by an EFF audit?


Unless they have constant access to your servers to make sure you don't make any changes afterward... But even then, the only way to be sure about one's privacy on the net is to run your own server, no web service can beat that kind of privacy. I personally prefer Google to Duck Duck Go by the way (especially using chromium search feature by pressing TAB) but if I really really cared about privacy to the point of giving up on Google, Duck Duck Go wouldn't be enough, sorry. Good luck though.


What is the big deal with sending the referrer header? It's been common practice for many years. Why the upheaval now? And shouldn't this type of thing be a browser setting instead of a website hack?


It's no 'big deal', it's just that people that are not technically oriented don't even realize that it exists.

The problem here is - or rather was - that duckduckgo is differentiating itself from other search engines in a number of ways, and one of the more important ways in which this is done is that at duckduckgo.com you can search for things in the knowledge that they take your privacy seriously and that they will not sell your soul to their highest bidder.

On top of that they try to limit inadvertent exposure of your private information to third parties.

And that's where the problem came from, if you state loud and clear that you are secure, and then you're found to be insecure you have an immediate problem.

So, since duckduckgo.com is setting itself apart from the 'pack' by providing this useful feature and because it is part of their image of being trustworthy they had to address it.

If browser manufacturers would think a bit they would drop the 'Referer' header on https requests, but since it's been there for years now you can expect it to be there for years to come, to step forward and fix it was the right thing to do, and I'm very impressed with the way Gabriel responded to the publicity around the issue and how quick and thoroughly he fixed it.

I wished all companies would be that professional in their dealings with the public.


Is there a way to add Duck Duck Go to the firefox search bar? I really like using keywords to launch searches (since I use VIMperator) but I can't seem to add arbitrary keywords to my knowledge.


To add it to the search bar, go to http://duckduckgo.com/ or https://duckduckgo.com/ (depending on if you want SSL or not), and click 'Add to Firefox' at the bottom.

To add it to the address bar, type about:config into the address bar, search for keyword.url and then change it to http://duckduckgo.com/?q= (or https://duckduckgo.com/?q= for SSL).


This will just replace one of the current searches whereas I want to add one. But I figured it out - in the searchplugins directory in the firefox install are a bunch of xml files. I added a new one with the relevant info and now "duck" is the keyword that maps to a duck duck go search.


For the information of anyone else looking to do this, you can do it in the GUI:

Add it to the search bar

In the search bar dropdown, click "Manage search engines"

Select DDG and set a keyword. I use "d".

Then, search using e.g. "d Hacker News"


People using vimperator can access the dialog with

  :dialog searchangines


"To add it to the address bar, type about:config into the address bar, search for keyword.url"

Actually, it's much simpler than that: just right click on the search field and choose "Add a Keyword for this Search…".


You can also use the search plugin at http://nfriedly.com/stuff/duckduckgoogle/ if you want search suggestions. Note, however, that this kills some of the privacy as it sends your searches to both Google and Duck Duck Go.


Try the Add to Search Bar extension: https://addons.mozilla.org/en-US/firefox/addon/3682/

I also use Vimperator, and have a DDG keyword search.


Excellent work. Thank you for taking this problem seriously.


Clicking on a search result now redirects through the DDG server.

So, Gabriel, if you woke up tomorrow as your Evil twin, you could do click tracking with no further user-visible change of behavior.

Google used to do stealth click tracking via javascript. I have not kept up with whether they still do, but it was that kind of thing that influenced me to switch to DDG.


Maybe I'm missing something here, but I don't see the privacy leak given that I don't know who you are.


This did get me thinking about if there was a way to obscure the referrer without hitting the server side. Apparently in webkit-based browsers if you create an iframe, and inject some javascript to redirect to the desired link from inside the iframe the referrer doesn't get passed. Unfortunately it doesn't work in IE or FF, but I suppose there might be some trick. Quick proof of concept for webkit:

http://junk.wehrenberg.us/t1.html

EDIT: Apparently both FF and IE work fine if you redirect using a meta refresh rather than js from inside the iframe. But that doesn't give you any way to bust out of the iframe.

http://junk.wehrenberg.us/t3.html


I think I got this to work. Email me if you'd like to test it out with me!


Well, Evil Gabe probably would know who I am based on all the logs of my DDG searches. Adding click tracking to that is just more data. :)


Hey Gabriel,

Fantastic work, a case study in dealing with problems in a product.

If there was a prize for stuff like this you ought to get it :)


The site now(?) uses some redirection instead of sending me directly to the URL I click on. This screws up copy'n'pasting. I really do not like this.

I found kd=-1 on the params page but that parameter is gone as soon as I enter a new search term.


Unless I'm missing something, it shouldn't screw up copy and pasting at all. It's done via JS when you click on the link, but the link itself is the original URL.

You can set the setting indefinitely on the settings page (http://duckduckgo.com/settings.html). To use the URL params to do so, you can add that param to your search bar settings.


I am using the latest Opera snapshot on Linux. If I rightclick on a link on a result page and copy the link address, I get the redirection link.

The parameters are not "transfered" to another DDG page for me. Eg I manually crafted http://duckduckgo.com/?kf=-1&kz=-1&kd=-1&q=hello... which works great but if I now enter "hello epi0Bauqu" at the site and hit Enter I end up at http://duckduckgo.com/?q=hello+epi0Bauqu&v=

I have cookies disabled, referrers too.


Gotcha, yes, I can confirm. It happens on mousedown so I guess that is triggering before the copy.

Let me try and understand the params issue because I think that will fix it for you. So why aren't you creating a custom search in opera with those parameters that you one?

Or are you saying that once you do a search and use the search box again, your parameters should be present in that second search? I think you're right that they are not and they should be. I can fix this, but I want to understand exactly what is going on first.


You rock!

Ah, sorry, I thought I explained it good enough, German here. I made a custom search with those parameters. It works fine. But if I then enter a new search term on the website's search box itself, not through my custom search, the parameters are dropped. That also happens if I reach a site where I have to choose the category and then get to a result page for that specific category, for example if I search for "test": http://duckduckgo.com/?kf=-1&kz=-1&kd=-1&q=test

Then I have to choose a category, I click on http://duckduckgo.com/Test_(student_assessment) and there the parameters are gone.


Added to bug list--will fix.


What's wrong with https://www.google.com


I did a search and I was served a domain parking Ad as a landing page. WTF am I missing? This search engine looks like a spam to me.and I wouldn't have cared if it hadn't shown up at HN.


"WTF am I missing?"

You've obviously missed a lot of previous HN posts related to the privacy features of DDG, multiple posts from Gabriel's blog concerning the search engine, testimonials from people who have tried DDG for a week and never looked back. Let's just say you're missing a lot.


Sorry to hear that. It generally has less spam, and less parking pages in particular. If you email me your search I would love to look into it. http://duckduckgo.com/feedback.html


k, thanks, have added it to the firefox, will give it a try. The first impression was pretty bad with that Ad showing up.


Thx. I understand the bad first impression. I would still appreciate you emailing me the bad link.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: