> As Google and other tech giants adopted the technology, no one quite realized it was learning the biases of the researchers who built it.
Woah, this came out of nowhere and it’s completely wrong. The problem isn’t that deep learning is picking up biases of the researchers, it’s that it picks up biases from the training data.
Algorithmic bias is not ONLY a data problem, it’s also a model problem [0]. The bias from model developers are coded via learning rates, network hyper parameters and objective functions.
Model bias is not a huge issue. Maybe something about class imbalance or regularization. The huge issue is deployment - what is the model used for? How is it affecting people in reality? What metric is it optimizing?
Between all these the degree of L1 regularization or the class weights are minor things. Most models will perform similarly given the same data. It's mostly the data that makes the difference.
There's an interplay between the two insofar as a model built to handle a specific dataset will involve design decisions informed by the data. E.g. you might pick a certain level of L1 regularization because it maximizes performance on the data you have, which can lead to bias against data you don't have.
But if you take "model" to mean the pure mathematical description without parameters or hyperparameters that need to be determined by experimentation, then I agree that optimizing the model on a dataset will not lead to bias against specific groups of humans unless the data used contains such a bias.
Hyperparameters play a significant role in bias when you're dealing with imbalanced classes, or long tail samples.
But this ties back to the original data problem, right? If you don't have enough training samples for (known or unknown) unknowns, your model is likely to be biased against them.
While this is true, "learning the biases of the researchers who built it" is a very misleading way of putting it, because it is still very unclear if and how certain design decisions impact the bias of the resulting model.
Given that reducing bias while not giving up other desirable properties is a young and open research direction, researchers in general should not be faulted for using the current (imperfect) state of the art or for working on something that is not (yet) focused on bias.
Yes, the optimization goal (the objective function) is a major factor in the function of algorithmic systems. I'm not sure bias is the best word to use here, however.
It is a known challenge to align the designed purpose of an algorithm with actual optimization metrics. For instance, recommendation systems may have the purpose of improving user experience, but if time-on-site metrics are used as the optimization function, there can be unexpected results.
Yeah, but it is indirectly the biases of the researchers. A researcher is more likely to notice and correct for training data problems that conflict with their biases.
This requires him to be aware of each pattern/“bias” in the data, which he isn’t, which is the reason we use the algos in the first place.
Come to think of it, isn’t that an interesting venue for GAN-esque methods to detect the relation of patterns falling into these categories of biases? Or is that recursive problem? If not, put me in the paper :-)
> This requires him to be aware of each pattern/“bias” in the data, which he isn’t, which is the reason we use the algos in the first place.
That's not strictly true. In a lot of cases you start out oblivious to biases in the data, and then when you evaluate the model you notice problems.
But your point about obliviousness to bias is exactly what I'm speaking towards. One might be oblivious to bias that aligns with your own biases, but notice bias that conflicts with it.
People doing ML “research” are not the ones applying it to specific data sets day to day. “I pointed a neural net at our sales data” is not research in the normal sense.
Yeah, I look at the semantic problem with calling it "ML research" and just throw up my arms. These discussions aren't generally driven by people who care about semantics.
That's not the only mechanism. Even if you want to stick to technical issues, it can be biased because you didn't train it long enough or the model is too small. And of course the entire question depends on what the model is being used for.
Everything you say is true, but in context the important point is that the sort of biases ML models pick up are overwhelmingly related to training insufficiencies and are often incredibly difficult to spot unless you already know they exist. For a practical example, see the recent Twitter image cropping oddities (https://twitter.com/bascule/status/1307440596668182528).
The idea (as quoted) that models are routinely picking up biases directly from researchers is complete nonsense.
Right, that's some kind of human interest journalist fudging and it's not true. But bias/surprising wrong answers in ML is obviously a real problem and fixing the data is not always the right answer. You might not be able to tell what's wrong with the data, or where you could get any more of it, and you might be reusing a model for a new problem and not have the capability to retrain it.
I would argue that DNNs would not work as well if they weren't picking up biases. Sometimes we need to learn the biases in order to better detect them.
Even humans need to know about swear words in order to consciously avoid using them, or need to learn about reproduction in order to avoid teenage pregnancies. Not knowing does not make us or the AI better.
For example, what GPT-3 needs is a "conscience", a separate model monitoring and rejecting harmful outputs. If I am not mistaken the demo is already displaying warnings when it goes off into weird places.
Woah, this came out of nowhere and it’s completely wrong. The problem isn’t that deep learning is picking up biases of the researchers, it’s that it picks up biases from the training data.