Wasn’t that they got hung up on the details of whether the average of ints should be an int, float, or double, or how to handle overflow, but just couldn’t get started at all.

After that, I stopped shying away from wanting to see code from even senior technical candidates.

(It was not a standard question for senior roles; I got a strong whiff of “this guy might be entirely full of it; let’s have a look-see”.)

from the other side, while reviewing designs (not code) of various components I can spot locations where code handles sockets incorrectly or a couple of dozen of scenarios that system will fall apart under because engineers don't think that far.

value that I bring is not in averaging list of numbers. my value is 25 years of making mistakes, learning from them and knowing how to avoid them.

To me, this seems more like an executive chef who doesn’t know (or has forgotten) how a knife works rather than an Elon Musk or Henry Ford who doesn’t know how to weld a body panel.

But if a company the size of Google can get by without software architects, why does your company need to be organized in such a way that you require that role?

Back when I worked there, one of the universal shocks when people go to work for Google is that the overall architecture was light years ahead of anywhere else. The person usually given credit for that is https://research.google/people/jeff/. Based on results, he may well be the best software architect alive.

If you want to learn from his example, I recommend that you start with "a running prototype beats a whiteboard design".

hence, i am not musk or ford that don't know how to weld.

What sheer irony.

BOTH Musk and Ford knew how to weld, and considered that knowledge essential to being able to do their jobs. Sure, they didn't do a lot of welding day to day. But how could they make correct decisions about how to build new machines if they didn't know how those machines are put together?

I have a few notes here:

0. Google maybe is exception. But i never worked in google, so i don't know how good it is really.

1. Not all companies can pay same wages and get same talent as Google

2. Every time there is talks about GCP/AWS/Azure a lot of people say that all of those services seems to designed by completely different people and don't mesh together nicely.

3. I worked in one of the FAANGS , and while it's widely known for it's excellent engineering it was a massive dumpster fire. There were excellent people that did amazing things on lower infra level, but in the moment that you move a bit higher up it was disorganized mess of multiple teams not knowing how to collaborate properly what resulted in outages, sometimes rather big, which were not noticed from outside mostly due to size of production environments that allowed to shift users to still functioning parts of the system. Those outages were mostly result of not having somebody who will look at system end to end and will identify potential cascading failures or weak points in general.

>BOTH Musk and Ford knew how to weld, and considered that knowledge essential to being able to do their jobs. Sure, they didn't do a lot of welding day to day. But how could they make correct decisions about how to build new machines if they didn't know how those machines are put together?

I didn't say that I don't know how to code. I said that I won't pass coding interview. Companies that I work in have coders much better than me, but I am much better than them in understanding how system supposed to work end to end and how to prevent it from collapsing under various unpleasant scenarios.

I acknowledge that Google is able to hire incredible people. They also got a lucky break in hiring Jeff Dean early. However it is my considered belief that when you call Google an exception, you're reversing cause and effect. When people who come up with the architecture have to write code, work with their system, and see from experience what is wrong with it, they produce better architectures.

It is not whether coding produces more value than architecture. It is that continuing to code informs their architectural decisions.

For a similar idea in a very different industry, when Robert Townsend was CEO of Avis back in the 1960s he turned the company around. He made it profitable, and made it grow.

One of the things he did that he says made a huge difference was to make it a rule that everyone worked the rental desk. Didn't matter whether you were the CEO, VP, big manager or whatever. One day a month you stood behind the rental desk and had to deal with live customers. And having that regular experience meant that problems in the organization that would otherwise go missed became instantly visible. Just like how actually having to code and debug things in your architecture makes architecture problems visible that otherwise you'd discount. It made Robert Townsend a better CEO. The corresponding exercise would make you a better architect.

See https://hbr.org/2010/02/make-the-change-be-an-undercov for more.

[0] https://en.wikipedia.org/wiki/Operations_support_system

Totally agree with this. I personally started with installing trumpet winsock and netscape navigator on win3.11. I did customer support while been developer. I went through most of positions that are usually part of software development org and dealt with with most aspects of software. And I consider myself lucky to been able to do so.

But the unfortunate truth it's that in majority of companies that develop software (lets get outside of silicone valley for a moment), developers tend to be very far removed from production environments and from clients. As example, I was hired to work on a new, from scratch product, in one of biggest suppliers of software for telecoms (8B in sales). Over there "core engineering" was tasked with developing "core functionality" and there were totally separate delivery organization that will take this code and adapt it to client needs and will write installers/deployment systems, figure out how it actually integrate, run, etc. Clients weren't accessible to core engineering and internal product management couldn't talk to them either.

Also, the overall approach was that it's perfectly normal that software will crash and leave everything in inconsistent mess, hence there were mandatory tooling to be developed for manual operation to "fix things".

I spent first few months arguing with my boss that it's not normal and that we must deliver software that can be installed and work without crashing left and right. I got to built first devops org in company, first private cloud lab, pretty much first in company integrated ci/cd, jira+confluence (because company used internally developed "agile" tools that were unusable. and i actually got visit from internal auditors questioning me why we spent money on something that already exists in company), lobby to create track for engineers to interact with clients so they will understand what is real operational environment of software they are writing and what are the real requirements so they will be able to do better job, get training for developers so they will learn that there is something outside of "enterprise java" (as anecdote, when i presented rabbitmq as part of architecture, i got asked to what java standard it corresponds and when i said to none, i was told that they don't really understand why would I use it). Project was amazing success and delivered more in less time than was common in the company. (manager got fast tracked to vp/svp/gen manager overseeing 2B in business within 4 years. I got relocation to states :).

my point i guess, it's that most of software development in most companies is disorganized mess with developers been completely detached from realities production environments. And when I am hired, pardon the pathos, my job is to bring order into the chaos or light into the darkness or whatever. In cases when I am out of my depth in some areas (i well aware about limitations of my knowledge) , i usually have enough pull to hire some domain specialists that I can outsource to them designs/implementation.

And I'm saying that the band-aid doesn't work that well, and you do better if you fix the organizational structure to not need it. And furthermore, as a developer, those organizations are no fun to be in and you can't pay me enough to want to deal with them. Particularly not when I have the option of finding better organizations to be in.

Which adds to the problem. When good developers choose not to be in your company because your company is a disaster for developers, you don't get good developers. Which reinforces the idea that developers aren't that good, and causes them to try to put guard-rails around developers to prevent the problems that bad ones cause. Thereby creating a negative reinforcement cycle.

. Developers that are mature but lack proficiency because they don't care about software development

. Inexperienced developers that lack, well, experience to make proper designs

. Absence of domain specific knowledge when company tries to shift it's technological gears or switch to new domain/market all together

. Problems in communication and planning between dozens of teams/services that build together 1 product that must work in unison perfectly, yet it's hard for each team to know architecture/plans of all of the other teams and properly consider it during development

. Absence of ability to see "the big picture" by individual teams and take it properly into consideration

. Just a general jack of all trades that will compensate for bad product/project/program management, will establish proper processes, etc.

As part of this you occasionally can revamp organizational structure and leave company in much better shape, but it requires a lot of work that you, as architect have no authority to do :)

> And furthermore, as a developer, those organizations are no fun to be in and you can't pay me enough to want to deal with them

One of my last jobs was in company that I literally sweared to never work for. They gave me a shiny title, 50% raise of already top salary in industry that I had, yet what really sealed the deal was manager who wanted to try to do something new. Even though it was at times horror show, at the end of the day it was probably most fulfilling position that I had due to a sheer volume of things that I managed to change. And we also delivered totally new product, first in industry beating all the competition to market, including companies that were supposed to be market leaders in this area.

>Which adds to the problem. When good developers choose not to be in your company because your company is a disaster for developers, you don't get good developers.

The issue is that really good developers are rare. Developers that you can delegate to them and sleep well at night are a rare commodity on the market. I been working in dev for 25 years, both in startups and in established companies and I maybe know only dozen of people like this. Unfortunately, most of the developers on the market are in it only for money and don't really care about what they are doing.

Also, as anecdote, at this ^ job place that I described, in order to work around bad reputation of the company I suggested to create daughter company with a different name that will be operated independently in a different location in order to try to get better developers. We were a few steps away from closing on office lease before management bailed out because they were afraid that it will upset employees of the company

My rule of thumb for developers goes like this. I mentally divide developers into junior, mid-level and senior. This has nothing to do with their titles. The breakdown is that a junior developer is still learning basic programming consistency and good habits. They generally struggle on programs that go over a couple of thousand lines. A mid-level developer has internalized consistency but is unable to remain aware of the big picture while focusing on details. They tend to top out and struggle at 10k-20k line projects. A senior developer is aware of the big picture while dealing with details, and remembers details while dealing with the big picture. They tend to top out around 200k line projects. And a principal software engineer adds an awareness of subtleties about how the big picture is likely to evolve, and how to guide that in a long-term positive way. These people can manage architectures in the millions of lines.

On this scale, I rate myself senior. To move to the next level, I'd need to spend more time at large companies with very large codebases. But dealing with the associated politics is not my preference so I'm happy where I am.

The issue is that really good developers are rare. Developers that you can delegate to them and sleep well at night are a rare commodity on the market. I been working in dev for 25 years, both in startups and in established companies and I maybe know only dozen of people like this.

Judging by what I saw of environments with software architects as a role, I guess I shouldn't be surprised at this.

However I know a heck of a lot more than a dozen programmers that I consider good like that. OK, throw out top-notch organizations like Google, and just go for relatively small companies that I've worked at. Let's narrow it farther and just do places that I've worked in the last 10 years. I still have no problem getting past a dozen.

Why? Because I choose to work at organizations where software architecture is owned by developers. With other developers who have made the same choice.

(Incidentally, 3 of those programmers were co-founders at ZipRecruiter. Who hired a lot of mutual acquaintances on my list, then grew to be a billion dollar company. Somehow I don't think that this is coincidence.)

BTW, principal software engineer in many places == architect. Titles are very country dependent, company dependent. When startup where I worked was bought by US company I became principal. When I was acquired by German, I was retitled... don't even remember to what.

>Judging by what I saw of environments with software architects as a role, I guess I shouldn't be surprised at this.

I wasn't always working in companies that had this title. Yet, this still remains true. Also we may have different standards and operate in different environment on different types of products :)

anything more complex, not sure. i touched last time linked lists in 2001. never learned algorithms and in fact didn't even graduated from school. and guess what - everybody is okay with all of it. because this is not the value that I am expected to deliver to the company.

edit: when people are interviewed for system architect positions, algo/coding interviews are not part of the process. because it's not skill set that company is looking for.

this was also the only whiteboard coding interview in my life

Moving on, here is an excellent example of an important architecture system that almost everyone gets wrong. In any distributed system you should transparently support having rpc calls carry an optional tracing_id that causes them to be traced. Which means that you log the details of the call with a tracing id, and cause all rpcs that get emitted from that one to carry the same flag. You then have a small fraction of your starting requests set that flag, and collect the logs up afterwards in another system so that you can, live, see everything that happened in a traced rpcs. To make this easy for the programmer, you build it in to the rpc library so that programmers don't even have to think about it.

You then flag a small random small fraction of rpcs at the source for tracing. This minimizes the overhead of the system. But now when there is a production problem that affects only a small percentage of RPCs you just look to see if you have a recent traced RPC that shows the issue, look at the trace, and problems 3 layers deep are instantly findable.

Very few distributed systems do this. But those that do quickly discover that it is a critical piece of functionality. This is part of the secret sauce that lets Google debug production problems at scale. But basically nobody else has gotten the memo, and no standard library will do this.

Now I don't know why they reinvented xml-rpc for themselves. But if they had that specific feature in it, I am going to say that it wasn't a ridiculous thing to do. And the reason why not becomes obvious the first time you try to debug an intermittent problem in your service that happens because in some other service a few calls away there is an issue that happens 1% of the time based on some detail of the requests that your service is making.

It was not way before microservices at Google. But it was before there was much general knowledge about them.

Sadly the internal lessons learned by Google have not seeped into the outside world. Here are examples.

1. What I just said about how to make requests traceable through the whole system without excess load.

2. Every service should keep statistics and respond at a standard URL. Build a monitoring system which has scrapes of that operational data as a major input.

3. Your alerting system should supports rules like, "Don't fire alert X if alert Y is already firing." That is, if you're failing to save data, don't bother waking up the SRE for every service that is failing because you have a backend data problem. Send them an email for the morning, but don't page people who can't do anything useful anyways.

4. Every service stood up in multiple data centers with transparent failover. At Google the rule was n+2/n+1. Meaning that your service had to globally be in at least 2 more data centers than it needed for normal load, and in every region it had to be in at least one extra data center. With the result that if any data center goes out, no service should be interrupted, and if any 2 data centers go out the only externally visible consequence should be that requests might get slow.

Now compare that to what people usually do with Docker and Kubernetes. I just have to shake my head. They're buzzword compliant but are failing to do any of what they need to do make a distributed system operationally tractable. And then people wonder why their "scalable system" regularly falls over and nobody can fix it.

and maybe you are questioning my ability, but engineering teams that I work with - not. They come to me so i'll help them with problems that they have.

And for these kinds of more complex algorithms problems I would not expect most architects to be able to solve them, since they're utterly irrelevant for their jobs. Heck, a huge amount of software developers can't solve them and bitterly (and at least partly legitimately) complain about them to whoever will listen.

It is the interviewer's job to check the possible reasons to not hire someone and validate that the they aren't going to be problems. One of the possible reasons to not hire someone whose last job was "architect" is that they have lost contact with coding. If they have, then from experience I can tell you that they won't want to go back to coding, and their architectures are losing contact with reality. They are a liability.

This observation is not original to me. The first time that I saw it made explicit was when I took interviewing training while I was at Google. Google's experience is that a certain fraction of architects avoid coding. That fraction makes really bad hires. So they explicitly told people to look for that in the interview so that don't wind up accidentally hiring them.

But if someone's title is "architect" and they can code, then that isn't a concern. Companies do tend to promote good programmers to architect, and you don't want to miss out on those programmers.

Firstly, do you believe that if you have not coded for 4-5 years you become useless just becauss you don't know the latest %fad% framework?

This is a straw man argument. I said nothing about "latest fad framework". The exercise is to write some code, not write code in any particular framework or language. Go ahead and be offended at something that I didn't say, but please remember that I didn't say it.

Secondly, the whole phrasing just conveys disrespect for cadidate, where the only responce to 'jump' is 'how high'.

If an interviewee objects to writing code in the interview, how are they going to respond when they're asked to write code on the job? Seriously, if they get offended in the way that you just did, that's a good reason not to hire.

If someone is interviewing for non-code postition they might tell you to take a hike.

I do not want to work in an organization where "architect" is a non-coding position. Seen that, left in horror. So if I'm interviewing you, you are almost certainly being hired for a coding position.

The idea that a contrived 30 minute coding session tells you more about a candidate that studying code they've previously written or debugging together is idiotic.

It doesn't tell you more, but it tells you something different. In particular it tells the difference between someone who shows their own code versus shows someone else's code and lies about the source. (Yes, I have seen that.)

Everything in your responce sounds to me like in your view, architect is 'extra senior' developer, whereas I understand it to be a different job. Isn't what you are describing a principal engineer?

"In particular it tells the difference between someone who shows their own code versus shows someone else's code and lies about the source. (Yes, I have seen that.)"

To eliminate that possibility, you can ask questions about the code, how it was designed, etc. Presumably if they stole the code, they are not going to know it inside-out, like the author would.

In my view, software architecture should be part of the job of a principal engineer, and the role "software architect" should not exist as a separate thing.

Separately, you would be amazed at how many developers will happily give you code that they wrote a year or more ago, and then not remember their design decisions. So not knowing it inside and out doesn't imply that they are not the author.