Just because it's GC'd doesn't mean it's constantly allocating memory. Optimizing the most common heap allocation sources isn't very difficult (especially compared to the nightmare of manual memory management).
I recommend running the Roslyn Clr Heap Allocation Analyzer extension for VS for a few days - it helps to learn which language features and APIs implicitly allocate objects (or box value type objects).
Plus, modern GC implementations are super clever and efficient (as opposed to the implementation used by Unity, which unfortunately rather sucks, but I hear they're working on swapping it out).
I wrote games and engines in C++ for 15 years, and Unity for about 5. I strongly prefer manual mem management, hands down. Optimizing for Unity's GC, in pathological situations, is the nightmare.
If the game is relatively simple, and the heap relatively small, it's not a big problem. But if you're trying to make a large, complex simulation that's actually pushing boundaries, the GC becomes a constant adversary. People always talk about allocations, but not allocating during a frame is just the minimum price of admission. The real trouble is the static characteristics of the heap, like size and graph complexity. With manual alloc, at least the problem is straightforward: don't leak, and don't dangle. With a black-box GC, where you can't even hint to it about the lifetimes of objects, you have to be much, much more aware of memory management, then go through a lot of different non-idiomatic contortions regarding every single thing you put on the heap.
I recommend running the Roslyn Clr Heap Allocation Analyzer extension for VS for a few days - it helps to learn which language features and APIs implicitly allocate objects (or box value type objects).
Plus, modern GC implementations are super clever and efficient (as opposed to the implementation used by Unity, which unfortunately rather sucks, but I hear they're working on swapping it out).