If you're willing to deal with enterprise pricing, Undo [0] implements something reasonably comparable to rr without the hardware timer requirements.

[0] https://undo.io/

Are you aware of solutions for multi-threaded and relative time-accurate recordings, for example by using user-selected synchronization points? Afaiu, rr and undo do simulation of threads on one CPU core, which rules out detecting timing issues.

Co-founder of Undo here. This is a common misunderstanding, and just not true -- neither for Undo nor rr. Most races will reproduce at least as easily in Undo, especially if you use our "thread fuzzing" feature (rr has something similar, called chaos mode).

Sure, there will always be some races/timing issues that just won't repro under recording (Heisenberg principal and all that), but in fact most races are _more likely_ to occur under recording. Part of this is because you slow down the process being recorded, which is equivalent to speeding up the outside world.

And of course, when you do have your gnarly timing issue captured in a recording, it's usually trivial to root-cause exactly what happened. Our customers tell us that races and timing issues are a major use-case.

Thanks for clarification. I have not been aware of thread fuzzing and chaos mode before.

Multiplexing onto a single thread is sufficient to observe and record concurrency errors. If that is not sufficient, then I am assuming you want to observe and record errors caused by true parallelism. If so, then you need a full memory trace. That restricts you to either hardware that supports full memory trace connected to a hardware trace probe or instrumented software full memory trace.

The former basically only exists for embedded boards and the latter does not exist (at say less than a 10x slowdown) for Linux or any other common desktop operating system as far as I am aware.

So the only way to trace probe consumer desktop CPUs and possibly GPUs is by the hardware vendors them-self or specialized facilities/laboratories? For Intel I can find https://www.lauterbach.com/supported-platforms/architectures..., but nothing for trace probing AMD or later versions.

You can trace consumer desktop CPUs using instrumented software full memory trace, but that requires OS + debugger + compiler support which is not available for Linux, Windows, Mac, etc.

You can trace hardware that exposes trace functionality usually via a debug port of some kind. Many chips have trace functionality in their production design, but no debug connector is physically present in off-the-shelf boards (to reduce manufacturing cost). You can usually physically modify the board to get access to this functionality which is routinely done when porting software to a new chip/board.

Trace functionality comes in two major flavors, control flow trace and memory trace. Control flow trace only records control flow, so the contents of memory are unknown which is not very useful for your desired use case. Memory trace records memory accesses, so the contents of memory are known. Unfortunately, memory trace is very resource intensive, so most systems that support trace only implement control flow trace. As far as I am aware, it is very unlikely that any desktop or server CPU has memory trace.

The major manufacturers of trace probes and solutions that I know of are Green Hills Software, Lauterbach, and Segger.

Thanks for your write-up, this is very helpful for understanding.

While the single-threaded execution means that issues from thread interleaving on the scale of nanoseconds will effectively not happen, multiple threads are still allowed and will be context-switched between. rr also has a chaos mode to intentionally make the context switching unfair.

What kind/class of issues are caused by "thread interleaving on the scale of nanoseconds"? Faulty CPU bit flips due to radiation/quantum effects or what are you referring to? Just curious.

Not doing things atomically when they should be (incl. missing locks around tiny ops) would be a pretty large class.

With native multithreading data can pass from thread to thread millions of times per second, and you're much less likely to hit obscure interactions when limited instead to maybe a couple hundred context switches per second.

Exactly right. Undo has "thread fuzzing" which is similar concept to chaos mode, but more targeted.

Antithesis is the only general purpose system I've seen for that. It takes the same single threaded approach, but can scale to N separate systems and fault injecting possible orderings.

eh, thanks

> How can I persuade my boss to pay for Undo?

fun quote.. but my boss will never pay for it because I am the only one who use gdb in our company, unfortunately)

For what it's worth we do offer a more printf-y interface for people who don't like a debugger - https://docs.undo.io/PostFailureLogging.html

And CLion / VS Code for people who prefer an IDE interface.

But a lot of people do really want to stick with their printf debugging.

If your boss won't buy you an Undo you can still use https://rr-project.org/ - or on Windows the built in time travel debug of WinDbg.

  > gdb(..) smashing tui with app output

this got me losing my mind. How/why propose this tempting TUI mode when the result looks like a broken arcade game ??

How do you people comfortably debug C in Linux ? I know VSCode looks nice but by principle I can't accept to use such a beast to basically edit code..

> How do you people comfortably debug C in Linux?

same way I debug everything, everywhere: logging.

That not very environmentally friendly.

> How do you people comfortably debug C in Linux ?

I just got comfortable using gdb/lldb from the terminal. Once you get used to it, it's fine (albeit not pretty).

The advantage for text based interfaces is that they work over ssh/tmate over trans atlantic connections whereas the GUI tools would suffer greatly. It just works everywhere and if you learn it, it will be just as good as a gui debugger. Potentially even more ergonomic without hunting for options with a mouse.

I‘m using cgdb, which is a minimal ncurses wrapper around gdb. It‘s a lot better than the TUI of gdb.

Termdebug works great with gdb, and I get my usual editor features as well as the full functionality of gdb. Seems fine to me.

Before I switched from emacs I had an equivalently good setup with dap-mode.

Use Emacs to drive gdb (through the GUD mode) gives a much more ergonomic interface. It highlights the line of code being traced in the code window.

as I said - this easy to fix once and for ever

Also not all apps write to stdout/stderr by default.

> How do you people comfortably debug C in Linux ?

It depends on what comfortable is for you. Most of my pc experience is terminal and browser and this is comfortable for me. I just use gdb for debugging. Sometimes trying lldb

emacs. Worked great for decades and works great today. Learn it.

rr should work on any remotely modern Intel system, and generally on AMD's Zen CPUs too. Unless you're in a virtualized environment (some of which are supported) or a more esoteric architecture rr probably works on your silicon.

I have AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8) @ 2.100GHz.

As I remember - it is should work according to documentation, but I couldn't launch it. Probably I'm not spend enough time to solve errors

You probably need the fix from this page: https://github.com/rr-debugger/rr/wiki/Zen

Which visual studio are you using?

It’s been a number of years since I’ve used it but Visual Studio PRO could do all these things - at least as long as I was using it (since visual c++ 5).

VS Code on the other hand is no where near as featured or powerful.

I use VS2022 Enterprise

If you know solutions, I will be very thankful for any info.

P.S.

Note, though I meet all of this problems, probably I don't spent enough time to find a solution (maybe tried first links at google and so on). E.g. tried `strcmp` for breakpoints, tried to write .natstepfilter.

So, if VS really can do all of this, I'm sorry for my hurry.

I don't know for c but for C# you can write a custom expression that get evaluated for the condition.

Man, if you read my comments lately you would think I'm a Microsoft fanboy, sheesh.

But honestly, in all my years, Visual Studio has been (by far) the best non-commercial (or should I say built-in?) debugger that I've used, and that includes gdb.

I am not a huge c++ on Windows guy though, so YMMV.

Here are a few guides that you may find helpful (and I am also going to include the beginner one, but please do not take that as an indictment of your skill level, I am including only for completeness).

These are all for VS2022:

C++ Debugging Tutorial: https://learn.microsoft.com/en-us/visualstudio/debugger/gett...

C++ Breakpoint Debugging: https://learn.microsoft.com/en-us/visualstudio/debugger/usin...

Breakpoint/Watch Expressions (pay attention to the debugger intrinsics): https://learn.microsoft.com/en-us/visualstudio/debugger/expr...

High Level Debugger Tour: https://learn.microsoft.com/en-us/visualstudio/debugger/debu...

VS2022 Debugging TOC: https://learn.microsoft.com/en-us/visualstudio/debugger/?vie...

My apologies if you've already found these references and they don't do you any good, but your issues just don't sound like the types of issues I've ever experienced with the debugger, and sometimes MS' documentation is just disorganized and incomplete.

I'm a UNIX guy through and through, literally everything I ever do is writing code on Linux or macOS which targets Linux and/or macOS. I loathe almost everything about the whole Windows ecosystem, from Microsoft as a company to the way they build their UIs to the technical foundations of Windows.

However I have to give them one thing: their developer tooling with Visual Studio and other first-party tools seem vastly superior than anything on macOS/Linux when it comes to debugging. I would never use it as a code editor, but it's clear that a lot of effort has been invested into the debugging experience.

very thanks, this will be useful to simplify my debugging!

> and I am also going to include the beginner one, but please do not take that as an indictment of your skill level

Don't worry, I am really feel myself as newbie in windows

Out of interest what are your main use cases for needing conditional breakpoints?

For example for debugging gui apps. Let's say we have function `on_event(event_type)` and you want to examine execution if `event_type == mouse_up`.

In reality conditional breakpoint is the same as simple, but simple require support from code:

``` void on_event(event_type type) { if (type == mouse_up) { // set breakpoint here } } ```

Also useful to break depending on call stack[0]

[0] https://sourceware.org/gdb/current/onlinedocs/gdb.html/Conve...

Yeah, I was specifically wondering about cases where you can't or don't want to change the source code. Sometimes that happens when I need to explore the state after a long process e.g. loading a large file. Generally though I avoid them like the plague as they make the code so slow. Curious to know what other people use it for.