It's exactly that: the trickiness around debugging is the main thing that feels like it's got some barriers compared to a turnkey solution in an IDE. I heard VS Code and Godot was available until I realized that the LSP and debugger for Godot 4.x was unusable for months until the recent refactor.
Don't get me wrong though, I am totally using VS Code for my Rust projects. It just isn't a turnkey solution that I'd recommend to someone if they just want to hit "New project" and do the whole write-compile-debug loop without needing to understand anything. (I had also used it a while back prior to rust-analyzer being the main go-to extension, I think...)
I don't like it, but if they're part of the project files, then they belong in version control. I do worry about the challenges of combining the difficult-to-merge nature of binaries with the distributed workflows that Git encourages. While data doesn't get lost, the inability to merge them may mean that someone needs to spend extra time re-performing their changes if they "lose" the push/merge race.
Game engines have been doing a better job of transitioning away from large monolithic binaries by either serializing them in somewhat mergeable text files or at least splitting them into large numbers of smaller binaries to reduce file contention.
Git LFS does offer the ability to off-load them from the repository, reduce download and checkout times as well as the ability to lock files (which does introduce centralization...), but it doesn't seem to be as ubiquitous and can be more expensive to use, depending on the team's options for Git repo providers.
Note: I assume you mean binaries as in "non-text files", not build artifacts, which definitely don't belong in version control at all.