It turns out there's finally a simple and efficient way to swap two variables without using a temporary variable, thanks to this StackOverflow post. As of C# 7.0, you can use tuples like in the example below.
int a = 1; int b = 0; (a, b) = (b, a);
Evidently it's just as efficient as using a temporary variable as well. The generated IL (machine code) is exactly the same as if you used a temporary variable.