Symptoms:
Delphi XE3 sometimes fails to run 64-bit applications under a debugger. Code would compile, but the part where Delphi switches to debug layout never happens, Delphi just pops a message saying "Cannot run the debugger".
32-bit debugging continues to work normally, and so does "Run without debugging".
The funny part is that this happens irregularly. Sometimes the first attempt would succeed, and then the debugger would run all the time in all instances of Delphi. But if it fails the first time then it would always fail even if you restart Delphi.
I also noticed that the earlier I launch Delphi + debugger, the higher is the chance it would run (and then continue working). It seemed like there was something I was doing or the computer was doing sometime after boot that broke the debugger if I hadn't launched it yet.
Solution:
Stop the "Internet connection sharing" service and restart Delphi.
What might have contributed:
– Uninstalling older versions of Delphi on the same PC.
– Disabling Windows Firewall
– Disabling Windows Defender
Diagnostics process:
Looking at the successful and failed debugger launches with Process Monitor, in both cases Delphi runs a remote debugger. But on the successful run it's dbkw64_17_0.exe
(64 bit) while failed runs spawn rmtdbg170.exe
(32 bit). Both are Delphi debuggers, but I suspected that the second one is only supposed to be used for 32 bit debugging.
Further investigation showed that in both cases dbkw64_17_0.exe
launches initially, but in the second case it terminates shortly afterwards. Delphi then tries to connect to it through TCP, unable to do so, and restarts it automatically. But the code that does the restart probably wasn't updated to 64 bit and launches 32-bit rmtdbg170.exe
instead.
Anyway, the problem lies in the initial instance of dbkw64_17_0.exe
terminating. Comparing Process Monitor logs, both successful and failed runs load the libraries and then work with winsock. Stack in the final calls indicates ws2_32.dll
's socket()
is running – the debugger is probably trying to open it's command socket for listening – after which failed instance abruptly terminates (Thread Exit, Process Exit). I figured socket()
probably returns with an error.
Using rohitab's Api Monitor I tried to find out the error code, but this didn't work out. Api Monitor successfully traced all the calls until roughly WSAStartup()
, but no further – the last bunch of calls just before the termination always got lost, perhaps the injected driver wasn't being able to send it back to the main app in time before the application terminated.
Then I opened dbkw64_17_0.exe
for debugging in Visual Studio. I set a breakpoint to {,,ws2_32.dll}socket
, caught the execution there and studied what happens step by step. Turns out, socket()
was successful. It was followed by setsockopt
call, also successful (to know which functions we were stepping into, I used VS's standard ability to load Windows DLL symbols from Microsoft servers). Then dbkw64_17_0.exe
called bind()
which failed.
My initial guess was that someone else occupied the port it needed. Checking bind()
parameters at MSDN, I looked into RDX, RCX, R8, R9
registers which host parameters in x64 calls, namely the memory referenced by RCX
, which kept the requested family and port number. It turned out to be 0xC0F3 but it was unoccupied.
I then traced the call to bind()
and from the internal call to WSPBind()
got the error code: 0x1D27, that is 10013 (WSAEACCES: Permission denied. An attempt was made to access a socket in a way forbidden by its access permissions).
This code has no single specific reason for it. From the internet it looks like it appears when some driver or network-related service misbehaves. I tried stopping network related services one by one, until finally bind()
succeeded. The infringing service was "Internet connection sharing (ICS)". As long as I stop this service, the debugger launches normally, and so long as ICS is running, the debugger would not start.
The reason why sometimes the debugger would run and then run always, is probably that ICS hadn't yet been started or did not yet harm the network stack at the time. If the debugger run at that point, it would bind the socket, and for whatever reason binding at that port would then continue working later. But if the debugger was initially launched after the harm has been done, it wouldn't be able to bind to the port neither once nor at all.