WinDbg kernel debugging symbol pdb download issues #5192

2023-04-04T13:12:09Z

abcde-r
Apr 4, 2023

To separate this out from my previous thread on Windows kernel debugging: Once I get connected, I can't seem to download PDBs.

I will first note that I've got the symbol lookup configured and working fine for static analysis

And C:\SymbolCache is what I'm using in WinDbg too for a cache, so all the relevant symbols should already be cached locally anyway.

When I get into the successful WinDbg connection, I do .sympath to set the download and cache locations for Ghidra (I'm not sure if there's a more correct location to set this.)

0: kd>.sympath cache*c:\SymbolCache;srv*https://msdl.microsoft.com/download/symbols
Symbol search path is: cache*c:\SymbolCache;srv*https://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: cache*c:\symbolcache;srv*https://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       cache*c:\SymbolCache
Deferred                                       srv*https://msdl.microsoft.com/download/symbols
0: kd>.reload /f
Connected to Windows 10 19041 x64 target at (Tue Apr  4 08:56:13.046 2023 (UTC - 4:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
................*** WARNING: Unable to verify timestamp for win32k.sys
......*** WARNING: Unable to verify timestamp for win32kbase.sys
.*** WARNING: Unable to verify timestamp for win32kfull.sys
...*** WARNING: Unable to verify timestamp for cdd.dll
.......................
Loading User Symbols

Loading unloaded module list
.....

************* Symbol Loading Error Summary **************
Module name            Error
ntkrnlmp               The system cannot find the file specified
hal                    The system cannot find the file specified
kdnet                  The system cannot find the file specified
<cut>
condrv                 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.

But basically as you can see it's complaining that it can't find symbols. lm also confirms that at best I have export symbols, not pdb symbols:

0: kd>lm
start             end                 module name
ffff9ed5`e2400000 ffff9ed5`e26d3000   win32kbase T (no symbols)           
ffff9ed5`e26e0000 ffff9ed5`e2a96000   win32kfull T (no symbols)           
ffff9ed5`e2aa0000 ffff9ed5`e2ae9000   cdd      T (no symbols)           
ffff9ed5`e2b10000 ffff9ed5`e2baa000   win32k   T (no symbols)           
fffff807`3bfc0000 fffff807`3c24f000   mcupdate_GenuineIntel   (export symbols)       mcupdate_GenuineIntel.dll
fffff807`3c250000 fffff807`3c256000   hal        (export symbols)       hal.dll
fffff807`3c260000 fffff807`3c2bc000   kd_02_8086   (export symbols)       kd_02_8086.dll
fffff807`3c2c0000 fffff807`3c309000   kdcom      (export symbols)       kdnet.dll
fffff807`3c310000 fffff807`3c337000   tm         (export symbols)       tm.sys
fffff807`3c340000 fffff807`3c3aa000   CLFS       (export symbols)       CLFS.SYS
fffff807`3c3b0000 fffff807`3c3bb000   BOOTVID    (export symbols)       BOOTVID.dll
fffff807`3da00000 fffff807`3ea46000   nt         (export symbols)       ntkrnlmp.exe
<cut>

I tried using !sym noisy to see more verbose information...

0: kd>!sym noisy
SECURE: File not allowed to be loaded - C:\Windows\SYSTEM32\dbghelp.dll
Error code: Win32 error 0n5
The call to LoadLibrary(ext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.
No export sym found

But it looks like I'm running into this earlier issue, (which was believed fixed in 2020?).

I tried a workaround from there:

0: kd>.load C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll
0: kd>!sym noisy
noisy mode - symbol prompts on

But when I do .reload /f, there's no difference in the output.

So is there a more correct way that I should be specifying the symbol download server / cache location?

Answered by d-millar

Apr 4, 2023

Hmmm, I see "cacheC:\SymbolCache" but (and maybe you have tried it - my eyes are old) I mean literally "C:\SymbolCache" w/o the "cache".

View full answer

2023-04-04T17:30:48Z

d-millar
Apr 4, 2023
Collaborator

I have one idea, but may be way off track - in your comments above are you issuing ".sympath", ".load", and ".reload" from inside Windbg or from the Interpreter in Ghidra? Am hoping the latter.

3 replies

abcde-r Apr 4, 2023
Author

The interpreter in Ghidra

Pasted Graphic 11

d-millar Apr 4, 2023
Collaborator

Hmmm, two guesses (but definitely guesses):

(1) It's possible that the symbol load is timing out. I think this is unlikely, especially if you pre-loaded C:\SymbolCache by using it from Windbg first, so let's ignore that for the moment.
(2) Possible that dbghelp.dll has not gotten loaded from anywhere, which the !sym_noisy response you pasted suggests:
SECURE: File not allowed to be loaded - C:\Windows\SYSTEM32\dbghelp.dll
I like the ".load" solution but that might not have cut it. Try editing support/launch.properties and adding the line:
VMARGS=-Djna.library.path="C:\Program Files (x86)\Windows Kits\10\Debuggers\x64"
then re-launch Ghidra.

abcde-r Apr 4, 2023
Author

Adding that VMARGS line did eliminate the dbghelp.dll error message, but didn't change the actual loading of symbols:

Using NET for debugging
Opened WinSock 2.0
Waiting to reconnect...
Connected to target 172.16.126.139 on port 50505 on local IP 172.16.126.137.
You can get the target MAC address by running .kdtargetmac command.
Interrupt sent
Connected to Windows 10 19041 x64 target at (Tue Apr  4 15:30:52.538 2023 (UTC - 4:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is: 
Windows 10 Kernel Version 19041 MP (2 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Kernel base = 0xfffff805`4ac00000 PsLoadedModuleList = 0xfffff805`4b82a1b0
Debug session time: Tue Apr  4 15:30:52.503 2023 (UTC - 4:00)
System Uptime: 0 days 1:45:07.221
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
*                                                                             *
*   You are seeing this message because you pressed either                    *
*       CTRL+C (if you run console kernel debugger) or,                       *
*       CTRL+BREAK (if you run GUI kernel debugger),                          *
*   on your debugger machine's keyboard.                                      *
*                                                                             *
*                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
*                                                                             *
* If you did not intend to break into the debugger, press the "g" key, then   *
* press the "Enter" key now.  This message might immediately reappear.  If it *
* does, press "g" and "Enter" again.                                          *
*                                                                             *
*******************************************************************************
*******************************************************************************
*                                                                             *
*   You are seeing this message because you pressed either                    *
*       CTRL+C (if you run console kernel debugger) or,                       *
*       CTRL+BREAK (if you run GUI kernel debugger),                          *
*   on your debugger machine's keyboard.                                      *
*                                                                             *
*                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
*                                                                             *
* If you did not intend to break into the debugger, press the "g" key, then   *
* press the "Enter" key now.  This message might immediately reappear.  If it *
* does, press "g" and "Enter" again.                                          *
*                                                                             *
*******************************************************************************
0: kd>lm
start             end                 module name
fffff805`4ac00000 fffff805`4bc46000   nt         (export symbols)       ntkrnlmp.exe

Unloaded modules:
fffff805`50fe0000 fffff805`50fef000   dump_storport.sys
fffff805`50030000 fffff805`5005d000   dump_stornvme.sys
fffff805`50080000 fffff805`5009e000   dump_dumpfve.sys
fffff805`50a70000 fffff805`50a8c000   dam.sys 
fffff805`4f9e0000 fffff805`4f9f1000   hwpolicy.sys
0: kd>.sympath cache*c:\SymbolCache;srv*https://msdl.microsoft.com/download/symbols
Symbol search path is: cache*c:\SymbolCache;srv*https://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: cache*c:\symbolcache;srv*https://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       cache*c:\SymbolCache
Deferred                                       srv*https://msdl.microsoft.com/download/symbols
0: kd>.reload /f
Connected to Windows 10 19041 x64 target at (Tue Apr  4 15:31:26.277 2023 (UTC - 4:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
................*** WARNING: Unable to verify timestamp for win32k.sys
......*** WARNING: Unable to verify timestamp for win32kbase.sys
.*** WARNING: Unable to verify timestamp for win32kfull.sys
...*** WARNING: Unable to verify timestamp for cdd.dll
.......................
Loading User Symbols

Loading unloaded module list
.....

************* Symbol Loading Error Summary **************
Module name            Error
ntkrnlmp               The system cannot find the file specified
hal                    The system cannot find the file specified
kdnet                  The system cannot find the file specified
<cut>
condrv                 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0: kd>!sym noisy
noisy mode - symbol prompts on
0: kd>.reload /f
Connected to Windows 10 19041 x64 target at (Tue Apr  4 15:31:37.649 2023 (UTC - 4:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
................*** WARNING: Unable to verify timestamp for win32k.sys
......*** WARNING: Unable to verify timestamp for win32kbase.sys
.*** WARNING: Unable to verify timestamp for win32kfull.sys
...*** WARNING: Unable to verify timestamp for cdd.dll
.......................
Loading User Symbols

Loading unloaded module list
.....

************* Symbol Loading Error Summary **************
Module name            Error
ntkrnlmp               The system cannot find the file specified
hal                    The system cannot find the file specified
kdnet                  The system cannot find the file specified
<cut>
condrv                 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0: kd>lm
start             end                 module name
fffff218`96c00000 fffff218`96ed3000   win32kbase T (no symbols)           
fffff218`96ee0000 fffff218`97296000   win32kfull T (no symbols)           
fffff218`972a0000 fffff218`972e9000   cdd      T (no symbols)           
fffff218`977e0000 fffff218`9787a000   win32k   T (no symbols)           
fffff805`48810000 fffff805`48a9f000   mcupdate_GenuineIntel   (export symbols)       mcupdate_GenuineIntel.dll
fffff805`48aa0000 fffff805`48aa6000   hal        (export symbols)       hal.dll
fffff805`48ab0000 fffff805`48b0c000   kd_02_8086   (export symbols)       kd_02_8086.dll
fffff805`48b10000 fffff805`48b59000   kdcom      (export symbols)       kdnet.dll
fffff805`48b60000 fffff805`48b87000   tm         (export symbols)       tm.sys
<cut>    
fffff805`524c0000 fffff805`52541000   cldflt     (no symbols)           
fffff805`52550000 fffff805`5256a000   storqosflt   (no symbols)           
fffff805`52570000 fffff805`52598000   bindflt    (no symbols)           
fffff805`525a0000 fffff805`525f6000   msquic     (export symbols)       msquic.sys

Unloaded modules:
fffff805`50fe0000 fffff805`50fef000   dump_storport.sys
fffff805`50030000 fffff805`5005d000   dump_stornvme.sys
fffff805`50080000 fffff805`5009e000   dump_dumpfve.sys
fffff805`50a70000 fffff805`50a8c000   dam.sys 
fffff805`4f9e0000 fffff805`4f9f1000   hwpolicy.sys

Also, just to confirm, I found the Edit -> Symbol Server Config menu bar option, and confirmed that it shows the same config I pasted an image of previously, that I had set up for static analysis. I also tried just ".sympath cache*c:\SymbolCache" but that didn't yield different results either.

2023-04-04T21:41:35Z

d-millar
Apr 4, 2023
Collaborator

Still trying to figure this out, but definitely not making sense to me. If Windbg lists the modules with symbols and you've point .sympath to that cache in Ghidra, the dbgeng should be reading the same symbols. (Edit->Symbol Server Config doesn't buy you anything, unfortunately, as that's only used by the Static engine.) This is going to sound dumb, but try ".sympath c:\SymbolCache", ".reload /f", and then "lm".

1 reply

abcde-r Apr 4, 2023
Author

Yep, I knew it was worth a try (to eliminate your theory about timeout) and had actually tried that already (and appended the result to previous message), but no luck.

2023-04-04T22:01:53Z

d-millar
Apr 4, 2023
Collaborator

Hmmm, I see "cacheC:\SymbolCache" but (and maybe you have tried it - my eyes are old) I mean literally "C:\SymbolCache" w/o the "cache".

6 replies

abcde-r Apr 4, 2023
Author

Apparently it's my eyes that are bad, because I didn't even see the difference. And it turns out that worked! (I didn't even realize that was valid syntax.)

1: kd>.sympath c:\SymbolCache
Symbol search path is: c:\SymbolCache
Expanded Symbol search path is: c:\symbolcache

************* Path validation summary **************
Response                         Time (ms)     Location
OK                                             c:\SymbolCache
1: kd>.reload /f
Connected to Windows 10 19041 x64 target at (Tue Apr  4 18:09:11.273 2023 (UTC - 4:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
.................................................
Loading User Symbols

Loading unloaded module list
.....

************* Symbol Loading Error Summary **************
Module name            Error
clipsp                 The system cannot find the file specified
vsock                  The system cannot find the file specified
vmci                   The system cannot find the file specified
vmrawdsk               The system cannot find the file specified
vmmouse                The system cannot find the file specified
vm3dmp_loader          The system cannot find the file specified
vm3dmp                 The system cannot find the file specified
drmk                   The system cannot find the file specified
vmusbmouse             The system cannot find the file specified
vmmemctl               The system cannot find the file specified
peauth                 The system cannot find the file specified
vmhgfs                 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
1: kd>lm
start             end                 module name
fffff218`96c00000 fffff218`96ed3000   win32kbase # (pdb symbols)          c:\symbolcache\win32kbase.pdb\7C7B899DC8ABA790212BDBF0B422A7801\win32kbase.pdb
fffff218`96ee0000 fffff218`97296000   win32kfull   (pdb symbols)          c:\symbolcache\win32kfull.pdb\8F65B46C091ABCC8D15DCACFACE478F21\win32kfull.pdb
fffff218`972a0000 fffff218`972e9000   cdd      # (pdb symbols)          c:\symbolcache\cdd.pdb\D25E87920F34983CCC758E03726361181\cdd.pdb
fffff218`977e0000 fffff218`9787a000   win32k   # (pdb symbols)          c:\symbolcache\win32k.pdb\AD6D3E11D496C8837D0142839435EE071\win32k.pdb
<cut>

abcde-r Apr 4, 2023
Author

quick question before I perhaps pull it onto a different thread: Now that symbols are working, I set a hardware breakpoint on a function in win32kfull.sys, and I have that same code already analyzed in my project. When the breakpoint fires and I hit decompile, it doesn't decompile. Should it? (If so I'll make a new thread.)

I was able to open win32kfull.sys and then manually go to the address of the breakpoint and then decompile and that works, but I'm thinking there's probably a way to follow the rip register in the dynamic view and track the decompiled code?

d-millar Apr 4, 2023
Collaborator

Maybe - another can of worms, but depends on whether the program name (in the static listing) and the name in the Modules list are a match. If they're not, you have to map the two manually in Modules.

abcde-r Apr 4, 2023
Author

OK, thanks. That was hint enough. I had noticed that the Modules list only had nt in it, so once I was at a breakpoint I manually hit the "refresh" icon in the Objects window and that caused all the other kernel modules to show up. Thanks again, you're super helpful and awesome!!!

d-millar Apr 5, 2023
Collaborator

My pleasure - I should really be thanking you. Appreciate your working through this and providing insight into things we could improve. Feel free to check any time!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WinDbg kernel debugging symbol pdb download issues #5192

{{title}}

Replies: 3 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

WinDbg kernel debugging symbol pdb download issues #5192

abcde-r Apr 4, 2023

Replies: 3 comments · 10 replies

d-millar Apr 4, 2023 Collaborator

abcde-r Apr 4, 2023 Author

d-millar Apr 4, 2023 Collaborator

abcde-r Apr 4, 2023 Author

d-millar Apr 4, 2023 Collaborator

abcde-r Apr 4, 2023 Author

d-millar Apr 4, 2023 Collaborator

abcde-r Apr 4, 2023 Author

abcde-r Apr 4, 2023 Author

d-millar Apr 4, 2023 Collaborator

abcde-r Apr 4, 2023 Author

d-millar Apr 5, 2023 Collaborator

abcde-r
Apr 4, 2023

Replies: 3 comments 10 replies

d-millar
Apr 4, 2023
Collaborator

abcde-r Apr 4, 2023
Author

d-millar Apr 4, 2023
Collaborator

abcde-r Apr 4, 2023
Author

d-millar
Apr 4, 2023
Collaborator

abcde-r Apr 4, 2023
Author

d-millar
Apr 4, 2023
Collaborator

abcde-r Apr 4, 2023
Author

abcde-r Apr 4, 2023
Author

d-millar Apr 4, 2023
Collaborator

abcde-r Apr 4, 2023
Author

d-millar Apr 5, 2023
Collaborator