Windows Platform Design Notes

Design Information for the Microsoft® Windows® Family of Operating Systems

Debugging NDIS Drivers

Abstract

This paper provides information about debugging Network Driver Interface Specification (NDIS) drivers for the Microsoft® Windows® family of operating systems. It provides guidelines for the NDIS driver developer to identify commonly encountered issues in network drivers and specifies best practices that will help the driver developer avoid network driver issues in the first place. This paper also provides information on how to use Ndiskd.dll, the NDIS kernel debugger extension, to gather information about the state of NDIS and the NDIS drivers.

Contents

Introduction. 3

NDIS Overview.. 3

Using Ndiskd.dll to Debug NDIS Drivers. 4

Debugger Extensions. 6

NDIS Tracing. 9

NDIS Bug Checks. 10

PnP Overview.. 11

Debugging a Miniport Initialization Problem.. 12

Debugging the Failure of a Miniport to Halt13

Debugging Power Management Issues. 14

Debugging Interrupt Storms. 15

MiniportReset Overview.. 16

Best Practices for Developing Miniport Drivers. 16

Tools – Using NDISTest and Driver Verifier16

Using Debug Messages. 16

Tracking Spin Locks. 16

Best Practices for Developing Intermediate Drivers. 17

Common Intermediate Driver Issues. 17

Best Practices for Developing Protocol Drivers. 18

Call to Action and Resources. 18

This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred.

® 2003 Microsoft Corporation. All rights reserved.

Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Introduction

This paper provides information about debugging NDIS drivers for the Microsoft® Windows® family of operating systems. It also provides coding practices that driver developers can use to avoid common problems encountered by NDIS drivers such as initialization failures in NDIS miniports, NDIS Intermediate drivers whose miniports do not get halted, and other common issues reported to the NDIS team at Microsoft. The paper discusses:

• Ndiskd.dll—NDIS kernel debugger extensions
• NDIS tracing
• PnP overview
• Power Management overview
• NDIS intermediate drivers
• NDIS protocol drivers

Note: The information presented in this paper is applicable to Microsoft Windows Server™ 2003 driver development. Most of the information also applies to Microsoft Windows XP and Microsoft Windows 2000.

NDIS Overview

This paper assumes that you are familiar with the basic NDIS model. In addition to the miniports and protocols that have been discussed in the Microsoft Windows Driver Development Kit (DDK), this paper introduces the MOpen structure that represents a binding between a protocol and a miniport. The model, including the MOpen structure, appears in figure 1.

Figure 1: NDIS diagram including the MOpen structure

Using Ndiskd.dll to Debug NDIS Drivers

Developers and testers at Microsoft Corporation use Ndiskd.dll to quickly analyze potential network driver issues. Ndiskd.dll is a part of the Microsoft Debugging Tools package for kernel-mode debugging. Ndiskd.dll runs on both WinDBG.exe and Kd.exe, and it provides basic and detailed information about a miniport driver and the protocol drivers that are bound to it.

The most commonly used Ndiskd.dll commands are !miniports and !miniport <MiniportAdapterHandle>. Use the !miniports command to print all of the NDIS miniports that are present in the system. Use the !miniport <MiniportAdapterHandle> command to print detailed information about a miniport adapter, including the protocols it is bound to and the miniport’s PnP and power management states.

The following example shows how to get detailed information about a particular protocol that is bound to the Sample Ethernet Driver miniport. The Sample Ethernet Driver is a real NDIS miniport driver that has been given a fictitious name.

To print detailed information about the Sample Ethernet Driver, use the !miniportcommand. The pointer in bold in the preceding example is the argument to !miniport.

Use the !mopen command to print detailed information about an Mopen structure, or binding, between the Sample Ethernet Driver and a protocol. Use the pointers in bold from the preceding example as arguments for the !mopen command.

To print information about the protocol structures use the !protocol command. Use the pointer in bold from the preceding example as the argument to the !protocol command.

The preceding example shows how to use the list of miniport drivers in the system to get detailed information about the protocols that are bound to a miniport adapter. The example printed four sets of information:

• Detailed information about each NDIS miniport adapter in the system.
• Specific information about the Sample Ethernet Driver.
• Detailed information about each of the Mopen structures, or bindings, on the Sample Ethernet Driver miniport.
• Detailed information about the NdisUio protocol.

Note: For detailed information about Ndiskd.dll, see the Help file provided with the Microsoft Debugging Tools, and the Microsoft Windows Driver Development Kit (DDK).

Debugger Extensions

Three useful debugger extensions exist that are not well known:

• !pkt
• !findpacket -p
• !stacks

Ndiskd.dll includes the !pkt and !findpacket commands. The !stacks command is part of the Microsoft Debugging Tools package and is not included in Ndiskd.dll.

!pkt Command

The !pkt command prints out detailed information about an NDIS_PACKET structure. You can specify a verbosity level to control the amount of information that the debugger prints, as shown in the following code.

To print every NDIS_BUFFER structure in an NDIS_PACKET structure, use a verbosity level of 4:

The preceding example prints every NDIS_BUFFER structure in a particular NDIS_PACKET structure, as well as the Out-Of-Band(OOB) information in the NDIS_PACKET structure.

!findpacket p Command

The !findpacket -p command finds all the packets that are allocated from a particular packet pool and that are currently in use by the miniport driver. Intermediate (IM) driver developers can use the findpacket -p command to list all of the packets that are allocated by the IM driver and indicated up to the protocols.

The following example shows how to locate all of the packets that are currently allocated in a packet pool. You need to pass the packet pool’s PoolHandle, that was returned to the NDIS driver when it called the NdisAllocatePacketPoolEx function, as the argument to the !findpacket -p command, as shown below.

!stacks Command

Use the !stacks command to find hung threads and to debug other difficult problems. This extension is present in the Debugger package but is not a part of ndiskd.dll.

You can specify a character string as an argument to the !stacks command. The debugger prints all the threads that contain the character string in their stacks. For example, the command !stacks 2 ndis! causes the debugger to print every thread that contains the “ndis!” string in its stack.

The threads printed by the debugger can provide important clues that will help you solve the problem you are investigating. For example, the output in the following code was taken from an IM driver that was not completing any requests to send packets. When the miniport adapter was disabled, the underlying miniport adapter was never halted as TCP/IP waited for the IM driver to complete the sends. The output, which is the result of the !stacks 2 ndis! command, contains two threads: one that shows TCP/IP waiting for outstanding sends to complete, and another that is an NDIS thread reserved for system use.

The preceding example shows the benefit of running the !stacks 2 ndis! command, which provides useful clues about the current state of the computer. In the example, you can see that TCP/IP is waiting for an event. It is reasonable to assume that it is waiting for outstanding sends. Later in this paper, you will see how this assumption can be verified.

The second thread printed in the preceding example is reserved for system use.

Use the !stacks 2 <miniport driver name> command to print all the threads with the miniport driver’s name in their stacks.

NDIS Tracing

NDIS has debug messages built into its checked version. By default, these are not printed in the debugger and need to be turned on. Use NDIS tracing to analyze problems that are not in the data path, such as PnP and power-management problems. NDIS tracing works on both free and checked builds of the operating system.

The following example demonstrates how to turn on the NDIS tracing feature in the REQUEST code path at the INFO level within NDIS. There are four levels of debug messages available to the developer: Info, Warn, Err and Fatal. The Info level is the most verbose, and the Fatal level is the least verbose. (The Log level has not been implemented.)

The Dbglevel and Dbgsystems settings are toggle settings. To turn off the debug print feature for the REQUEST and BIND code paths, you must re-enter the previous command, as shown here:

The preceding example shows how you can set and reset the verbosity and component values, which control the level of debugging information that NDIS prints in the debugger.

NDIS Bug Checks

The system performs a bug check when it detects an error caused by an NDIS miniport driver. The bug check is performed as soon as an error is detected. If the system did not stop, these errors would appear later, making it difficult to pinpoint the cause of the problem at that stage.

The bug checks that are caused by an NDIS miniport driver use a special bug check code. On Windows XP and Windows 2000, the bug check code is BUGCODE_ID_DRIVER. On Windows Server 2003, the bug check code is BUGCODE_NDIS_DRIVER. If a debugger is connected to the system being tested, a sentence describing the bug check is printed in the debugger. The !analyze –v command provides detailed information on the bug check.

Examples of NDIS miniport driver problems that cause the system to perform a bug check include the following:

• Before the ownership of a receive packet, that the miniport driver indicated by calling the NdisMIndicateReceivePacket function, is transferred back to the miniport driver, the driver indicates the packet in another call to NdisMIndicateReceivePacket.
• The miniport driver frees shared memory that it has not allocated.
• The miniport driver unloads without deregistering an interrupt that it previously registered.

Note: For more information about bug check codes, see the Debugger.chm help file in the Microsoft Debugging Tools package.

PnP Overview

NDIS implements the following Plug and Play (PnP) functionality on behalf of the miniport:

• NDIS intercepts all the PnP driver and device IRPs that are sent to the miniport driver.
• NDIS and the driver together comprise the functional device object (FDO) of the miniport adapter.
• NDIS implements the AddDevice routine on behalf of the minport driver. In response to the StartDevice IRP, NDIS calls the MiniportInitialize function handler of the miniport driver.
• NDIS propagates the Remove/Stop IRP to the miniport driver’s MiniportHalt function. Before calling this function, NDIS unbinds all the protocols bound to that miniport.

The !ndiskd.miniport command prints information that is useful for debugging PnP problems, as shown in the following example.

The !miniport extension prints a lot of useful information. The items that the previous example prints include the following:

• The PnP state of the miniport adapter.
• The physical device object (PDO) and the FDO of the miniport adapter.
• The protocols that are bound to the miniport adapter.
• The hardware resources that are assigned to the miniport adapter.

Debugging a Miniport Initialization Problem

Miniport adapters can fail to initialize for a variety of reasons. If the problem is reproducible, the best way to proceed is to turn on NDIS tracing and reproduce the problem. The debug messages printed in the debugger makes it easy to identify the point of failure.

To debug a miniport initialization problem:

1. Turn on NDIS tracing.
2. Look for the statement specifying the failure code that the MiniportInitialize function handler returns.
3. Using the failure code as a starting point, find the last NDIS function that the miniport called before it entered the failure code path.
4. Inspect the source code of the miniport driver to find the cause of the fault.

The following example contains the debugging messages that are printed during the initialization of a miniport adapter.

The preceding example shows a miniport adapter failing its initialization routine. In the beginning of the example, NDIS gets called at its AddDevice routine for the miniport, followed by a StartDevice IRP. As part of the IRP, NDIS calls the MiniportInitialize function handler, which fails because of an error that occurred after it successfully called the NdisMRegisterIoPortRange function. You can use this information as the starting point for debugging the initialization problem. NDIS debug messages also tell you if the NDIS API failed and possibly why it failed.

Debugging the Failure of a Miniport to Halt

If NDIS does not call a miniport driver’s MiniportHalt function when the miniport adapter is disabled through the user interface or an automated test, it is usually because an operation that the miniport driver must complete before being halted has not completed. For example, NDIS will not halt a miniport adapter until the driver finishes sending all the packets that were sent to it for transmission.

To debug the failure of a miniport to halt:

1. Run the !stacks 2 ndis! command in the debugger to print all the threads that contain the “ndis!” string in their stacks.
2. Inspect the threads to see if they offer any clues. For example, a thread that shows TCP/IP waiting implies that the protocol is waiting for the miniport driver to complete send operations.
3. To find the number of outstanding requests to send packets in the miniport, print all of the Mopen structureson the miniport as explained earlier in this paper. If the miniport driver is not completing packets back to the protocol, the Referencesvalue never returns to one. Calling the NdisCloseAdapter function also decreases this value by one.
4. Investigate the miniport driver to see why the miniport is not completing the packets. The miniport driver calls the NdisMSendComplete function to complete such send operations.

The following example shows an MOpen structure between the NDISUIO protocol and the Sample Ethernet Driver miniport.

The preceding example shows an Mopen structure between the NDISUIO protocol and the Sample Ethernet Driver. NDIS has called the protocol’s ProtocolUnbindAdapter handler.The protocol has called the NdisCloseAdapter function. NDIS will not call the protocol’s CloseAdapterComplete handler until the number of references reaches zero. The references are caused by outstanding send commands in the miniport.

Debugging Power Management Issues

As the device power policy owner of the miniport, NDIS is responsible for querying the bus driver, the miniport driver, and the system settings before deciding on the power management policy for the miniport adapter.

To find the values returned by the bus driver and the miniport driver, turn on NDIS tracing or look at the output of the !miniport command:

In the preceding example, NDIS sets the DISABLE_WAKE_UP flag, because the user has not enabled Wake-On-LAN (WOL) for this miniport adapter. To debug a WOL problem, you need to inspect the power management values returned by the system, the bus driver, and the miniport to determine which of the three entities is disabling WOL. These values are all displayed in the output of the !miniport command.

Debugging Interrupt Storms

Interrupt storms occur when either a miniport driver fails to disable device interrupts or the device continues to assert an interrupt after that interrupt has been disabled. In either case, the computer is not usable until the device stops asserting interrupts.

To diagnose an interrupt storm problem:

1. Set a breakpoint on the MiniportISR function.
2. If MiniportISR returns TRUE, set a breakpoint on the MiniportDpc function. MiniportISR should not return TRUE until the MiniportDpc functionenables interrupts.
3. Investigate why the device is asserting an interrupt before the DPC has completed.

MiniportReset Overview

When a miniport driver calls the NdisMResetComplete function, NDIS completes all outstanding requests on behalf of the driver. If that driver is a serialized driver, NDIS also returns all the send packets that NDIS has queued for the driver to the appropriate protocol drivers.

Best Practices for Developing Miniport Drivers

The following practices are followed by developers at Microsoft Corporation when developing NDIS drivers:

• Running tools such as NDISTest and Driver Verifier.
• Using debug messages.
• Tracking Spin Locks.

Tools – Using NDISTest and Driver Verifier

NDISTest and Verifier.exe are good tools to use when developing NDIS drivers. NDISTest aids and verifies the completeness of a miniport driver. You should use it on both miniports and IM drivers. NDISTest cannot test protocols. Use Driver Verifier to ensure that common errors are easily found and rectified early in the development cycle.

Use Driver Verifier from the beginning of the development effort. Use NDISTest to test the functional completeness and robustness of the driver. You can run an individual test from the test suite to test specific features of the NDIS driver.

Using Debug Messages

The __LINE__ compiler directive is a useful tool. The compiler replaces each __LINE__ directive with the actual line number in the file, which is useful for logging or printing debugging messages.

Tracking Spin Locks

You can enhance a deserialized driver to track the acquisition and release of spin locks by recording each file and line number that acquired a spin lock. This record is cleared just before releasing the spin lock.

At the beginning of each .c file, include a #define statement that uniquely identifies the file, as shown in the following code. Be sure that the defined value is larger than the number of lines that a file could contain, because the line number and the file number will be combined into a single unsigned integer.

Define the following structures and macros in a header file of the driver:

The preceding structure and two macros enable you to quickly find the line of code that last acquired the spin lock. To verify that no spin-lock issues exist in the code, run Driver Verifier with the Deadlock Detection option turned on. Each acquisition of a spin lock will be tested to make sure that it occurred in the right order with respect to all the other spin locks in your driver.

Best Practices for Developing Intermediate Drivers

An NDIS IM driver is complex, because any of the entry points in the driver’s protocol module or miniport module can be called regardless of the state of the other module.

The IM driver must ensure that a notification, such as a send or receive command to one of its modules, does not result in an incorrect action from its other module. For example, when NDIS requests that an IM driver’s miniport module should send a packet, the driver must then determine whether the protocol module can propagate the packet down or if the packet needs to fail.

You should maintain a reference count on all outstanding actions that the IM driver will be called back for—actions such as send commands and requests that the driver initiates to the miniport below it. This reference count should be synchronized with the unbind code in the protocol. The IM miniport should also maintain a reference count of the number of outstanding receive commands that the IM driver has indicated. This count must be synchronized with the PnP code path in the IM miniport driver, to ensure that no receive commands occur after the IM miniport adapter has been halted.

Common Intermediate Driver Issues

When developing an IM driver, you might encounter the problem of TCP/IP not sending packets to the IM driver for transmission. Common causes of this problem include the following:

• The IM driver is reporting an invalid 802.3 physical (MAC) address.
• The IM driver is indicating corrupted packets to TCP/IP.
• The IM driver’s MiniportTransferData function handler is returning corrupted packets.

You should run NDISTest on the IM driver to solve such problems.

Another problem is that outstanding send and request commands in the IM miniport module can prevent NDIS from calling an IM driver’s MiniportHalt routine after the driver has called the NdisIMDeInitializeDeviceInstance function. In this situation, the IM driver could have passed such a packet to the miniport adapter below it, or queued the packet internally. The Mopen structure can determine whether any packets were passed to the underlying miniport.

Best Practices for Developing Protocol Drivers

The problems that commonly occur with protocol drivers are similar to those that occur with IM drivers. As described earlier, it is important to synchronize the protocol driver’s PnP code path with the driver’s send code path.

Use the appropriate I/O Control (IOCTL) codes when designing the protocol driver’s IOCTL interface. IOCTLs should specify the METHOD_BUFFERED option as documented in the DDK and have the appropriate read or write privilege set in the IOCTL code. If an IOCTL is going be used only when an Administrator is logged on, the protocol driver should create a device object by calling the IoCreateDeviceSecure routine.

Call to Action and Resources

Call to Action:

• For driver developers:
• Run NDISTest and Verifier on NDIS drivers during development. These tools verify functional completeness and find common errors in a reproducible manner.
• Build kernel debugger extensions to quickly extract useful information from the driver’s data structures.
• Build diagnostic information into your NDIS driver to quickly identify issues.
• Post questions on the public Microsoft device driver newsgroup. It is an excellent forum for both novice and expert developers.

For questions about debugging and developing NDIS drivers, post questions on the public Microsoft device driver newsgroup.

For specific questions about this paper, send e-mail to ndis6fb@microsoft.com.

Resources:

Microsoft Hardware and Driver Developer Information

http://www.microsoft.com/hwdev/

Microsoft Windows Driver Development Kit (DDK)

http://www.microsoft.com/ddk/

Microsoft Debugging Tools

http://www.microsoft.com/ddk/debugging/default.asp

Microsoft’s public device driver newsgroup

microsoft.public.development.device.drivers