Solaris Driver Models.1298

Solaris (x86 Platform Edition) Device Driver Writers Orientation
A Solaris (x86 Platform Edition) Driver Model Overview for UNIX Kernel Programmers
(A summary of some differences between traditional SVR4 and Solaris)
January 1998 (Revised 12/98)
Copyright 1998 Sun Microsystems, Inc. All rights reserved.
1/68
A Driver Model Overview for UNIX Kernel Programmers
This presentation provides a contrast between the traditional (single-threaded) UNIX kernel driver environment and the Solaris kernel driver environment. The audience is assumed to have: A good understanding of operating system principles Familiarity with traditional (single-threaded) UNIX kernel and driver models Familiarity with UNIX kernel implementation on x86 based systems This presentation is intended to: Introduce UNIX kernel driver writers to some Solaris driver models Build on assumed existing knowledge of traditional UNIX kernel models This presentation is NOT intended as: A general class on writing device drivers A thorough coverage of Solaris device driver models A substitute for the Writing Device Drivers book A substitute for Section 9 of the man pages
2/68
Disclaimer
In order to achieve the objective of rapidly introducing Solaris models to programmers of traditional UNIX kernels, we discuss some aspects of the Solaris implementation, or plausible implementation. Many of these implementation descriptions are simplified, and in any case are architecture specific. What is important for Solaris driver writers is the set of Solaris driver interfaces, and the models that they operate within, rather than the implementations underlying them. These models and interfaces are described more precisely in Writing Device Drivers and Section 9 of the man pages, which you are urged to study. In this presentation, we sometimes discuss the implementation, or plausible implementation, so that we can take advantage of the pre-existing, implementation oriented knowledge that the audience is assumed to already hold about traditional UNIX kernels. The audience should use these discussions in the spirit in which they are intended: to facilitate a quicker grasp of the Solaris models. In all cases, drivers written for Solaris should be based on the Solaris models and interface definitions, without further assumptions about the underlying implementation. This will result in a higher likelihood of producing architecture-independent software that continues to work in future releases of the system.
3/68
Topics: DDI/DKI Specification Dynamic Loading of Kernel Modules Device Instances, Properties, and Configuration Files Multithreaded Kernel, Mutex and Condition Variables Interrupt Handling Accessing Devices DMA Services SCSI Drivers: SCSA Network Drivers: DLPI and GLD Realmode (x86 Boot) Drivers General Advice
4/68
DDI/DKI Specification Device Driver Interface/Driver Kernel Interface (DDI/DKI) Two Versions of UNIX SVR4 DDI/DKI Specification: Uniprocessor version, published in 1990 Multiprocessor version, published in 1992 Solaris DDI/DKI Origins: Derived from SVR4 uniprocessor version Added MP support Solaris MP support developed independently from SVR4 MP support SVR4 implemented MP support differently from Solaris Solaris Versus Traditional SVR4 DDI/DKI Consistencies: Functions found in both specifications generally work the same way Few upward-compatible exceptions Solaris functions are designed to be identical or upward-compatible
Solaris (x86 Platform Edition) Device Driver Writers Orientation 5/68 A Driver Model Overview for UNIX Kernel Programmers
DDI/DKI Specification (cont) Some Solaris DDI/DKI Additions: MP support Dynamic loading of kernel modules Device autoconfiguration Architecture independent abstraction layers for hardware functions such as: Device addressing Interrupt handling DMA handling Interface between SCSI target drivers and HBA drivers Support in these areas is significantly different from traditional UNIX systems
6/68
DDI/DKI Specification (cont) See Also: Section 9 of the Reference Manual (man pages) provides exact specifications Summary of Solaris 7 DDI/DKI Services in Writing Device Drivers provides a summary of non-STREAMS DDI/DKI functions grouped by functional area STREAMS Programming Guide discusses STREAMS DDI/DKI functions
7/68
Dynamic Loading of Kernel Modules Device Drivers Are Kernel Modules: Solaris supports dynamic loading of kernel modules Modules are automatically loaded when references are made to them e.g. when one of the devices controlled by a driver module is opened Many kernel modules are dynamically loadable Device drivers must be dynamically loadable Device drivers should also be unloadable Module information is maintained by kernel in module linkage structure DDI routines mod_install(9F) and mod_remove(9F) add and remove modules from the list of currently loaded modules
8/68
Dynamic Loading of Kernel Modules (cont) Entry Points: Each module contains entry points _init(9E), _fini(9E), _info(9E) _init(9E) entry point is called when module is loaded _init(9E) entry point must call mod_install(9F) _init(9E) routine must pass a modlinkage(9S) structure to mod_install(9F) modlinkage(9S) structure describes various module attributes Driver module attributes include a pointer to a dev_ops(9S) structure dev_ops(9S) structure specifies the drivers traditional entry points Module _fini(9E) entry point is called to request that the module unload Module _fini(9E) entry point should call mod_remove(9F) Module _info(9E) entry point must return specific module information
9/68
Dynamic Loading of Kernel Modules (cont) Contrast With Traditional SVR4: These facilities replace SVR4 init(), start(), and halt() entry points Driver entry points are specified in the mod_install(9F) operation There is no longer a bdevsw[] or cdevsw[] array See Also: Autoconfiguration in Writing Device Drivers _init(9E), _fini(9E), mod_install(9F), modlinkage(9S), modldrv(9S), modlstrmod(9S), dev_ops(9S), cb_ops(9S)
10/68
Device Instances, Properties, and Configuration Files Device Information (dev info) Tree: Stores information about devices in the system Each node (dev_info node) in the tree represents one device instance System stores all information it knows about a device in the dev_info node Leaf nodes generally represent devices that can be opened An internal (non-leaf) node generally represents a bus (or bus nexus) The root node of the tree represents the entire system
11/68
Sample Device Information Tree

root nexus node
pseudo nexus node
PCI bus nexus node
EISA bus nexus node
PCI-PCI nexus node
dnet
adp HBA nexus node
dnet
dnet
dnet
Leaf Nodes
sd0
sd6
12/68
Device Instances, Properties, and Configuration Files (cont) Bus Nexus Node: Provides architecture independence Contains knowledge of the bus itself, and how its services are implemented e.g. EISA DMA services, SCSI HBA transport services Isolates leaf drivers from bus dependencies, allowing better portability Many DDI/DKI functions called from leaf drivers result in calls to their parent nexus driver to provide the bus-dependent service See Also: SunOS Kernel and Device Tree in Writing Device Drivers
13/68
Device Instances, Properties, and Configuration Files (cont) Properties: Each dev_info node has an associated set of properties Each property has a name and a value Some properties are understood by the system e.g. interrupts used by ddi_add_intr(9F) reg used by ddi_regs_map_setup(9F) Some properties are driver-specific A driver can access its properties using ddi_prop_op(9F) routines Dynamic properties can be implemented using a prop_op(9E) entry point However, usually a driver specifies system standard prop_op(9F) routine Origin of Properties: Most properties for modern devices are passed up from the bootstrap Properties can also originate in a driver configuration (.conf) file A driver can also create its own properties using ddi_prop_create(9F)
14/68
Device Instances, Properties, and Configuration Files (cont) Driver .conf File: Can specify device properties, including physical configuration For most modern devices, this file is obsolete and unnecessary Sometimes this is used to communicate user-selectable driver options See Also: isa(4), eisa(4), pci(4) Self-Identifying Devices: SPARC devices have firmware to inform system of device configuration Solaris (x86 Platform Edition) uses realmode drivers for the same effect Devices are detected during system booting The dev info tree is created by the bootstrap and passed to the kernel Each devices configuration stored in dev_info node properties in that tree As far as the kernel is concerned, these devices are already identified For such devices, no probe(9E) routine is needed
15/68
Device Instances, Properties, and Configuration Files (cont) Autoconfiguration: In Solaris (x86 Platform Edition), the x86 bootstrap identifies all devices The device tree is already complete when the kernel takes control Driver probe(9E) routines are no longer necessary Device drivers do not poke around looking for their devices The prtconf(1M) command may be used to examine the device tree The kernel will call a driver to attach(9E) a device instance when needed
16/68
Device Instances, Properties, and Configuration Files (cont) Device Attachment: A device driver is associated with each node in the dev info tree A device driver may be associated with multiple nodes (instances) The kernel framework may call upon the driver one or more times to attach the various instances of the device Entry Points: identify(9E) probe(9E) attach(9E) detach(9E)
obsolete; entry point should be specified as nulldev no longer required for most modern devices; use nulldev attach an instance of a device to the driver detach an instance of a device from the driver
17/68
Device Instances, Properties, and Configuration Files (cont) attach(9E): Prepares the driver to control a device instance Must succeed before the device may be opened Uses ddi_get_instance(9F) to obtain instance number assigned by system May call ddi_soft_state_zalloc(9F) to allocate a per-instance state structure State structure is defined by the driver Contains whatever information driver must keep for device instance e.g. pointer to dev_info node, mutex or condition variables, devices address, state of the device, etc. State structure pointer subsequently available via ddi_get_soft_state(9F) Dynamic allocation per instance avoids compiled-in arrays No limits on number of instances supported by a driver Sets up any required interrupt routines or device memory mapping Initializes any required mutex and condition variables Creates the minor device nodes associated with that instance Nodes are created only for devices actually present in the system
Device Instances, Properties, and Configuration Files (cont) Major Numbers: Each device type (driver) has an associated major number Mapping of driver name to major number is defined in /etc/name_to_major Major numbers are assigned automatically by add_drv(1M) Instance Numbers: System assigns a unique instance number to each device of a given type Driver can discover assigned instance number using ddi_get_instance(9F) Instance numbers are small integers and may be used as minor numbers Instance numbers persist across reboots and reconfigurations So instance numbers are not necessarily consecutive integers
19/68
Device Instances, Properties, and Configuration Files (cont) Minor Numbers: A device instance may have one or more associated minor device numbers Common to have one minor device per instance Sometimes convenient to have a set of minor devices per instance e.g. tape driver: selection of density or rewind behavior Mapping from minor number to instance number is defined by driver The driver normally performs a simple calculation to map one to the other System may query driver about this mapping using getinfo(9E) entry point Decoding each minor number must always yield the same instance number getinfo(9E) may be called before attach(9E) to map minor number to instance number, so cannot assume soft state exists in this case getinfo(9E) may also be called to map minor number to dev_info node; this case will only occur after successful attach(9E), so attach(9E) must save its dev_info pointer, usually in the soft state structure
20/68
Device Instances, Properties, and Configuration Files (cont) Device Special Files: Driver specifies file system node names using ddi_create_minor_node(9F) Actual device special files are stored under /devices The /devices file system tree represents the internal dev info tree Bus nexus nodes are represented by directories under /devices Leaf nodes are represented by special files in those directories Entries in traditional /dev directory are symbolic links into /devices Current Implementation: /devices is part of the root file system During a reconfiguration reboot Utility programs read the internal dev info tree and create corresponding device nodes in /devices Utility programs create symbolic links in /dev Entries in /dev and /devices persist across a reboot See Also: drvconfig(1M), add_drv(1M), disks(1M), tapes(1M), devlinks(1M)
Sample Device File System Hierarchy (abridged)
/devices pci@0,0 pci1011,21@f pseudo mm@0:mem mm@0:kmem mm@0:null mem kmem null dnet0 dnet1 iprb0
/dev dsk rdsk
pci10b8,2003@4:dnet0 pci10b8,2003@5:dnet1 pci8086,1229@6:iprb0

[symbolic links]
22/68
Device Instances, Properties, and Configuration Files (cont) Attach Ordering: A driver should not make any assumptions about: Order of calls to attach(9E) for different instances Order of calls to probe(9E) and attach(9E) for different instances Number of times probe(9E) may be called for an instance Number of times attach/detach pairs may be called for an instance detach(9E): Drivers detach(9E) routine is called to undo a previous attach(9E) detach(9E) routine must deallocate all resources allocated in attach(9E) e.g. interrupts, mutex and condition variables, device mappings, soft state structure, minor device nodes Driver may not retain any state for device instance after detach(9E) returns detach(9E) may not return while driver has callback functions pending Pending callbacks must be cancelled, or waited for Otherwise callback could occur after the driver is unloaded!
Device Instances, Properties, and Configuration Files (cont) open(9E) and attach(9E) Ordering: Solaris supports automatic attachment when an open is attempted A drivers open(9E) routine may be called before attach(9E) succeeds In this case open(9E) must fail and return ENXIO The system will then attempt to attach the device using attach(9E) If the call to attach(9E) succeeds, the open(9E) will be retried Current Implementation: Automatic attachment can happen because: Entries in /dev and /devices persist across a reboot So device special files remain available for open(2) after reboot Such an open can cause automatic driver module loading and device attachment described above
24/68
Multithreaded Kernel, Mutex and Condition Variables Threads: A thread of control, or thread, is a sequence of instructions in a program Solaris supports multithreading: A program may have multiple threads of control Threads can share data and code with other threads On MP systems, multiple threads execute simultaneously There are application threads, which run in application programs There are kernel threads, which run in the kernel Drivers are concerned with kernel threads
25/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Multithreaded Kernel: Solaris has a multithreaded kernel There are kernel threads associated with user processes There are system kernel threads such as those created to handle interrupts Device driver routines run in kernel threads On a multiprocessor system, several kernel threads can run simultaneously Contrast With Traditional SVR4: The Solaris kernel may preempt a thread at any time to run another thread Solaris kernel code can no longer assume that a process in the kernel will continue to run until it voluntarily calls sleep() or swtch() Traditional systems only allowed interrupts to gain control from kernel code, and that could be prevented using calls to spl() routines In MP systems, multiple threads may execute simultaneously in the kernel
26/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Mutual Exclusion Locking: Sometimes it is necessary to perform atomic operations in the kernel e.g. updating link fields in a queue; checking condition and setting a flag Solaris thread preemption and multithreading requires a mechanism to prevent simultaneous use of related data by multiple threads Mutual exclusion (mutex) lock functions provide such a mechanism A mutex lock is associated with a collection of data to be protected A routine accessing any of the protected data should acquire mutex first When the routine is finished with the data, it releases the mutex Only one thread can hold the mutex at a time A thread blocks if it attempts to acquire a mutex held by another thread When thread releases mutex, a thread blocked on that mutex is unblocked Acquiring a mutex not currently held is very fast Releasing a mutex not currently being waited for is very fast
27/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Device Drivers and Mutex Locks: A driver usually allocates a mutex for each driver instance data structure A mutex to regulate access to a structure is usually a field in the structure In this case each device instance has its own mutex, so Threads accessing one instance of a device do not contend with threads accessing a different device instance Mutex locks also protect data in interrupt routines
28/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Waiting for an Event: Sometimes it is necessary for a thread to wait for an event to occur e.g. completion of some operation by another thread or interrupt Contrast With Traditional SVR4: Traditional UNIX systems accomplish this using sleep() and wakeup() In traditional UNIX systems, a process specifies a particular wait channel (typically the memory address of a variable associated with the event) in a call to sleep(), which blocks the processwhen the event occurs in an interrupt routine or in another process, that routine specifies the same address in a call to wakeup(), which causes all processes sleeping on the same wait channel (address) to unblock
29/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Condition Variables: Solaris condition variables provide the facility to wait for an event A condition variable may have wait and signal operations performed on it A thread can wait (block) on a condition variable using cv_wait(9F) A signal operation can unblock the first process waiting on a condition variable using cv_signal(9F) A signal operation can instead unblock all processes waiting on a condition variable using cv_broadcast(9F)
30/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Interaction of Mutex Locks and Condition Variables: When calling cv_wait(9F), driver specifies a condition variable and mutex Calling thread must be holding the mutex at the time it calls cv_wait(9F) The cv_wait(9F) routine releases the mutex while the thread is blocked When thread is unblocked, cv_wait(9F) reacquires the mutex before returning This allows other threads to acquire the mutex while the blocking thread waits for the condition to be signalled The cv_wait(9F) routine prevents race conditions by blocking the thread and releasing the mutex in one atomic operation, so that it is not possible for another thread to acquire the mutex and signal the condition variable before the first thread is suspended, resulting in the loss of the signal To prevent the race, the signalling thread must acquire the mutex before signalling the condition
31/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Contrast With Traditional SVR4: In traditional UNIX systems, the above race can only occur when the signaller is an interrupt routine It is handled by the caller using an spl() routine to block the interrupt before calling sleep() The sleep() routine atomically suspends the process and effects spl0()
32/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Other Mutex Rules: A mutex can only be acquired once by the same thread Attempts to acquire a mutex already held by that thread result in panic A thread may not release a mutex it doesnt hold Acquire multiple mutex locks in the same order on all threads Otherwise deadlock can occur If you initialize a mutex or condition variable, destroy it later
33/68
Multithreaded Kernel, Mutex and Condition Variables (cont) Locking Schemes: Mutexes and condition variables are found in most Solaris device drivers Another locking scheme is called readers/writer locks (rwlocks) rwlocks permit multiple readers but exclusive writing to a set of data But rwlocks are more expensive than mutex locks A set of semaphore routines is also available See Also: Multithreading in Writing Device Drivers Advanced Topics (locking performance, deadlocking) in Writing Device Drivers
34/68
Interrupt Handling Registering Interrupt Handlers: Most drivers implement an Interrupt Service Routine (ISR) DDI/DKI routine ddi_add_intr(9F) registers the drivers ISR ddi_add_intr(9F) is usually called from drivers attach(9E) routine ddi_remove_intr(9F) is usually called from drivers detach(9E) routine Interrupt characteristics (IRQ, priority) embedded in interrupts property Each device instance has its own interrupts property Driver specifies ISR to call when the device might have interrupted Driver also specifies argument to be passed to that ISR The argument allows the interrupt routine to distinguish between multiple device instances that may have been registered to the same ISR Argument is conventionally a pointer to the instances soft state structure A device may have more than one interrupt, though most have only one If the device has more than one interrupt, the driver adds each one in a separate call to ddi_add_intr(9F)
35/68
Interrupt Handling (cont) Interrupt Context: Interrupts are triggered asynchronously by external events Interrupts do not run in user context Process-related data is not available from within ISRs Classes of Interrupt: Two broad classes of interrupt: High-level and normal interrupts Most interrupts on the system are normal interrupts Seldom must a device be configured to a high-level interrupt High-level interrupts are handled much like traditional UNIX interrupts A high-level interrupt has no process or thread context of its own A high-level interrupt may not block for any reason A high-level ISR may only call mutex_enter(9F), the associated mutex_exit(9F), and ddi_trigger_softintr(9F)
36/68
Interrupt Handling (cont) Normal Interrupts: System allocates an interrupt thread in which to execute the ISR Existence of thread allows the ISR to block for a short time Even a normal (non-high-level) ISR should not block for very long Typically the only potentially blocking action is to call mutex_enter(9F)
37/68
Interrupt Handling (cont) Implementation Effects: While executing the ISR, the system blocks that interrupt from recurring As a side effect it may also block other interrupts from occurring (in some implementations interrupts are assigned to priority levels; blocking one level also blocks interrupts at lower levels; on x86 based systems, any other devices sharing the same IRQ will certainly be blocked) For this reason even normal ISRs should not block for very long Restrictions on Interrupt Service Routines (ISRs): ISRs should not call cv_wait(9F) or related blocking functions DDI/DKI routines should not be called in a way that they might block The exception is mutex_enter(9F) Some DDI routines accept a flag specifying whether sleeping is allowed ISRs should always specify that sleeping is not allowed See Also: CONTEXT in various Section 9 man pages
Interrupt Handling (cont) Implementation of High-Level Versus Normal Interrupts: Distinguishing characteristic is whether the thread scheduler can run while the interrupt routine is executing Interrupts that block the clock are considered high-level Interrupts that do not block the clock are considered normal ddi_intr_hilevel(9F) returns whether an interrupt is high-level or not Protection of Data in an Interrupt Service Routine: The same mutex mechanism described in Mutex Exclusion Locking is used
39/68
Interrupt Handling (cont) Action If Mutex Is Held When a Normal Interrupt Occurs: ISR runs on an interrupt thread Interrupt thread will block in mutex_enter(9F) Thread holding mutex temporarily inherits priority of interrupt thread (priority inheritance prevents the high-priority (interrupt) thread from being blocked by a low-priority thread holding the mutex) Thread holding mutex runs and eventually releases it Interrupt thread unblocks and succeeds acquiring the mutex
40/68
Interrupt Handling (cont) Action When a High-Level Interrupt Occurs: ISR runs on whatever thread happens to be on the CPU at the time It would be a gross error to block within the high-level ISR Blocking would block the random thread that was previously running Blocking would be incorrect and could cause a system deadlock System guarantees high-level ISR can acquire mutex without blocking ISR always succeeds in acquiring mutex without blocking
41/68
Interrupt Handling (cont) High-Level Mutex Implementation: The driver requests from ddi_add_intr(9F) or ddi_get_iblock_cookie(9F) an interrupt block cookie or iblock_cookie, which contains the information the system needs in order to block the associated interrupt The iblock_cookie must be passed to mutex_init(9F) If a mutex has been initialized with a high-level iblock_cookie, then subsequent calls to mutex_enter to acquire that mutex cause the system to block the associated interrupt from the CPU holding the mutex Other interrupts may also be blocked as a side effect A mutex initialized with a high-level iblock_cookie is a high-level mutex Any thread holding a high-level mutex may not block for any reason Any thread holding a high-level mutex may only call mutex_exit(9F) or ddi_trigger_softintr(9F) A high-level mutex should only be held by any thread for a very short time
42/68
Interrupt Handling (cont) High-Level Interrupt / Mutex Action on a Uniprocessor System: mutex_enter(9F) blocks the interrupt from occurring while mutex is held If interrupt occurs, mutex is therefore guaranteed not currently held Thus system guarantees mutex is not held at the time interrupt occurs ISR is therefore free to acquire mutex without contention High-Level Interrupt / Mutex Action on a Multiprocessor System: mutex_enter(9F) blocks interrupt from occurring on CPU where mutex held If interrupt occurs, it must be on a different CPU ISR calls mutex_enter(9F) mutex_enter(9F) must spin (wait in a loop without giving up control) until mutex is freed by thread holding it (on the other CPU) A high-level mutex should only be held by any thread for a very short time
43/68
Interrupt Handling (cont) High-Level ISR Rule: Any mutex to be acquired in a high-level interrupt routine must be initialized with the iblock_cookie associated with that interrupt Otherwise the interrupt routine could be entered while the mutex is held On a uniprocessor system this would be a detectable deadlock condition, and the system would panic
44/68
Interrupt Handling (cont) Claiming the Interrupt: ISR may be called, even if device did not generate interrupt On x86 based systems, a shared IRQ is one way this could happen On x86 based systems, all ISRs registered to an IRQ are called at least once when that IRQ interrupts ISR must ascertain whether or not its device requires service ISR must return DDI_INTR_CLAIMED or DDI_INTR_UNCLAIMED Failure to properly return DDI_INTR_UNCLAIMED when the device is not causing the interrupt can cause system hangs under certain circumstances When ISR May Be Called: ISR may be called at any time after driver calls ddi_add_intr(9F) A shared IRQ can cause an ISR call even before device is initialized Driver must ensure all data structures, mutexes, etc. required by the ISR are initialized before calling ddi_add_intr(9F)
45/68
Interrupt Handling (cont) Soft Interrupts: Soft interrupt system works similarly to hardware interrupt system But soft interrupts are triggered by a DDI/DKI call rather than a device Soft interrupts are not high-level, and run on interrupt threads A high-level interrupt routine might delegate most of its processing work to a soft interrupt routine it triggers See Also: ddi_add_softintr(9F), ddi_trigger_softintr(9F) See Also: Interrupt Handlers in Writing Device Drivers
46/68
Accessing Devices Accessing Device Memory and Registers: SVR4 physmap() function is not supported Solaris instead provides an architecture independent device register model This model works on all supported buses and architectures This model supports both memory-mapped I/O and x86 port I/O A driver need not know whether it is using memory-mapped or port I/O The same device may use MMIO on one machine and port I/O on another! Driver does not know the specific physical address of its device Device address information is embedded in dev_info node reg property Multiple sets of device addresses may be specified for a device DDI/DKI functions are used to access device registers Access routines can be configured to do automatic byte swapping if required See Also: ddi_regs_map_setup(9F), ddi_get8(9F), ddi_put8(9F), ddi_device_acc_attr(9S)
DMA Services Direct Memory Access (DMA): Solaris provides a software framework for driver access to DMA DMA allows some devices to directly read and write system memory This frees the CPU from transferring each piece of data Solaris DMA services are somewhat different from SVR4 Generic DMA functions break a DMA object into DMA-able pieces DMA engine functions deal with motherboard DMA controller programming
48/68
DMA Services (cont) Generic DMA Functions: A DMA operation may be performed to or from an object in memory A DMA handle is an opaque data type representing such an object Generic DMA functions establish bindings of handles to objects Other DMA functions extract DMA cookies from information embedded in a DMA handle A DMA cookie represents a contiguous portion of a DMA object that is entirely addressable by the device On x86 based systems, a DMA cookie comprises a physical address and a length
49/68
DMA Services (cont) DMA Break-up: On x86 based systems, DMA addresses are physical addresses An object may span multiple non-contiguous pages in physical memory Such an object cannot be represented by a single DMA cookie Such an object must be broken into multiple pieces, each represented by a separate DMA cookie A DMA attribute structure associated with the handle is used to determine the capabilities of the DMA device, to determine how the object must be broken up The attribute structure is also for DMA devices that may not be able to access all of physical memory If system or device resources are insufficient to allow DMA access to the entire object at one time, the system will break the object up into multiple DMA windows, each of which is represented by one or more DMA cookies
50/68
DMA Mapping
DMA object (contiguous virtual addresses) DMA handle (represents DMA object) DMA window DMA cookies DMA window
Physical address and length
Physical Memory Pages

DMA Services (cont) Non Bus Master DMA: Most modern DMA devices are bus masters Some older devices made use of a DMA controller on the system board These devices use the DMA Engine DDI/DKI functions to program DMA See Also: ddi_dmae(9F) Bus Masters: Most modern DMA devices are bus masters Drivers of bus masters use generic DMA functions to extract cookies Drivers extract the physaddr and count and program the device directly See Also: DMA in Writing Device Drivers ddi_dma_alloc_handle(9F), ddi_dma_cookie(9S), ddi_dma_attr(9S), ddi_dma_buf_bind_handle(9F), ddi_dma_addr_bind_handle(9F), ddi_dma_nextcookie(9F), ddi_dma_getwin(9F), ddi_dma_sync(9F)
SCSI Drivers: SCSA Small Computer Systems Interface (SCSI): Solaris supports SCSI for disks, tapes, and other devices SCSI defines a protocol to communicate between initiators and targets Initiators request service on a SCSI bus Targets are devices on the SCSI bus Host Bus Adapter (HBA): A host bus adapter is a SCSI bus controller Solaris communicates over the SCSI bus using the HBA The HBA sends commands to target devices on the SCSI bus The target devices perform the requested operations The responses are communicated back to Solaris via the HBA
53/68
SCSI Drivers: SCSA (cont) Sun Common SCSI Architecture (SCSA): Divides SCSI drivers into two major classes: SCSI HBA drivers control the HBA SCSI bus controller SCSI target drivers control the actual device (e.g. disk, tape) that is attached to the SCSI bus Makes target drivers independent of the HBA devices
54/68
Relationship Between SCSI and SCSA
Target Driver
SCSA
HBA Driver I/O
DISK
SCSI
HBA
55/68
SCSI Drivers: SCSA (cont) SCSI Target Drivers: Know how to control the particular target device on the SCSI bus using SCSI commands Are independent of the HBA being used Are not concerned with low-level SCSI bus protocol Construct target device commands and pass them to an HBA driver for transport to the target device SCSI HBA Drivers: Know how to transport the target drivers SCSI commands through the HBA and across the SCSI bus to the target device Can transport SCSI commands on behalf of different target drivers Are not concerned with the programming of the specific target device
56/68
SCSI Drivers: SCSA (cont) Architecture: The SCSI target is a child node of the HBA node in the dev info tree The SCSA functions in the DDI/DKI route SCSI transport requests from a SCSI target driver to the parent SCSI HBA driver Multichannel SCSI controllers are represented in the dev info tree by a bus nexus node and a child HBA node for each channel See Also: Writing Device Drivers SCSI Target Drivers SCSI Host Bus Adapter Drivers Advanced Topics Sun Disk Device Drivers SCSA Global Data Definitions (for debugging) SCSA Tagged Queuing
57/68
Multichannel SCSI Device Info Tree Example
root nexus node EISA bus nexus node mscsi bus nexus node esa HBA nexus node sd0 sd1 sd2 esa HBA nexus node sd0 sd1 sd2
58/68
Network Drivers: DLPI and GLD STREAMS / DLPI Network I/O: Solaris network drivers are cloning STREAMS drivers Network protocol modules (e.g. TCP, IP) are STREAMS modules Interface between driver and protocol stack is DLPI over STREAMS Data Link Provider Interface (DLPI) is industry standard Network drivers must conform to: DDI/DKI specification STREAMS specification DLPI specification
59/68
Network Drivers: DLPI and GLD (cont) Generic LAN Driver (GLD): Solaris (x86 Platform Edition) implements GLD module GLD provides a device-independent upper layer for a network driver Device-dependent lower-layer drivers can interface with GLD GLD handles most of the DLPI and STREAMS protocols Lower layer can focus on hardware specific portion of the driver GLD implements both Style 1 and Style 2 DLPI opens
60/68
Network Drivers: DLPI and GLD (cont) Device-Dependent GLD Drivers: Interface between GLD and device-dependent driver is at the MAC layer Drivers share a macinfo data structure with GLD for each device instance Drivers must implement a small set of MAC-layer functions: reset(), start(), stop() to start and stop the card saddr(), sdmulti(), prom() to program addresses and promiscuous mode gstat() to update kernel statistics send() to send a packet intr() to handle a receive interrupt
61/68
Network Drivers: DLPI and GLD (cont) GLD Service Routines: register/unregister functions for drivers to link with GLD gld_recv() for a driver interrupt routine to report a received packet GLD Not SunDDI Standard: GLD is not yet a committed Solaris interface and is evolving However, all x86 Ethernet drivers are GLD drivers Projected changes are intended but not guaranteed upward compatible See Also: Data Link Provider Interface Specification, UNIX International Generic LAN Driver (GLD) template and documentation
62/68
Realmode (x86 Boot) Drivers Solaris Boot System: Must be capable of booting off CD-ROM (via HBA) and network devices SPARC implements Open Boot Prom (OBP) on each device to allow booting x86 based systems boot using ROM BIOS Unfortunately ROM BIOS does not typically support net/CD-ROM booting Solaris therefore requires auxiliary boot drivers for these devices Solaris (x86 Platform Edition) Boot Drivers: These drivers operate as an extension to the system BIOS These drivers are called BIOS Extension Files (BEFs) These drivers run in real-address mode of x86 based systems These drivers are often called realmode drivers BEF drivers interface to the rest of the boot system via defined BEF interface See Also: External BEF Interface Specification for x86 Realmode DDI (white paper) Note: Versions of the x86 boot system prior to Solaris 2.6 operate differently
Realmode (x86 Boot) Drivers (cont) Detection of Devices: Boot system automatically detects all self-identifying devices: PCI devices EISA devices MCA (Micro Channel Architecture) devices Plug and Play ISA devices Non-Plug and Play ISA devices are referred to as legacy devices Boot system cannot detect legacy devices Boot system relies on realmode drivers to detect legacy devices
64/68
Realmode (x86 Boot) Drivers (cont) Functions of Boot System: Identify non-legacy devices present in the system Configure system devices Construct hardware dev info tree to pass to kernel (Each dev_info node embeds all the devices configuration information) Allow user selection of boot device Use realmode drivers to read in the required kernel modules Transfer control to the kernel Functions of Realmode Drivers: Identify legacy devices present in the system Device configuration Kernel boot I/O support
65/68
Realmode (x86 Boot) Drivers (cont) Three General Types of Boot Drivers: Network drivers SCSI HBA drivers Generic Underneath, all of these use the same BEF interface Network and SCSI driver libraries are available Libraries significantly simplify driver implementation Kit for Building Realmode Drivers: Realmode driver documentation and sample code exist Separate documentation exists for network, SCSI, and generic drivers Libraries and documentation aimed specifically at network and SCSI drivers Kit available on DDK website See Also: Realmode Drivers (white paper)
General Advice Read intro(9), intro(9E), intro(9F), intro(9S) Write Architecture Independent Drivers: Use only documented DDI/DKI functions Dont make assumptions about the underlying implementation Use symbolic data types e.g. use major_t and minor_t, not short, for device numbers Run DDI Compliance Tool (DDICT) on your drivers Test drivers on different architectures Watch out for 64-bit issues Write Configuration Independent Drivers: Addresses, IRQ numbers, etc. should never be compiled in Configuration information should be passed in via properties
67/68
General Advice (cont) Deallocate Resources You Allocated: Anything allocated in _init(9E) should be deallocated in _fini(9E) Normally things allocated in attach(9E) should be deallocated in detach(9E) Normally things allocated in open(9E) should be deallocated in close(9E) probe(9E) must deallocate anything it allocates A call to mutex_destroy(9F) should match a call to mutex_init(9F) A call to cv_destroy(9F) should match a call to cv_init(9F) A call to ddi_remove_intr(9F) should match a call to ddi_add_intr(9F) Memory that was allocated or mapped should be freed or unmapped etc... Failure to properly deallocate resources results in memory leaks Memory leaks can result in degraded system performance over time
68/68

Solaris Driver Models.1298

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Solaris Driver Models.1298

Încărcat de

Drepturi de autor:

Formate disponibile

Solaris (x86 Platform Edition) Device Driver Writers Orientation

January 1998 (Revised 12/98)

Copyright 1998 Sun Microsystems, Inc. All rights reserved.

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Sample Device Information Tree

pseudo nexus node

PCI bus nexus node

EISA bus nexus node

PCI-PCI nexus node

adp HBA nexus node

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Sample Device File System Hierarchy (abridged)

/dev dsk rdsk

pci10b8,2003@4:dnet0 pci10b8,2003@5:dnet1 pci8086,1229@6:iprb0

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers

Solaris (x86 Platform Edition) Device Driver Writers Orientation

A Driver Model Overview for UNIX Kernel Programmers