USB Debugging and Profiling

Techniques
Kishon Vijay Abraham I and Basak Partha

Agenda
Introduction
USB Generic Linux System Architecture
USB Mass Storage Architecture
Challenges in debugging
USB Debugging Techniques (sysfs, usbmon, dynamic debug interface,
tracepoint, protocol analyzer)
Gadget Zero
Other profiling tools

Introduction
Widespread use of USB in embedded space
HOST MODE:
To connect Ethernet/Hub
To connect Modem
To connect mass storage devices
DEVICE MODE
Acts as mass storage device
USB Speakers
USB serial device
USB webcam

Introduction
Ease of use
Hot pluggable
Speed
Reliability
Power devices from the bus
Bus expansion using hub
USB On-The-Go

USB Generic Linux System Architecture

Application
Input, Sound, FS..
Subsystem

DEVICE

libusb

Gadget Driver

VFS

UDC

Class Driver

USB Device Stack

HCD/usbcore

Firmware Driver

Firmware Driver

Device Controller
ulpi/utmi
PHY
Device

HOST

HSIC/TLL
Host Controller
ulpi/utmi
PHY
Host

SWcomp
HWcomp

USB Mass Storage Architecture

Block Layer
Backing Store

VFS

VFS

Mass Storage
Function Driver

SCSI
Block Layer

UDC

Mass storage
Class driver

USB Device Stack

HCD/usbcore

Firmware Driver

Firmware Driver

Device Controller
ulpi/utmi

HSIC
/TLL

File System

SWcomp

Host Controller
ulpi/utmi

PHY

PHY

Device

Host

HWcomp

Challenges in debugging
Data on the bus is encoded
Timing issues
Out-of spec signaling errors
Protocol errors
Multiple layers makes it difficult to identify where exactly the problem
originates
Too many formatted prints lead to skewed results

USB debugging
Using linux kernel facilities
sysfs/debugfs
usbmon
Dynamic debug interface
Tracepoints
Using debug tools and Analyzers
Elisys/Lecroy/total Phase analyzer tools
ETM

Sysfs Entry in host

ls/sys/bus/usb/devices/
10:1.01111.111.1:1.011:1.0usb1

The names that begin with "usb" refer to USB controllers

The devices are named by a scheme
bus-port.port.port
The interfaces are indicated by suffixes having this form
:config.interface
All information about the device will be under this entry

TheDocumentationinformationforthisentrycanbeobtainedfrom
http://www.linuxusb.org/FAQ.html#i6

Debugfs Entry in host

cat/sys/kernel/debug/usb/devices

T:Bus=01Lev=00Prnt=00Port=00Cnt=00Dev#=1Spd=480MxCh=3
B:Alloc=0/800us(0%),#Int=1,#Iso=0
D:Ver=2.00Cls=09(hub)Sub=00Prot=00MxPS=64#Cfgs=1
P:Vendor=1d6bProdID=0002Rev=3.06
S:Manufacturer=Linux3.7.0rc2next2012102600003g72580a5ehci_hcd
S:Product=OMAPEHCIHostController
S:SerialNumber=ehciomap.0
C:*#Ifs=1Cfg#=1Atr=e0MxPwr=0mA
I:*If#=0Alt=0#EPs=1Cls=09(hub)Sub=00Prot=00Driver=hub
E:Ad=81(I)Atr=03(Int.)MxPS=4Ivl=256ms
T:Bus=01Lev=01Prnt=01Port=00Cnt=01Dev#=2Spd=480MxCh=5
D:Ver=2.00Cls=09(hub)Sub=00Prot=02MxPS=64#Cfgs=1
P:Vendor=0424ProdID=9514Rev=2.00
C:*#Ifs=1Cfg#=1Atr=e0MxPwr=2mA
I:If#=0Alt=0#EPs=1Cls=09(hub)Sub=00Prot=01Driver=hub
E:Ad=81(I)Atr=03(Int.)MxPS=1Ivl=256ms
I:*If#=0Alt=1#EPs=1Cls=09(hub)Sub=00Prot=02Driver=hub
E:Ad=81(I)Atr=03(Int.)MxPS=1Ivl=256ms
TheDocumentationinformationforthisentrycanbeobtainedfrom
Documentation/usb/proc_usb_info.txt

10

Debugfs Entry in host contd..

T:Bus=01Lev=02Prnt=02Port=00Cnt=01Dev#=3Spd=480MxCh=0
D:Ver=2.00Cls=ff(vend.)Sub=00Prot=01MxPS=64#Cfgs=1
P:Vendor=0424ProdID=ec00Rev=2.00
C:*#Ifs=1Cfg#=1Atr=e0MxPwr=2mA
I:*If#=0Alt=0#EPs=3Cls=ff(vend.)Sub=00Prot=ffDriver=smsc95xx
E:Ad=81(I)Atr=02(Bulk)MxPS=512Ivl=0ms
E:Ad=02(O)Atr=02(Bulk)MxPS=512Ivl=0ms
E:Ad=83(I)Atr=03(Int.)MxPS=16Ivl=1ms

TheDocumentationinformationforthisentrycanbeobtainedfrom
Documentation/usb/proc_usb_info.txt

11

Debugfs (in device controller)

drivers/usb/dwc3/debugfs.c
Show the entire register dump
Change the port mode (device, host, OTG)
Force the controller to work at a particular speed (super, high..)

12

Debug example using debugfs

PROBLEM STATEMENT
Added PM support for dwc3 and realized once the system goes to off
mode and comes back the device is not enumerating. Adding some
debug prints showed the we are not getting any interrupts in dwc3.

DWC3WRAPPER

DWC3CORE

interupt

HWWrite

MPU

SWRead
EVENTBUFFER

Configuration Diagram
13

Debug example using debugfs

Took the register dump before the system goes to suspend and after
resuming using cat /debug/dwc3.0/regdump

WorkingRegdump
(BeforeSuspend)

NonWorkingRegdump
(AfterResume)

After going to OFF mode and coming back, DEVTEN register lost the
contents.
DEVTEN controls the generation of device specific events.
The fix is to restore the contents of DEVTEN after coming back from
OFF mode.
14

usbmon
Facility in kernel to be used to collect URB traces
USB monitoring facilities for Linux consists of a kernel part and user
part
Reports requests made by peripheral-specific drivers to Host
Controller Drivers (HCD)
Requires a reliable working HCD
Has a "text" and "binary" API

15

usbmon Architecture

binary

Character device
(for each host
controller)

usbmon main
urb_submit/
urb_complete/
add/remove bus
HCD

/dev/usbmonX

text

Debugfs entry
(for each host
controller)

/sys/kernel/debug/
/usb/usbmon/Xu

tcpdump
vusb-analyzer

wireshark
Linuxkernelcomp
Userspacetools

16

usbmon ASCII capture

mounttdebugfsnone_debugs/sys/kernel/debug
modprobeusbmon
ls/sys/kernel/debug/usb/usbmon/
0s0u1s1t1u
cat/sys/kernel/debug/usbmon/1u>usbmon.mon
./vusbanalyzerusbmon.mon

17

usbmon binary capture

modprobeusbmon
ls/dev/usbmon<TAB>
usbmon0usbmon1
sudotcpdumpiusbmon1wusbmon.pcap&
./wiresharkusbmon.pcap

18

Decoding usbmon captures

Usbmontrace
d2263780469874560SCi:3:003:0s80060100
000000088<
d2263780469875939CCi:3:003:008=12010002
ef020140
URB
Address

Time
stamp

Urb
Event

d2263780

469874560

wValue

wIndex

wLength

URB
Address

Time
stamp

d2263780

469875939

Urb
Event
C

Transfer
Bus
&
Number
Direction
Ci

Transfer
Bus
&
Number
Direction
Ci

Device
Number
003

Device
Number
003

Endpoint
URB
Number
Status
0

Endpoint
URB
Number
Status
0

bmRequest bRequest
type

80

Length

06

data

= 12010002
ef020140
19

Debug examples using usbmon

PROBLEM STATEMENT1
usb stick works fine under older kernels (tested on 2.6.34 and
2.6.37) but stopped working for 3.0 and 3.1
Obtain usbmon traces for working and non working setup
Filter the usbmon traces for only the required devices

catusb_nw.mon|grep2:011>usb_nw_dev.mon
catusb_w.mon|grep1:019>usb_w_dev.mon
./vusbanalyzerusb_nw_dev.monusb_w_dev.mon
The non working log has some additional commands compared to
working log
20

Debug examples using usbmon contd.

NONWORKINGLOG

From the diff, there are additional commands in the non-working case
like the UDISK shown in the command
Then the URB return status shows broken PIPE error.
The host sends CLEAR ENDPOINT HALT and this happens again and
again
So it's concluded some user space program send bogus commands
causing the issue.

21

Debug examples using usbmon

PROBLEM STATEMENT2
Webcam does not work when connected to USB 2 port but works on
USB 3
Obtain usbmon traces for working and non working setup
Filter the usbmon traces for only the required devices

catusb2.mon|grep1:006>usb2_nw.mon
catusb3.mon|grep3:003>usb3_w.mon
./vusbanalyzerusb2_nw.monusb3_w.mon

22

Debug examples using usbmon contd.

NONWorkingusbmonlog

Workingusbmonlog

Both traces show that the webcam stopped being used for a short time
and was suspended
when it was resumed again, it worked okay in USB3 but did not work in
USB-2
The workaround was to disable auto-suspend (and debug the device)

23

Dynamic Debug Interface

Extended version of printk
Use dev_dbg and dev_vdbg to control debug messages
Enabled using DDEBUG and DVERBOSE_DEBUG compiler options
If CONFIG_DYNAMIC_DEBUG is NOT set, everything under dev_dbg
turns to normal printk
If CONFIG_DYNAMIC_DEBUG is set
a new debugfs entry /sys/kernel/debug/dynamic_debug/control
gets created
Writing to this file will enable or disable specific debugging
functions
e.g., echo file gadget.c line 269 +p > .../dynamic_debug/control
24

Debug Example using Dynamic Debug

Interface
GOOD: out transfer on dwc3 device mode:
[274.694458]=>>>queingrequested7cdea0toep2outbulklength512
[274.694458]dwc3dwc3.0:ep2outbulk:reqed7cdea0dmaac9f0000
length512last
[274.713378]dwc3dwc3.0:ep2outbulk:cmd'StartTransfer'params
00000000ad5ab11000000000
[274.713378]dwc3dwc3.0:CommandComplete>0
[274.726989]dwc3dwc3.0:ep2outbulk:TransferComplete
[274.732482]=>>requested7cdea0fromep2outbulkcompleted16/512
===>0

BAD: out transfer on dwc3 device mode:

[217.927062]=>>>queingrequestecb1ecc0toep1outbulklength24
[219.333374]dwc3dwc3.0:ep1outbulk:reqecb1ecc0dmaaca1b000
length24last
[219.333374]dwc3dwc3.0:ep1outbulk:cmd'StartTransfer'params
00000000ad55e00000000000
[219.349822]dwc3dwc3.0:CommandComplete>0
[219.360168]dwc3dwc3.0:ep1outbulk:endpointbusy
25

Debug Example using Dynamic Debug

Interface
The log shows OUT transfers of 512bytes is getting succeeded but
OUT transfer of size 24 is failing
Confirms maxpacket aligned transfers for out direction are succeeded
and there is a problem with short packet transfer
The problem is that the controller requires OUT transfers to be aligned
on wMaxPacketSize, even if we *know* the correct size
So the fix is done in the gadget driver to give maxpacket aligned buffers
to the controller.

26

Protocol Analyzer
A protocol analyzer decodes, filters, and displays USB data
Some analyzers also have an exerciser along with it that can generate
data on the bus
PROTOCOLANALYZER
SWPC

PROTOCOLANALYZER

UsbDevice

UsbHost

27

Debug Example using Protocol Analyzer

USB tethering is not working on omap5
Captured CATC trace in omap4(working) and in omap5(non-working)

NonWorking(OMAP5)

Working(OMAP4)

28

Debug Example using Protocol Analyzer

The host sends a QUERY message for
OID_802_3_PERMANENT_ADDRESS
In the working case the device responds with a QUERY_COMPLT
message with some DATA in it.
In the non working case the device responds with a QUERY_COMPLT
without any DATA in it.
So the host sends a HALT message
The issue was around copying data from the bounce buffer to the
gadget layer.

29

tracepoint
Used to record data at a specific point in the kernel for later retrieval
Light weight hooks added to the kernel
Two types: static tracepoint and dynamic tracepoint
can be used by a number of tools for kernel debugging and
performance problem diagnosis
They have zero overhead when disabled and minimal overhead when
enabled

30

Defining Static tracepoint

Macros existing to create a static tracepoint include
TRACE_EVENT()
TP_PROTO()
TP_ARGS()
TP_STRUCT__entry()
TP_fast_assign()
TP_printk()
The TRACE_EVENT() is created by
TRACE_EVENT(name, proto, args, struct, assign, print)
33

Defining Static tracepoint

Macros existing to create a static tracepoint to do more than printing
DECLARE_TRACE()
DECLARE_TRACE_NOARGS()
DEFINE_TRACE()
To hook into the tracepoints
register_trace_[tracepoint_func]()
To unregister the hook
unregister_trace_[tracepoint_func]()

34

Tracepoint Sample
Patch to log usb_request from gadget layer

http://gitorious.org/linux-usb/linux-usb/commit/a7e7fb69808c
Patch to log every read/write in dwc3

http://comments.gmane.org/gmane.linux.usb.general/64821
([PATCH] usb: dwc3: add trace support)

35

Gadget Zero
Obtain performance characteristics
Two configuration device
sinks and sources bulk data
loops back a configurable number of transfers
A kernel driver (host), drivers/usb/misc/usbtest.c, where the
actual test cases live
Userland software to call that driver, such as testusb.c and
test.sh
Can't be used for class or vendor-specific functionality
36

Sample test.sh output

**Controltestcases:
test9:ch9postconfig
/dev/bus/usb/001/027test9,64.183046secs
test10:controlqueueing
/dev/bus/usb/001/027test10,11.738630secs
test14:controlwrites
/dev/bus/usb/001/027test14,5.432296secs
assumingsinksrcconfiguration
**HostWrite(OUT)testcases:
TestCaseNo1,3,5,7are
test1:5000transfers,samesize
Writetestcases
/dev/bus/usb/001/027test1,1.624243secs
TestCaseNo2,4,6,8are
test3:5000transfers,variable/shortsize
readtestcases
/dev/bus/usb/001/027test3,1.090523secs
test5:2000scatterlists,samesizeentries
/dev/bus/usb/001/027test5,13.404251secs
test7a:2000scatterlists,variablesize/shortentries
/dev/bus/usb/001/027test7,8.839292secs
37

Sample test.sh output contd.

test7b:2000scatterlists,variablesize/biggerentries
/dev/bus/usb/001/027test7,6.701336secs
test7c:2000scatterlists,variablesize/microentries
/dev/bus/usb/001/027test7,4.173024secs
**HostRead(IN)testcases:
test2:5000transfers,samesize
/dev/bus/usb/001/027test2,0.803351secs
test4:5000transfers,variablesize
/dev/bus/usb/001/027test4,0.784580secs
test6:2000scatterlists,samesizeentries
/dev/bus/usb/001/027test6,5.442718secs
test8:2000scatterlists,variablesizeentries
/dev/bus/usb/001/027test8,3.621307secs

38

Other Profiling Tools

Iozone
Can be used to profile mass-storage devices
Iozone can perform 13 types of test: Read, write, reread, re-write, read backwards, read strided, fread etc.,
Sudo iozone -Rab massstorage.xls -i0 -i1 -e -f /dev/sdb
FFSB
dd
Copy a file, converting and formatting according to the
operands
dd if=/dev/zero of=/dev/sdb bs=128 count=1024

39

References
Documentation/usb/*
Documentation/trace/*
Drivers/usb/*
lwn.net
Bootstrap Yourself with Linux-USB Stack
linux-usb list
http://vusb-analyzer.sourceforge.net/

40

THANK YOU
For Queries and Feedback
kishon@ti.com, kishonvijayabraham@gmail.com
partha_basak2000@yahoo.com

41