Troubleshooting Terminal Server

Licensing Issues (Part 1)

Once licensing has been properly set up in the environment, there are few licensing-
related problems you are likely to encounter. In this four-part series, we will look at the
most popular licensing-related issues and how to deal with them. Parts one through
three will focus on popular server-related licensing issues while part four will look at
common issues from the client side.

Verifying License Server Installation

The most popular issues are the result of incorrect setup or configuration, or failure to
fully understand the licensing process and requirements. Believe it or not, the cause of
many licensing issues comes down to simply overlooking the fact that a license server is
necessary. Because terminal servers have an automatic 180-day grace period, people
sometimes forget to install and activate a license server. Once the grace period runs out,
the terminal servers will no longer accept connections. It is usually at this point that
someone realizes they neglected to install a license server.

If you know a license server is not yet installed, it’s easy to do so. Any Windows Server
2003 instance can be a license server. The Terminal Server Licensing component is
installed through Add/Remove programs. When installing a license server,
discoverability is critical, so check out my articles on License Server Discovery and
License Server High Availability.

Common event log entries when no license server has been installed are below:

Event Source Description

ID
1008 TermService The terminal services licensing grace period has expired and
the service has not registered with a license server.
1009 TermService The terminal services licensing grace period is about to expire
on <date> and the service has not registered with a license
server with installed licenses.

These events indicate that the terminal server has not yet registered with a license server
and the grace period will be expiring soon (ID 1009) or has already expired (ID 1008).

Be sure a license server has been installed in your environment and check that the
Terminal Server Licensing service is started and set to Automatic. Also make sure there
is no network connectivity or routing issues between the terminal servers and the license
server. If the license server is located on the opposite side of a firewall from the
terminal servers, be sure standard RPC communication is open between the two.

Event ID 1009 may also be observed when Windows Server 2003 terminal servers are
installed in per-user mode from the start, resulting in the LicensingGracePeriodEnded
registry key not being set. This key suppresses the event log entries from being logged.
The fix is to add the LicensingGracePeriodEnded key on the license server.

1. Locate the following registry key on the Windows Server 2003 terminal server:

HKLM\System\CurrentControlSet\Services\TermService\Parameters
2. Add a new DWORD value named LicensingGracePeriodEnded with a NULL
(0) data value.

When terminal servers are in per-user mode, the grace period has no bearing on
connectivity to the event log entries are innocuous. The above steps just suppress the
messages.

One last note on verifying license server installations - be sure you have the correct
version of a license server installed to support the terminal servers. Windows Server
2003 terminal servers require a Windows Server 2003 license server; they cannot
leverage a Windows 2000 license server.

Terminal Servers Cannot Locate a License Server

Probably the most common licensing-related issue is the failure to discover the license
server. My article on License Server Discovery covers this topic in depth, including
some troubleshooting steps, so I’ll just review the key points.

When a license server cannot be contacted the following messages may be logged to the
Windows event log:

Event Source Description

ID
1000 TermService Unable to acquire a license for user <user>, domain
<domain>
1004 TermService The terminal server cannot issue a client license.
1010 TermService The terminal server could not locate a license server

Event IDs 1000 and 1004 are logged when the terminal server attempts to allocate a
CAL to a device upon connection, but cannot locate a license server or no licenses are
available. Event ID 1010 is typically seen with 1004 when no license server can be
contacted. The most common root cause for these event log entries is a failure to
discover the license server.

LSVIEW
License Servers must be discovered by Terminal Servers. Once located, a digital
certificate exchange takes place so the terminal server can check CALs allocated to
clients by the license server. Discoverability is critical. The Windows Server 2003
Resource Kit utility LSVIEW is an administrator’s view into the discovery process and
can log key information to a text file to aid in troubleshooting. When launched on a
terminal server, this utility will display any license servers that are discovered, and list
the type of license server (Domain or Enterprise), as seen in Figure 1.
Figure 1: LSVIEW

If no servers are displayed in LSVIEW, either the license servers are offline, or they
cannot be discovered by this terminal server.

Enterprise License Servers

License Servers are discovered a few different ways, depending on which mode the
license server was installed. When an Enterprise License Server is installed, an Active
Directory registration is created under the configuration container in the site to which
the license server host belongs. When the license server is installed, the administrator
performing the installation must have the appropriate rights in AD to create the
object. If not, the registration may not complete and the terminal server may not be able
to discover the license server.

To verify the entry was successfully created, use ADSIEDIT from the Windows Server
2003 Support Tools to check for its existence in the configuration container:

LDAP://CN=TS-Enterprise-License-
Server,CN=<sitename>,CN=sites,CN=configuration, DC=<domainname>,DC=com

Under the properties of this object, look for the siteServers attribute and check the
values. This identifies the registered license server(s) for the Active Directory site.

Recommendations for Discoverability

For the full details on license server discovery, check out my articles on the subject, also
available on this site. To summarize, follow these suggestions to ensure your license
servers are easily discovered:

• Install Enterprise-mode license servers in the same Active Directory domain or

site as your terminal servers to ensure they are always discovered.
• If an Enterprise-mode license server is not possible in your environment, load
the license service on a domain controller where it is more likely to be
discovered.
• An easy way to ensure the license server is discovered is to leverage the
LicenseServers registry key located here on each terminal server:

HKLM\System\CurrentControlSet\Services\TermService\Parameters\LicenseSer
vers
 This overrides the discovery process and will ensure the license server is
discovered.
• Install more than one activated license server in your environment to ensure
redundancy.

Finally, some additional things to check here are basic network and RPC connectivity,
and be sure the license server is online and the license service is started.

License Server Security Group GPO

Windows Server 2003 introduced the ability to control which terminal servers can
access a particular license server through the use of a new group policy setting –
License Server Security Group. If you are leveraging this policy, be sure that the
terminal server in question has been added to the policy so it can access the licensing
services.

Also check to see if the terminal server in question is the license server. There is a
known issue in the new functionality where if the server that requests the license is the
same machine as the license server, the operation fails. A hotfix is available to correct
this issue and can be obtained from Microsoft Product Support Services. See Microsoft
KB 834803 for more information.

Conclusion
In this segment, we looked at a few common problems that can lead to licensing-related
issues. We’ll continue this discussion in parts two and three where we will cover
additional server-related issues and conclude with possible client-side issues in part
four.

Troubleshooting Terminal Server

Licensing Issues (Part 2)
No Licenses Available on License Servers
Another common issue is simply running out of licenses. This is most often due to
inadequately planning for the proper number of connections to the farm.

Available licenses can be observed in the Terminal Server Licensing program, so that’s
the first place to look. The different pools, or types, of licenses will be displayed along
with the total number of licenses in each pool, the number allocated and how many
remain. If you double-click on a particular pool, more details will be displayed
including the machine name to which it was issued, when it was issued and when the
CAL expires.
Two event IDs common when the license server can be reached but is out of licenses
include:

Event Source Description

ID
1000 TermService Unable to acquire a license for user <user>, domain
<domain>
1004 TermService The terminal server cannot issue a client license.
1011 TermService The terminal services client <clientname> has been
disconnected because its temporary license has expired.

Event IDs 1000 and 1004 have been mentioned previously; however event ID 1011 may
be logged if the client has a temporary CAL and it has expired. If this occurrs, then
licensing counts are not sufficient to handle the actual number of connections. To
resolve this, purchase and add more device CALs.

Using LSREPORT
LSREPORT, another great utility in the resource kit, can display detailed information
about the issue status of the CALs in the licensing database. The following command
line arguments can be used with LSREPORT:

/F filename
Output file name

/D [start] [end]
Only licenses from start to end

/T
Only temporary licenses

/W
Include HardwareID in report

serverlist
List of license servers

Usage: LSREPORT /W /F c:\report.txt server1 server2

Figure 2: LSREPORT

In some circumstances, it’s possible that a client may lease more than one CAL thereby
prematurely consuming all the CALs in the database, as seen in the above
example. This is typical when the MSLicensing key has been deleted on the client. In
most cases, a new client can connect using a temporary CAL until the allocated (but
unused) CAL is returned to the pool through normal attrition. More detailed
information on the MSLicensing key will be covered in part four.

Mixed-Licensing Environments
There is a known issue with the licensing service in Windows Server 2003 that affects
mixed-licensing environments - those that have both Windows 2000 and Windows
Server 2003 license servers. Normally, Windows 2000 license servers would detect
when per-device CALs were available on Windows Server 2003-based license servers
so they can forward unfulfilled requests to those license servers. However, a bug in the
Windows Server 2003 version of LServer.exe prevents this from occurring. This could
lead to a failure to accept the connection and may appear to be intermittent. Typically,
the following circumstances accompany this issue:

• Your environment consists of a mix of both Windows 2000 and Windows

Server 2003 license servers.
• The Windows 2000 Server-based license servers have no available permanent
CALs.
• The Windows Server 2003-based license servers do have available permanent
CALs.
• The connecting client is running an older version of Windows, such as Microsoft
Windows 98.

This issue has been fixed in Windows Server 2003 SP2, but a hotfix is available from
Microsoft to address the problem on Windows Server 2003 SP1 and earlier; more
details are available in Microsoft KB Q911288. Contact Microsoft Product Support
Services to obtain the hotfix.

Terminal Servers in the Wrong Licensing Mode

This is another common issue. It used to be much more popular before Windows Server
2003 SP1 as there was a bug in the Add/Remove programs process. If you added a
Windows component at any time after installing Terminal Services, the registry value
PolicyAcOn would get set to an invalid value of 0 and would cause the server to revert
to per-device mode. This issue was corrected in SP1.

The two modes still seem to confuse a lot of people. Windows Server 2003 Terminal
Servers can be in one of two licensing modes - either-per user or per-device. The choice
pertains to whom or what leases the CAL, or more importantly, which CALs were
purchased. Per-device CALs are leased to machines or devices, while per-user CALs
are allocated to named users.

Per-device CALs are explicitly leased to clients and are recorded in the license server’s
database. Upon first connection, clients are given a temporary access license, which is
good for 90 days. The second time the client connects, and on each subsequent
connection until successful, the terminal server will attempt to upgrade the temporary
CAL to a permanent one. Permanent CALs are leased for a limited amount of time (up
to 89 days) and do expire.

Per-user CALs are not leased as there is currently no tracking mechanism available to
tie a terminal services CAL to a user account. Therefore, terminal servers in per-user
mode only check for license server availability, but never actually lease a CAL to the
user. If the terminal server can contact the license server, it simply accepts the
connection.

The important point here is the licensing mode that is in effect is determined by the
terminal server, not the license server or the CALs installed. If you are licensing
Terminal Services in per-user mode (allocating CALs per named user), then be sure you
are in the appropriate mode, and have verified this is set correctly on all terminal servers
in your environment. This can be adjusted in the Terminal Services Configuration
program from Administrative Tools at any time and does not require a reboot or a
restart of services.

Common symptoms of an incorrect licensing mode are event log entries for event ID
1000 or 1004 (described earlier), or the sudden inability for clients to connect. If you
see this on only one terminal server out of the farm, then definitely check the licensing
mode on the misbehaving box. You can also automate the process of checking or setting
this value by creating a script to query the value in the following registry key:

HKLM\System\CurrentControlSet\Control\TerminalServer\Licensing Core\PolicyAcOn

The two valid entries are 2 for per-device mode and 4 for per-user mode. A value of 0
indicates that this was set as a result of the pre-SP1 bug noted earlier.

Large Licensing Database Issues

Although uncommon, some terminal server licensing databases may become large. If
the database reaches around 80MB or more, there may be problems with the licensing
service. This issue is usually accompanied by the following event in the Windows event
log:
 Event Source Description
ID
43 TermServLicensing Work Manager error Can't Startup work scheduler, Error
code -1072167891

There is a known issue with licensing databases that are larger than 80MB in size. The
licensing service uses a Microsoft Jet-based database engine, and the transactions
involved in a large database can consume a lot of memory. If a transaction hangs or runs
for an extended period of time (for instance, it can’t complete because the disk is full),
this can cause Jet to consume the entire Version Store, crashing the service.

In case you are wondering, the Version Store is used in Jet-based databases as an in-
memory “cache” used to hold pending transactions until they can be written to disk.

Microsoft included a new registry key in Windows Server 2003 to be able to tweak the
default memory resources associated with the Version Store in Jet transactions. If you
find you have a large database and are seeing event ID 43, locate the following key in
the registry on the license server:

HKLM\System\CurrentControlSet\Services\TermServLicensing\Parameters

Add a new Value called MaxVerPages (REG_DWORD) with a value data of 1024
(decimal).

Most likely, you will not see this issue except in extremely large terminal server
deployments of several hundred nodes. For more information on this issue, check out
Microsoft KB article Q310122.

Final Thoughts
Whenever problems arise with connectivity, always check the event logs as they may
contain valuable information. I also make it a point to perform basic connectivity tests
to be sure the terminal servers can contact the license server, and leverage the tools at
your disposal.

In part three, we’ll review some of the more-obscure issues seen in terminal server
licensing from the server side, and part four will cover issues from the client’s
perspective.

Troubleshooting Terminal Server

Licensing Issues (Part 3)
Corrupt Licensing Database
Although rare, it is possible for the licensing database to become corrupt. Symptoms of
a corrupt database include not being able to start the licensing service, failure of the
service to stay running, or possibly the inability of terminal servers to lease CALs
(although this last one is most likely due to some other underlying issue).

Corruption can occur for a number of reasons, but the most popular is making the
mistake of placing the licensing database on a compressed drive. The drive compression
could cause a write-delay, thereby corrupting the database. Per Microsoft, Jet-based
databases should not be located on compressed drives. See Microsoft KB Q318116 for
more information.

Other ways corruption can occur is incorrect shutdown of the server or a licensing
service crash, or a hardware related issue such as a problematic RAID controller or
faulty cache.

Of course, there is no better protection against corruption than a good backup; however
if you find yourself with a corrupt database you can recreate a new (empty) database
pretty easily. Keep in mind that the new database will not have any of your CALs
installed; you will need to contact the Microsoft Clearinghouse to recover them.

Recreating the database is as simple as removing or renaming the existing database

folder and restarting the service:

1. Stop the Terminal Server Licensing service.

2. Rename the LServer directory in Windows\System32 (default location) to
something such as LServer_Old.
3. Create a new folder where the old one was (\Windows\System32 by default)
called LServer.
4. Start the Terminal Server Licensing service.

When the service is started, it will automatically create a new TLSLic.edb database file
along with a set of transaction logs. You can verify success by checking the LServer
directory for the existence of the database file, and the event log should list the
following events:

Event Source Description

ID
5 TermServLicensing Policy Module %SystemRoot%\system32\tls236.dll for
company Microsoft Corporation has been loaded.
0 TermServLicensing Terminal Services Licensing was started.

The LServer directory should contain the following files when the service is started:

File Name Description

edb.log License database transaction log
edb.chk Checkpoint file used by Jet to identify where in the
File Name Description
transaction log that transactions have been committed to the
database
res1.log Reserve log file in case the disk runs out of space. Limited
to 5MB
res2.log Second reserve log with the same purpose of res1.log
TLSLic.edb The actual licensing database file
temp.edb Used to store information about in-progress transactions

Once you have verified the database has been successfully recreated, you will need to
contact the Microsoft Clearinghouse to have them reissue your CALs. To locate the
phone number for the Clearinghouse, change the License server’s activation method to
Phone in the Terminal Server Licensing program (right-click on the license server in the
interface and select Properties). When you then go through the Install Licenses wizard,
the phone number for your region will be displayed. You can also find all region phone
numbers in the following registry location:

HKLM\Software\Microsoft\TermServLicensing\LrWiz\CSNumbers

If you received an error message when you restarted the Terminal Server Licensing
service in step 4 above, make sure you created the LServer directory and it is in the
correct location. The service looks to the following registry location to find the
licensing database:

HKLM\System\CurrentControlSet\Services\TermServLicensing\Parameters\DBP
ath

The following event log entries may also be observed if the database could not be
created:

Event Source Description

ID
44 TermServLicensing General database error occurred, Can't initialize ESE
instance - error -1811 JET_errFileNotFound, File not
found.
7024 Service Control The Terminal Services Licensing service terminated with
Manager service-specific error 31.

Activation Issues and Problems Installing CALs

If you are having problems activating the license server, there may be a few causes.

You could have an incomplete License Service installation or a set of corrupt/invalid

certificates. Check the event logs for the following event:
 Event Source Description
ID
38 TermServLicensing Can't generate a license for client because of error 'Can't
add certificate to store, error c0010020.

You may also observe the following error:

Internal Error: 0xc0110011

If so, the easiest resolution is to remove the Terminal Server Licensing service from the
server and reinstall it.

1. If necessary, stop the Terminal Server Licensing service and rename the LServer
directory (located in \Windows\System32 by default).
2. In the Windows Components section of Add/Remove Programs, uncheck
Terminal Server Licensing, and complete the wizard.
3. After rebooting the server, return to the Windows Components section of
Add/Remove Programs and check Terminal Server Licensing. Complete the
wizard by selecting a location to hold the LServer directory (anywhere but a
compressed drive; the default location of \Windows\System32 is fine).

Check that the service has started and that the LServer directory was created properly.
You can also check the Windows event log for the following entries that indicate a
successful start of the service:

Event Source Description

ID
5 TermServLicensing Policy Module %SystemRoot%\system32\tls236.dll for
company Microsoft Corporation has been loaded.
0 TermServLicensing Terminal Services Licensing was started.

Once that’s verified, use the Licensing Wizard from the Terminal Server Licensing
program to reactivate the license server.

CAL Pack Installation Issues

If you are having troubles installing CAL packs, there are a few things to check. You
may receive an error message similar to the following when you attempt to install the
token pack:

Licensing Wizard was unable to install the client license key pack. Please verify your
entry and try this operation again. Message Number 0x13A7.

The message number may also be either 0xFA1 or 0x13A4. In any case, there are a few
possible causes. First, if you are installing a license token pack on a Windows 2000
License Server, be sure the CALs are not intended for Windows Server 2003. You
cannot install a Windows Server 2003 license pack on a Windows 2000 license server;
you need to install a Windows Server 2003 license server in your enterprise to use
Windows Server 2003 CAL packs.

Another possible cause is a problem with the TermServLicensing Policy key. Check the
following registry key on the license server:

HKLM\Software\Microsoft\TermServLicensing\Policy\Microsoft Corporation\A02

There should be a string data type of DLL with a value of %systemroot

%\System32\tls236.dll. If you notice that this value doesn’t match, then correct it. If
the DLL string data is missing entirely, then create it and set the value to %systemroot
%\System32\tls236.dll.

If all else fails, remove the license service from the server and reinstall it. This will
correct any issues with the service installation or the certificates associated with the
server.

If you are having issues with the Terminal Server Licensing program crashing, there is a
known issue which has been corrected in Windows Server 2003 SP2. You will typically
see this when you open Terminal Server Licensing, click on a per-user CAL pack and
refresh the display. There’s a bug in the Licmgr.exe binary that causes an access
violation. If you aren’t in a position to upgrade your terminal servers to SP2 to correct
it, you can obtain a hotfix for SP1 installations by contacting Microsoft Product Support
Services. See Microsoft KB 910088 for more information. Pre-SP1 installations didn’t
typically exhibit this behavior.

Finally, sometimes trouble with activating a license server or installing CALs can be
resolved simply by attempting to activate via phone. It’s not quite as convenient as
activating over the Internet, but the process is straightforward and the Clearinghouse is
usually very responsive (in other words, little or no hold times). The process for
selecting Phone as your method of activation is detailed in the section “Corrupt
Licensing Database” above.

Contact the Clearinghouse or PSS?

The question will ultimately arise as to which side of Microsoft should you contact for a
specific problem – the Microsoft Clearinghouse or Product Support Services.

Contact the Microsoft Clearinghouse when your issue is one of the following:

• Trouble activating the license server or adding CAL packs

• You are reinstalling the license server, or recovering from a failed license server
or corrupt database and need to recover lost CALs
• You believe you have the incorrect CAL types (i.e. Per-Device CALs instead of
Per-User CALs); the Clearinghouse can assist you with swapping out CALs for
the correct type, provided that have not yet been activated

Basically, if the trouble is something administrative pertaining to the Terminal Server

Licensing program, the Clearinghouse can help you.
If you are having technical difficulties with any of the following, contact Microsoft
Product Supports Services (PSS):

• Troubles discovering license servers

• Issues with CALs not being allocated to clients correctly
• License service crashes or won’t stay running

Generally, any technical issues after the license service has been successfully activated
and CAL packs are installed will need to be referred to Microsoft PSS, including
obtaining any hotfixes mentioned in this article series.

Final Thoughts
I covered several “not-so-common” issues with licensing in this segment. Be sure to
check out parts one and two for more common issues that you will likely encounter. In
the final segment of this four-part series, we’ll look at some of the common client-
related issues with licensing.