Wireless Netw
DOI 10.1007/s11276-017-1551-9
A hierarchical learning approach to anti-jamming channel
selection strategies
Fuqiang Yao1,2 • Luliang Jia2,3 • Youming Sun2,3 • Yuhua Xu2,3 • Shuo Feng4
Yonggang Zhu1,2
 Springer Science+Business Media, LLC 2017
Abstract This paper investigates the channel selection
problem for anti-jamming defense in an adversarial envi-
ronment. In our work, we simultaneously consider mali-
cious jamming and co-channel interference among users,
and formulate this anti-jamming defense problem as a
Stackelberg game with one leader and multiple followers.
Specifically, the users and jammer independently and
selfishly select their respective optimal strategies and
obtain the optimal channels based on their own utilities. To
derive the Stackelberg Equilibrium, a hierarchical learning
framework is formulated, and a hierarchical learning
algorithm (HLA) is proposed. In addition, the convergence
performance of the proposed HLA algorithm is analyzed.
& Luliang Jia
jiallts@163.com
Fuqiang Yao
yfq2030@163.com
Youming Sun
sunyouming10@163.com
Yuhua Xu
yuhuaenator@gmail.com
Shuo Feng
fengs13@mcmaster.ca
Yonggang Zhu
zhumaka1982@163.com
1
Nanjing Telecommunication Technology Institute,
Nanjing 210007, China
2
College of Communication Engineering, PLA University of
Science and Technology, Nanjing 210007, China
3
Science and Technology on Communication Networks
Laboratory, Shijiazhuang 050002, China
4
Computational Science and Engineering at McMaster
University, Hamilton, ON L8S 4L8, Canada
•
Finally, we present simulation results to validate the
effectiveness of the proposed algorithm.
Keywords Anti-jamming  Stackelberg game  Channel
selection  Stochastic learning
1 Introduction
The security of the spectrum availability (SSA) is a critical
issue in wireless communication networks under the con-
dition of complicated electromagnetic environment. It
mainly faces the following challenges. Firstly, malicious
jamming attack poses a significant security threat to the
SSA. Secondly, mutual interference, which comes from the
users with the same channel and unwanted emissions in
spurious and out-of-band domain, also severely degrades
network performance, especially in the area with enormous
number of wireless devices [1]. In our work, for the mutual
interference, we mainly consider the co-channel interfer-
ence, which is the main source of mutual interference. For
the malicious jamming attack, we focus on the smart
jammer, which can adjust its strategies adaptively accord-
ing to users’ strategies as well as dynamic environment to
maximize its damage.
Anti-jamming defense is of great importance in the SSA
of wireless communications and has attained growing
attention in recent years [2, 3]. It is known that various
countermeasures have been proposed in order to deal with
the threats of jamming attacks, such as Frequency Hopping
Spread Spectrum (FHSS) and Uncoordinated Frequency
Hopping (UFH). However, these approaches need wide-
band spectrum or continuous broadband, and they are
spectrally inefficient. Typically, these methods are infea-
sible in the dynamic and scarce spectrum environment
123

[4–6]. Therefore, more flexible and efficient anti-jamming
techniques are needed.
Cognitive radio technology can re-configure communi-
cation parameters intelligently, and it can be exploited by
jamming attacks to launch more complicated and
threatening attacks [7, 8]. Since users and jamming
attacks independently and selfishly select their strategies,
game theory [9–14] can be employed to analyze the
interactions between the users and jamming attacks. In
[15], an anti-jamming stochastic game framework was
proposed in the presence of cognitive attackers. In [16], a
dogfight game was modeled to analyze the interactions
between defending secondary users and attacks in cog-
nitive radio systems. The authors in [17] investigated the
security problem in cognitive radio ad hoc networks and
proposed a stochastic game to defend against sweep
attack. In [18], a Bayesian game was formulated to
analyze the competitive interactions between a transmit-
ter–receiver pair and a jammer. A zero-sum game was
formulated in [19] to make the strategic decision-making
for both AWGN channels and frequency selective fading
channels. In [20] and [21], a Markov game was formu-
lated to model the interactions between the legitimate
transmitter and jammer, and the optimal defense strategy
was obtained. In [22], a game-theoretic framework was
adopted to investigate the anti-jamming rendezvous
problem, and the known and unknown rendezvous
channel cases were considered.
Considering the hierarchical competition among players,
Stackelberg game [23, 24] is a suitable framework to
capture the sequential interactions between the users and
jammers. In [25] and [26], the anti-jamming defense
problem was formulated as a Stackelberg game, and the
optimal power control strategy was achieved. Anti-jam-
ming Stackelberg game with observation error was ana-
lyzed in [27]. The uncertainties of the channel state
information and transmission cost information were con-
sidered, and a Bayesian anti-jamming Stackelberg game
was proposed in [28]. The anti-reactive-jamming problem
was investigated, and the transmitter-jammer interactions
were modeled as a Stackelberg game in [29]. In [30] and
[31], the authors formulated the sequential interactions
among a mobile device, a smart attacker, and a security
agent as a Stackelberg game, and the Stackelberg
Equilibrium (SE) was obtained. However, few work has
devoted to the channel selection problem in the anti-jam-
ming field.
Channel selection plays an important role in anti-
jamming defense. There are some studies for channel
selection to mitigate the co-channel interference among
users. Specifically, in [32–34], the interference mitigation
game was formulated and proved to be an exact potential
game. These studies, however, did not consider the
123
Wireless Netw
malicious jamming, which can severely deteriorate the
intended transmission and pose a significant threat to
wireless communications. In [4], a game-theoretic anti-
jamming scheme was proposed, and a channel selection
strategy was provided to combat jamming attacks. Wu
et al. [8] investigated the anti-jamming defense in cog-
nitive radio networks with a secondary user and proposed
a flexible channel selection scheme. However, the studies
mentioned above paid little attention to the mutual
competition among users for both limited spectrum
resources and the sequential actions between users and
external jamming attacks.
Although there exist some related works for interference
mitigation and anti-jamming, the two important aspects
were separately investigated in previous studies. In this
paper, the malicious jamming and co-channel interference
among users are simultaneously considered, and we for-
mulate the channel selection problem for anti-jamming
defense as a Stackelberg game, which is an appropriate
method to model the hierarchical interactions and make the
sequential decision-making between the users and smart
jammer. However, it is challenging to obtain the SE solu-
tions. Inspired by [32–36], we can incorporate learning
technologies into the proposed game and obtain the
learning solution.
Since game theory is a well-developed and flexible tool
to analyze interactions among players, and learning tech-
nologies can cope with the constraints of lacking infor-
mation exchange and random environment, incorporating
learning technologies into game theory is a promising and
important research direction [32–36]. In this paper, we
extend the stochastic learning into a hierarchical learning
framework and propose a hierarchical learning algorithm
(HLA) based on the stochastic learning theory to acquire
the solutions of the proposed game. The main contributions
of this paper are given as follows.
• We investigate the channel selection problem in
wireless communication networks when malicious
jamming and co-channel interference among users are
simultaneously considered, and formulate this anti-
jamming defense problem as a Stackelberg game.
Moreover, its properties are investigated. The network
utility is the expected weighted aggregate interference
and jamming. Specifically, the smart jammer and users
independently and selfishly select their optimal strate-
gies, and determine their optimal channels to maximize
their own utilities.
• To obtain the solution of the proposed game, based on
the stochastic learning theory, a hierarchical learning
framework is formulated, in which each player chooses
its channel strategy based on a probability distribution.
Furthermore, a hierarchical and learning-based channel
Wireless Netw
selection algorithm is proposed, which can converge to
the SE solution of the proposed game.
The rest of this paper is organized as follows. In Sect. 2, we
introduce the system model and problem formulation. In
Sect. 3, an anti-jamming channel selection game is for-
mulated, and its properties are investigated. In Sect. 4, a
hierarchical stochastic learning algorithm is proposed. In
Sect. 5, simulation results are presented. Finally, Section 6
draws the conclusions.
2 System model and problem formulation
2.1 System model
As shown in Fig. 1, there is one jammer and N users in our
system. Denote the user set as N ¼ f1; 2; . . .; N g and
users’ available channel set as M ¼ f1; 2; . . .; M g. The
jammer is smart, and it can sense the users’ available
channels and adjust its strategies adaptively according to
the users’ strategies as well as dynamic environment to
achieve maximum damage. On the other hand, each user is
intelligent such that it adopts flexible channel selection
strategy, which can minimize its received external jam-
ming and co-channel interference among users.
The channel selection profile of all users is
a ¼ fa1 ; a2 ; . . .; aN g. Let an ¼ ½a0 ; a1 ; . . .; an1 ; anþ1 ;
. . .; aN  denote the channel selection profile of all the users
except user n. For users, if two or more users choose the
same channel, mutual co-channel interference emerges. For
the smart jammer, it chooses h channels to launch the
Fig. 1 System model
jamming attack, and the jamming channel set is
C ¼ fc1 ; c2 ; . . .; cH g. In this paper, to simplify analysis, we
assume that only one channel cj 2 C is jammed at a time,
and the jamming channel set C is the same as the available
channel set M.
Similar to [32], it is assumed that the available channels
undergo block fading, and the Rayleigh fading model is
considered. Moreover, we assume that each user selects
one channel for transmission. The instantaneous interfer-
ence gain from user m to n is defined as Hmn an
¼ ðdmn Þa1
an
bmn , where superscript an represents the selected channel of
user n, dmn , a1 and bamnn denote the distance, path-loss
exponent and instantaneous fading coefficient between user
m and user n, respectively. The instantaneous jamming
c c
gain from the smart jammer to user n is Hjnj ¼ ðdjn Þa2 bjnj ,
where superscript cj denotes the selected channel of the
c
smart jammer, djn , a2 and bjnj are distance, path-loss
exponent and instantaneous fading coefficient between the
smart jammer and user n, respectively.
2.2 Problem formulation
To improve network performance, efficient channel selec-
tion faces the following challenges in the anti-jamming
field. Firstly, it is indispensable to defend against the
malicious jamming. Secondly, it needs to mitigate the co-
channel interference among users due to the conflicting
competition for limited channel resources. The selected
channel of user n and smart jammer are denoted as an 2 M
and cj 2 C, respectively. Then, the received interference
and jamming of user n is given by:
X c
an
Dn ¼ In þ Jn ¼  Pm Hmn f ðam ; an Þ  Pj Hjnj f ðcj ; an Þ;
m2fN=fngg
ð1Þ
where In denotes the received co-channel interference from
other users, Jn represents the malicious jamming, Pm and Pj
denote the transmitting power of the user m and smart
jammer, respectively, and f ðÞ is the Kronecker delta
function, which is expressed as:

1; an ¼ a m ;
f ðam ; an Þ ¼ ð2Þ
0; an 6¼ am :
Referring to [32–34], the network utility function is the
expected weighted aggregate interference and jamming
given by:
123
 Wireless Netw

X
U¼ Pn E½Dn ; From the perspective of the users, the follower sub-game
n2N can be defined as:
X X  
¼ mn
Pn P m H an
f ðam ; an Þ Gf ¼ N; M; un ðan ; an ; cj Þ ; ð6Þ
ð3Þ
n2N m2fN=fngg
X c where all the users are players, and the strategy set is the
þ jnj f ðcj ; an Þ;
Pn Pj H
n2N
available channel set M. Each user selfishly and inde-
h i pendently chooses its optimal channel, which can maxi-
 a 
where Hmn
an
¼ E Hmn
n
¼ ðdmn Þa1 bamnn , H jncj ¼ E Hjn
cj
¼ mize its utility.
a2 cj For the smart jammer, it aims at achieving its maximum
ðdjn Þ bjn , E½ denotes the statistical expectation. The
damage, and its utility function can be expressed as:
object of efficient channel selection strategy is to minimize
the received interference and jamming, thus, improving the   X
uj a; cj ¼ jncj f ðcj ; an Þ:
Pn Pj H
network performance, i.e., ð7Þ
n2N
X
ðP1Þ : aopt ¼ arg min U ¼ arg min Pn E½Dn : ð4Þ The smart jammer’s optimization problem is
n2N  
cj ¼ arg maxcj uj a; cj .
However, due to the randomness of environment, the com- From the smart jammer’s perspective, the leader sub-
binatorial optimization problem P1 is intractable through game can be defined as:
standard optimization methods. In the following, this prob-   
lem is formulated as a Stackelberg game, and a learning- Gl ¼ J; C; uj a; cj ; ð8Þ
based algorithm is proposed to obtain its SE. where the strategy set is the jamming channel set C,
which is the same as the available channel set M in this
paper.
3 Anti-jamming channel selection game
3.2 Stackelberg game solution
3.1 Game model
Definition 1 (Exact potential game [37]) The game Gf is
We formulate the channel selection problem for anti-jam-
an exact potential game, if there is a potential function U
ming defense as a Stackelberg game. Mathematically, the
which holds that the variation in potential function equals
anti-jamming channel selection game is denoted as
  to the variation in the utility function due to any player’s
G ¼ N; J; M; C; un ; uj , where N is the user set, J
unilateral deviation. Mathematically,
denotes the smart jammer, M and C respectively represent      
the strategy set of the users and smart jammer, un and uj are U a~n ; an ; cj  U an ; an ; cj ¼ un a~n ; an ; cj
  ð9Þ
the utility function of user n and smart jammer, respec-  un an ; an ; cj ;
tively. To be specific, the smart jammer acts as the leader,
where a~n is the user n’s action after unilateral deviation.
whereas the users are followers.
From the user-side, it aims to minimize the received inter- Lemma 1 ([37]) Every exact potential game possesses at
ference and jamming, and its utility can be defined as [32]: least one pure strategy Nash Equilibrium (NE).
un ðan ; an ; cj Þ ¼ L  Pn E½Dn ; Theorem 1 The follower sub-game Gf with given strat-
X
¼L mn
Pn Pm H an
f ðam ; an Þ egy cj of the smart jammer is an exact potential game.
ð5Þ
m2fN=fngg
Proof Motivated by [32–34], given the smart jammer’s
 jncj f ðcj ; an Þ;
Pn Pj H strategy cj , a potential function of the follower sub-game
can be formulated as:
where L denotes a predefined positive constant and will be
explained in Sect. 4.1. The user n’s object is to tune its      
U an ; an ; cj ¼ U1 an ; an ; cj þ U2 an ; an ; cj ;
channel selection to maximize its utility, and its opti-
1X X ð10Þ
mization problem can be written as an ¼ arg maxan un ¼ Pn E½In   Pn E½Jn ;
2 n2N n2N
ðan ; an ; cj Þ.

123
Wireless Netw

   
where U a~n ; an ; cj  U an ; an ; cj
   
  ¼ U1 a~n ; an ; cj þ U2 a~n ; an ; cj
U1 an ; an ; cj    
 U1 an ; an ; cj  U2 an ; an ; cj
1X 1X X mn
an X X
¼ Pn E½In  ¼  Pn Pm H f ðam ; an Þ ¼ Pn Pm Hmn
an
f ðam ; an Þ þ
cj
jn
Pn Pj H f ðcj ; an Þ
2 n2N 2 n2N m2fN=fngg
8 m2fN=fngg j2C
X X
1< X mn
an  mn
Pn Pm H a~n
f ðam ; an Þ  jncj f ðcj ; a~n Þ
Pn Pj H
¼ Pn Pm H f ðam ; an Þ
2 :m2fN=fngg m2fN=fngg j2C
   
9 ¼ un a~n ; an ; cj  un an ; an ; cj
X X =
a
mii f ðam ; ai Þ
þ Pi Pm H ð15Þ
i2fN=fngg m2fN=figg
;
8 Thus, according to Definition 1, the follower sub-game Gf
1< X mn
an
is an exact potential game. h
¼ Pn Pm H f ðam ; an Þ
2 :m2fN=fngg
In the following, we will analyze the existence of SE in
X X
þ Pi Pm H ai
mi f ðam ; ai Þ the formulated Stackelberg game.
i2fN=fngg m2fN=fi;ngg
9 Theorem 2 Given the smart jammer’s strategy cj , the
X = follower sub-game Gf always exists a strategy that satisfies
þ niai f ðan ; ai Þ :
Pi Pn H
; un ðan ; an ; cj Þ  n ðan ; an ; cj Þ, which is a pure strategy NE
i2fN=fngg
point.
ð11Þ
Proof Given the smart jammer’s strategy cj , then the
Similar to [32–34], interference symmetry is considered. users, which are rational and independent, play a non-co-
By applying Hmn
an
¼H nm
an
, we have operative channel selection sub-game.
X X Based on Theorem 1, the follower sub-game Gf is an
mn
Pn Pm H an
f ðam ; an Þ ¼ niai f ðan ; ai Þ:
P i Pn H exact potential game, and it has at least a pure strategy NE
m2fN=fngg i2fN=fngg point. h
ð12Þ
For the leader sub-game Gl , which is a non-cooperative
 
Hence, we can rewrite U1 an ; an ; cj as follows: game. Every finite strategic game has a mixed strategy
equilibrium [9]. Therefore, there exists a SE in the sense of
  X
U1 an ; an ; cj ¼  mn
Pn Pm H an
f ðam ; an Þ  Tðan Þ; stationary strategy.
m2fN=fngg Based on the above analysis, we assume that a smart
jammer’s mixed strategy is h0 . In the following, we define
ð13Þ
the SE for the proposed game [9].

P P ai Definition 2 The strategy profile a; h0 constitutes the

mi
where Tðan Þ ¼ i2fN=fngg m2fN=fi;ngg Pi Pm H f ðam ;
ai Þ is independent of an . SE for the proposed game, if the following conditions hold:
In addition,

uj a; h0  uj ða; h0 Þ; ð16Þ

 
U2 an ; an ; cj

X X un an ; an ; h0  n an ; an ; h0 : ð17Þ

¼ Pn E½Jn  ¼  jncj f ðcj ; an Þ
Pn Pj H
n2N n2N Theorem 3 In the proposed game, there exists a smart
X ð14Þ
jncj f ðcj ; an Þ 
¼ Pn Pj H jicj f ðcj ; ai Þ
P i Pj H jammer’s stationary policy and a users’ NE policy that
i2fN=fngg consist a SE.
c
jnj f ðcj ; an Þ  C2 ðan Þ:
¼ Pn Pj H
Proof Inspired by [35], if a smart jammer’s stationary
policy h0 is given, the proposed game reduces to a non-
P cooperative game. Based on Theorem 1, the follower sub-
jicj f ðcj ; ai Þ is also inde-
where C2 ðan Þ ¼ i2fN=fngg Pi Pj H
game is an exact potential game. According to [37], every
pendent of an .
finite exact potential game possesses at least one NE.
Therefore, we have

123

Therefore, NEðh0 Þ always exists in the proposed game
given the smart jammer’s policy h0 . However, the smart
jammer’s stationary policy can be given by:
h0 ¼ arg maxuj ðh0 ; NEðh0 ÞÞ:
h0
According to [9], every finite strategic game has a mixed
strategy equilibrium. Hence, h0 ; NE h0 constitutes a
SE in the sense of stationary strategy.
4 Hierarchical learning solution for anti-jamming
channel selection game
4.1 Algorithm description
In this subsection, to achieve the solution of the pro-
posed game, a hierarchical learning framework based on
the stochastic learning theory is formulated, and a hier-
archical learning algorithm (HLA) is proposed. We
extend the proposed anti-jamming channel selection
game to a mixed strategy form in order to characterize
the proposed HLA. A mixed strategy of user n at time
slot t is defined as hn ðtÞ ¼ ðhn1 ðtÞ; . . .; hnm ðtÞ; . . .; hnM ðtÞÞ
P
and m2M hnm ðtÞ ¼ 1, where hnm ðtÞ means the proba-
bility that the user n selects channel m from users’
available channel set M ¼ f1; 2; . . .; M g at time slot
t. For the smart jammer, a mixed strategy at epoch k is
denoted by h0 ðkÞ ¼ ðh01 ðkÞ; . . .; h0h ðkÞ; . . .; h0H ðkÞÞ and
P
h2C h0h ðkÞ ¼ 1, where h0h ðkÞ means the probability that
the smart jammer selects channel h from the jamming
channel set C ¼ fc1 ; c2 ; . . .; cH g at epoch k. In the hier-
archical learning framework, the smart jammer and users
update their policies at different time scales. The smart
jammer updates its policies at each epoch k, whereas
users update their policies at each time slot t, and each
epoch contains T time slots.
In the follower sub-game, based on the stochastic
learning automata (SLA) [38], we propose a learning-
based anti-jamming channel selection algorithm, which is
an uncoupled and distributed algorithm. Specifically, it
does not need the information exchange, and each user
updates its policy based on its individual payoff. The
stochastic learning automata [38] can learn the optimal
strategies from a set of strategies through repeated
interactions with random environment, and has been
employed for adaptive decision-making in wireless
communications [32, 39–42]. For the tth time slot, we
assume that the users’ channel selection profile is
aðtÞ ¼ fa1 ðtÞ; a2 ðtÞ; . . .; aN ðtÞg. The user n’s random
payoff can be written as:
123
Wireless Netw
ð18Þ
h
Fig. 2 The diagram of the proposed HLA
X X
an ðtÞ
mn
un ðtÞ ¼ L  Pn Pm H f ðam ðtÞ; an ðtÞÞ
n2N m2fN=fngg
X c ðtÞ
jnj f ðcj ðtÞ; an ðtÞÞ:
 Pn Pj H
n2N
ð19Þ
where the predefined positive constant L is introduced to
ensure that the random payoff is nonnegative, and the
proposed algorithm in the following is effective.
In the leader sub-game, an online channel selection
algorithm based on Q-learning [35, 36, 43, 44] is proposed
for the smart jammer. In the learning process, the strategy
is updated by repeated interaction with surrounding envi-
ronment, and the policies that yield high payoff are rein-
forced through a trial and error exploration. The overall
goal of learning is to learn to optimize its long-term
cumulative payoff. The smart jammer’s random payoff can
be estimated by exploration. For the kth epoch, the esti-
mated random payoff of the smart jammer is given by:
X
uj ð k Þ ¼ jncj ðkÞ f ðcj ðkÞ; an ðkÞÞ:
Pn Pj H ð20Þ
n2N
An illustrative diagram of the proposed HLA is shown in
Fig. 2, and the proposed algorithm is presented in Algo-
rithm 1. At each time slot t, based on the probability dis-
tribution hn ðtÞ, each user chooses a channel an ðtÞ 2 M at
random. And the probability distribution hn ðtÞ is updated at
each time slot t according to the feedback from the envi-
ronment. The smart jammer takes actions at each epoch
k in a similar way. For the smart jammer, the stop criterion
is either the channel selection probability vector holds
h0 ðkÞ ¼ h0 ðk þ 1Þ, or the maximum iteration number is
reached. For users, we have similar stop criterions.
Wireless Netw

Algorithm 1: Hierarchical learning algorithm (HLA)

Step 1: Set t=0, k=0 and initialize the mixed strategy for both the users and smart jammer, and θ0h (k) =

1/ |C |, θnm (t) = 1/ |M | , ∀h ∈ C , ∀m ∈ M .

Step 2: In the kth epoch, the smart jammer stochastically selects jamming channel ch (k) according to

its policy θ0 (k).

Step 3: Stochastic learning process of all users, for each epoch k.

(1) In the tth slot, each user n stochastically selects channel an (t) according to its current strategy

θn (t).

(2) Each user n measures its utility un (t).

(3) Each user n updates its strategy according to the following rules:

θnm (t + 1) = θnm (t) + b1 ũn (t) (1 − θnm (t)) ,m = an (t),

(21)
θnm (t + 1) = θnm (t) − b1 ũn (t)θnm (t) ,m = an (t),

where 0 < b1 < 1 is the learning step size of users, and ũn (t) is the normalized utility.

ũn (t) = un (t)/L. (22)

Step 4: The smart jammer measures its utility u j (k).

Step 5: The smart jammer updates its Q values according to the following rule.

Q0k+1 (ch ) = (1 − κ0k )Qk0 (ch ) + κ0k u j (k) , (23)

2
where κ0k ∈ [0,1) is the learning rate, which holds ∑∞ k ∞ k
k=0 κ0 = ∞, ∑k=0 κ0 < ∞. And the smart jammer

updates its policy based on the Boltzmann distribution.

exp Qk0 (ch )/τ0

θ0h (k) = , (24)
∑ exp Qk0 (ch )/τ0
C

where the temperature τ0 controls the tradeoff of exploration-exploitation. Specifically, for τ0 → 0, the

smart jammer tends to select the policy with the maximum Q value, whereas for τ0 → ∞, the smart

jammer’s policy is completely random.

Step 6: If the stopping criterion holds, stop. Otherwise, update k=k+1 and go to step 2.

123
 Wireless Netw

(" #
4.2 Convergence analysis h0h ðkÞ jk X
¼ h0h ðkÞ 0 uj ð k  1Þ  h0g ðkÞuj ðk  1Þ
dk s0
Let aðtÞ ¼ fa1 ðtÞ; a2 ðtÞ; . . .; aN ðtÞg represent channel g2C

selection profile of all users at time slot t, and hðtÞ ¼ X h0h ðkÞ
s0 h0g ðkÞ ln g:
fh1 ðtÞ; h2 ðtÞ; . . .; hN ðtÞg denote a mixed strategy profile of g2C
h0g ðkÞ
users, where hn ðtÞ ¼ ðhn1 ðtÞ; hn2 ðtÞ; . . .; hnM ðtÞÞ, 8n 2 N
ð30Þ
denotes the channel selection probability of user n at time
slot t. Moreover, the function hns ðhÞ is defined by: In the following, we resort to the ODE to analyze the
X convergence of h0 ðkÞ, which converges to the solution of
hns ðhÞ ¼ un ða1 ; . . .; an1 ; s; anþ1 ; . . .; aN Þ P htat ;
at ;t6¼n
t6¼n the ODE [38]. The right-hand side of (30) can be denoted
as f ðh0 Þ. h0 ðkÞ will converge weakly, as jk0 ! 0, to
ð25Þ

h0 ; NE h0 , which is the solution of dh dk ¼ f ðh0 Þ,

0

where hns ðhÞ represents the user n’s expected utility func-
tion when user n selects action s and other users adopt
mixed strategy hn .
Motivated by [32, 35, 36, 41], the asymptotic conver-
gence behavior of the proposed HLA algorithm is analyzed
in the following Theorems 4, 5 and 6.
Lemma 2 As b1 ! 0, the follower-SLA converges to a
pure NE point in the follower learning sub-game.
Proof Based on Theorem 3.1 in [38], as b1 ! 0, the
sequence hn ðtÞ converges to the solution of the ordinary
differential equation (ODE):
dh
¼ f ðhÞ; hð0Þ ¼ h0 ;
dt
where f ðhÞ ¼ E½Gðhn ðtÞÞ; an ðtÞ; un ðtÞjhn ðtÞ ¼ hn , and hð0Þ
is the initial condition.
Moreover, similar to [32, 41], define function
HðhÞ ¼ E½UðhÞ, where UðÞ denotes potential function
specified by Eq. (10). Therefore,
X
Hðm; hn Þ ¼ Un ða1 ; . . .; an1 ; s; anþ1 ; . . .; aN ÞPt6¼n htat :
at ;t6¼n
Applying (15), (25) and (27), we have
Hðm1 ; hn Þ  Hðm2 ; hn Þ ¼ hnm1 ðhÞ  hnm2 ðhÞ:
According to Theorem 3.2 in [38] and Theorem 5 in
[41], Theorem 4 is proved. h a1 ¼ a2 ¼ 2, the positive constant L = 0.005, and the
Lemma 3 The leader Q-learning converges
h0 ð0Þ ¼ h0 , where h0 ð0Þ is the initial condition. h
Theorem 4 The proposed HLA can always find a SE.
Proof Inspired by [35], we prove it by contradiction. It is
assumed that the learning process converges to a point that
is not a SE. According to Theorem 3.1 in [38], the learning
process converges to the solution of ODE that are
stable points. Therefore, the proposed HLA only converge
to stable points.
Thus, the stationary point is a non-SE, which is
contradictory with Theorem 3. This completes the proof. h
ð26Þ
5 Numerical results and discussions
In this subsection, the simulation results are performed to
show the performance of the proposed HLA. In the simu-
lation, the system involves four channels, the users are
randomly located in a region of 100 m  100 m, and the
jammer located in a region of ½150; 200 m. An illustrative
diagram of the placement of users and smart jammer is
ð27Þ shown in Fig. 3. The fading of the channels is Rayleigh
fading model, and the fading parameters are exponentially
distributed with unit mean. Referring to [32, 34], the
parameters for this paper are given as: The user n’s
ð28Þ
transmitting power Pn ¼ 2W, the smart jammer’s trans-
mitting power Pj ¼ 25W, the path loss exponent
learning step size b1 ¼ 0:08. Unless otherwise stated, these
to

parameters are fixed.

h0 ; NE h0 in the leader learning sub-game.
Proof Motivated by [35, 36, 44], we describe the evolu-
tion of the Q values in the following differential equation.
dQk0 ðch Þ  
¼ jk0 uj ðkÞ  Qk0 ðch Þ :
dk
Compared to the dynamics of Q values, we would like to
express the dynamics in terms of strategies. By differen-
tiating (24) with respect to k and use (29), we have
The convergence of the proposed HLA for a single
simulation run is shown in Figs. 4 and 5. A system with
five users and one smart jammer is considered. Take user 1
for example, its convergence behavior in the first epoch is
presented in Fig. 4. At time slot t ¼ 0, user 1 randomly
ð29Þ
selects its channel with equal probabilities
(h11 ¼ h12 ¼ h13 ¼ h14 ¼ 1=4). The channel selection
probability h14 converges to 1 in about 300 iterations, h11 ,
h12 and h13 converge to 0. Figure 5 shows the smart jam-
mer’s convergence behavior. At k ¼ 0, the smart jammer

123
Wireless Netw

1
θ
01
0.9
θ
02
0.8 θ03

Channel selection probabilities

θ04
0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 10 20 30 40 50
Epoch numbers

Fig. 5 Evolution of channel selection probabilities of the smart

Fig. 3 The diagram of the placement of the users and smart jammer jammer for a single simulation run

1 0.06

Expected weighted aggregate interference and

θ HLA
11
0.9 Random selection
θ
12 0.05
0.8 θ13
Channel selection probabilities

θ
14
0.7 0.04
jamming (U)

0.6
0.03
0.5

0.4
0.02

0.3

0.01
0.2

0.1
0
4 5 6 7 8 9 10
0
0 100 200 300 400 500 Number of users (N)
Iteration numbers
Fig. 6 Performance comparison of expected weighted aggregate
Fig. 4 Evolution of channel selection probabilities of the user 1 in the interference and jamming for different channel selection strategies of
first epoch for a single simulation run the users

selects the channels randomly with equal probabilities
(h01 ¼ h02 ¼ h03 ¼ h04 ¼ 1=4). As the algorithm iterates,
the channel selection probabilities can converge in about
13 epochs.
In order to evaluate the performance of the proposed
HLA, it is compared with the random selection scheme, in
which each user randomly selects one channel at each time
slot. The presented results are obtained by simulating 1000
independent trials and then taking the mean. As can be seen
from Fig. 6, the proposed HLA is superior to the random
selection scheme and yields lower expected weighted
aggregate interference and jamming. Specifically, the
proposed HLA reduces the expected weighted aggregate
interference and jamming by 56% for 8 users versus a
random selection. Moreover, the performance gap will be
more significant with the number of users increasing. This
is due to the fact that the proposed HLA may converge to a
desirable solution, whereas the random selection scheme is
an instinctive approach.
In Fig. 7, we have shown the influence of the trans-
mitting power of the users and smart jammer. Figure 7
indicates that the expected weighted aggregate interference
and jamming will increase with the growing transmitting
power of the users. The reason is that higher transmitting
power of the users leads to more serious mutual interfer-
ence. Moreover, as higher transmitting power of the smart
jammer results in more serious damage, the expected
weighted aggregate interference and jamming will also

123
 Wireless Netw

x 10
−3 0.025

Expected weighted aggregate interference and

11 HLA
HLA, P =20W No jamming
j
Expected weighted aggregate interference and

10 HLA, Pj=25W Random jamming

0.02
HLA, P =30W
9 j

jamming (U)
8 0.015
jamming (U)

0.01
6

5
0.005
4

3
0
4 5 6 7 8 9 10
2 Number of users (N)
1 1.5 2 2.5 3
Transmitting power of users (W)
Fig. 9 Performance comparison of expected weighted aggregate
interference and jamming for different types of jamming attacks
Fig. 7 The expected weighted aggregate interference and jamming
versus transmitting power of the users (N ¼ 5)

time. For the no jammer scenario, the algorithm is con-

0.4 sistent with the algorithm in [32], in which only mutual co-
Expected weighted aggregate interference and

Rayleigh (0dB) channel interference is considered. As indicated in Fig. 9,

0.35 Rayleigh (1dB)
Rayleigh (3dB) compared to random jammer and no jammer scenario, the
0.3 network undergoes the highest expected weighted aggre-
gate interference and jamming in the smart jammer sce-
jamming (U)

0.25
nario. The reason is that the smart jammer can adjust its
0.2 strategies adaptively according to the users’ strategies as
well as dynamic environment, and achieve maximum
0.15
damage.
0.1

0.05
0
5 10 15
Number of users (N)
Fig. 8 Performance comparison of expected weighted aggregate
interference and jamming for different Rayleigh fading parameters
increase with the growing transmitting power of the smart
jammer. The performance comparison for different Ray-
leigh fading parameters is shown in Fig. 8. Specifically, it
is noted in Fig. 8 that the expected weighted aggregate
interference and jamming increases with the value of
Rayleigh fading growing.
In Fig. 9, performance evaluation is presented with
different types of jamming attacks. For the proposed HLA,
the type of jamming attack is smart, which can adjust its
strategies adaptively to maximize its damage. For com-
parison, we present the performance for random jammer
and no jammer scenario. In the random jammer scenario,
the random jammer chooses one channel randomly at a
6 Conclusion
20 25
In this paper, the channel selection scheme for anti-jam-
ming defense had been investigated in an adversarial
environment. First, in order to analyze the sequential
interactions between the users and smart jammer, a
Stackelberg game was formulated, and its properties were
analyzed. Then, based on the stochastic learning theory, a
hierarchical learning algorithm (HLA) was proposed.
Moreover, the proposed HLA was proven to converge to
Stackelberg Equilibrium (SE) solution. Finally, simulation
results were presented to validate the effectiveness of the
learning solution.
Acknowledgements This work was supported in part by the Natural
Science Foundation for Distinguished Young Scholars of Jiangsu
Province under Grant BK20160034, in part by the National Science
Foundation of China under Grant 61631020, Grant 61671473, Grant
61401508, and Grant 61401505, in part by Jiangsu Provincial Natural
Science Foundation of China Grant BK20130069, and Grant
BK20151450, and in part by the Open Research Foundation of Sci-
ence and Technology in Communication Networks Laboratory.

123
Wireless Netw
References
1. International Telecommunication Union (2015). Technical and
operational principles for HF sky-wave communication stations
to improve the man-made noise HF environment(ITU-R 258/5).
ITU.
2. Zou, Y., Zhu, J., Wang, X., & Hanzo, L. (2016). A survey on
wireless security: Technical challenges, recent advances, and
future trends. Proceedings of the IEEE, 104(9), 1727–1765.
3. Sagduyu, Y. E., Berry, R. A., & Ephremides, A. (2011). Jamming
games in wireless networks with incomplete information. IEEE
Communications Magazine, 49(8), 112–118.
4. Chen, C., Song, M., Xin, C., & Backens, J. (2013). A game-
theoretical anti-jamming scheme for cognitive radio networks.
IEEE Network, 27(3), 22–27.
5. Zhang, L., Guan, Z., & Melodia, T. (2016). United against the
enemy: anti-jamming based on cross-layer cooperation in wire-
less networks. IEEE Transactions on Wireless Communications,
15(8), 5733–5747.
6. Zhu, H., Fang, C., Liu, Y., et al. (2016). You can jam but you
can’t hide: defending against jamming attacks for Geo-location
database driven spectrum sharing. IEEE Journal on Selected
Areas in Communications,. doi:10.1109/JSAC.2016.2605799.
7. Pietro, R. D., & Oligeri, G. (2013). Jamming mitigation in cog-
nitive radio networks. IEEE Network, 27(3), 10–15.
8. Wu, Y., Wang, B., Liu, K., & Clancy, T. (2012). Anti-jamming
games in multi-channel cognitive radio networks. IEEE Journal
on Selected Areas in Communications, 30(1), 4–15.
9. Han, Z., Niyato, D., Saad, W., Basar, T., et al. (2012). Game
theory in wireless and communication networks. Cambridge:
Cambridge University Press.
10. Xu, Y., Wang, J., Wu, Q., et al. (2015). A game-theoretic per-
spective on self-organizing optimization for cognitive small cells.
IEEE Communications Magazine, 53(7), 100–108.
11. Sun, Y., Wu, Q., Wang, J., Xu, Y., & Anpalagan, A. (2016).
Veracity: Overlapping coalition formation based double auction
for heterogeneous demand and spectrum reusability. IEEE
Journal on Selected Areas in Communications, 34(10),
2690–2705.
12. Shao, H., Sun, Y., Zhao, H., et al. (2016). Locally cooperative
traffic-offloading in multi-mode small cell networks via potential
games. Transactions on Emerging Telecommunications Tech-
nologies, 27(7), 968–981.
13. Sun, Y., Wang, J., et al. (2016). Local altruistic coalition for-
mation game for spectrum sharing and interference management
in hyper-dense cloud-RANs. IET Communications, 10(15),
1914–1921.
14. Sharma, R. K., & Rawat, D. B. (2015). Advances on security
threats and countermeasures for cognitive radio networks: A
survey. IEEE Communications Surveys & Tutorials, 17(2),
1023–1043. (Second Quarter) .
15. Wang, B., Wu, Y., Liu, K., & Clancy, T. (2011). An anti-jam-
ming stochastic game for cognitive radio networks. IEEE Journal
on Selected Areas in Communications, 29(4), 877–889.
16. Li, H., & Han, Z. (2010). Dogfight in spectrum: Combating pri-
mary user emulation attacks in cognitive radio systems, part I:
Known channel statistics. IEEE Transactions on Wireless Com-
munications, 9(11), 3566–3577.
17. Oskoui, M. G., Khorramshahi, P., & Salehi, T. A. (2016). Using
game theory to battle jammer in control channels of cognitive
radio ad hoc networks. In Proceedings of the IEEE ICC (pp. 1–5).
18. El-Bardan, R., Brahma, S., & Varshney, P. K. (2016). Strategic
power allocation with incomplete information in the presence of
jammer. IEEE Transactions on Communications, 64(8),
3467–3479.
19. Song, T., Stark, W. E., Li, T., & Tugnait, J. K. (2016). Optimal
multiband transmission under hostile jamming. IEEE Transac-
tions on Communications, 64(9), 4013–4027.
20. Hanawal, M. K., Abdel-Rahman, M. J., & Krunz, M. (2014).
Game theoretic anti-jamming dynamic frequency hopping and
rate adaptation in wireless systems. In Proceedings of the WiOpt
Conference (pp. 247–254).
21. Hanawal, M. K., Abdel-Rahman, M. J., & Krunz, M. (2016).
Joint adaptation of frequency hopping and transmission rate for
anti-jamming wireless systems. IEEE Transactions on Mobile
Computing, 15(9), 2247–2259.
22. Abdel-Rahman, M. J., & Krunz, M. (2014). Game-theoretic
quorum-based frequency hopping for anti-jamming rendezvous in
DSA networks. In Proceedings of the IEEE DYSPAN (pp.
248–258).
23. Sun, Y., Wang, J., Sun, F., & Zhang, J. (2016). Energy-aware
joint user scheduling and power control for two-tier femtocell
networks: A hierarchical game approach. IEEE Systems Journal,.
doi:10.1109/JSYST.2016.2580560.
24. Kang, X., Zhang, R., & Motani, M. (2012). Price-based resource
allocation for spectrum-sharing femtocell networks: A Stackel-
berg game approach. IEEE Journal on Selected Areas in Com-
munications, 30(3), 538–549.
25. Yang, D., Zhang, J., Fang, X., & Richa, A., et al. (2012). Optimal
transmission power control in the presence of a smart jammer. In
Proceedings of the IEEE Globecom (pp. 5506–5511).
26. Yang, D., Xue, G., Zhang, J., Richa, A., & Fang, X. (2013).
Coping with a smart jammer in wireless networks: A stackelberg
game approach. IEEE Transactions on Wireless Communications,
12(8), 4038–4047.
27. Xiao, L., Chen, T., Liu, J., & Dai, H. (2015). Anti-jamming
transmission stackelberg game with observation errors. IEEE
Communications Letters, 19(6), 949–952.
28. Jia, L., Yao, F., Sun, Y., et al. (2016). Bayesian Stackelberg game
for anti-jamming with incomplete information. IEEE Communi-
cations Letters, 20(10), 1991–1994.
29. Tang, X., Ren, P., Wang, Y., et al. (2015). Securing wireless
transmission against reactive jamming: A Stackelberg game
framework. In Proceedings of the IEEE GLOBECOM (pp. 1–6).
30. Xiao, L., Xie, C., Chen, T., et al. (2016). Mobile offloading game
against smart attacks. In Proceedings of the IEEE INFOCOM (pp.
403–408).
31. Xiao, L., Xie, C., Chen, T., et al. (2016). A mobile offloading
game against smart attacks. IEEE Access, 4, 2281–2291.
32. Wu, Q., Xu, Y., Wang, J., et al. (2013). Distributed channel
selection in time-varying radio environment: Interference miti-
gation game with uncoupled stochastic learning. IEEE Transac-
tions on Vehicular Technology, 62(9), 4524–4538.
33. Zheng, J., Cai, Y., Yang, W., et al. (2013). A fully distributed
algorithm for dynamic channel adaptation in canonical commu-
nication networks. IEEE Wireless Communications Letters, 2(5),
491–494.
34. Zheng, J., Cai, Y., Xu, Y., & Anpalagan, A. (2014). Distributed
channel selection for interference mitigation in dynamic envi-
ronment: A game theoretic stochastic learning solution. IEEE
Transactions on Vehicular Technology, 63(9), 4757–4762.
35. Chen, X., Zhang, H., Chen, T., & Lasanen, M.: Improving energy
efficiency in green femtocell networks: a hierarchical reinforce-
ment learning framework. In Proceedings of the IEEE ICC 2013
(pp. 2241–2245).
36. Sun, Y., Shao, H., Liu, X., et al. (2015). Traffic offloading in two-
tier multi-mode small cell networks over unlicensed bands: A
hierarchical learning framework. KSII Transactions on Internet
and Information Systems, 9(11), 4291–4310.
37. Monderer, D., & Shapley, L. S. (1996). Potential games. Games
and Economic Behavior, 14(1), 124–143.
123
 Wireless Netw

38. Sastry, P., Phansalkar, V., & Thathachar, M. (1994). Decentral- Youming Sun received his B.S.
ized learning of Nash equilibria in multi-person stochastic games degree in electronic and infor-
with incomplete information. IEEE Transactions on Systems, mation engineering from Yan-
Man, and Cybernetics, 24(5), 769–777. shan University, Qinhuangdao,
39. Zhong, W., Xu, Y., Tao, M., et al. (2010). Game theoretic mul- China, in 2010 and M.S. degree
timode precoding strategy selection for MIMO multiple access from National Digital Switching
channels. IEEE Signal Processing Letters, 17(6), 563–566. System Engineering and Tech-
40. Zheng, J., Cai, Y., Lu, N., et al. (2015). Stochastic game-theoretic nological Research Center
spectrum access in distributed and dynamic environment. IEEE (NDSC), Zhengzhou, China, in
Transactions on Vehicular Technology, 64(10), 4807–4820. 2013, respectively. He is cur-
41. Xu, Y., Wang, J., Wu, Q., et al. (2012). Opportunistic spectrum rently working toward the Ph.D.
access in unknown dynamic environment: A game-theoretic degree in communications and
stochastic learning solution. IEEE Transactions on Wireless information system in NDSC.
Communications, 11(4), 1380–1391. His research interests include
42. Xu, Y., Xu, Y., & Anpalagan, A. (2015). Database-assisted resource allocation in small cell
spectrum access in dynamic networks: A distributed learning networks, cognitive radio networks, game theory and statistical
solution. IEEE Access, 3, 1071–1078. learning. He currently serves as a regular reviewer for many technical
43. Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine journals, including IEEE Journal on Selected Areas in Communica-
Learning, 8, 279–292. tions, IEEE Systems Journal, IEEE Access, Wireless Networks, IET
44. Kianercy, A., & Galstyan, A. (2012). Dynamics of Boltzmann Communications, KSII Transaction on Internet and Information
Q-learning in two-player two-action games. Physical Review E, Systems. He has acted as Technical Program Committees (TPC)
85(4), 1–10. member for IEEE International Conference on Wireless Communi-
cations and Signal Processing 2015 (WCSP 2015).

Fuqiang Yao received his M.S. Yuhua Xu received his B.S.

degree and Ph.D. degree in degree in Communications
Communications and Electronic Engineering, and Ph.D. degree
Systems from Xidian Univer- in Communications and Infor-
sity, Xi’an, China, in 1990 and mation Systems from College of
1993, respectively. Currently, Communications Engineering,
he is a research fellow at the PLA University of Science and
Nanjing Telecommunication Technology, in 2006 and 2014
Technology Institute, China. respectively. He has been with
His current research interests College of Communications
include wireless communica- Engineering, PLA University of
tions and communication anti- Science and Technology since
jamming. 2012, and currently as an
Assistant Professor. His
research interests focus on
opportunistic spectrum access, learning theory, game theory, and
distributed optimization techniques for wireless communications. He
Luliang Jia received his B.S.
has published several papers in international conferences and reputed
degree in communications
journals in his research area. He served as Associate Editor for Wiley
engineering from Lanzhou
Transactions on Emerging Telecommunications Technologies and
Jiaotong University, Lanzhou,
KSII Transactions on Internet and Information Systems, and the
China, in 2011 and M.S. degree
Guest Editor of Special Issue on ‘‘The Evolution and the Revolution
in Communications and Infor-
of 5G Wireless Communication Systems’’ for IET Communications.
mation Systems from College of
In 2011 and 2012, he was awarded Certificate of Appreciation as
Communication Engineering,
Exemplary Reviewer for the IEEE Communications Letters. He was
PLA University of Science and
selected to receive the IEEE Signal Processing Society’s (SPS) 2015
Technology, Nanjing, China, in
Young Author Best Paper Award, and the Funds for Distinguished
2014, respectively. He is cur-
Young Scholars of Jiangsu Province in 2015.
rently working toward the Ph.D.
degree in College of Commu-
nication Engineering, PLA
University of Science and
Technology. His current research interests include game theory,
learning theory and communication anti-jamming technology.

123
Wireless Netw
Shuo Feng (S’15) received the
B.Sc. degree (Hons.) in electri-
cal engineering from the
University of Electronic Science
and Technology of China,
Chengdu, China, in 2011, and
the M.Sc. degree in communi-
cations and information systems
from the College of Communi-
cations Engineering, PLA
University of Science and
Technology, Nanjing, China, in
2014. He is currently pursuing
the Ph.D. degree in computa-
tional science and engineering
at McMaster University, Hamilton, ON, Canada. His research inter-
ests include cognitive radio networks, machine learning, cognitive
dynamic system, and information geometry. He is an invited reviewer
for several journals such as IEEE Journal on Selected Areas in
Communications, IEEE Communications Letters, etc. He served as
the Technical Program Committee (TPC) member of IEEE WCSP
2015 and WCCN 2015. He was a co-recipient of the Best Paper
Award from IEEE VTC 2014-Fall.
Yonggang Zhu received his
B.S. degree in electrical engi-
neering and Ph.D. degree in
communications engineering
from College of Communica-
tion Engineering, PLA Univer-
sity of Science and Technology,
Nanjing, China, in 2004 and
2009, respectively. From 2011
to 2015, he was a postdoctoral
fellow at Nanjing Telecommu-
nication Technology Research
Institute of CESEC. His current
research interests include wire-
less communication anti-jam-
ming, compressive sampling and adaptive signal processing.
123