to frequency values, the freqToLetter dictionary will map frequency keys to list of

letter
values.
Line 38 creates a blank dictionary. Line 39 loops over all the letters in LETTERS.
The if
statement on line 40 checks if the letter�s frequency (that is,
letterToFreq[letter])
already exists as a key in freqToLetter. If not, then line 41 adds this key with a
list of the
letter as the value. Or else, line 43 appends the letter to the end of the list
that is already at
letterToFreq[letter].
If we continue to use our �Alan Mathison Turing�� example value of letterToFreq
then
freqToLetter would end up looking like this: {1: ['Z'], 2: ['J', 'Q'], 3: ['X'],
135: ['A'], 8: ['K'], 139: ['I'], 140: ['T'], 14: ['V'], 21: ['Y'], 30:
['B', 'W'], 36: ['P'], 37: ['F', 'U'], 39: ['G'], 58: ['D', 'M'], 62:
['L'], 196: ['E'], 74: ['C'], 87: ['H'], 89: ['S'], 106: ['R'], 113:
['O'], 122: ['N']}
The sort() Method�s key and reverse Keyword Arguments
freqAnalysis.py
45. # third, put each list of letters in reverse "ETAOIN" order, and then
46. # convert it to a string
47. for freq in freqToLetter:
48. freqToLetter[freq].sort(key=ETAOIN.find, reverse=True)
49. freqToLetter[freq] = ''.join(freqToLetter[freq])
The third step of getFrequencyOrder() to is sort the letter strings in each list in
freqToLetter in reverse ETAOIN order (as opposed to alphabetical order).
Remember that freqToLetter[freq] will evaluate to a list of letters that have a
frequency
count of freq. A list is used because it�s possible that two or more letters have
the exact same
frequency count, in which case this list will have two-or-more-letters strings in
it.
When multiple letters are tied for frequency, we want these tied letters to be
sorted in the reverse
order that they appear in the ETAOIN string. We need this so that we have a
consistent way of
breaking ties. Otherwise messages with the same letter frequencies might produce
different return
values from getFrequencyOrder()!
For example, if E appears 15 times, D and W appear 8 times each, and H appears 4
times, we
would want them to be sorted as 'EWDH' and not 'EDWH'. This is because while E is
the most
frequent, D and W have the same frequency count but W comes after D in the ETAOIN
string.
Chapter 20 � Frequency Analysis 311
Python�s sort() function can do this sorting for us if we pass it a function or
method for its
key keyword argument. Normally the sort() function simply sorts the list it is
called on into
alphabetical (or numeric) order. However, we can change this by passing the find()
method of
the ETAOIN string as the key keyword argument. This will sort the items in the
freqToLetter[freq] list by the integer returned from the ETAOIN.find() method, that
is, the order that they appear in the ETAOIN string.
Normally the sort() method sorts the values in a list in ascending order (that is,
lowest to
highest or the letter A first and letter Z last). If we pass True for the sort()
method�s
reverse keyword argument, it will sort the items in descending order instead. The
reason we
want to sort the letters in reverse ETAOIN order is so that ties result in lower
match scores in the
englishFreqMatchScore() function rather than higher match scores. (This function is
explained later.)
If we continue using our �Alan Mathison Turing�� example value for freqToLetter,
then
after the loop finishes the value stored in freqToLetter would be: {1: 'Z', 2:
'QJ', 3:
'X', 135: 'A', 8: 'K', 139: 'I', 140: 'T', 14: 'V', 21: 'Y', 30: 'BW', 36:
'P', 37: 'FU', 39: 'G', 58: 'MD', 62: 'L', 196: 'E', 74: 'C', 87: 'H', 89:
'S', 106: 'R', 113: 'O', 122: 'N'}
Notice that the strings for the 30, 37, and 58 keys are all sorted in reverse
ETAOIN order.
Passing Functions as Values
freqAnalysis.py
48. freqToLetter[freq].sort(key=ETAOIN.find, reverse=True)
If you look on line 47, you�ll notice that we are not calling the find() method but
instead using
the find method itself as a value that is passed to the sort() method call. In
Python, functions
themselves are values just like any other values. For example, try typing this into
the interactive
shell:
>>> def foo():
... print('Hello!')
...
>>> bar = foo
>>> bar()
Hello!
In the above code, we define a function named foo() that prints out the string
'Hello!'. But
this is basically defining a function and then storing it in a variable named foo.
Then we copy
312 http://inventwithpython.com/hacking
Email questions to the author: al@inventwithpython.com
the function in foo to the variable bar. This means we can call bar() just like we
can call
foo()! Note that in this assignment statement we do not have parentheses after foo.
If we did,
we would be calling the function foo() and setting bar to its return value. Just
like spam[42]
has the [42] index operating on spam, the parentheses means, �Call the value in foo
as a
function.�
You can also pass functions as values just like any other value. Try typing the
following into the
interactive shell:
>>> def doMath(func):
... return func(10, 5)
...
>>> def adding(a, b):
... return a + b
...
>>> def subtracting(a, b):
... return a - b
...
>>> doMath(adding)
15
>>> doMath(subtracting)
5
>>>
When the function in adding is passed to the doMath() call, the func(10, 5) line is
calling adding() and passing 10 and 5 to it. So the call func(10, 5) is effectively
the
same as the call adding(10, 5). This is why doMath(adding) returns 15.
When subtracting is passed to the doMath() call, func(10, 5) is the same as
subtracting(10, 5). This is why doMath(subtracting) returns 5.
Passing a function or method to a function or method call is how the sort() method
lets you
implement different sorting behavior. The function or method that is passed to
sort() should
accept a single parameter and returns a value that is used to alphabetically sort
the item.
To put it another way: normally sort() sorts the values in a list by the
alphabetical order of the
list values.. But if we pass a function (or method) for the key keyword argument,
then the values
in the list are sorted by the alphabetical or numeric order of the return value of
the function when
the value in the list is passed to that function.
You can think of a normal sort() call such as this:
Chapter 20 � Frequency Analysis 313
someListVariable.sort()
�as being equivalent to this:
def func(x):
return x # sorting based on the value itself
someListVariable.sort(key=func)
So when the sort() method call is passed ETAOIN.find, instead of sorting the
strings like
'A', 'B', and 'C' by the alphabetical order the sort() method sorts them by the
numeric
order of the integers returned from ETAOIN.find('A'), ETAOIN.find('B'), and
ETAOIN.find('C'): that is, 2, 19, and 11 respectively. So the 'A', 'B', and 'C'
strings
get sorted as 'A', 'C', and then 'B' (the order they appear in ETAOIN).
Converting Dictionaries to Lists with the keys(), values(), items()
Dictionary Methods
If you want to get a list value of all the keys in a dictionary, the keys() method
will return a
dict_keys object that can be passed to list() to get a list of all the keys. There
is a similar
dictionary method named values() that returns a dict_values object. Try typing the
following
into the interactive shell:
>>> spam = {'cats': 10, 'dogs': 3, 'mice': 3}
>>> spam.keys()
dict_keys(['mice', 'cats', 'dogs'])
>>> list(spam.keys())
['mice', 'cats', 'dogs']
>>> list(spam.values())
[3, 10, 3]
>>>
Remember, dictionaries do not have any ordering associated with the key-value pairs
they
contain. When getting a list of the keys or values, they will be in a random order
in the list. If you
want to get the keys and values together, the items() dictionary method returns a
dict_items
object that can be passed to list(). The list() function will then return a list of
tuples
where the tuples contain a key and value pair of values. Try typing the following
into the
interactive shell:
>>> spam = {'cats': 10, 'dogs': 3, 'mice': 3}
>>> list(spam.items())
314 http://inventwithpython.com/hacking
Email questions to the author: al@inventwithpython.com
[('mice', 3), ('cats', 10), ('dogs', 3)]
We will be using the items() method in our getFrequencyOrder() function, but you
should know about the keys() and values() methods too. Remember, in order to use
the
return values from these methods as lists, they must be passed to the list()
function. The
list() function then returns a list version of the dict_keys, dict_values, or
dict_items object.
Chapter 20 � Frequency Analysis 315
Sorting the Items from a Dictionary
freqAnalysis.py
51. # fourth, convert the freqToLetter dictionary to a list of tuple
52. # pairs (key, value), then sort them
53. freqPairs = list(freqToLetter.items())
The fourth step of getFrequencyOrder() is to sort the strings from the freqToLetter
dictionary by the frequency count. Remember that the freqToLetter dictionary has
integer
frequency counts for the keys and lists of single-letter strings for the values.
But since dictionaries
do not have an ordering for the key-value pairs in them, we will call the items()
method and
list() function to create a list of tuples of the dictionary�s key-value pairs.
This list of tuples
(stored in a variable named freqPairs on line 53) is what we will sort.
freqAnalysis.py
54. freqPairs.sort(key=getItemAtIndexZero, reverse=True)
The sort() method call is passed the getItemAtIndexZero function value itself. This
means the items in the freqPairs will be sorted by the numeric order of the value
at index 0 of
the tuple value, which is the frequency count integer. Line 54 also passes True for
the reverse
keyword argument so that the tuples are reverse ordered from largest frequency
count to smallest.
If we continue using the previous �Alan Mathison Turing�� example, the value of
freqPairs
will be [(196, 'E'), (140, 'T'), (139, 'I'), (135, 'A'), (122, 'N'), (113,
'O'), (106, 'R'), (89, 'S'), (87, 'H'), (74, 'C'), (62, 'L'), (58,
'MD'), (39, 'G'), (37, 'FU'), (36, 'P'), (30, 'BW'), (21, 'Y'), (14,
'V'), (8, 'K'), (3, 'X'), (2, 'QJ'), (1, 'Z')]
freqAnalysis.py
56. # fifth, now that the letters are ordered by frequency, extract all
57. # the letters for the final string
58. freqOrder = []
59. for freqPair in freqPairs:
60. freqOrder.append(freqPair[1])
The fifth step is to create a list of all the strings from the sorted list in
freqPairs. The variable
freqOrder will start as a blank list on line 58, and the string at index 1 of the
tuple in
freqPairs will be appended to the end of freqOrder.
If we continue with the �Alan Mathison Turing was a British mathematician...�
example from
before, after this loop has finished, freqOrder will contain the value ['E', 'T',
'I',
316 http://inventwithpython.com/hacking
Email questions to the author: al@inventwithpython.com
'A', 'N', 'O', 'R', 'S', 'H', 'C', 'L', 'MD', 'G', 'FU', 'P', 'BW', 'Y',
'V', 'K', 'X', 'QJ', 'Z']
freqAnalysis.py
62. return ''.join(freqOrder)
Line 62 creates a string from the list of strings in freqOrder by joining them
together with the
join() method. If we continue using the previous exampleto frequency values, the
freqToLetter dictionary will map frequency keys to list of letter
values.
Line 38 creates a blank dictionary. Line 39 loops over all the letters in LETTERS.
The if
statement on line 40 checks if the letter�s frequency (that is,
letterToFreq[letter])
already exists as a key in freqToLetter. If not, then line 41 adds this key with a
list of the
letter as the value. Or else, line 43 appends the letter to the end of the list
that is already at
letterToFreq[letter].
If we continue to use our �Alan Mathison Turing�� example value of letterToFreq
then
freqToLetter would end up looking like this: {1: ['Z'], 2: ['J', 'Q'], 3: ['X'],
135: ['A'], 8: ['K'], 139: ['I'], 140: ['T'], 14: ['V'], 21: ['Y'], 30:
['B', 'W'], 36: ['P'], 37: ['F', 'U'], 39: ['G'], 58: ['D', 'M'], 62:
['L'], 196: ['E'], 74: ['C'], 87: ['H'], 89: ['S'], 106: ['R'], 113:
['O'], 122: ['N']}
The sort() Method�s key and reverse Keyword Arguments
freqAnalysis.py
45. # third, put each list of letters in reverse "ETAOIN" order, and then
46. # convert it to a string
47. for freq in freqToLetter:
48. freqToLetter[freq].sort(key=ETAOIN.find, reverse=True)
49. freqToLetter[freq] = ''.join(freqToLetter[freq])
The third step of getFrequencyOrder() to is sort the letter strings in each list in
freqToLetter in reverse ETAOIN order (as opposed to alphabetical order).
Remember that freqToLetter[freq] will evaluate to a list of letters that have a
frequency
count of freq. A list is used because it�s possible that two or more letters have
the exact same
frequency count, in which case this list will have two-or-more-letters strings in
it.
When multiple letters are tied for frequency, we want these tied letters to be
sorted in the reverse
order that they appear in the ETAOIN string. We need this so that we have a
consistent way of
breaking ties. Otherwise messages with the same letter frequencies might produce
different return
values from getFrequencyOrder()!
For example, if E appears 15 times, D and W appear 8 times each, and H appears 4
times, we
would want them to be sorted as 'EWDH' and not 'EDWH'. This is because while E is
the most
frequent, D and W have the same frequency count but W comes after D in the ETAOIN
string.
Chapter 20 � Frequency Analysis 311
Python�s sort() function can do this sorting for us if we pass it a function or
method for its
key keyword argument. Normally the sort() function simply sorts the list it is
called on into
alphabetical (or numeric) order. However, we can change this by passing the find()
method of
the ETAOIN string as the key keyword argument. This will sort the items in the
freqToLetter[freq] list by the integer returned from the ETAOIN.find() method, that
is, the order that they appear in the ETAOIN string.
Normally the sort() method sorts the values in a list in ascending order (that is,
lowest to
highest or the letter A first and letter Z last). If we pass True for the sort()
method�s
reverse keyword argument, it will sort the items in descending order instead. The
reason we
want to sort the letters in reverse ETAOIN order is so that ties result in lower
match scores in the
englishFreqMatchScore() function rather than higher match scores. (This function is
explained later.)
If we continue using our �Alan Mathison Turing�� example value for freqToLetter,
then
after the loop finishes the value stored in freqToLetter would be: {1: 'Z', 2:
'QJ', 3:
'X', 135: 'A', 8: 'K', 139: 'I', 140: 'T', 14: 'V', 21: 'Y', 30: 'BW', 36:
'P', 37: 'FU', 39: 'G', 58: 'MD', 62: 'L', 196: 'E', 74: 'C', 87: 'H', 89:
'S', 106: 'R', 113: 'O', 122: 'N'}
Notice that the strings for the 30, 37, and 58 keys are all sorted in reverse
ETAOIN order.
Passing Functions as Values
freqAnalysis.py
48. freqToLetter[freq].sort(key=ETAOIN.find, reverse=True)
If you look on line 47, you�ll notice that we are not calling the find() method but
instead using
the find method itself as a value that is passed to the sort() method call. In
Python, functions
themselves are values just like any other values. For example, try typing this into
the interactive
shell:
>>> def foo():
... print('Hello!')
...
>>> bar = foo
>>> bar()
Hello!
In the above code, we define a function named foo() that prints out the string
'Hello!'. But
this is basically defining a function and then storing it in a variable named foo.
Then we copy
312 http://inventwithpython.com/hacking
Email questions to the author: al@inventwithpython.com
the function in foo to the variable bar. This means we can call bar() just like we
can call
foo()! Note that in this assignment statement we do not have parentheses after foo.
If we did,
we would be calling the function foo() and setting bar to its return value. Just
like spam[42]
has the [42] index operating on spam, the parentheses means, �Call the value in foo
as a
function.�
You can also pass functions as values just like any other value. Try typing the
following into the
interactive shell:
>>> def doMath(func):
... return func(10, 5)
...
>>> def adding(a, b):
... return a + b
...
>>> def subtracting(a, b):
... return a - b
...
>>> doMath(adding)
15
>>> doMath(subtracting)
5
>>>
When the function in adding is passed to the doMath() call, the func(10, 5) line is
calling adding() and passing 10 and 5 to it. So the call func(10, 5) is effectively
the
same as the call adding(10, 5). This is why doMath(adding) returns 15.
When subtracting is passed to the doMath() call, func(10, 5) is the same as
subtracting(10, 5). This is why doMath(subtracting) returns 5.
Passing a function or method to a function or method call is how the sort() method
lets you
implement different sorting behavior. The function or method that is passed to
sort() should
accept a single parameter and returns a value that is used to alphabetically sort
the item.
To put it another way: normally sort() sorts the values in a list by the
alphabetical order of the
list values.. But if we pass a function (or method) for the key keyword argument,
then the values
in the list are sorted by the alphabetical or numeric order of the return value of
the function when
the value in the list is passed to that function.
You can think of a normal sort() call such as this:
Chapter 20 � Frequency Analysis 313
someListVariable.sort()
�as being equivalent to this:
def func(x):
return x # sorting based on the value itself
someListVariable.sort(key=func)
So when the sort() method call is passed ETAOIN.find, instead of sorting the
strings like
'A', 'B', and 'C' by the alphabetical order the sort() method sorts them by the
numeric
order of the integers returned from ETAOIN.find('A'), ETAOIN.find('B'), and
ETAOIN.find('C'): that is, 2, 19, and 11 respectively. So the 'A', 'B', and 'C'
strings
get sorted as 'A', 'C', and then 'B' (the order they appear in ETAOIN).
Converting Dictionaries to Lists with the keys(), values(), items()
Dictionary Methods
If you want to get a list value of all the keys in a dictionary, the keys() method
will return a
dict_keys object that can be passed to list() to get a list of all the keys. There
is a similar
dictionary method named values() that returns a dict_values object. Try typing the
following
into the interactive shell:
>>> spam = {'cats': 10, 'dogs': 3, 'mice': 3}
>>> spam.keys()
dict_keys(['mice', 'cats', 'dogs'])
>>> list(spam.keys())
['mice', 'cats', 'dogs']
>>> list(spam.values())
[3, 10, 3]
>>>
Remember, dictionaries do not have any ordering associated with the key-value pairs
they
contain. When getting a list of the keys or values, they will be in a random order
in the list. If you
want to get the keys and values together, the items() dictionary method returns a
dict_items
object that can be passed to list(). The list() function will then return a list of
tuples
where the tuples contain a key and value pair of values. Try typing the following
into the
interactive shell:
>>> spam = {'cats': 10, 'dogs': 3, 'mice': 3}
>>> list(spam.items())
314 http://inventwithpython.com/hacking
Email questions to the author: al@inventwithpython.com
[('mice', 3), ('cats', 10), ('dogs', 3)]
We will be using the items() method in our getFrequencyOrder() function, but you
should know about the keys() and values() methods too. Remember, in order to use
the
return values from these methods as lists, they must be passed to the list()
function. The
list() function then returns a list version of the dict_keys, dict_values, or
dict_items object.
Chapter 20 � Frequency Analysis 315
Sorting the Items from a Dictionary
freqAnalysis.py
51. # fourth, convert the freqToLetter dictionary to a list of tuple
52. # pairs (key, value), then sort them
53. freqPairs = list(freqToLetter.items())
The fourth step of getFrequencyOrder() is to sort the strings from the freqToLetter
dictionary by the frequency count. Remember that the freqToLetter dictionary has
integer
frequency counts for the keys and lists of single-letter strings for the values.
But since dictionaries
do not have an ordering for the key-value pairs in them, we will call the items()
method and
list() function to create a list of tuples of the dictionary�s key-value pairs.
This list of tuples
(stored in a variable named freqPairs on line 53) is what we will sort.
freqAnalysis.py
54. freqPairs.sort(key=getItemAtIndexZero, reverse=True)
The sort() method call is passed the getItemAtIndexZero function value itself. This
means the items in the freqPairs will be sorted by the numeric order of the value
at index 0 of
the tuple value, which is the frequency count integer. Line 54 also passes True for
the reverse
keyword argument so that the tuples are reverse ordered from largest frequency
count to smallest.
If we continue using the previous �Alan Mathison Turing�� example, the value of
freqPairs
will be [(196, 'E'), (140, 'T'), (139, 'I'), (135, 'A'), (122, 'N'), (113,
'O'), (106, 'R'), (89, 'S'), (87, 'H'), (74, 'C'), (62, 'L'), (58,
'MD'), (39, 'G'), (37, 'FU'), (36, 'P'), (30, 'BW'), (21, 'Y'), (14,
'V'), (8, 'K'), (3, 'X'), (2, 'QJ'), (1, 'Z')]
freqAnalysis.py
56. # fifth, now that the letters are ordered by frequency, extract all
57. # the letters for the final string
58. freqOrder = []
59. for freqPair in freqPairs:
60. freqOrder.append(freqPair[1])
The fifth step is to create a list of all the strings from the sorted list in
freqPairs. The variable
freqOrder will start as a blank list on line 58, and the string at index 1 of the
tuple in
freqPairs will be appended to the end of freqOrder.
If we continue with the �Alan Mathison Turing was a British mathematician...�
example from
before, after this loop has finished, freqOrder will contain the value ['E', 'T',
'I',
316 http://inventwithpython.com/hacking
Email questions to the author: al@inventwithpython.com
'A', 'N', 'O', 'R', 'S', 'H', 'C', 'L', 'MD', 'G', 'FU', 'P', 'BW', 'Y',
'V', 'K', 'X', 'QJ', 'Z']
freqAnalysis.py
62. return ''.join(freqOrder)
Line 62 creates a string from the list of strings in freqOrder by joining them
together with the
join() method. If we continue using the previous example