REVERSING ANDROID MARKET PROTOCOL

1
INDEX
INDEX...................................................................................................................................2
PREFACE ..............................................................................................................................3
INTRODUCTION..................................................................................................................4
HTTPS REQUESTS...........................................................................................................4
POST REQUESTS .............................................................................................................5
GET REQUEST .................................................................................................................6
DECODING PROTOCOL BUFFERS....................................................................................7
REQUESTS ...........................................................................................................................9
FEATURED REQUESTS ................................................................................................10
SEARCH REQUEST........................................................................................................11
ICON REQUEST .............................................................................................................12
RESPONSES .......................................................................................................................13
SUMMARY .........................................................................................................................14

2
PREFACE
This project started as my frustration of not having unlimited data plan. So instead of using
money to download software from android market i wanted to download it to my PC. I also
found out that on my phone a lot of free software doesnt show up, and non free programs
doesnt show up at all.

I've been googling and couldnt find any good information about market application. Also i
was unable to find desktop client. I decided i'll take a look, and try to reverse the protocol.
As it turned out it was pretty easy.

3
INTRODUCTION
First i followed instructions posted at to install emulator with market. I used wireshark to
capture packets. Start wireshark and then run emulator, with wipe user data option on. Sign in
with your gmail account and then start market. Finaly do a search, select one of the results and
download it. You might get packet capture similar to this:

192.168.123.175 209.85.135.102 HTTPS traffic

192.168.123.175 209.85.135.102 HTTPS traffic
192.168.123.175 209.85.135.102 HTTPS traffic

192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1

192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1
192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1
192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1
192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1
192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1
192.168.123.175 209.85.135.102 HTTP POST /market/api/ApiRequest HTTP/1.1

192.168.123.175 209.85.135.102 HTTP GET

/market/download/Download?assetId=10000291265&userId=70745944&deviceId=12163409

I divided results in 3 sections. First section is HTTPS traffic, probably used to login to gmail
account. Second section are various HTTP POST requests to market API. These are used to
obtain featured products, do searches, get pictures, comments. Finaly there's one HTTP GET
request used to download the actual program.

HTTPS REQUESTS

We are unable to analyze HTTPS request as all the traffic is encrypted. I tried various
techiques of sniffing HTTPS, but loging in to google fails with message »Can't establish a
reliable data connection to the server«.

UPDATE: I've found website where you can find information about one of this HTTPS
requests. Its a simple request which sends your username and password to gmail, and as result
recieves the authentication token which we'll see used later.

4
POST REQUESTS

POST /market/api/ApiRequest HTTP/1.1

Content-Length: 645
Content-Type: application/x-www-form-urlencoded
Host: android.clients.google.com
Connection: Keep-Alive
User-Agent: Android-Market/2 (dream DRC83); gzip

version=2&request=CskCCuABRFFBQUFKWUFBQUFEbnZvalN1eWNCUl9KandnMFFRa
29BeFl6TjBhbUktajlKWWkxSVVjVnFvUnFfSWJESDQwT1pFWlNUOFFlTHBOSzZvaGt
WczR3cGlsR3lHT2MxeU04dWxuc2dNOTVuTVlRdi04OEZTdDJJMlVyeTRMbDZibDF0V
2p5S2M1TVlKSkNUc1l1dTl2OWQxSXNTendreVZ3WGNFNVFGYUhSXzNoLWNCVzk4
WEhsSmtrLUJLQzRna2NzaGRoR3YtT19xSEhMcE9KbW5HbWh2dWlwMV9QYzFBN1A
QABjUDCIPMWIwMjE5MGJkMjg5ODJkKgdkcmVhbTo0MgJlbjoCVVNCB0FuZHJvaWR
KB0FuZHJvaWRSBjMxMDI2MFoGMzEwMjYwYglhbS1nb29

If we look at the details of the POST requests, we can see, that we have 2 variables posted.
First one is version, which is fixed. Second is request. Its base64 encoded, so to analyze data
we first need to decode it. Also important is user agent, which tell the server that we are
android market, and requests gzip compression of content.

If we decode the base64 encoded string we might get something similar to this:

É
a
DQAAAJUAAADC-
KGBxuN2BDgPcG6c8g5jcyYVTq31zoTiooQV32mnuGe3wqvqyjF47qZa8wjLODg15ql3PC
5j1qrNp8jsHxs_tBIRWIBqPzy7DNfY-
8imcwQQLg31_NuTbl0rhuHfDYxD3itE1CS64TzkuxcQR8aWydPus40Lc_Gs09685tqTxjU
NONC1zmvPkNjxjq6AA1qWe ? Ô" 1b020909d28982d* dream:42 en: USB Androi
dJ AndroidR 310260Z 310260b am-googlej 560098a6f4b1925 "
  ftp0
@?H

We can see there are several string inside, but data seems to be using some kind of binary
encoding. Becouse this is google service, its not hard to guess, that they are using ther
Protocol Buffers. In next section we will see how easy it is to decode them.

Also if we ungzip reponse, we find similary structured data, so response is constructed using
protocol buffers as well.

5
GET REQUEST

GET
/market/download/Download?assetId=10075565&userId=7012944&deviceId=195471564845

Cookie: ANDROID=DQAAAJYAAAADnvojSuycBR_Jjwg0QQkoAxYzN0amI-
j9JYi1IUcVqoRq_IbDH40OZ88FSt2I2Ury4Ll6bl1tWjyKc5MYJJCTsYuu9v9d1IsSzwkyVw
XcE5QFaHR_3h-cBW98XHlJkk-BKC4gkcshdhGv-O_qHHLpOJmnGmhvuip1_Pc1A7P

Host: android.clients.google.com
Connection: Keep-Alive
User-Agent: AndroidDownloadManager

Here we can see, that along GET variables there's also cooking with authentication token
being sent. AssetId is id of the program we are trying to download. deviceId is unimportant,
we can change it and we will still get valid responses.

6
DECODING PROTOCOL BUFFERS
You can get information about protocol buffers on google's project website. Specialy
interesting is this document:
http://code.google.com/intl/sl/apis/protocolbuffers/docs/encoding.html

Protocol Buffer message is sturctured in following way:

message Test1 {
required int32 a = 1;
}

We specify the data type, its name and its tag. When we convert this to byte code, we only tell
the wire type and tag. Lower 3 bits tell us the wire type, which can be one of the following:
Type Meaning Used For
0 Varint int32, int64, uint32, uint64, bool, enum
1 64-bit fixed64, sfixed64, double
2 Length-delimited string, bytes, embedded messages
3 Start group groups (deprecated)
4 End group groups (deprecated)
5 32-bit fixed32, sfixed32, float

The other bytes tell us tag number. For ta numbers from 1 to 16 one byte is used, for larger
tag numbers more bytes are used. (read about varints in project documentation)

Type 2 is the most interesting one, as it can represent embeded message.

In this case, bytes identifying embeded message are like in all length delimited fields followed
by field length. Next embeded message is started. This would be the only hard thing to create
using automated tool, as its hard to know wether type 2 is normal string or embeded message.
Message Test2 {
required string a = 1 [default=«hello«];
required int32 b = 2 [default=1];
}

message Test1 {
required int32 a = 1 [default=5];
group NewGroup = 2 {
required int32 a_ingroup = 1 [default=4];
required string b = 2 [default=«hello1«];
required Test2 t = 3;
}

If we try to convert this .proto message to byte code (using default values) we get:

08 05 [08 represents 1000 (tag = 1, type = 0), 05 is the value of a]

13 [13 represents 10011 (tag = 2, type = 3), start of group]
08 04 [08 represents 1000 (tag = 1, type = 0), 04 is the value of a_ingroup]
12 06 48 45 4C 4C 50 31 [12 represents 10010 (tag = 2, type = 2), 06 is length of string
followed, and after that we have the message hello1 ]
1A 09 [1A represents 11010 (tag 3, type = 2), 09 is length of nested message]
0A 05 48 45 4C 4C 50 [0A represents 1010 (tag = 1, type = 2), 05 is length of string
Followed by string hello. ]

7
10 01 [10 represents 10000 (tag = 2, type = 0), 01 is value of b]
14 [14 represents 10100 (tag = 2, type = 4), end of group]

In next chapters we will simply try to reverse the proccess. We'll take a look at the binary
data, and try to recreate the .proto file, with which we can generate java, php, c#, c++, ...
classes to read and write our data.

8
REQUESTS
All requests are sent using 1 protobuffer message. We can notice great similarity among them.
They all start with some kind of header, followed by some data, which tells the server what do
we want.

Lets take a look at this request. It starts with 0A (1010, tag=1, type=2). Next we see byte CA
(11001010). As first bit is set, we know we must read another byte (varints). So we read 02,
and those two bytes together gives us the length. Next we have another 0A ( tag=1, type=2).
Its followed by its length E0 01, and then by some long string.

So lets try to write down what we have till now. One nested message, with a string inside.

message nestedMessage {
required string authentication=1;
}

message mainMessage {
required nestedMessage msg=1;
}

Now we must actualy count all string bytes, so we can know where string ends. If we convert
E0 01 from varint, we get result E0. String starts at address 06, E0 + 06 = E6. At address E6
we find bytes: 10 00. This is another varint (tag=2, type=0, value=0). However the question
is, wether this varint belongs to mainMessage or to the nestedMessage. We can see that length
of nestedMessage is CA 02, converted to number (from varint) we get 14A. 03 + 14A = 14D.
So it still belongs to the nested message.

9
We can update nestedMessage (lets call it header from now on) with all the data we read until
address 14D:

message header {
required string authentication
required int32 x1 = 2 [default = 0];
required int32 x2 = 3 [default = 1620];
required string serial1 = 4;
required string phone = 5[default = "dream:4"];
required string lang = 6 [default = "en"];
required string locale = 7 [default = "US"];
required string s1 = 8 [default = "Android"];
required string s2 = 9 [default = "Android"];
required string s3 = 10 [default = "310260"];
required string s4 = 11 [default = "310260"];
required string s5 = 12 [default = "am-google"];
required string serial2 = 13;
}

If we continue our analysis on addredd 14D. There is byte 13 there. We look it up and get
tag=2, type=3. Type 3 is start group. So we can continue with our mainMessage in this way:

message mainMessage {
required header myheader = 1;
required group myGroup = 2 {

}
}

Our group expands until the end group byte (14). This is basic structure of the request packet.,
which all requests share. We will look into more details for different requests now.

FEATURED REQUESTS

This request, like all starts with header. Then its followed by myGroup with string, type=2,
tag=21. Be carefull, when tag numbers are higher than 16, two bytes are used for them. So in
the above hex code: AA 01 02 08 01 this says: AA 01 (tag=21, type=2) 02 (length = 2) value
= 08 01. After that we have 14 (end group byte).

But right after it we have another myGroup. So we have to change definition of myGroup
above from required to repeated.

Here is complete structure of featured request:

repeated group myGroup = 2 {

required string string1 = 21;
required string string2 = 22;
required query query1 = 2;
}

10
 mainMessage {
header
myGroup {
string1 = »\0x08\0x01«;
}
myGroup {
string2 = »«;
}
myGroup {
query1 = nestedMessage2;
}
}

nestedMessage2 {
int1 = 1;
int2 = 3;
int3 = 0;
int4 = 30
}

We will call this nestedMessage2 searchrequest. Int4 is the number of results we want, int3
tells the server if we also want non-free results. I'm not sure about int1 and int2, nor am i
about string1 and string2 (in above groups), so we just leave that data intact.

SEARCH REQUEST

Our search request looks like this :

mainMessage {
header;
myGroup {
query1 = nestedMessage2;
}
}

nestedMessage2 {
string1 = »what we search for«;
int1 = 1;
int3 = 0;
int4 = 30;
}

We tell it what to search for, if we want non free results or not, and how many results we
want.

11
ICON REQUEST

mainMessage {
header;
myGroup {
icon = iconNestedMsg;
}
myGroup { .... one for each icon }
}

iconNestedMsg {
id = »idofourproduct«;
int1 = 0;
string1 = »0«
}

12
RESPONSES
I wont go in any detail about responses. You can just use .proto files found with this project.

13
SUMMARY
Based on information obtained i created simple demo application. Its a desktop client for
android market. You can take a look at .proto files in the archieve and at the source code for
further information.

This work is not complete. There are still a lot of unknown fields in requests which i know
nothing about, but we have enought information to do searching and downloading.

14