Shell

and SQL Baseball and airport analysis

Austin Kinion
Part 1 (i): Compute the number of outbound flights for each of the five airports OAK, SMF, LAX,

SFO and JFK, and sort these counts from largest to smallest.

Done in Shell
#Get the data from the website
$ curl -O http://eeyore.ucdavis.edu/stat141/Data/Airline2012_13.tar.gz
#output for the amount of time it took to get it
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 291M 100 291M 0 0 8803k 0 0:00:33 0:00:33 --:--:-
- 7512k

so it took 33 seconds to get the data from the website in Shell

#Grab each of the months csv files
tar -zxvf Airline2012_13.tar.gz
#Create a file named ORGIN.txt with all of the origin columns pulled
out for each month
time cut -d , -f 15 201[23]_*.csv | sort | uniq -c | sort -nr >
ORIGIN.txt

The time it took was: user 1m55.666s

#Grab the names of the five airports out of the ORGIN file and count
for each airport
# and orer the counts Then put into new file ORIGIN2.txt
time egrep 'OAK|SMF|LAX|SFO|JFK' ORIGIN.txt > ORIGIN2.txt

The time it took was: real 0m0.004s

So that's a total time of about 2 minutes for the shell.

Taking a look at the file created from shell:
File: ORIGIN2.txt:
222029 "LAX"
169734 "SFO"
105097 "JFK"
44911 "OAK"
43145 "SMF"

Part 1 (i): Done in R

setwd("~/Desktop/Airline2012_13")

list.files()
#concatinate all five airport names into variable: airports
airports = c("LAX", "SFO", "JFK", "OAK", "SMF")

#call variable origincounts a blank vector with five 0's to fill in

later
origincounts = numeric(5)

#Use filenames to later iterate over all files

filenames = list.files()

#loop over all csv files and pull out origin column
system.time(for (i in 1:length(filenames))
{
#Read in csv files 1:12
cur.csv = read.csv(filenames[i])
#Create a table with the origin counts for each airport
origintable = table(cur.csv$ORIGIN)
#Create a table with just the five airports that were asked for
origincounts = origincounts + origintable[airports]
})

The output matches that of the output from shell, so that is a very good thing.
#R output to show it matches the shell output:
origincounts
#LAX SFO JFK OAK SMF
#222029 169734 105097 44911 43145

And the time it took was:

# user system elapsed
# 438.322 15.523 458.014

So, if we use the user time, that is about 7.3 minutes, which is over 3 times as long as
the time it took in shell. So I have determined, that when you need to grep a few
items out of a very large database, that It is faster to do it in Shell over R.
Part 1 (ii) Compute the total number of flights in and out of the five airports, i.e., the sume of both

the inbound and outbound flights. You can do this however you want using a mix of the shell and
R code. One way is to first obtain the lines in the files which involve any of these five airports.
Then obtain a count for each pair of airports, i.e., ORIGIN, DESTINATION pairs. At most, how
many will there be? Then read these counts by ORIGIN, DESTINATION pairs into R and
compute the total number of flights for each of the 5 airports.

First, the code that was done in shell:

#This grabs each of the airport names out of all the csv files from the
data:
egrep 'OAK|SMF|LAX|SFO|JFK' 201[23]*.csv > 12_13.txt
#Then this takes the content from the file created above and
#pull outs both the origin and destination column, and sorts the counts
cut -d , -f 15,25 12_13.txt | sort | uniq -c | sort -nr >
ORIGIN_DEST.txt

So the counts for all five airports, whether a destination or an origin are all in the file
above. I will now go use R to finish cleaning up the data and get the counts for each
of the five airports
setwd("~/")
data=readLines("ORIGIN_DEST.txt")

#Grab each of the 3 letter codes for the airports out of the file
LAX= data[grep("LAX", data)]
JFK=data[grep("JFK", data)]
SMF=data[grep("SMF", data)]
SFO=data[grep("SFO", data)]
OAK=data[grep("OAK", data)]

#Name regular expression to make it easier for coding later

regex='([^0-9])'

#Substitute all the nonsense with nothing to just obtain the numbers
LAX_NUM=gsub(regex, '', LAX)
#Sum up those numbers obtained
LAX_TOTAL=sum(as.numeric(LAX_NUM))

#Do the same thing as above for the rest of the airports
JFK_NUM=gsub(regex, '', JFK)
JFK_TOTAL=sum(as.numeric(JFK_NUM))

OAK_NUM=gsub(regex, '', OAK)

OAK_TOTAL=sum(as.numeric(OAK_NUM))

SMF_NUM=gsub(regex, '', SMF)

SMF_TOTAL=sum(as.numeric(SMF_NUM))

SFO_NUM=gsub(regex, '', SFO)

SFO_TOTAL=sum(as.numeric(SFO_NUM))

#List each of the airports with their sums from above

A= list(LAX=LAX_TOTAL, SFO=SFO_TOTAL, JFK=JFK_TOTAL, OAK=OAK_TOTAL,
SMF=SMF_TOTAL)
#Make result from above into data frame for readability
result_air=as.data.frame(A)

And the output for this is:

LAX SFO JFK OAK SMF
444080 339463 210175 89820 86293

Baseball with SQL
Number 1
What years does the data cover? are there data for each of these years?
#Find the years in which the data ranges, This function is from Nick's
OH's
year_range= function(tbl, db){

query= 'SELECT yearID FROM '
query = paste0(query, tbl, ';')
cat(paste0('Query:', query, '\n'))
# Use tryCatch() to catch errors.
tryCatch(dbGetQuery(db, query),
error = function(e) NULL)
}

tables= dbListTables(db)

years = lapply(tables, year_range, db)
u= unlist(years)

first_year = min(u)
last_year = max(u)

So the data begins in the year:

>first_year
1871

And ends in the Year:

>last_year
2013 #So the data ranges from 1871-2013

Are there data for al the years?

#If returns TRUE, then yes:

all(seq(min(u), max(u)) %in% u)
[1] TRUE

So there are data for each of the years in the Set

Number 2
How many (unique) people are included in the database? How many are players, managers, etc?
#help from Piazza was given to make sure count was correct
number_unique= function(tbl, db){

query= 'SELECT playerID FROM '
query = paste0(query, tbl, ';')
cat(paste0('Query:', query, '\n'))
# Use tryCatch() to catch errors.
tryCatch(dbGetQuery(db, query),
error = function(e) NULL)
}
uni_people = lapply(tables, number_unique, db)
playerIDs = unlist(people)
unique_people = length(unique(playerIDs))
unique_people
managers = dbGetQuery(db, 'SELECT COUNT(DISTINCT playerID) FROM
Managers')
managers
Number_players = unique_people-managers

Found 18359 unique in the set, 682 Managers, and 17677 players. Some players
were managers at some point, and found that there might be some overlap in the
tables, but I believe that this is a very good estimate of the number of players and
managers from all of the years.
Number 3
What team won the World Series in 2000?
Win_2000= dbGetQuery(db, "SELECT name FROM Teams WHERE WSWin = 'Y' and
yearID = '2000';")
>Win_2000
name
1 New York Yankees

The winner of the World Series in 2000 was the New York Yankees
Number 4What team lost the World Series each year?

World_Series_Losers = dbGetQuery(db, "SELECT yearID, name FROM Teams

WHERE LGWin = 'Y' and WSWin = 'N' GROUP BY yearID;")

For the sake of saving paper, I will just show the first 5 years of world series losers
and the last five years:
yearID name
1 1884 New York Metropolitans
2 1885 Chicago White Stockings
3 1886 Chicago White Stockings
4 1887 St. Louis Browns
5 1888 St. Louis Browns
... ... ...
112 2009 Philadelphia Phillies
113 2010 Texas Rangers
114 2011 Texas Rangers
115 2012 Detroit Tigers
116 2013 St. Louis Cardinal
Number 5

Do you see a relationship between the number of games won in a season and winning the World
Series?
#I recieved a lot of help from Charles on this problem.
World_Series_Winners = dbGetQuery(db, "SELECT WSWin,teamID,W, yearID
FROM Teams WHERE LGWin = 'Y' AND WSWin = 'Y' GROUP BY TeamID;")

World_Series_win= World_Series_Winners[,3]
World_Series_year= World_Series_Winners[,4]

plot(World_Series_year,World_Series_win, xlab="Year", ylab="Number of
Wins")

World_Series_Losers = dbGetQuery(db, "SELECT WSWin,teamID,W, yearID
FROM Teams WHERE WSWin = 'N' GROUP BY TeamID;")
World_Series_win2= World_Series_Losers[,3]
World_Series_year2= World_Series_Losers[,4]

plot(World_Series_year2,World_Series_win2, xlab="Year", ylab="Number of

Wins")

So there does seem to be a relationship for numbers of wins and winning the world
series, since the plots differ substantially. It is apparent that there is a higher
number of wins for the teams who won the world series, so therefore they differ.
just to make sure that My plot reading skills are correct, I computer the mean and
median for the number f wins in each catagory:
median(World_Series_win)
95
mean(World_Series_win)
95.16
median(World_Series_win2)
70
mean(World_Series_win2)
67

So this just clarifies my point from above that three are a higher number of wins for
the teams who won the world series.
Number 6

In 2003, what were the three highest salaries?

high_salary = dbGetQuery(db, "SELECT salary FROM Salaries WHERE yearID
= '2003' ORDER BY salary DESC limit 3; ")

>high_salary
salary
1 22000000
2 20000000
3 18700000

So the three highest slaries are: $22,000,000, $20,000,000, and $18,700,000.

Number 7

For 1999, compute the total payroll of each of the different teams. Next compute the team
payrolls for all years in the database for which we have salary information.
Payroll_99= dbGetQuery(db, "SELECT teamID,sum(salary) FROM Salaries
WHERE yearID='1999' GROUP BY teamID;")

So the payroll for the year 1999 for each of the teams is:
>Payroll_99
teamID sum(salary)
1 ANA 55388166
2 ARI 68703999
3 ATL 73140000
4 BAL 80605863
5 BOS 63497500
6 CHA 25620000
7 CHN 62343000
8 CIN 33962761
9 CLE 72978462

10 COL 61935837
11 DET 36489666
12 FLO 21085000
13 HOU 54914000
14 KCA 26225000
15 LAN 80862453
16 MIL 43377395
17 MIN 21257500
18 MON 17903000
19 NYA 86734359
20 NYN 65092092
21 OAK 24431833
22 PHI 31692500
23 PIT 24697666
24 SDN 49768179
25 SEA 54125003
26 SFN 46595057
27 SLN 49778195
28 TBA 38870000
29 TEX 76709931
30 TOR 45444333
``````````````````````````````````````````````````````````````````
Payroll= dbGetQuery(db, "SELECT teamID,sum(salary), yearID FROM
Salaries GROUP BY teamID, yearID;")

For the sake of paper, I will just display the first 5 salaries from th first five teams,
and the last 5:

teamID sum(salary) yearID

1 ATL 14807000 1985
2 BAL 11560712 1985
3 BOS 10897560 1985
4 CAL 14427894 1985
5 CHA 9846178 1985
... ... ... ...
824 SLN 92260110 2013
825 TBA 52955272 2013
826 TEX 112522600 2013
827 TOR 126288100 2013
828 WAS 113703270 2013

Number 8

Study the change in salary over time. Have salaries kept up with inflation, fallen behind, or grown
faster?

#bring all dollars to 1985 dollars.
CPI=read.table("CPI.txt")

#Multiply the cpi values by salary and then plot.
#Much help From Nick was recieved to answer this question
year_salary=dbGetQuery(db, "SELECT yearID, sum(salary) AS salary FROM
Salaries GROUP BY yearID")

CPI2=year_salary/CPI
CPI2=t(CPI2) #Take the transpose so it will work
year=as.vector(year$yearID) #For the lines statement

plot(year_salary, type='l', col="blue", lwd=2.5, xlab="Year",
ylab="Salary")
lines(year,CPI2, type='l',col="red", lwd=2.5)
legend(1985,3.0e+09, c("Salary","Salary with no inflation"), # puts
text in the legend
lty=c(1,1), # gives the legend appropriate symbols (lines)
lwd=c(1,1),col=c("blue","red"), cex=.55)


Above is a plot which shows the increase in salary since 1985 (in blue), and the
increase in salary if the salary were computed in 1985 dollars (in red). Another way
of exapling the red line is that it is the salary with no inflation rate. It is clear that the
salary for MLB players in increasing much faster than inflation, so players are
getting paid a lot more money than they did in 1985.
Number 9
Compare payrolls for the teams that are in the same leagues, and then in the same divisions. Are
there any interesting characteristics? Have certain teams always had top payrolls over the years?
Is there a connection between payroll and performance?

#American League Salary

library(reshape)
library(reshape2)
American_L_Sal= dbGetQuery(db, "SELECT teamID, sum(salary), yearID FROM
Salaries WHERE lgID= 'AL' GROUP BY teamID, yearID")

Teams= unique(American_L_Sal[,1])
names(American_L_Sal)= c("Team", "Salary", "Year")
m=melt(American_L_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)
matplot(c, type="l", col=1:68, xlab="Year", ylab="Salary",
main=American League)
legend("topleft", legend=Teams, cex=.4, col=1:68, pch=.5, lty=.5,
lwd=1)


Above, we can see the salaries for each of the American League teams. This graph
and th one's following took me several hours and I am very proud of them. The
graph above is not the easiest to read, but I think it is still plenty readable. We can
see that the highest paid team for the American league (the recent years) is
undoubtably the Texas Rangers.



#National League Salary
National_L_Sal= dbGetQuery(db, "SELECT teamID, sum(salary), yearID FROM
Salaries WHERE lgID= 'NL' GROUP BY teamID, yearID")

Teams= unique(National_L_Sal[,1])
names(National_L_Sal)= c("Team", "Salary", "Year")
m=melt(National_L_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)
matplot(c, type="l", col=55:68, xlab="Year", ylab="Salary", main="Nat.
League")
legend("topleft", legend=Teams, cex=.4, col=40:68, pch=.5, lty=.5,
lwd=1)


Above we can see that Mostly in the last five years, for the National League, the LA
Dogers have been the highest paid tem, until the last 2 years, where we can see that
the New York Yankees salaries have boosted greatly.


#Now the devisions:
#American League West: Used Nick's code from discussion, recieved help
from Michael in OH's
American_LW_Sal= dbGetQuery(db, "SELECT a.teamID, sum(a.salary),
a.yearID FROM Salaries AS a, Teams as b WHERE a.teamID = b.teamID AND
a.yearID = b.yearID AND a.lgID = b.lgID AND
a.lgID= 'AL' AND b.divID= 'W' GROUP BY a.yearID, a.teamID;")

Teams= unique(American_LW_Sal[,1])

names(American_LW_Sal)= c("Team", "Salary", "Year")
m=melt(American_LW_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)

matplot(c, type="l", col=50:60, xlab="Year", ylab="Salary", main="AL
WEST")
legend("topleft", legend=Teams, cex=.6, col=50:60, pch=.5, lty=.5,
lwd=1)


Highest paid team in last 5 years looks to be Seatle Mariners for AL West






#American League East
American_LE_Sal= dbGetQuery(db, "SELECT a.teamID, sum(a.salary),
a.yearID FROM Salaries AS a, Teams as b WHERE a.teamID = b.teamID AND
a.yearID = b.yearID AND a.lgID = b.lgID AND
a.lgID= 'AL' AND b.divID= 'E' GROUP BY a.yearID, a.teamID;")

Teams= unique(American_LE_Sal[,1])
names(American_LE_Sal)= c("Team", "Salary", "Year")

m=melt(American_LE_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)

matplot(c, type="l", col=50:60, xlab="Year", ylab="Salary", main="AL
EAST")
legend("topleft", legend=Teams, cex=.6, col=50:60, pch=.5, lty=.5,
lwd=1)


Highest paid team in last 5 years looks to be NY Yankees for AL East








#American League Central
American_LC_Sal= dbGetQuery(db, "SELECT a.teamID, sum(a.salary),
a.yearID FROM Salaries AS a, Teams as b WHERE a.teamID = b.teamID AND
a.yearID = b.yearID AND a.lgID = b.lgID AND
a.lgID= 'AL' AND b.divID= 'C' GROUP BY a.yearID, a.teamID;")

Teams= unique(American_LC_Sal[,1])
names(American_LC_Sal)= c("Team", "Salary", "Year")

m=melt(American_LC_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)

matplot(c, type="l", col=1:68, xlab="Year", ylab="Salary", main="AL
Central")
legend("topleft", legend=Teams, cex=.6, col=1:68, pch=.5, lty=.5,
lwd=1)


Highest paid team in last 5 years looks to be Kansas City Royals for AL Central







#National League West
National_LW_Sal= dbGetQuery(db, "SELECT a.teamID, sum(a.salary),
a.yearID FROM Salaries AS a, Teams as b WHERE a.teamID = b.teamID AND
a.yearID = b.yearID AND a.lgID = b.lgID AND
a.lgID= 'NL' AND b.divID= 'W' GROUP BY a.yearID, a.teamID;")

Teams= unique(National_LW_Sal[,1])
names(National_LW_Sal)= c("Team", "Salary", "Year")

m=melt(National_LW_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)

matplot(c, type="l", col=1:68, xlab="Year", ylab="Salary", main=NL
WEST)
legend("topleft", legend=Teams, cex=.6, col=1:68, pch=.5, lty=.5,
lwd=1)


Highest paid team in last 5 years looks to be Arizona Diamondback for NL West, but
SF Giants seem to have become the highest from 2012-2013.






#National League East
National_LE_Sal= dbGetQuery(db, "SELECT a.teamID, sum(a.salary),
a.yearID FROM Salaries AS a, Teams as b WHERE a.teamID = b.teamID AND
a.yearID = b.yearID AND a.lgID = b.lgID AND
a.lgID= 'NL' AND b.divID= 'E' GROUP BY a.yearID, a.teamID;")

Teams= unique(National_LE_Sal[,1])
names(National_LE_Sal)= c("Team", "Salary", "Year")

m=melt(National_LE_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)

matplot(c, type="l", col=1:68, xlab="Year", ylab="Salary", main="NL
EAST")
legend("topleft", legend=Teams, cex=.6, col=1:68, pch=.5, lty=.5,
lwd=1)


Highest paid team in last 5 years looks to be Maimi Marlins for NL East.







#National League Central
National_LC_Sal= dbGetQuery(db, "SELECT a.teamID, sum(a.salary),
a.yearID FROM Salaries AS a, Teams as b WHERE a.teamID = b.teamID AND
a.yearID = b.yearID AND a.lgID = b.lgID AND
a.lgID= 'NL' AND b.divID= 'C' GROUP BY a.yearID, a.teamID;")

Teams= unique(National_LC_Sal[,1])
names(National_LC_Sal)= c("Team", "Salary", "Year")

m=melt(National_LC_Sal,id=c("Team", "Year"))
c=cast(m, Year~Team)

matplot(c, type="l", col=1:68, xlab="Year", ylab="Salary", main="NL
Central")
legend("topleft", legend=Teams, cex=.6, col=1:68, pch=.5, lty=.5,
lwd=1)


Highest paid team in last 5 years looks to be Chicago Cubs for NL Central, but
CINCINNATI REDS seem to have become the highest from 2012-2013.







NUMBER 10
Has the distribution of home runs for players increased over the years?

#Number 10
home_run= dbGetQuery(db, "SELECT yearID,HR AS homerun FROM Batting ")

a=split(home_run$homerun, home_run$yearID)
boxplot(a, outwex=.2, outline=FALSE)


From the plot, we can see that the distrobution of homeruns HAS changed over the
years, with a peak in the 90's and early 2000's most likely due to steroid use being
unregulated.

BONUS QUESTIONS!
Have the RBI's in the last 13 years gone down due to Steroid use being enforced?
RBI= dbGetQuery(db, "SELECT yearID,sum(RBI) FROM Batting WHERE yearID
BETWEEN 2000 AND 2013 GROUP BY yearID ")
plot(RBI, type='l', ylab="RBI's", xlab="Year", main="RBI's since 2000"
)

#Verify that Batting has worstened with Steroid decline by looking at

Homeruns
home_runs= dbGetQuery(db, "SELECT yearID,sum(HR) AS homerun FROM
Batting WHERE yearID BETWEEN 2000 AND 2013 GROUP BY yearID")
plot(home_runs, type='l', ylab="Homeruns", xlab="Year", main="Homeruns
since 2000" )


So it is apparent that the RBI's have gone down recently, and the amount of
homeruns, and this is likely due to the decrease of steroids over the past 10-15
years.
NUMBER 2 Look at the number of strikeouts over the years, have pitchers gotten
better?
strike_outs= dbGetQuery(db, "SELECT yearID,sum(SO) AS homerun FROM
Pitching GROUP BY yearID")
plot(strike_outs, type='l', ylab="Strikeouts", xlab="Years",
main="Strikeouts over the years")

It does appear that pitchers have gotten better over the the years, but that could
also mean that batters have gotten worse while pitchers remained the same.
Question 3
Who are the 5 top managers (managerID's) with the highest number of wins in this
dataset?
top_managers= dbGetQuery(db, "SELECT playerID,yearID,W AS wins FROM
Managers ORDER BY W DESC limit 5")

top_managers

#From baseballreference.com
#Frank Chance, Lou Piniella, Joe Torre, AL Lopez, Fred Clarke

Number 4
In which years were there tie games? and how many were there?
dbGetQuery(db, "SELECT sum(ties), yearID AS year FROM SeriesPost WHERE
ties=1 GROUP BY yearID;")

sum(ties) year
1 1 1885
2 1 1890
3 1 1892

Number 5
Who are all of the pitchers in the MLB for the year 2013 and which team did they
play for?
pitchers=dbGetQuery(db, "SELECT teamID, playerID,yearID, Pos FROM
FieldingPost WHERE yearID=2013 AND Pos='P';")

There are 166 Pitchers, so I will just list the first 5 and the last 5:
teamID playerID yearID POS
1 DET albural01 2013 P
2 DET albural01 2013 P
3 CLE allenco01 2013 P
4 DET alvarjo02 2013 P
5 OAK anderbr04 2013 P
... ... ...... ... ..
162 BOS workmbr01 2013 P
163 BOS workmbr01 2013 P
164 BOS workmbr01 2013 P
165 TBA wrighja01 2013 P
166 TBA wrighwe01 2013 P