Accelerating File Transfers

Increase File Transfer Speeds

in Poorly-Performing Networks

www.filecatalyst.com
March 1, 2009


Contents
Introduction..............................................................................................................................................2
CommonFileDeliveryMethods...............................................................................................................2
UnderstandingFTP...................................................................................................................................3
LatencyanditseffectonFTP.............................................................................................................3
PacketlossanditseffectonFTP........................................................................................................4
ExistingFileTransferProcesses................................................................................................................5
FTPServerSolution............................................................................................................................5
EmailFileAttachments......................................................................................................................5
InstantMessaging..............................................................................................................................6
PhysicalMedia...................................................................................................................................6
OvercomingFileTransferIssueswithAcceleration..................................................................................7
OptimizingthroughputwithUDP.......................................................................................................7
OptimizingTCPwithMultipleStreams..............................................................................................7
ReducingDatawithCompression......................................................................................................8
MinimizingdatawithDeltaTransfers................................................................................................8
TheCostofSlowFileTransfers.................................................................................................................9
Thenumbercrunching.......................................................................................................................9
FactorstoConsiderforyourFileTransferSolution................................................................................11
AboutFileCatalyst...................................................................................................................................12

2009UnlimiTechSoftware,Inc. [1] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Introduction
According to a Hilty Moore & Associates study, companies have increased
their reliance on digital file transfer via File Transfer Protocol (FTP) by 66%
from2006to2007.

These companies were in not only hightech, but also health, financial,
FTPresultedin
incompletedeliveryas manufacturing and service industries. Most respondents experienced
muchas1in5times. deliverystoppagesorincompletetransmissionswithFTPasmuchas1outof
5times.Amorereliablealternativeisneeded.

Thispaperaddressestechnicalissuesfacedbyenterpriseswhentransferring
largefiles.ItwillalsopresentabusinesscasebehindswitchingtoanonFTP
digitaldeliverysystem,andguidestheenterprisetofindtheoptimalsolution
foritsrequirements.

Common File Delivery Methods

Companieswithlargedatasetscontinuetousephysicalmethods,recording
toDVD,Flash,orHDDandusingadeliveryservice.Thisworksonanadhoc
basisbutcanleadtohighcosts.

Standarddigitalalternativesincludeemailorinstantmessagingservices,but
thesehaveissuesincludingfilesizelimitationsorrelianceonthesameTCP/IP
backbonecausingFTPtofail.

Digitaldeliveryisstilltheultimatesolutionformostenterprises.However,a
Alternativesshould
overcomelimitations solutionshouldovercomethestoppagesandslowdownscommontoFTP.It
ofFTP,maximize should allow organizations to maximize their existing infrastructure, saving
existinginfrastructure, moneythatwouldotherwisehavebeenwastedacquiringhigherbandwidths.
andbesimple.
Finally,itshouldbeeasytoimplementandmoreimportantlyeasytouse.

2009UnlimiTechSoftware,Inc. [2] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Understanding FTP
FileTransferProtocol(FTP)isoneoftheoldestprotocolsontheInternet,but
FTPhasexistedsince is ubiquitous for file transfer. FTP resulted from an RFC first drafted in the
the1970s,andis 1970s,andhassincegonethroughrelativelyfewchanges.FTPwasdesigned
ubiquitous
tobecompletelyagnosticindependentfromoperatingsystems,filestorage
systemsandfiletypes.

FTPrunsonTCP/IPprotocol,usingtwoTCPconnections.Thefirstisacontrol
channel used to exchange instructions, and is always open. The other is a
data channel, used for file transfer and directory listings. The channel is
openedorclosedasneeded.

2waycontrol channelalways open

Client datachannelopened asneeded

Server

TCP/IP is the most widely deployed protocol on the internet; consequently,

FTP is the de facto standard for file transfer. However, it also inherits the
drawbacksofTCP.

Latency and its effect on FTP

Inordertoprovidereliabledatatransmission,TCP/IPrequiresthereceiverto
acknowledge each packet being sent, in sequential order. Each such
communicationismeasuredasroundtriptime(RTT),orthetimeittakesfor
apackettobesentandacknowledged.

Latency(RTT)causes
TCP responds to latency by adjusting the amount of unacknowledged data
FTPtoslowdown that can be on the link before waiting for a reply. The optimal amount of
considerably unacknowledged data en route should equal the endtoend bandwidth
multiplied by the RTT, also called the bandwidthdelay product. TCP
continuallyestimatesthisvalue,settingaTCPwindowtocontrolhowmuch
data should be sent. TCP has limits on the size of this value, so when the
bandwidthdelay product exceeds a certain threshold, the result is a lot of
waitingordeadair.Satelliteconnectionscanbeintothehundredsoreven
thousandsofmillisecondsofRTT.

2009UnlimiTechSoftware,Inc. [3] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Throughputis
reduced
dramaticallyat
only80msRTTto
justover5,000kbps

Packet loss and its effect on FTP

Network congestion typically causes buffer overflows of intermediate
routers,ausingpacketloss.Sincepacketsaresentsequentially,thiscancause
aholdupinthecycle.UnacknowledgedpacketsalsocausetheTCPwindow
toshrinkorevenclosecompletelyforperiodsoftime.Wirelessandsatellite
transfers can have even higher packet loss due to sources of interference
suchascloudsorphysicalstructures.Packetlosscombinedwithhighlatency
createsevenworseperformanceforFTPtransfers.

After0.03%the
throughputdrops
exponentially.
Publicinternet
averages0.5%
packetloss.

2009UnlimiTechSoftware,Inc. [4] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Existing File Transfer Processes

TCP is used as the backbone behind many existing corporate transfer

processes. Other methods of sharing or distributing files include email
attachments,shippingphysicalmedia,andinstantmessenger(IM)transfers.
Theseareusedindependentlyorconcurrentlyforgettingdatafromsourceto
sink.Eachofthesehaspotentiallimitations.

FTP Server Solution

Includingtransferspeedissues,thereareotherpotentialproblemswithFTP:
FTPserversareoften
setupbymediaand LearningCurve:TheaverageenduserisnotfamiliarwithFTPclients.
designagenciesto FirewallandSecurity:NotallITnetworksallowforunrestrictedFTPaccess;
sharelargefiles. someusersmaynotbeabletoconnecttotheFTPserver.
Maintenanceis
difficultandFTP FTP Administration: IT administrators have to manage endusers on the
suffersfromTCP/IP FTPserverandprovidesupportandmaintenancetotheFTPserver.
drawbacks TCP/IP Performance: Poorlyperforming networks compromise transfer
speedandencumberanorganizationsworkflow.

Email File Attachments

Email is perhaps the most common method of sharing files others, both at
workandathome.However,therearesomeconsiderablelimitations:

Reliability: Email servers are configured to handle attachments up to a

Emailserverssimply certain size; usually 10 MB or less. Users cannot depend on their email
cannothandlelarge arrivingwithoutbeingbouncedforexceedingthislimit.
filetransfers
effectivelyorreliably. Archival: Emails are archived by the mail server. Attachments add to the
Archivingisexpensive. storagerequirementsofthemailserver,increasingITcost.
PoorNetworkUtilization:EmailusesSimpleMailTransferProtocol(SMTP)
fortransfers,whichisnotthemostefficientprotocoltotransferlargefiles
overanetwork.

2009UnlimiTechSoftware,Inc. [5] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Instant Messaging
According to PC World , Instant Messaging has become the most common
formofelectroniccommunications,surpassingemail.Therearesomeissues
tobeawareof:

Interoperability:AlthoughdifferentIMclientsareoftenabletosharetext
messages with one another, file transfer is not usually possible between
Interoperability, differentclients.
TCP/IPdrawbacksand
securityrisksmakeIM Security:PopularIMclientscontinuetobetargetsfortrojansandexploits.
lessthanidealfor Transmissionsarenotusuallyoversecurechannels.
enterprises. No tracking: Transferring files over IM makes it difficult or impossible to
trackfilesmovingbetweendifferentendpoints.
Poor Network utilization: TCP/IP is typically used as the transport layer.
ThesamenetworkissuesfacingFTPalsofaceIMfiletransfer.

Physical Media
Organizations may record data to a DVD, hard drive, or flash storage, then
physicallyshipittotherecipient.Thisprocesscanbeusefulwhenthereare
hugedatasetsorwhenthereispoorconnectivitybetweenthetwoparties,
butithassomeobviousdrawbacks:

Postproduction
PreparationTime:Thisprocessrequireshumaninteraction,fromrecording
housesexchange thedatatoprintingoutmailinglabelsandshippingthemedia.
terabytesofdataby Cost: Depending on the urgency of the files, costs for using a courier
shippingphysical
media.Thisislabor
servicecaneasilyaddup.
intensiveand DeliveryTime:Sincephysicaldeliveryisinvolved,theprocesscantake1to
expensive. 5 business days. This may be an unwelcome delay and makes time
sensitivedeliveryanunreliableproposition.

2009UnlimiTechSoftware,Inc. [6] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Overcoming File Transfer Issues with Acceleration

Most companies realize that transferring files online is more costeffective
OvercomingTCP/IP than shipping physical media. By overcoming TCP/IP bottlenecks, digital
bottlenecksallows
digitaldeliveryto delivery becomes a realistic option. File transfer acceleration is possible
becomeviable through two complementary but separate methods. Generally speaking,
theseareoptimizingthroughputandreducingdatabeingsent.

Optimizing throughput with UDP

UserDatagramProtocol(UDP)squeezesperformancefromanIPnetworkby
UDPtakeslatencyout notimplementingallfeaturesofTCP.UDPisa"connectionless"protocolthat
oftheequationbecause does not depend on sequenced acknowledgements and retransmissions.
itdoesnotrequire Althoughthisimpliespotentialforincompletetransfers,vendorsarefreeto
acknowledgment
buildtheirownreliabilityintotheirapplications.

SinceUDPallowspacketstobereceivedinadifferentorderthantheywere
sentitdoes notstallifpacketsaredropped.Lostpacketscan berequested
with a command channel: data flows over UDP while errorcorrecting
commands are periodically exchanged over a TCP/IP stream. Without the
sameacknowledgmentrequirementsasTCP,latencyiseffectivelyeliminated.

Optimizing TCP with Multiple Streams

Multiplestreams WithstandardFTP,onlyonedatachannelisused.Thischannelbecomesidle
meanslessdeadair
duringatransfer,butis
duringacknowledgementandotherhousekeeping.Byincreasingthenumber
resourceintensive ofdatastreams,thereislessdeadair,asoneormorestreamsislikelyto
besendingdataatanygiventime.

Thismethodoftransferringfilesiseffectivewhennetworkdegradationisnot
extreme. However, the process is extremely processor intensive due to the
largenumberofconcurrentthreadsthatmustberunning.

2009UnlimiTechSoftware,Inc. [7] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Reducing Data with Compression

Datacompressionisageneraltermforagroupoftechnologiesthatencode
large files in order to shrink them down in size. Fewer bytes to send will
naturallyresultinafastertransfer,whetherthenetworkistheinternet,an
intranet,oralocalareanetwork(LAN).

Smallerfilesizes Data compression involves applying an algorithm to data that stores

requirelesstimeto repetitive bits as a shorthand. The receiving end uses an appropriate
transfer
decoder to restore these short forms to their original state, resulting in an
identicalcopyoftheoriginalfile.

Some files can be compressed more than others. Textbased files, such as
documents, reports, database files are highly compressible. On the other
hand, many media files such as MP3 or JPG are already compressed and
cannotbecompressedfurther.

Minimizing data with Delta Transfers

Afilemayalreadyexistonbothsidesofthetransfer,buthasbeenmodified
Transferringafile atthesource.Whenadifferenceisdetected,analgorithmcancalculatethe
deltacanreducedata differences and store them in discrete files. These small delta files are
sentbyastaggering
amount transferred instead of resending the entire file, then applied as a patch,
updatingthedestinationfileinordertomakeanidenticalcopy.Thebenefit
is clear: imagine being able to send a 4MB delta instead of a 2TB, for
example.OnepopularUnixapplicationthatsupportsdeltatransfersisrsync.

2009UnlimiTechSoftware,Inc. [8] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



The Cost of Slow File Transfers

Most enterprises do not realize the cost of slow file transfers. The most
Slowtransfersincur
significanthardand significanthardcostsisbandwidth;softcostsincludelosthoursintermsof
softcosts employee productivity and project completion. Fixing the problem of slow
transfersyieldssignificantreturnsbothonhardandsoftcosts.

The number crunching

Accelerationtechnologiesprovidereturnoninvestment(ROI)bymaximizing
existing infrastructures. Spending money on more bandwidth makes little
sense if current bandwidth is underutilized. Based on the following chart,
consider some scenarios in which acceleration provides cost or productivity
benefits:

MonthlyInternetCosts
Bandwidth(mbps) 1.5 3 5 10 50 100

Cost(month) $75 $250 $500 $750 $1000 $1300

PricingbasedonGoWebManandRogersCable

Scenario 1reclaim lost time: A postproduction house in New Zealand
transfers files to and from its LA office. Their 50mbps link costs $1000/mo,
and is theoretically very fast. However, the average latency of panPacific
transfers is 200ms, with an average packet loss of 1%. Under these
Improvingfiletransfer conditions,FTPwillonlytransferat490kbps,amere1%ofthelargepipe.
returnssignificant
timesavingstothe With UDPbased acceleration, transfer speed will reach near link speed, or
organization
100XfasterthanFTP.Art thatformerlytookahundredminutes nowtakes
one.Withoutspendinganymoremoney,thecompanyhasregainedninety
nine valuable minutes that can be applied to productivity and project
completion.

Scenario 2save IT expense: A company transfers files between North

AmericaandEuropeovera10mbpsconnection.Duetolatencyandpacket
loss, the effective throughput is 613kbps or 6% of bandwidth. The internet
serviceprovidersuggeststhecompanypurchaseafasterlink.

2009UnlimiTechSoftware,Inc. [9] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Insteadofaddingtotheirmonthlyoperatingexpense,thecompanydeploys
Maximizing
bandwidthisless accelerationtechnology.Byusingtheirbandwidthtoitsfullest,thecompany
expensivethanbuying experiencesfastertransfersandhasavirtualsavingsof$670/mocompared
afasterlink totheupgradeoption.

SummaryofScenarios
Scenario Line Latency/ Previous Acceleration
Speed Loss speed Advantage
1reclaim 50 290msRTT 490Kbps Increasetransferspeed
losttime Mbps 1%packet (1%) by100X,reclaiming
loss manhoursand
improvingturnaround
2saveIT 10 160msRTT 613Kbps Increasetransferspeed
expense Mbps 0%packet (6%) by15X,givingcompany
loss higherperformance
andavoid$670/mo
expense.

Optimizing bandwidth generates both virtual and actual returns on

investmentmadeinimplementingafileaccelerationsolution.

2009UnlimiTechSoftware,Inc. [10] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



Factors to Consider for your File Transfer Solution

Theoptimumfiletransfersolutionshouldbeabletoconnectthevariousfile
storage systems of the enterprise regardless of network topology, physical
location or operating system. Many commercial vendors provide a file
TheIdealSolution transfer solutionsome offer a comprehensive platform, while others
provideasubscriptionservicesolution.Thereareuniversalconsiderationsto
Scalable
Thesolutionshould bemadeforeachoption:
scaleinacosteffective
way.Thesolution Acceleration: How important are accelerated file transfers to your
shouldbeeasily enterprise?Doanyofyourprocessesinvolvelargefiletransfersthatmay
expandable
bereduced?

Reliable Automation and Scheduling: Will you benefit from automating routine
Onceafiletransferis filetransfers?Wouldautomaticschedulinghelpyourbusinesscomplete
initiated,usersshould
projectsandimproveitsbottomline?
expectdelivery.The
systemshouldrecover WebBasedInterface:Shouldendusershaveaccessviawebbrowser,or
fromerrors. isaclientsideapplicationpreferred?

Easeofdeployment:Howmanyserversandclientsdoyouneedtoinstall
Simple
Thesystemmusthave andwhere?Isinstallationfeasibleforremotelocations?
agentlelearningcurve Security and Compliance: What are the major security issues that your
forendusers
organizationfaces?Arethereanysecuritymandatesyoumustmeet,and

Secure doestheproposedsolutionmeetthem?
Thesolutionshouldbe Tracking and Reporting: How much tracking functionally do you need?
abletomeetexisting
Whattypeofreportswouldyouliketogenerate,anddoesthesolution
orpotentialsecurity
mandates supportthis?
Aftersales support and service: Does the vendor provide reasonable
supportandservicefortheproduct?

2009UnlimiTechSoftware,Inc. [11] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009



About FileCatalyst
FileCatalyst software solutions are developed by UnlimiTech Software, Inc.
Learnmoreaboutour Foundedin2000,UnlimiTechSoftwareisaprivatelyheldcorporationbased
technologyat in Ottawa, Canada, operating with a global reseller network. UnlimiTech
www.filecatalyst.com
products are focused on solving file transfer challenges in diverse
environments, from end user desktops to sophisticated WAN and satellite
basedmulticastsystems.

LearnmoreaboutUnlimiTechSoftwareandFileCatalyst:

www.filecatalyst.com
info@filecatalyst.com

tf +1.877.327.9387
t +1.613.667.2439
f +1.613.667.2439

2009UnlimiTechSoftware,Inc. [12] AcceleratingFileTransfers

www.filecatalyst.com 3/1/2009