http://gee.cs.oswego.

edu

Scalable IO in Java

Doug Lea State University of New York at Oswego dl@cs.oswego.edu http://gee.cs.oswego.edu

Outline
http://gee.cs.oswego.edu
"

Scalable network services Event-driven processing Reactor pattern

asic version !ultit"readed versions Ot"er variants

"

"

"

#alkt"roug" of $ava%nio nonblocking &O '(&s

Network Services
"

http://gee.cs.oswego.edu

#eb services) Distributed Ob$ects) etc !ost "ave sa*e basic structure+
Read request Decode request Process service Encode reply Send reply

"

"

ut differ in nature and cost of eac" step

,!L parsing) -ile transfer) #eb page generation) co*putational services) %%%

Classic Service Designs

http://gee.cs.oswego.edu
decode co*pute encode

client client client Server

read

send

"andler
read decode co*pute encode send

"andler
read decode co*pute encode send

"andler

Eac" "andler *ay be started in its own t"read

Classic ServerSocket oop

http://gee.cs.oswego.edu
class Server implements Runnable { public void run() { try { ServerSocket ss = new ServerSocket(PORT); while (!Thread.interrupted()) new Thread(new Handler(ss.accept())).start(); // or, single-threaded, or a thread pool } catch (IOException ex) { /* ... */ } } static class Handler implements Runnable { final Socket socket; Handler(Socket s) { socket = s; } public void run() { try { byte[] input = new byte[MAX_INPUT]; socket.getInputStream().read(input); byte[] output = process(input); socket.getOutputStream().write(output); } catch (IOException ex) { /* ... */ } } private byte[] process(byte[] cmd) { /* ... */ } } } Note: most exception handling elided from code examples

Scalability !oals
"

http://gee.cs.oswego.edu

.raceful degradation under increasing load /*ore clients0 1ontinuous i*prove*ent wit" increasing resources /1(U) *e*ory) disk) bandwidt"0 'lso *eet availability and perfor*ance goals
S"ort latencies !eeting peak de*and 2unable 3uality of service

"

"

"

Divide-and-con3uer is usually t"e best approac" for ac"ieving any scalability goal

Divide and Conquer

"

Divide processing into s*all tasks

Eac" task perfor*s an action wit"out blocking

http://gee.cs.oswego.edu

"

E4ecute eac" task w"en it is enabled

5ere) an &O event usually serves as trigger
read decode compute encode send

"

asic *ec"anis*s supported in $ava%nio

Non-blocking reads and writes Dispatc" tasks associated wit" sensed &O events

"andler

"

Endless variation possible

' fa*ily of event-driven designs

Event"driven Designs
"

Usually *ore efficient t"an alternatives

-ewer resources
"

http://gee.cs.oswego.edu

Don6t usually need a t"read per client Less conte4t switc"ing) often less locking !ust *anually bind actions to events

Less over"ead
"

ut dispatc"ing can be slower

"

"

Usually "arder to progra*

!ust break up into si*ple non-blocking actions
" "

Si*ilar to .U& event-driven actions 1annot eli*inate all blocking+ .1) page faults) etc

!ust keep track of logical state of service

#ackground$ Events in %&'

http://gee.cs.oswego.edu '#2 Event 7ueue Event ... Event '#2 t"read 'ctionListener
public void actionPerformed(...) { doSomething(); }

utton

click8

Event-driven &O uses si*ilar ideas but in different designs

Reactor Pattern
"

http://gee.cs.oswego.edu

Reactor responds to &O events by dispatc"ing t"e appropriate "andler

Si*ilar to '#2 t"read

"

(andlers perfor* non-blocking actions

Si*ilar to '#2 'ctionListeners

"

!anage by binding "andlers to events

Si*ilar to '#2 add'ctionListener

"

See Sc"*idt et al) Pattern-Oriented Software Architecture, Volume 2 /(OS'90

'lso Ric"ard Stevens6s networking books) !att #els"6s SED' fra*ework) etc

#asic Reactor Design

http://gee.cs.oswego.edu
client client client Reactor dispatc"
read read decode co*pute encode decode co*pute encode decode co*pute encode send send send

acceptor

read

Single t"readed version

)ava*nio Support
"

http://gee.cs.oswego.edu

C+annels
1onnections to files) sockets etc t"at support non-blocking reads

"

#u,,ers
'rray-like ob$ects t"at can be directly read or written by 1"annels

"

Selectors
2ell w"ic" of a set of 1"annels "ave &O events

"

Selection-eys
!aintain &O event status and bindings

Reactor .$ Setup
http://gee.cs.oswego.edu
class Reactor implements Runnable { final Selector selector; final ServerSocketChannel serverSocket; Reactor(int port) throws IOException { selector = Selector.open(); serverSocket = ServerSocketChannel.open(); serverSocket.socket().bind( new InetSocketAddress(port)); serverSocket.configureBlocking(false); SelectionKey sk = serverSocket.register(selector, SelectionKey.OP_ACCEPT); sk.attach(new Acceptor()); }
/* Alternatively, use explicit SPI provider: SelectorProvider p = SelectorProvider.provider(); selector = p.openSelector(); serverSocket = p.openServerSocketChannel(); */

Reactor /$ Dispatc+ oop

http://gee.cs.oswego.edu
// class Reactor continued public void run() { // normally in a new Thread try { while (!Thread.interrupted()) { selector.select(); Set selected = selector.selectedKeys(); Iterator it = selected.iterator(); while (it.hasNext()) dispatch((SelectionKey)(it.next()); selected.clear(); } } catch (IOException ex) { /* ... */ } } void dispatch(SelectionKey k) { Runnable r = (Runnable)(k.attachment()); if (r != null) r.run(); }

Reactor 0$ %cceptor
http://gee.cs.oswego.edu
// class Reactor continued class Acceptor implements Runnable { // inner public void run() { try { SocketChannel c = serverSocket.accept(); if (c != null) new Handler(selector, c); } catch(IOException ex) { /* ... */ } } } }
client client client acceptor
read read read

Reactor dispatc"
decode co*pute encode decode co*pute encode decode co*pute encode send send send

Reactor 1$ (andler setup

http://gee.cs.oswego.edu
final class Handler implements Runnable { final SocketChannel socket; final SelectionKey sk; ByteBuffer input = ByteBuffer.allocate(MAXIN); ByteBuffer output = ByteBuffer.allocate(MAXOUT); static final int READING = 0, SENDING = 1; int state = READING; Handler(Selector sel, SocketChannel c) throws IOException { socket = c; c.configureBlocking(false); // Optionally try first read now sk = socket.register(sel, 0); sk.attach(this); sk.interestOps(SelectionKey.OP_READ); sel.wakeup(); } boolean inputIsComplete() { /* ... */ } boolean outputIsComplete() { /* ... */ } void process() { /* ... */ }

Reactor 2$ Request +andling

// class Handler continued public void run() { try { if (state == READING) read(); else if (state == SENDING) send(); } catch (IOException ex) { /* ... */ } } void read() throws IOException { socket.read(input); if (inputIsComplete()) { process(); state = SENDING; // Normally also do first write now sk.interestOps(SelectionKey.OP_WRITE); } } void send() throws IOException { socket.write(output); if (outputIsComplete()) sk.cancel(); } }

http://gee.cs.oswego.edu

Per"State (andlers
"

' si*ple use of .o- State-Ob$ect pattern

Rebind appropriate "andler as attac"*ent
class Handler { // ... public void run() { // initial state is reader socket.read(input); if (inputIsComplete()) { process(); sk.attach(new Sender()); sk.interest(SelectionKey.OP_WRITE); sk.selector().wakeup(); } } class Sender implements Runnable { public void run(){ // ... socket.write(output); if (outputIsComplete()) sk.cancel(); } } }

http://gee.cs.oswego.edu

3ultit+readed Designs
"

Strategically add t"reads for scalability

!ainly applicable to *ultiprocessors

http://gee.cs.oswego.edu

"

#orker 2"reads
Reactors s"ould 3uickly trigger "andlers
"

5andler processing slows down Reactor

Offload non-&O processing to ot"er t"reads

"

!ultiple Reactor 2"reads

Reactor t"reads can saturate doing &O Distribute load to ot"er reactors
"

Load-balance to *atc" 1(U and &O rates

&orker '+reads
"

http://gee.cs.oswego.edu

Offload non-&O processing to speed up Reactor t"read

Si*ilar to (OS'9 (roactor designs

"

Si*pler t"an reworking co*pute-bound processing into event-driven for*

S"ould still be pure nonblocking co*putation
"

Enoug" processing to outweig" over"ead

"

ut "arder to overlap processing wit" &O

est w"en can first read all input into a buffer

"

Use t"read pool so can tune and control

Nor*ally need *any fewer t"reads t"an clients

&orker '+read Pools

http://gee.cs.oswego.edu
client client client acceptor Reactor

read read read

send send send

2"read (ool
decode co*pute encode

decode co*pute encode decode co*pute encode

worker t"reads

3ueued tasks

(andler wit+ '+read Pool

http://gee.cs.oswego.edu
class Handler implements Runnable { // uses util.concurrent thread pool static PooledExecutor pool = new PooledExecutor(...); static final int PROCESSING = 3; // ... synchronized void read() { // ... socket.read(input); if (inputIsComplete()) { state = PROCESSING; pool.execute(new Processer()); } } synchronized void processAndHandOff() { process(); state = SENDING; // or rebind attachment sk.interest(SelectionKey.OP_WRITE); } class Processer implements Runnable { public void run() { processAndHandOff(); } } }

Coordinating 'asks
"

5andoffs
Eac" task enables) triggers) or calls ne4t one Usually fastest but can be brittle

http://gee.cs.oswego.edu

"

1allbacks to per-"andler dispatc"er

Sets state) attac"*ent) etc ' variant of .o- !ediator pattern

"

7ueues
-or e4a*ple) passing buffers across stages

"

-utures
#"en eac" task produces a result 1oordination layered on top of $oin or wait:notify

4sing PooledE5ecutor
"

' tunable worker t"read pool !ain *et"od execute(Runnable r) 1ontrols for+
2"e kind of task 3ueue /any 1"annel0 !a4i*u* nu*ber of t"reads !ini*u* nu*ber of t"reads ;#ar*; versus on-de*and t"reads <eep-alive interval until idle t"reads die
"

http://gee.cs.oswego.edu

" "

to be later replaced by new ones if necessary block) drop) producer-runs) etc

Saturation policy
"

3ultiple Reactor '+reads

"

Using Reactor (ools

Use to *atc" 1(U and &O rates Static or dyna*ic construction
"

http://gee.cs.oswego.edu

Eac" wit" own Selector) 2"read) dispatc" loop

!ain acceptor distributes to ot"er reactors

Selector[] selectors; // also create threads int next = 0; class Acceptor { // ... public synchronized void run() { ... Socket connection = serverSocket.accept(); if (connection != null) new Handler(selectors[next], connection); if (++next == selectors.length) next = 0; } }

4sing 3ultiple Reactors

http://gee.cs.oswego.edu
client client client acceptor *ainReactor subReactor

read read read

send send send

2"read (ool
decode co*pute encode

decode co*pute encode decode co*pute encode

worker t"reads

3ueued tasks

4sing ot+er )ava*nio ,eatures

"

!ultiple Selectors per Reactor

2o bind different "andlers to different &O events !ay need careful sync"roni=ation to coordinate

http://gee.cs.oswego.edu

"

-ile transfer
'uto*ated file-to-net or net-to-file copying

"

!e*ory-*apped files
'ccess files via buffers

"

Direct buffers
1an so*eti*es ac"ieve =ero-copy transfer ut "ave setup and finali=ation over"ead est for applications wit" long-lived connections

Connection"#ased E5tensions
"

&nstead of a single service re3uest)

1lient connects 1lient sends a series of *essages:re3uests 1lient disconnects

http://gee.cs.oswego.edu

"

E4a*ples
Databases and 2ransaction *onitors !ulti-participant ga*es) c"at) etc

"

1an e4tend basic network service patterns

5andle *any relatively long-lived clients 2rack client and session state /including drops0 Distribute services across *ultiple "osts

%PI &alkt+roug+
"

uffer yte uffer

/1"ar uffer) Long uffer) etc not s"own%0

http://gee.cs.oswego.edu

"

" " " " " " "

1"annel Selectable1"annel Socket1"annel ServerSocket1"annel -ile1"annel Selector Selection<ey

#u,,er
http://gee.cs.oswego.edu
abstract class Buffer { int capacity(); int position(); Buffer position(int newPosition); int limit(); Buffer limit(int newLimit); Buffer mark(); Buffer reset(); Buffer clear(); Buffer flip(); Buffer rewind(); int remaining(); boolean hasRemaining(); boolean isReadOnly(); }

a b c position li*it *ark capacity

#yte#u,,er 6.7
http://gee.cs.oswego.edu
abstract static static static static class ByteBuffer extends Buffer { ByteBuffer allocateDirect(int capacity); ByteBuffer allocate(int capacity); ByteBuffer wrap(byte[] src, int offset, int len); ByteBuffer wrap(byte[] src); isDirect(); order(); order(ByteOrder bo); slice(); duplicate(); compact(); asReadOnlyBuffer(); get(); get(int index); get(byte[] dst, int offset, int length); get(byte[] dst); put(byte b); put(int index, byte b); put(byte[] src, int offset, int length); put(ByteBuffer src); put(byte[] src); getChar(); getChar(int index); putChar(char value); putChar(int index, char value); asCharBuffer();

boolean ByteOrder ByteBuffer ByteBuffer ByteBuffer ByteBuffer ByteBuffer byte byte ByteBuffer ByteBuffer ByteBuffer ByteBuffer ByteBuffer ByteBuffer ByteBuffer char char ByteBuffer ByteBuffer CharBuffer

#yte#u,,er 6/7
short short ByteBuffer ByteBuffer ShortBuffer int int ByteBuffer ByteBuffer IntBuffer long long ByteBuffer ByteBuffer LongBuffer float float ByteBuffer ByteBuffer FloatBuffer double double ByteBuffer ByteBuffer DoubleBuffer } getShort(); getShort(int index); putShort(short value); putShort(int index, short value); asShortBuffer(); getInt(); getInt(int index); putInt(int value); putInt(int index, int value); asIntBuffer(); getLong(); getLong(int index); putLong(long value); putLong(int index, long value); asLongBuffer(); getFloat(); getFloat(int index); putFloat(float value); putFloat(int index, float value); asFloatBuffer(); getDouble(); getDouble(int index); putDouble(double value); putDouble(int index, double value); asDoubleBuffer();

http://gee.cs.oswego.edu

C+annel
http://gee.cs.oswego.edu
interface Channel { boolean isOpen(); void close() throws IOException; } interface ReadableByteChannel extends Channel { int read(ByteBuffer dst) throws IOException; } interface WritableByteChannel extends Channel { int write(ByteBuffer src) throws IOException; } interface ScatteringByteChannel extends ReadableByteChannel { int read(ByteBuffer[] dsts, int offset, int length) throws IOException; int read(ByteBuffer[] dsts) throws IOException; } interface GatheringByteChannel extends WritableByteChannel { int write(ByteBuffer[] srcs, int offset, int length) throws IOException; int write(ByteBuffer[] srcs) throws IOException; }

SelectableC+annel
http://gee.cs.oswego.edu
abstract class SelectableChannel implements Channel { int validOps(); boolean isRegistered(); SelectionKey keyFor(Selector sel); SelectionKey register(Selector sel, int ops) throws ClosedChannelException; void boolean Object } configureBlocking(boolean block) throws IOException; isBlocking(); blockingLock();

SocketC+annel
abstract class SocketChannel implements ByteChannel ... { static SocketChannel open() throws IOException;

http://gee.cs.oswego.edu

Socket int boolean boolean boolean boolean boolean boolean void void int int int int int int }

socket(); validOps(); isConnected(); isConnectionPending(); isInputOpen(); isOutputOpen(); connect(SocketAddress remote) throws IOException; finishConnect() throws IOException; shutdownInput() throws IOException; shutdownOutput() throws IOException; read(ByteBuffer dst) throws IOException; read(ByteBuffer[] dsts, int offset, int length) throws IOException; read(ByteBuffer[] dsts) throws IOException; write(ByteBuffer src) throws IOException; write(ByteBuffer[] srcs, int offset, int length) throws IOException; write(ByteBuffer[] srcs) throws IOException;

ServerSocketC+annel
http://gee.cs.oswego.edu
abstract class ServerSocketChannel extends ... { static ServerSocketChannel open() throws IOException; int validOps(); ServerSocket socket(); SocketChannel accept() throws IOException; }

8ileC+annel
abstract class FileChannel implements ... { int read(ByteBuffer dst); int read(ByteBuffer dst, long position); int read(ByteBuffer[] dsts, int offset, int length); int read(ByteBuffer[] dsts); int write(ByteBuffer src); int write(ByteBuffer src, long position); int write(ByteBuffer[] srcs, int offset, int length); int write(ByteBuffer[] srcs); long position(); void position(long newPosition); long size(); void truncate(long size); void force(boolean flushMetaDataToo); int transferTo(long position, int count, WritableByteChannel dst); int transferFrom(ReadableByteChannel src, long position, int count); FileLock lock(long position, long size, boolean shared); FileLock lock(); FileLock tryLock(long pos, long size, boolean shared); FileLock tryLock(); static final int MAP_RO, MAP_RW, MAP_COW; MappedByteBuffer map(int mode, long position, int size); } NOTE: ALL methods throw IOException

http://gee.cs.oswego.edu

Selector
http://gee.cs.oswego.edu
abstract class Selector { static Selector open() throws IOException; Set keys(); Set selectedKeys(); int selectNow() throws IOException; int select(long timeout) throws IOException; int select() throws IOException; void wakeup(); void close() throws IOException; }

Selection-ey
http://gee.cs.oswego.edu
abstract class SelectionKey { static final int OP_READ, OP_WRITE, OP_CONNECT, OP_ACCEPT; SelectableChannel channel(); Selector selector(); boolean isValid(); void cancel(); int interestOps(); void interestOps(int ops); int readyOps(); boolean isReadable(); boolean isWritable(); boolean isConnectable(); boolean isAcceptable(); Object attach(Object ob); Object attachment(); }