Process Integration (PI) & SOA Middleware:

File/FTP Adapter - Large File Transfer (Chunk

Mode)
Posted by Sivasubramaniam Arunachalam Dec 26, 2011
This blog is the extension of following blogs.
ChunkMode for Binary File Transfer within PI 7.1 EHP1
PI 7.3 processing of large files - teaser

Background
Whenever large (binary) file needs to be transferred by using PI, there might be a chance for Out of Memory
error which might results in server restart. Technically, it often fails in the sender adapter.

The most popular work around is to split the file in to multiple parts in Sender Adapter and Combining them in
Receiver Adapter (via OS Scripts)
From PI 7.30 onwards File/FTP adapter natively supports transferring large size files by splitting them in to
smaller chunks based on the configured size. Each chunk will be processed as an individual XI Message in
sender adapter and all the chunks are combined based on the sequence in receiver adapter. Since all the
chunks have to be combined in the actual order, QoS EOIO has to be used to enable this feature.

Generated by Jive on 2016-10-04Z

1

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Configuration Options
Select 'EOIO' as a QoS under Processing tab.

After that select 'Advanced Mode' under 'Advanced' Tab

Option 1
You can choose the required chunk size option from the given list of 1, 2, 5, 10, 20 and 50 MB.

Generated by Jive on 2016-10-04Z

2

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Option 2
Custom values which are not possible in default options can also be configured in additional parameters.
Parameter Name

Possible Values

chunkMode

Non emptry string will be considered as true

chunkSizeKB

Positive Numeric Value

Generated by Jive on 2016-10-04Z

3

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Processing in Sender Adapter

Basic Mode will be enabled only if QoS is EOIO

Chunk Mode related Advanced Mode parameters will be ignored if QoS not EOIO
If chunkSizeKB is not defined /incorrect value, default is 10KB
If both are configured, Basic mode will have higher precedence

Sender Adapter will create XI message with additional dynamic headers for each chunk and forwarded to
Messaging System for IRD processing.

Example
For this example, chunk size is configured as 2 MB and the below input file (~4.5 MB) is used.

Generated by Jive on 2016-10-04Z

4

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

For demo purpose and to identify/analyze how the actual split happens, the input file is created with the data
like below ( 1 to 600000)

As per the configuration, three chunks should be created with the size of 2 MB, 2MB and ~500 KB.

Chunk 1 (2 MB)
Transferring first 2 MB

Generated by Jive on 2016-10-04Z

5

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Chunk 2 (2 MB)
Transferring next 2 MB (Total 4 MB)

Chunk 3 (~500 KB)

Transferring the remaining ~500 KB (Total ~4.5 MB)

Dynamic Headers
The entire chunk mode related dynamic headers will be set under the namespace http://sap.com/xi/XI/System/
Generic
Header

Value

ChunkStart

Start of the Chunk (Position of bytes)

ChunkMode

Active/End

ChunkKey

Unique Key (Same for all the chunks for a particular

file)

The corresponding screenshots for all 3 chunks are given below.

Generated by Jive on 2016-10-04Z

6

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

First Chunk

Next Chunk

Last Chunk

If multiple files sent from different file sender channels, receiver adapter uses the chunk key to identify the
proper output file(for combining). During the creation(and append) of output file, it uses the chunk key like
below.

Generated by Jive on 2016-10-04Z

7

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)
In Temp mode, the temp files wont be deleted till the last chunk is received.

Overall Control Flow

Recovery Mode
If the input file is very large, it would normally take more time to be transferred completely. If there is any
server failure in the middle, the transfer can resumed from the chunk it stopped (Not from the beginning)

Individual Payloads after Split

Though this feature operates only on binary mode, text file is used to analyze how the split happens during
chunk creation.

Generated by Jive on 2016-10-04Z

8

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Payload of Chunk 1

Payload of Chunk 2

Payload of Chunk 3
As per the above screenshots, the split never cosiders the payload. It's just a binary split. So the following
limitations would apply
Only for File Sender to File Receiver
No Mapping
No Content Based Routing
No Content Conversion
No Custom Modules
Technically, other type of receiver adapters can be used. But it won't have the option to combine the chunks.

Does any special configuration required in File Receiver?

No. File receiver will automatically identify & combine the chunks using dynamic headers.

How to choose the chunk size? Smaller chunk size ensures more reliability?
Yes. But significantly affects the performance. Smaller chunk size creates more chunks (more XI messages)

Generated by Jive on 2016-10-04Z

9

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Troubleshooting
If chunks are stuck in the EOIO queue, it can be monitored with
http://host:port/MessagingSystem/monitor/sequenceMonitor.jsp

14624 Views Tags: pi_731

Joe Compton in response to Venu Gopal D on page 10

Oct 16, 2015 6:10 PM
Hi,
Have the same/similar question. File to Proxy/Proxy to File scenarios.
Thanks,
Joe
Venu Gopal D
Oct 2, 2015 7:00 PM
Hi Siva,
Chunk mode will be only useful for file to file scenarios? not useful for File to Proxy interfaces? Please let me
know
Thanks,
Venu
Martin Babayan
Nov 10, 2014 6:22 PM
Hi,
Have you ever been able to do the same within ICO?
I configured a classical scenario in Pi 7.40 and it worked fine. Then I decided to migrate it ICO (just for test)
and now it's failing with error in sender CC "Unable to parse XML message payload". But why it is trying to
parse the file?
Will appreciate your input.
RAVI REDDY
Mar 3, 2014 6:54 AM
hi thanks for innovative information
thanks & regards
E.Ravi chandra reddy

Generated by Jive on 2016-10-04Z

10

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Peter Hermanns
Jun 25, 2013 3:58 PM
Very good information. One additional point leading us to move to PI 7.31 in the near future i guess.
Mariana Mihaylova in response to Marilyn S on page 11
Jun 12, 2013 12:36 PM
Hi Jose, We will try to recover the missing images.
Marilyn S in response to Mariana Mihaylova on page 11
May 29, 2013 6:23 PM
Mariana
Still some images are missing.
Thanks,
Jose
Mariana Mihaylova
Aug 27, 2012 2:40 PM
Hi Siva, All
All images should be fine now.
Thanks,
Mariana
Sivasubramaniam Arunachalam in response to Goradindla Sreerama Reddy on page 11
Jul 12, 2012 3:55 PM
Hi Sreeram,
Due to SDN to SCN migration, few screenshots are corrupted. I will check with SCN team.
Thanks,
Siva
Goradindla Sreerama Reddy
Jul 12, 2012 3:52 PM
not able to view the screeb shots can you correct the screens shots
regards
Sreeram
Sivasubramaniam Arunachalam in response to Shabarish Vijayakumar on page 11
Jan 10, 2012 7:19 AM
Yes. But we can use it at our own risk.

Generated by Jive on 2016-10-04Z

11

Process Integration (PI) & SOA Middleware: File/FTP Adapter - Large File Transfer (Chunk Mode)

Shabarish Vijayakumar in response to Sivasubramaniam Arunachalam on page 12

Dec 28, 2011 1:59 AM
So this means we cant use it in productive PI 7.11 environments?
Well that would be a shame, isnt it!
Community User in response to Sivasubramaniam Arunachalam on page 12
Dec 28, 2011 1:27 AM
Thanks for the blog Siva...
Can u tell me for which latest SP's it will supoort in PI 7.1.
Sivasubramaniam Arunachalam in response to Bhavesh Kantilal on page 12
Dec 27, 2011 7:06 AM
This feature can be found in 7.1 latest SPs. But as of now, officially supported only in 7.3+
Sivasubramaniam Arunachalam in response to Bhavesh Kantilal on page 12
Dec 27, 2011 6:52 AM
Thanks. Updated the links.
Sivasubramaniam Arunachalam in response to Bhavesh Kantilal on page 12
Dec 27, 2011 6:46 AM
It will perfectly work with ASMA.
Bhavesh Kantilal
Dec 27, 2011 4:38 AM
Hello
Does this feature continue to support ASMA - FileName? Can I use it to transfer files from source to target with
same original filename?
I see no reason it shouldn't but would be good if this can be confirmed.
Regards
Bhavesh
Bhavesh Kantilal
Dec 27, 2011 4:36 AM
Hello, The blog links in your blog point to the same blog. Is this feature available on 7.1? I was under the
impression that this is a feature available only from 7.31
A very helpful feature! Great blog with good explanation!

Generated by Jive on 2016-10-04Z

12