2/7/13

Welcome, Guest Login Register Getting Started Newsletters Store

SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

Solutions Services & Support About SCN Downloads Industries Training & Education Partnership Developer Center Lines of Business University Alliances Events & Webinars Innovation

Selective Reading and Postprocessing

Added by Vicky Bolster, last edited by Vicky Bolster on Mar 05, 2012
Sometimes the task is not that easy. The pattern might change, e.g. read all files matching the pattern "plan_[BOBJ:todays date].txt", or you w ant to rename the files after successful load. There is no direct support for that but a w orkaround - you can use parameters as filename as w ell. So the idea w ould be to use a function to get the first filename w hile a filename exists loop w ithin the loop call the dataflow w ith that filename to load and afterw ards rename the file and find the next filename - if there is one The first step is to create a Workflow to hold the entire logic. Inside that w orkflow a variable is created called $filename to hold the next filename to be processed.

In a first script w e use the DI 11.7 function w ait_for_file() to set the variable $filename to the first file matching our pattern or NULL if there is none. As the function does either use '\' or '/' a path separator char w ith the latter being the default setting in the DSConfig.txt, w e use a replace_substr() function to change it back to backslash. The DI internal functions can deal w ith either character as path separator, but w e w ill use exec() to run a msdos command later.

And now w e use a w hile loop to call a dataflow reading the file repetitive unless there are no more files ("$filename is not null").

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing

1/4

2/7/13

SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

In the dataflow - a "procedure" of its ow n - w e w ant to use the value of the local variable of the w orkflow as w ell. So w e need to define a parameter for the dataflow by clicking on the variable w indow w hen view ing the dataflow and define it. For clarity I called it $df_filename.

Now the dataflow requires a parameter and everyw here you call that dataflow "procedure" you need to apply a value. Where do w e call it? In the w orkflow . So go back to the w orkflow , open the calls tab of the variable w indow and provide a value - the value of $filename.

Then w e can use the source file name $df_filename in the dataflow 's objects, in our case as a filename. But w atchout, the variable is used in the file reader call, not in the file reader class. So open the object inside the dataflow to set the variable, not form the object library -> file formats as there you define the object itself. Not the call properties. Obviously you could have used global variables to avoid that passing along of variables but...

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing

2/4

2/7/13

SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

Last object is another script for the post processing. If a dataflow did load a file successfully, w e w ant to rename the file into something else. Today there are still no native functions in DI so w e have to go via the exec function.
e x e c ( ' c m d . e x e ' ,' m o v e[ $ f i l e n a m e ][ $ f i l e n a m e ] . o l d ' ,0 ) ;

And finally w e reset $filename to the next available filename inside the w hile loop.

The function w ait_for_file() has the syntax w ait_for_file(filename pattern, timeout, poll interval, how many file names should be returned, a variable holding the returned filename list, returned size of list, list separator char). In our case w e do not need the last tw o optional parameters. We w ant the function to return immediately (timeout=0), as w e w ill not poll multiple times the poll interval can be anything, w e w ant just one file to be returned and into the variable $filename.
w a i t _ f o r _ f i l e ( ' c : \ \ t e m p \ \ p l a n _ [ y e a r ( s y s d a t e ( ) ) ] * . t x t ' ,0 ,0 ,1 ,$ f i l e n a m e ) ;

Note: This entire chapter is based on the DI 11.7 function w ait_for_file(). But you can get the same result by setting the variable using
$ f i l e n a m e = ' c : \ \ t e m p \ \ '| |w o r d ( e x e c ( ' c m d . e x e ' ,' d i r/ bc : \ \ t e m p \ \ p l a n * . t x t ' ,0 ) ,1 ) ;

Change the Row Delimiter Errorhandling in file formats Excel- Save to DI How to create a flat file format from a Query Multirecord Files Reading a large XML file Reading multiple files at once File Group Reader Selective Reading and Postprocessing Shared Directory access Writing a large XML file

Follow SCN Contact Us SAP Help Portal Privacy Terms of Use Legal Disclosure Copyright

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing

3/4

2/7/13

SAP Community Network Wiki - Enterprise Information Management - Selective Reading and Postprocessing

wiki.sdn.sap.com/wiki/display/EIM/Selective+Reading+and+Postprocessing

4/4