Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Getting Started Newsletters Store

Welcome, Guest Login Register

Products Services & Support About SCN Downloads

Activity Communications Actions
Industries Training & Education Partnership Developer Center

Lines of Business University Alliances Events & Webinars Innovation Browse

SAP NetWeaver Business Warehouse

Invalid characters in SAP BW 3.x: Myths and Reality.

Part 2.
Posted by Eugene Khusainov in SAP NetWeaver Business Warehouse on Jul 23, 2006 7:58:14 AM

Share 1 0 Tweet

Invalid Characters in Characteristic Values of Infoobjects

It’s very well known that the most errors during data load occur due to invalid characters in data inflow.

The previous my blog was dedicated to invalid characters in the texts of infoobjects. Hereinafter, all is to be said is
applied to invalid characters in characteristic values only. Though, provided here information is valid for non-unicode
systems, I believe that the big part of it might be applied to Unicode systems also.

It’s amazing that even some very experienced people share some delusions about invalid characters. So, here is a list of
myths related to invalid characters in characteristic values.

Myth 3: The SPACE symbol is a special one and has to be added to allowed characters in RSKC (if we
want to have the SPACE in characteristic values)

Not all characters are allowed in characteristic values of infoobjects. There is a predefined set of allowed characters. All
notes, HowTos and help documents state that this set include the following characters (see, for example, the OSS Note
#173241 – “Allowed characters in the BW System”):

!"%&'()*+,-/:;<=>?_0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ

As you can see, there is no SPACE symbol. And I remember that there were even threads in BI forums related to
inserting the SPACE in RSKC by pressing ALT-0160 or ALT-255.

Actually, the default set of allowed characters DOES INCLUDE the SPACE symbol. You can see it by yourself:

Execute SE37, type in RSKC_ALLOWED_CHAR_GET.

Double click in the statement

E_DEFAULT_CHAR = G_C_ALLOWED_CHAR

on the right part.

You'll be brought to LRSKCTOP include.

There you can see the default allowed chars, and the SPACE is the first one:

!https://weblogs.sdn.sap.com/weblogs/images/32317/w01.JPG|height=154|alt=image|width=567|src=https:
//weblogs.sdn.sap.com/weblogs/images/32317/w01.JPG|border=0!

There is also a list of clearly forbidden characters: from hex code 00 to 1F.

Additionally, you may just make a little experiment - typing in some characters with spaces in characteristic value of
infoobject of type CHAR:

You’ll see that SPACE symbol is allowed without any special settings in RSKC.

1 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Myth 4: Presence of # sign in RSKC allows all invalid characters

Since invalid characters are often represented in BW as # sign, many beginners believe in this myth.

It’s a delusion. The system uses # character for all hexadecimal values, for which a code page does not know the
appropriate character to use. This is the case for invalid characters, and hence, this sign may represent ANY invalid
characters in BW messages.

The presence of this sign in RSKC will not prevent failure of data load with invalid characters. It only will allow presenting
the # sign itself (hexadecimal code 23) if it is not alone in the value.

Myth 5: ALL_CAPITAL setting in RSKC along with the other special characters solves problems with
invalid characters
The example:

It’s wrong. Put NOTHING else than "ALL_CAPITAL" in transaction RSKC, otherwise the system won’t recognize it. Only
additional special characters set will be recognized. ALL_CAPITAL will be treated simply as letters A, L, _, C, P, I and T to
be allowed. Though, these symbols are already permitted by default.

Myth 6: ALL_CAPITAL setting in RSKC allows all (special) characters

As you can see in the picture to Myth 3, characters with hex codes from 00 to 1F are forbidden. These very characters
cause the most of problems during data load.
Remember that pressing the keys like TAB, ENTER, arrows, BACKSPACE creates exactly these not permitted codes.
Hence, some validation of incoming to BW data must be made if data is created in an application where a user types in
information and the application doesn’t check it.

By calling some ABAP functional modules returning l_userdef_char (user defined allowed characters) you’ll not get a set
of permitted symbols in case of ALL_CAPITAL setting. You’ll simply get this very ‘ALL_CAPITAL’ string.

‘ALL_CAPITAL’ is interpreted by the other modules that check languages installed in the system.

ALL_CAPITAL_option, as it is said in the How To "Permitted Characters in Characteristic Values” allows you to use
all the characters that are capitals in the local language (the language in which the batch processes run). The result is
that the loading process becomes language dependent.”

Two questions arise immediately:

1. Is it possible to use lowercase letters with ALL_CAPITAL option?

There is a common belief that with this option only uppercase letters are allowed. That’s not true! If you check
‘Lowercase Letters’ for an infoobject, you can load data with lowercase letters too! Even for foreign languages installed
in the system.

2. Since except ‘ALL_CAPITAL’ string you cannot set another characters in RSKC, how to allow the
other special characters like #, ~ etc. (not hex 00-1F codes)?

You do not need this. ALL_CAPITAL option takes care about such symbols too!

So, as we see, ALL_CAPITAL is very powerful option and allows the maximum number of signs in BW.

It cannot manage only a small set of exceptions. For completeness of information it makes sense to provide here a
quotation from the mentioned How to:

The following characteristic values are not permitted and lead to errors in the system:

Values that consist only of the character #. This is because blank entries in variables are marked with # (no entry =
space).
Values that begin with the character !. The system deletes these values.
Control characters with the hex-display 00 to 1F (valid as of BW 3.0A if small letters were previously allowed).
Characters are not allowed if they are represented by a hex code that has a small letter in one of the installed
languages. You could, for example, only allow the capital Ö if no language is installed where the corresponding hex
code is a small letter. In this example, it would not be possible to have Russian installed, because the hex code for
capital Ö means a small sch in this example.

Special currency indicators

The symbols for the US Dollar ($), British Pound (£), Japanese Yen (¥) and the Euro are not allowed by default.
The symbol for the US Dollar ($) is allowed in BW customizing. The other special currency indicators lead to errors in the
system.

In the InfoObject maintenance screens or in the transfer rules, you need to create a transfer routine for any currency
fields and other characteristic values that have special currency indicators. This transfer routine converts the invalid
characters into valid characters or character sets. This conversion is mandatory for currency fields. The currency codes,
into which you need to change the currencies, are stored in BW customizing under General Settings -> Currencies ->
Check Currency Codes (or ISO4217). The key must agree with the keys in the currency tables.

2 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Most of the special currency indicators can be assigned to three-character currency codes. For example, the $ dollar
symbol is converted into USD for US dollars or AUD for Australian dollars.

How to catch and eliminate invalid characters?

As we can see, the most errors during data load belong to the first three categories provided in the quotation.

How to deal with these invalid characters? The best option is to write an ABAP code and place it into a routine in transfer
rules. Here, in the forum I saw many examples of such a code.

I somewhat like the one posted in this thread:

Re: Invalid Characteristic data

Though, I don’t agree with the statement that “for characteristics to be used for navigation – lower case is not permitted”.

I’ve made an experiment:

created 3 infoobjects (type CHAR) with lowercase letters flag checked,

made one of the infoobjects as navigational attribute, and another one as a compounding attribute,
loaded master data with lowercase letters for this basis infoobject,
created an ODS (BEx relevant) and a cube with this infoobject,
loaded transaction data into infoproviders,
created 2 queries, for the ODS and the cube.

Queries showed results without problems. Drill down by navigational attribute also worked fine. I had ‘ALL_CAPITAL’
setting in RSKC.

I think that the main prerequisite for successful data load is a load of master data before transactional ones.

So, in case of loading data in infoobjects with lowercase letters flag checked, there is no necessity to translate
everything to upper case.

Unfortunately, as far as I remember, all the code examples for removing invalid characters posted in the forums deal with
any settings in RSKC, EXCEPT ALL_CAPITAL.

One More Code

The code below deals with any settings in RSKC. It doesn’t compare incoming text with permitted characters. It simply
replaces by space the following invalid characters:

forbidden symbols with hex codes 00 – 1F;

strings started with ! sign;
strings with the only # sign.

The code assumes that the field for which this routine is created is of CHAR type. Moreover, if lowercase letters may
come from the source system, flag ‘Lowercase Letters’ should be checked for the infoobject.

Replace ZZZ by the name of the infoobject in the transfer structure.

FIELD-SYMBOLS: &ltic&gt TYPE x, &lttc&gt TYPE c.

DATA: ch1(32) TYPE x VALUE '00200120022003200420052006200720082009200A200B200C200D200E200F20',

ch2(32) TYPE x VALUE '10201120122013201420152016201720182019201A201B201C201D201E201F20'.

RESULT = TRAN_STRUCTURE-ZZZ.

The only # sign is not permitted

IF STRLEN( RESULT ) = 1. IF RESULT(1) = '#'. RESULT(1) = ' '. ENDIF. ENDIF.

Exclamation mark is not permitted as a first symbol of the field

content
IF RESULT(1) = '!'. RESULT(1) = ' '. ENDIF.

Replace Invalid Characters by SPACE

ASSIGN ch1 TO .

returncode <> 0 means skip this record

RETURNCODE = 0.
abort <> 0 means skip whole data package !!!
ABORT = 0.

Unfortunately, the code will not work in Unicode system.

Some limited solution may be created by using method

3 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

CL_ABAP_CHAR_UTILITIES=>GET_SIMPLE_SPACES_FOR_CUR_CP for getting Unicode presentation of TAB, LF

and CR.

Hopefully, some ABAPers will come up with the solution for Unicode systems also.

Resume

The golden rule in data load in BW:

Use ‘ALL_CAPITAL’ option in RSKC.

Load master data first.

If master data SIDs were created successfully, then problems with ODS activation and loads into cubes may arise
due to:Lowercase letters. Solution: check the ‘Lowercase Letters’ for appropriate infoobject or apply
TRANSLATE RESULT TO UPPER CASE.
statement in a routine in transfer rules (it might be done in a formula too) if you don’t care that all texts will be in
upper case only.

Forbidden characters listed earlier. Solution: apply a routine either posted in forums or provided here.

8367 Views Topics: enterprise_data_warehousing/business_warehouse

Average User Rating

(2 ratings)

Share 1 0 Tweet

30 Comments

Dirk Herzog Jul 24, 2006 3:15 AM

Something you really need in any system. Here is a completely compatible variant that also checks if
the InfoObject allows lower case letters.

DATA: l_d_length like sy-index,

l_d_char type c,
l_d_index type sy-index.
l_d_length = strlen( result ).
result = COMM_STRUCTURE-/BIC/ZZZ.
DO l_d_length times.
l_d_index = sy-index - 1.
l_d_char = result+l_d_index(1).
CALL FUNCTION 'RSKC_CHAVL_OF_IOBJ_CHECK'
IMPORTING
I_CHAVL = l_d_char
I_IOBJNM = 'ZZZ'
*" I_S_COB_PRO =
*" I_T_COB_PRO_CMP =
EXCEPTIONS
CHAVL_NOT_ALLOWED = 1.
IF SY-SUBRC <> 0.
result+l_d_index(1) = ' '.
ENDIF.
ENDDO.

Simply leave all the logic to SAP, don't care about Unicode, etc. The only thing that is (a bit) wrong is
that my code eliminates inner # even if it would pass otherwise. Not a big thing to add.

Best regards
Dirk

Like (0)

Jul 24, 2006 3:49 AM (in response to Dirk Herzog)

Hi Dirk,

It's simply superb!

Finally, we have a code that eliminates invalid characters!

4 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Thank you so much for your support and valuable addition!

Best regards,
Eugene

Like (0)

Nicolas Van Mello Nov 9, 2006 1:43 AM (in response to )

Superb ... but ... why to have the call function in the DO ... ENDDO ? Not needed I
guess.

Like (0)

namuduri VISHALAKSHMI Jun 15, 2007 2:35 AM (in response to Nicolas Van Mello)

Hi,

i want to get this data into the cube....

ST-FÉLICIEN-QC-CA
QUÉBEC-QC-CA

even i have tried RSKC but no use any conversion roune should be written
for these symbols usually we get it in insert symbol in word so ..........pls
send me a mail visalakshi.sapbw@gmail.com

it's very urgent

thanks in advance

vishali...

Like (0)

BI Learner BI Learner Nov 30, 2007 1:13 AM (in response to )

Hi Eugene
I am new to BI..but still I am working in BI..Could you pls help me with regard to the
Invalid character..??
Before that I want to know ...NORMALY
1)SOURCE FIELD LEN CAN BE DIFFERENT FRM OUR TARGET FIELD LEN
2)i have an invalid character error while I am activating the Request..
So if I use ur code..is that I can resolve for the following error..
Error: Value 'Not Applicable' (hex. '4E6F74204170706C696361626C65') of
characteristic ZANS_LAB contains invalid characters
Error: Value 'Eda Lam' (hex. '456461204C616D') of characteristic ZPENAME
contains invalid characters
Could u pls revert me ASAP..
Thanx in advance

Like (0)

BI Learner BI Learner Nov 30, 2007 1:10 AM (in response to Dirk Herzog)

Hi Dirk
I am new to BI..but still I am working in BI..Could you pls help me with regard to the Invalid
character..??
Before that I want to know ...NORMALY
1)SOURCE FIELD LEN CAN BE DIFFERENT FRM OUR TARGET FIELD LEN
2)i have an invalid character error while I am activating the Request..
So if I use ur code..is that I can resolve for the following error..
Error: Value 'Not Applicable' (hex. '4E6F74204170706C696361626C65') of characteristic
ZANS_LAB contains invalid characters
Error: Value 'Eda Lam' (hex. '456461204C616D') of characteristic ZPENAME contains
invalid characters
Could u pls revert me ASAP..
Thanx in advance

Like (0)

Iris Siebrands Jul 18, 2008 9:05 AM (in response to Dirk Herzog)

Hi Dirk,

thanks for that really comfortable solution! Since I've been a bit busy with simply trying to find
this essential peace of information - 2 years after it had been posted - I'd very much
appreciate if SAP could include this central and basic information e.g. in the standard online
SAP-library - for example where the general settings for characteristics are described.

Thanks to all of you for your troubles to help the SAP developer community ... ;-)

Iris

5 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Like (0)

Ian Curtis Jul 26, 2006 7:39 AM

Hi Eugene,

I have been using this setting in RSKC since November 2000 in all BW installations, but it is good to
know that all my recommendations of using it were true. I now know more about it and can really justify
it's inclusion in RSKC.

Thanks

Ian

Like (0)

Jul 26, 2006 10:01 AM (in response to Ian Curtis)

Hi Ian,

Do you mean ALL_CAPITAL?

The most of information provided in the blog is of experimental/experience nature (in the docs
and help you may find just some peaces of truth). I don't exclude that some of the statements
might wrong in some particular circumstances.

So, I welcome any feedback on the matters (either confirming or the opposite).

Best regards,
Eugene

Like (0)

Pedro Pais Sep 14, 2006 4:04 AM

I've tried to use Dirk's code in a transfer rule and had to make a small correction so I thought I'd let you
know.

Instead of entering directly the infoobject's name to the function module I had to declare a 30 character
string, assign it with the infoobject's name and use that string when calling the function.

Thank you for your help.

Like (0)

Krishna Joshi Dec 12, 2006 8:28 AM (in response to Pedro Pais)

Hi Pedro,

Can you please send me working code at nutkhat@optonline.com ? I declared char string
and assinged to infoobject but still having some problems.

Thank you very much in advance.

Kind Regards,
Krishna

Like (0)

Krishna Joshi Dec 12, 2006 10:00 AM (in response to Pedro Pais)

Hi Pedro,

Nevermind, I fixed the code and it is working.

Krishna

Like (0)

Sergio Locatelli Oct 13, 2006 6:26 AM

Very good presentation about the argument. In the past I spent too much time to found ALL_CAPITAL.

Regards,
Sergio

Like (0)

varada raju Dec 21, 2006 8:58 PM

Really intresting.. Good work

Like (0)

6 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Xiuhai Yue Sep 6, 2007 7:59 PM

ABORT = 0.

Like (0)

Eugene Khusainov Sep 6, 2007 9:58 PM (in response to Xiuhai Yue)

It's really very helpful. Now we have a code working in unicode system also. Thank you very
much Xiuhai Yue!

Like (0)

Xiuhai Yue Sep 6, 2007 10:09 PM (in response to Xiuhai Yue)

These codes is not correct when TRAN_STRUCTURE-ZZZ's unicode is '0200'

Like (0)

Meher Dheram Nov 13, 2008 10:49 AM (in response to Xiuhai Yue)

Hi,

I am having an issue with invalid characters. We are on BI 7.0, Unicode. Could you
e-mail me the code at mehervinay_7@yahoo.com.

Thanks,
Meher

Like (0)

Xiuhai Yue Sep 6, 2007 10:34 PM (in response to Xiuhai Yue)

Fix '0200' bug: new code:<br/>--

<br/>FIELD-SYMBOLS: <ic> TYPE x.<br/><br/>DATA:<br/>ch1(64) TYPE x

VALUE<br/>'00000020000100200002002000030020000400200005002000060020000700200008002000
<br/>ch2(64) TYPE x
VALUE<br/>'00100020001100200012002000130020001400200015002000160020001700200018002000
<br/><br/><br/>DATA:<br/> x8(4) type x,<br/> x0(2) type x VALUE '0020',<br/> x0200(2)
type x value '0200'.<br/><br/>RESULT = TRAN_STRUCTURE-ZZZ.<br/><br/>* #
sign<br/>IF STRLEN( RESULT ) = 1.<br/> IF RESULT(1) = '#'.<br/> RESULT(1) = ' '.<br/>
ENDIF.<br/>ENDIF.<br/><br/>* ! sign<br/>IF RESULT(1) = '!'.<br/> RESULT(1) = '
'.<br/>ENDIF.<br/><br/>* 0000-001F

Like (0)

Sherman Ames Nov 4, 2010 10:07 AM (in response to Xiuhai Yue)

I tried the above code to remove invalid data and it did not work.

#N#6.75" X 2", 48VDC, THERMISTOR, REMOTE

The above character # is 0001 and 0002.

Like (0)

Dubravko Cunovic Jan 11, 2009 1:59 PM

Shouldn't it be included in some control on R3 system - that user can not enter invalid characters in
texts? It will solve 90% of problems like that!

Thanks for great blog

Like (0)

HS Kok Oct 1, 2013 7:51 AM (in response to Dubravko Cunovic)

Unfortunately a lot of SAP BI/BI consultants are only concerned with fixing the problem at the
surface level, instead of addressing it at the root cause.

Invalid characters are usually caused by bad entry of data in the source system (ECC, or
non-SAP sources). These must be fixed in the source system, instead of patching the
character values in BI, because:

1) Data mismatch: You won't be able to match up the values between BW and the source
systems, if you start patching up the illegal characters.

(You might as well be transforming every single $100-sales order into $1,000 sales order to
mark up your reported earnings!)

2) Audit: As mentioned in #1, this will be a cause of concern.

3) Future support: If the data from the source system is loaded into multiple DSOs or cubes,
are you going to add the function module and call it in each and every transformation involving
the affected InfoObject/field?

7 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

#3 is usually the reason why most consultants are not bothered by this issue - because they
never stick around the client's company long enough to see things in the long-term (support)
perspective.

Like (0)

Eugene Khusainov Oct 1, 2013 6:31 PM (in response to HS Kok)

Yes, it's exactly what I wrote above: checking of invalid characters in the incoming
datastream in the source system should be dominant.

Like (0)

D Nowley Feb 6, 2009 6:20 AM

These days I just search for help on Blog and not in forums.Thanks Eugene ! Your blogs have been of
great use earlier as well ! :-)

Like (0)

Brian Yang Apr 9, 2009 11:01 PM

I learn a lot from your article. Thanks...

Like (0)

Asrar Ahamed Aug 10, 2009 2:45 AM

Hi Eugene,
Thanks for your wonderful blog which cleared so many doubts. But i am still facing one issue even after
enablling ALL_CAPITAL in RSKC. In ECC some texts are maintained independent of languages as this
represent the name of a person(Data source 0RESPNO_TEXT). The text description includes special
characters like Ššžè..
These are not working in BW. I am getting # in the place all these characters.
Please help me ho to resolve this issue.

Thanks & Regards,

Asrar Ahamed MA

Like (0)

Arun Varadarajan Dec 15, 2010 11:31 AM

Eugene,
I have used the code given by Dirk and some local changes in my system.. however I have a unique
problem .. I have a file that has Chinese , Korean , Japanese etc and I am having problems in loading
this file. Do I have to :
2. Log into the system in Chinese or Japanese and load the file ( The file comes in to the app server
and I would eventually want to load the same through a process chain )
3. Is there anything I can do in terms of settings to be able to load this file while logged into the system
in good old English..?

Thanks ,
Arun Varadarajan

Like (0)

Eugene Khusainov Dec 15, 2010 11:35 AM (in response to Arun Varadarajan)

Sorry, Arun,

I have no experience of dealing with Chinese, Japanese or Korean. Cannot help.

Like (0)

Arun Varadarajan Dec 15, 2010 1:08 PM (in response to Eugene Khusainov)

Thanks Eugene for the response...

We are trying to see if installing the language packs will help. Once this is working , I
should be able to put together a nice blog on the same...

Like (0)

Arun Varadarajan Dec 16, 2010 7:24 AM (in response to Arun Varadarajan)

I set RSKC to ALL_CAPITALS and the data load went through without
issues... thanks for everything....

Like (0)

8 of 9 12-04-2014 12:41 PM
Invalid characters in SAP BW 3.x: Myths and Rea... | SCN http://scn.sap.com/community/data-warehousing/netweaver-bw/blog/200...

Site Index Contact Us SAP Help Portal

Follow SCN
Privacy Terms of Use Legal Disclosure Copyright

9 of 9 12-04-2014 12:41 PM