Determining and Providing the Appropriate CCSID for an API, External System, or Language

Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

What do you do when you need to convert back and forth between EBCDIC and ASCII? When dealing with systems other than the IBM i or communicating with languages other than RPG, you will need to use a common language to communicate. The native language of RPG is EBCDIC, and most non-IBM systems and languages work with the ASCII code set, so you will need to convert the characters going out of and coming into RPG. This is where the Coded Character Set Identifier (CCSID) value comes into play when you see it specified as a parameter of an API. The CCSID is not a set of character sets; it is a set of identifiers to refer to character sets. This allows you to work with mixed byte-lengths and encoding by using an identifier that refers to an existing set of characters, such as EBCDIC and ASCII. Some of the CCSID codes are interchangeable, like ASCII and ISO-8859, because the new code set was actually developed to be backward-compatible with the existing standard. Imagine that.... When the ISO-8859 standard was created, it repeated the complete set of the 128 ASCII characters and added another 128 characters after that. UTF-8 did the same thing to support ISO-8859 and extended it even further. UTF-16 did the same thing, but now we are talking about increasing the storage size to two bytes, so it's not physically equivalent. Here is a list of some commonly used codes: Encoding CCSID EBCDIC 37 ASCII 367 ISO-8859-1 819 UTF-8 1208 You can find the complete list of CCSIDs here . The iconv() API converts from one CCSID to another. There is a CCSID for the EBCDIC code set, and there is also a CCSID for the ASCII code set. So you can use the iconv() API to convert

1/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

between EBCDIC and ASCII. Using iconv() is a three-step process: 1. Open the conversion descriptor. 2. Convert a buffer of characters using the conversion descriptor. 3. Close the conversion descriptor. The QDCXLATE API is an alternative to using the iconv() API. The QDCXLATE API uses tables to convert from one character set to the other. The most recent tables used to support EBCDIC and ASCII are QTCPEBC and QTCPASC, respectively. This may be sufficient for simple cases, and you can also create custom tables for use with the QDCXLATE API. The iconv() API may be more complicated than QDCXLATE, but it is more flexible and updated. My objective for this article is to convert between CCSIDs using procedures that will encapsulate the APIs to provide simplified code. The following code illustrates what the final code will look like to convert some EBCDIC bytes into ASCII bytes after we have built some custom procedures that can easily be dropped into a service program. /free // EBCDIC to UTF-8 (ASCII Superset) toCCSID = 1208; ebcdicString = 'AS/400'; cd = Converter_open(toCCSID); asciiString = Converter_convert(cd: ebcdicString); Converter_close(cd); /end-free If you were to debug this program and place a breakpoint immediately after the Converter_convert procedure executes, you would see the following results: eval ebcdicString = 'AS/400 ' eval ebcdicString:x = C1E261F4 F0F04040 4040.... eval asciiString = ' ' eval asciiString:x = 41532F34 30304040 4040.... As you can see here, there is minimal complexity. I was able to remove the need for pointers and complex data structures, with the exception of the conversion descriptor implementation of the iconv_t data structure. I contemplated putting a global structure in the service program, but I changed my mind because I wanted to support multiple construction descriptors in the application. Another alternative was to open and close the conversion descriptor within every conversion call, but I wanted to reduce the overhead by opening and closing the conversion descriptor only once. There's always the balance of simplicity versus speed, and you may

2/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

choose a different alternative, but this code seems simple enough. You may notice that the Converter_open procedure is using only one CCSID in this example. That's because it is assuming that you're converting from EBCDIC and is using the default system value that is indicated by setting the value of zero. If you were going in the opposite direction--ASCII to EBCDIC--you would specify both the to and from CCSID values as follows: /free // UTF-8 to EBCDIC toCCSID = 0; fromCCSID = 1208; cd = Converter_open(toCCSID: fromCCSID); ebcdicString = Converter_convert(cd: asciiString); Converter_close(cd); /end-free You can download the complete code sample here .

The APIs

Now that the objective has been defined, let's dig into the APIs used to perform the conversion. QtqIconvOpen: Open the Conversion Descriptor The QtqIconvOpen API will provide you with a conversion descriptor that the iconv API can use to convert between CCSIDs. We will be converting between EBCDIC and ASCII CCSIDs. When specifying EBCDIC, you can use the value of 37 or you can use the value of 0 to specify the currently used CCSID. Note: There is an alternative API for the QtqIconvOpen API called iconv_open() that has the same functionality. The only difference is in the interface, which uses characters instead of the QtqCode_T data structure. These are the parameters you'll use: - toCode () specifies the destination CCSID of the conversion results. Attribute: QtqCode_t data structure - fromCode specifies the source CCSID of the characters to be converted. Attribute: QtqCode_t data structure The QtqCode_t data structure is defined in the QSYSINC/QRPGLESRC, QTQICONV source

3/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

file. DQtqCode_t DS D QTQCCSID 1 4B 0 D QTQCA 5 8B 0 D QTQSA 9 12B 0 D QTQSA00 13 16B 0 D QTQLO 17 20B 0 D QTQMEO 21 24B 0 D QTQERVED02 25 32 Return: The Conversion Descriptor in an iconv_t Data Structure The iconv_t data structure that is returned by the QtqIconvOpen API is defined in the QSYSINC/QRPGLESRC, ICONV source file. Diconv_t DS D ICORV 1 4B 0 D ICOC 5 52B 0 DIM(00012) This is the QtqIconvOpen prototype: D QtqIconvOpen PR ExtProc('QtqIconvOpen') D like(iconv_t) D argToCCSID like(QtqCode_t) const D argFromCCSID like(QtqCode_t) const iconv: Convert a Buffer of Characters Using the Conversion Descriptor The iconv API converts a set of characters from one CCSID to another using the conversion descriptor. The original characters to be converted will be placed into the input buffer, and the results will be returned in the output buffer. Parameters Input/Output Type

4/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

Conversion Descriptor Input iconv_t Pointer to Input Buffer Input Char(*) Input Buffer Bytes Left Input Integer(10) Pointer to Output Buffer Output Char(*) Output Buffer Bytes Left Output Integer(10) Here's the iconv prototype: D* Convert CCSID from Input Buffer to Output Buffer D iconv PR ExtProc('iconv') D argConvDesc like(iconv_t) value D argInBuffer * D argInBytes 10I 0 D argOutBuffer * D argOutBytes 10I 0 iconv_close: Closing the Conversion Descriptor The iconv_close API closes the conversion descriptor.

5/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

Here's the iconv_close prototype: D* Close the Conversion Descriptor D iconv_close PR 10I 0 ExtProc('iconv_close') D argConvDesc like(iconv_t) VALUE

Creating the Custom Procedures

Opening the Conversion Descriptor *---------------------------------------------------------------- * Converter_open: Opens the Conversion Descriptor for iconv *---------------------------------------------------------------- P Converter_open... P B export D Converter_open... D PI likeDs(iconv_t) D argToCCSID 10I 0 D argFromCCSID 10I 0 options(*nopass) D* Local Variables D from DS likeDs(QtqCode_t) D to DS likeDs(QtqCode_t) D cd DS likeDs(iconv_t) D******************************************************************** /free // Set the target CCSID to = *ALLx'00'; to.QTQCCSID = argToCCSID; to.QTQSA00 = 1; // If Specified, Set the From CCSID from = *ALLx'00';

6/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

if %PARMS < 2; from.QTQCCSID = 0; else; from.QTQCCSID = argFromCCSID; endif; from.QTQSA00 = 1; // If Specified, Set the From CCSID cd = QtqIconvOpen(to: from); if (cd.ICORV < *zeros); // FAILURE else; // SUCCESS endif; return cd; /end-free P E Converting Between CCSIDs Specified Within the Conversion Descriptor *---------------------------------------------------------------- * Converter_convert: Converts the CCSIDs of Conversion Descriptor *---------------------------------------------------------------- P Converter_convert... P B EXPORT D Converter_convert... D PI 65535A D argCd likeDs(iconv_t) D argInString 65535A const varying D******************************************************************** D inBuf S 65535A

7/8

Determining and Providing the Appropriate CCSID for an API, External System, or Language
Written by Thomas Snyder Tuesday, 03 February 2009 19:00 -

D inBufPtr S * D inBufSize S 10I 0 D outBuf S 65535A D outBufPtr S * D outBufSize S 10I 0 D******************************************************************** /free inBuf = argInString; inBufPtr = %addr(inBuf); outBufPtr = %addr(outBuf); // Size is only the initial value // Number of Bytes Left is the true value inBufSize = %len(%trimr(inBuf)); outBufSize = %size(outBuf);

8/8