ML Reference
Typedefs | Enumerations
mlConvertUTF.h File Reference

Internal conversion file between UTF32, UTF-16, and UTF-8 adopted for usage in mlUtils. More...

#include "mlTypeDefs.h"

Go to the source code of this file.

Macros

Some fundamental constants
#define UNI_REPLACEMENT_CHAR   (static_cast<UTF32>(0x0000FFFD))
 Used instead of invalid characters on lenient conversion.
 
#define UNI_MAX_BMP   (static_cast<UTF32>(0x0000FFFF))
 
#define UNI_MAX_UTF16   (static_cast<UTF32>(0x0010FFFF))
 
#define UNI_MAX_UTF32   (static_cast<UTF32>(0x7FFFFFFF))
 

Typedefs

typedef MLuint32 UTF32
 Note : This file has been adapted to the purpose of ML and MeVisLab usage.
 
typedef MLuint16 UTF16
 at least 16 bits
 
typedef MLuint8 UTF8
 typically 8 bits
 

Enumerations

enum  ConversionResult { conversionOK , sourceExhausted , targetExhausted , sourceIllegal }
 Enum to describe conversion results. More...
 
enum  ConversionFlags { strictConversion = 0 , lenientConversion }
 Enum to describe conversion strictness. More...
 

Functions

Standard conversion routines.
ConversionResult ConvertUTF32toUTF16 (const UTF32 **sourceStart, const UTF32 *sourceEnd, UTF16 **targetStart, UTF16 *targetEnd, ConversionFlags flags)
 Converts UTF32 string to UTF16 string.
 
ConversionResult ConvertUTF16toUTF32 (const UTF16 **sourceStart, const UTF16 *sourceEnd, UTF32 **targetStart, UTF32 *targetEnd, ConversionFlags flags)
 Converts UTF16 string to UTF32 string.
 
ConversionResult ConvertUTF16toUTF8 (const UTF16 **sourceStart, const UTF16 *sourceEnd, UTF8 **targetStart, UTF8 *targetEnd, ConversionFlags flags)
 Converts UTF16 string to UTF8 string.
 
ConversionResult ConvertUTF8toUTF16 (const UTF8 **sourceStart, const UTF8 *sourceEnd, UTF16 **targetStart, UTF16 *targetEnd, ConversionFlags flags)
 Converts UTF8 string to UTF16 string.
 
ConversionResult ConvertUTF32toUTF8 (const UTF32 **sourceStart, const UTF32 *sourceEnd, UTF8 **targetStart, UTF8 *targetEnd, ConversionFlags flags)
 Converts UTF32 string to UTF8 string.
 
ConversionResult ConvertUTF8toUTF32 (const UTF8 **sourceStart, const UTF8 *sourceEnd, UTF32 **targetStart, UTF32 *targetEnd)
 Converts UTF8 string to UTF32 string.
 
MeVis Extra Code
ConversionResult ConvertUTF8toLatin1 (const UTF8 *sourceStart, char *targetStart, char *targetEnd, ConversionFlags flags)
 Convert UTF8 to latin1 chars.
 
ConversionResult CalculateNumCharsInUTF8 (const UTF8 *sourceStart, unsigned int *length, ConversionFlags flags)
 Calculate the number of chars in an UTF8 string (may be less than strlen, because of multibyte encoded chars).
 
ConversionResult CalculateUTF16BufferSizeForUTF8 (const UTF8 *sourceStart, unsigned int *length, ConversionFlags flags)
 Calculate the number of chars required in UTF16 encoding to fit the given UTF8 string (may be less than strlen, because of multibyte encoded chars), but more than CalculateNumCharsInUTF8() because of UTF16 surrogates.
 

Detailed Description

Internal conversion file between UTF32, UTF-16, and UTF-8 adopted for usage in mlUtils.

Definition in file mlConvertUTF.h.

Macro Definition Documentation

◆ UNI_MAX_BMP

#define UNI_MAX_BMP   (static_cast<UTF32>(0x0000FFFF))

Definition at line 136 of file mlConvertUTF.h.

◆ UNI_MAX_UTF16

#define UNI_MAX_UTF16   (static_cast<UTF32>(0x0010FFFF))

Definition at line 137 of file mlConvertUTF.h.

◆ UNI_MAX_UTF32

#define UNI_MAX_UTF32   (static_cast<UTF32>(0x7FFFFFFF))

Definition at line 138 of file mlConvertUTF.h.

◆ UNI_REPLACEMENT_CHAR

#define UNI_REPLACEMENT_CHAR   (static_cast<UTF32>(0x0000FFFD))

Used instead of invalid characters on lenient conversion.

Definition at line 135 of file mlConvertUTF.h.

Typedef Documentation

◆ UTF16

typedef MLuint16 UTF16

at least 16 bits

Definition at line 128 of file mlConvertUTF.h.

◆ UTF32

typedef MLuint32 UTF32

Note : This file has been adapted to the purpose of ML and MeVisLab usage.

That includes

  • changing file type to .cpp,
  • changes in documentation,
  • added tracing code,
  • checks for NULL pointers
  • added exception handlers to get more stability and information in case of crashes,
  • changes of signed integer to unsigned if sign is not needed,
  • removal of warnings and
  • source code reformatting. The following 4 definitions are compiler-specific. The C standard does not guarantee that wchar_t has at least 16 bits, so wchar_t is no less portable than unsigned short! All should be unsigned values to avoid sign extension during bit mask & shift operations. at least 32 bits

Definition at line 127 of file mlConvertUTF.h.

◆ UTF8

typedef MLuint8 UTF8

typically 8 bits

Definition at line 129 of file mlConvertUTF.h.

Enumeration Type Documentation

◆ ConversionFlags

enum ConversionFlags

Enum to describe conversion strictness.

Enumerator
strictConversion 

Strict conversion, error on conversion problem.

lenientConversion 

Non-strict conversion, use '?' or UNI_REPLACEMENT_CHAR instead.

Definition at line 154 of file mlConvertUTF.h.

◆ ConversionResult

enum ConversionResult

Enum to describe conversion results.

Enumerator
conversionOK 

conversion successful

sourceExhausted 

partial character in source, but hit end

targetExhausted 

insuff. room in target for conversion

sourceIllegal 

source sequence is illegal/malformed

Definition at line 144 of file mlConvertUTF.h.

Function Documentation

◆ CalculateNumCharsInUTF8()

ConversionResult CalculateNumCharsInUTF8 ( const UTF8 * sourceStart,
unsigned int * length,
ConversionFlags flags )

Calculate the number of chars in an UTF8 string (may be less than strlen, because of multibyte encoded chars).

If any pointer is NULL then sourceIllegal is returned and *length is set to 0 if non NULL. The return value reports errors or success of conversion.

◆ CalculateUTF16BufferSizeForUTF8()

ConversionResult CalculateUTF16BufferSizeForUTF8 ( const UTF8 * sourceStart,
unsigned int * length,
ConversionFlags flags )

Calculate the number of chars required in UTF16 encoding to fit the given UTF8 string (may be less than strlen, because of multibyte encoded chars), but more than CalculateNumCharsInUTF8() because of UTF16 surrogates.

If any pointer is NULL then sourceIllegal is returned and *length is set to 0 if non NULL. The return value reports errors or success of conversion.

◆ ConvertUTF16toUTF32()

ConversionResult ConvertUTF16toUTF32 ( const UTF16 ** sourceStart,
const UTF16 * sourceEnd,
UTF32 ** targetStart,
UTF32 * targetEnd,
ConversionFlags flags )

Converts UTF16 string to UTF32 string.

For detailed parameter and function description see header comments in ConvertUTF.h.

◆ ConvertUTF16toUTF8()

ConversionResult ConvertUTF16toUTF8 ( const UTF16 ** sourceStart,
const UTF16 * sourceEnd,
UTF8 ** targetStart,
UTF8 * targetEnd,
ConversionFlags flags )

Converts UTF16 string to UTF8 string.

For detailed parameter and function description see header comments in ConvertUTF.h.

◆ ConvertUTF32toUTF16()

ConversionResult ConvertUTF32toUTF16 ( const UTF32 ** sourceStart,
const UTF32 * sourceEnd,
UTF16 ** targetStart,
UTF16 * targetEnd,
ConversionFlags flags )

Converts UTF32 string to UTF16 string.

For detailed parameter and function description see header comments in ConvertUTF.h.

◆ ConvertUTF32toUTF8()

ConversionResult ConvertUTF32toUTF8 ( const UTF32 ** sourceStart,
const UTF32 * sourceEnd,
UTF8 ** targetStart,
UTF8 * targetEnd,
ConversionFlags flags )

Converts UTF32 string to UTF8 string.

For detailed parameter and function description see header comments in ConvertUTF.h.

◆ ConvertUTF8toLatin1()

ConversionResult ConvertUTF8toLatin1 ( const UTF8 * sourceStart,
char * targetStart,
char * targetEnd,
ConversionFlags flags )

Convert UTF8 to latin1 chars.

If sourceStart is NULL then sourceIllegal is returned. If targetStart or targetEnd is NULL then targetExhausted is returned. The return value reports errors or success of conversion.

◆ ConvertUTF8toUTF16()

ConversionResult ConvertUTF8toUTF16 ( const UTF8 ** sourceStart,
const UTF8 * sourceEnd,
UTF16 ** targetStart,
UTF16 * targetEnd,
ConversionFlags flags )

Converts UTF8 string to UTF16 string.

For detailed parameter and function description see header comments in ConvertUTF.h.

◆ ConvertUTF8toUTF32()

ConversionResult ConvertUTF8toUTF32 ( const UTF8 ** sourceStart,
const UTF8 * sourceEnd,
UTF32 ** targetStart,
UTF32 * targetEnd )

Converts UTF8 string to UTF32 string.

For detailed parameter and function description see header comments in ConvertUTF.h.