Fundamentals of Data Representation: ASCII and unicode
ASCII[edit | edit source]
ASCII normally uses 8 bits (1 byte) to store each character. However, the 8th bit is used as a check digit, meaning that only 7 bits are available to store each character. This gives ASCII the ability to store a total of
2^7 = 128 different values.
|There is also extended ASCII that uses the 8th bit to store data, allowing for a much larger character set, but for the exam you'll probably be fine with 7 bit parity ASCII|
ASCII values can take many forms:
- Letters (capitals and lower case are separate)
- Punctuation (?/|\£$ etc.)
- non-printing commands (enter, escape, F1)
Take a look at your keyboard and see how many different keys you have. The number should be 104 for a windows keyboard, or 101 for traditional keyboard. With the shift function valus (a, A; b, B etc.) and recognising that some keys have repeated functionality (two shift keys, the num pad). We roughly have 128 functions that a keyboard can perform.
If you look carefully at the ASCII representation of each character you might notice some patterns. For example:
As you can see, a = 97, b = 98, c = 99. This means that if we are told what value a character is we can easily work out the value of subsequent or prior characters.
Without looking at the ASCII table above! If we are told that the ASCII value for the character '5' is 011 0101, what is the ASCII value for '8'.
We know that '8' is three characters after '5', as 5,6,7,8. This means that the ASCII value of '8' will be three bigger than that for '5':
011 0101 ASCII '5' + 011 -------- 011 1000 ASCII '8'
Checking above this is the correct value.
If you are worried about making mistakes with binary addition, you can deal with the decimal numbers instead. Take the example where you are given the ASCII value of 'g', 110 0111, what is 'e'?
We know that 'e' is two characters before 'g', as e, f, g. This means that the ASCII value of 'e' will be two smaller than that for 'g'.
64 32 16 8 4 2 1 1 1 0 0 1 1 1 = 10310 = ASCII value of 'g' 103 - 2 = 10110 64 32 16 8 4 2 1 1 1 0 0 1 0 1 = 10110 = ASCII value of 'e'
88 - as it is 2 characters down in the alphabet
27 or 26. If you said 23 you'd be wrong because you must include the non-printing characters at the end of each line. Each end of line needs a EOL command, and a new line needs a carriage return (CR), making the text like so:
Hello Pete,[EOL][CR] ASCII rocks![EOL]
For the Latin alphabet ASCII is generally fine, but what if you wanted to write something in Mandarin, or Hindi? We need another coding scheme!
You might have to use ASCII codes when reading from text files. To see what each ASCII code means we can use the folliwing function
For x = 0 To 127 Console.WriteLine("ASCII for " & x & " = " & ChrW(x)) Next Console.ReadLine()
Unicode[edit | edit source]
The problem with ASCII is that it only allows you to represent a small number of characters (~128 or 256 for Extended ASCII). This might be OK if you are living in an English speaking country, but what happens if you live in a country that uses a different character set? For example:
You can see that we quickly run into trouble as ASCII can't possibly store these hundreds of thousands of extra characters in just 7 bits. What we use instead is unicode. There are several versions of unicode, each with using a different number of bits to store data:
|UTF-8||8-bit is the most common unicode format. Characters can take as little as 8-bits, maximizing compatibility with ASCII. But it also allows for variable-width encoding expanding to 16, 24, 32, 40 or 48 bits when dealing with larger sets of characters|
|UTF-16||16-bit, variable-width encoding, can expand to 32 bits.|
|UTF-32||32-bit, fixed-width encoding. Each character takes exactly 32-bits|
With over a million possible characters we should be able to store every character from every language on the planet, take a look at these examples:
|code point||glyph*||character||UTF-16 code units (hex)|
|U+007A||z||LATIN SMALL LETTER Z||007A|
|U+6C34||水||CJK UNIFIED IDEOGRAPH-6C34 (water)||6C34|
|U+10000||LINEAR B SYLLABLE B008 A||D800, DC00|
|U+1D11E||MUSICAL SYMBOL G CLEF||D834, DD1E|
You can find out more about unicode encoding on Wikipedia
100 0111 - as it is 3 characters further on in the alphabet
110 1101 - as it is 6 characters down in the alphabet
Each character only takes up 8 bits, meaning that storing data in ASCII may take up less memory than unicode
ASCII stores a much smaller character set than unicode, meaning that you are limited to the Latin character set and cannot represent characters from other languages.
2^7 = 128
unicode as it would allow you to display non Latin character sets such as Hindi and Cyrillic