The tokens of a language are the basic building blocks which can be put together to construct programs. A token can be a reserved word (such as int or while), an identifier (such as b or sum), a constant (such as 25 or "Alice in Wonderland"), a delimiter (such as { or ;) or an operator (such as + or =).
For example, consider the following portion of the program we met in this article:
public class PrintSum {public static void main(String[] args) {int a, b, sum;a = 14;b = 25;sum = a + b;
Starting from the beginning, we can list the tokens (shown in bold) in order:
public - reserved word
class - reserved word
PrintSum - user identifier
{ - left brace, delimiter
public - reserved word
static - reserved word
void - reserved word
main - user identifier
( - left parenthesis, delimiter
String - class name identifier
[ - left square bracket, delimiter
] - right square bracket, delimiter
args - user identifier
) - right parenthesis, delimiter
{ - left brace, delimiter
int - reserved word
a - user identifier
, - comma, delimiter
b - user identifier
, - comma, delimiter
sum - user identifier
; - semicolon, delimiter
a - user identifier
= - equals sign, operator
14 - constant
; - semicolon, delimiter
and so on. Thus we can think of a program as a ‘stream of tokens’, which is precisely how the compiler views it. So that, as far as the compiler is concerned, the above could have been written:
public class PrintSum { public static void main(String[] args) {
int a, b, sum;
a = 14; b = 25; sum = a + b;
The order of the tokens is exactly the same; to the compiler, it is the same program. To the computer, only the order of the tokens is important. However, layout and spacing are important to make the program more readable to human beings.