StringTokenizer can be used for splitting a String into tokens.

There are three constructors of StringTokenizer.

(1.) StringTokenizer(String str)

– Constructs a string tokenizer for the specified string.
– The tokenizer uses the default delimiter set, which is ” \t\n\r\f”: the space character, the tab character, the newline character, the carriage-return character, and the form-feed character.
– Delimiter characters themselves will not be treated as tokens

Lets take an example to understand this.

package com.canvera;
import java.util.StringTokenizer;
public class Main {
	public static void main(String[] args) {
		 StringTokenizer strTokenizer = new StringTokenizer("This is test program for StringTokenizer");
		 while(strTokenizer.hasMoreTokens()){
		     System.out.println(strTokenizer.nextToken());
		 }
	}
}

Output of the above program is as below:

This
is
test
program
for
StringTokenizer

We can see here that the default deliminator taken is space character.

(2.) StringTokenizer(String str,String delim)

– Constructs a string tokenizer for the specified string.
– The characters in the delim argument are the delimiters for separating tokens. Delimiter characters themselves will not be treated as tokens.
– if delim is null, this constructor does not throw an exception. However, trying to invoke other methods on the resulting StringTokenizer may result in a NullPointerException.

package com.canvera;
import java.util.StringTokenizer;
public class Main {
	public static void main(String[] args) {
	        StringTokenizer strTokenizer = new StringTokenizer("This is test program :for StringTokenizer",":");
		 while(strTokenizer.hasMoreTokens()){
		     System.out.println(strTokenizer.nextToken());
		 }
	}
}

Here , the String will be tokenized according to the delimitor “:” .
Output of the above program is as below:

This is test program 
for StringTokenizer

What happens when the delimitor is null. It would throw NullPointerException. This is shown as below:

package com.canvera;
import java.util.StringTokenizer;
public class Main {
	public static void main(String[] args) {
		 StringTokenizer strTokenizer = new StringTokenizer("This is test program :for StringTokenizer",null);
		 while(strTokenizer.hasMoreTokens()){
 		     System.out.println(strTokenizer.nextToken());
		 }
	}
}

This should throw exception as the delimitor is null.
The output of the above program is as below:

Exception in thread "main" java.lang.NullPointerException
	at java.util.StringTokenizer.skipDelimiters(StringTokenizer.java:246)
	at java.util.StringTokenizer.hasMoreTokens(StringTokenizer.java:323)
	at com.canvera.Main.main(Main.java:10)

(3.) StringTokenizer(String str,String delim,boolean flag)

– Constructs a string tokenizer for the specified string.
– All characters in the delim argument are the delimiters for separating tokens.
– If the returnDelims flag is true, then the delimiter characters are also returned as tokens. Each delimiter is returned as a string of length one. If the flag is false, the delimiter characters are skipped and only serve as separators between tokens.
– If delim is null, this constructor does not throw an exception. However, trying to invoke other methods on the resulting StringTokenizer may result in a NullPointerException.

package com.canvera;
import java.util.StringTokenizer;
public class Main {
	public static void main(String[] args) {
	        StringTokenizer strTokenizer = new StringTokenizer("This is test program :for StringTokenizer",":",true);
		 while(strTokenizer.hasMoreTokens()){
			 System.out.println(strTokenizer.nextToken());
		 }
	}
}

Here, the flag is true and hence, the delimitor would also be taken as a token. This can be verified from the output of this program
Output of the above program is as below:

This is test program 
:
for StringTokenizer

If the flag is false, then delimitor would not be taken as a token. Then , the output of the above program will be as below:

This is test program 
for StringTokenizer

Leave a Reply