StringTokenizer and the Missing Token

StringTokenizer and the Missing Token

If we have a comma delimited string like “token0,token1,token2,,token4” that has some token(s) missing, and we try to use java.util.StringTokenizer to place each delimited token in a slot of an array A such that:

 A[0]="token0"; A[1]="token1"; A[2]="token2"; A[3]=""; A[4]="token4"; 

we might write code like:

 String s = "token0,token1,token2,,token4"; java.util.StringTokenizer stringTokenizer=    new java.util.StringTokenizer(s,","); int index = 0; int numTokens = stringTokenizer.countTokens(); System.out.println("num tokens: "+numTokens); String[] A = new String[numTokens]; if(numTokens>0) {    while(stringTokenizer.hasMoreTokens())    {       String token = stringTokenizer.nextToken();       A[index++] = token;    } } 

Notice that array A contains:

  "token0","token1","token2","token4";

We get no indication of the missing token. In fact, the variable numTokens will have the value of 4, representing the number of tokens that are actually in a tokenized string.

The way around this behavior, if we want an empty string indication of the missing token, lies in one of the constructors of the java.util.StringTokenizer class, namely:

 public StringTokenizer(String str,                        String delim,                        boolean returnTokens) 

Here, the boolean returnTokens indicates to the java.util.StringTokenizer instance that we’d like to have, in addition to the actual tokens, the delimiters to be returned as tokens.

The following method uses the above mentioned StringTokenizer constructor, and returns a java.util.Vector whose elements are the tokens of a tokenized string and empty strings for any missing tokens.

 Vector split(String input,String delimiter) {    boolean wasDelimiter=true;    String token=null;    Vector v=new Vector();    StringTokenizer st=new StringTokenizer(input,delimiter,true);    while(st.hasMoreTokens())    {       token=st.nextToken();       if(token.equals(delimiter))       {          if(wasDelimiter)          {             token="";          }          else          {             token=null;          }          wasDelimiter=true;       }       else       {          wasDelimiter=false;       }       if(token!=null)       {          v.addElement(token);       }    }    return v; } 
Share the Post:
XDR solutions

The Benefits of Using XDR Solutions

Cybercriminals constantly adapt their strategies, developing newer, more powerful, and intelligent ways to attack your network. Since security professionals must innovate as well, more conventional endpoint detection solutions have evolved

AI is revolutionizing fraud detection

How AI is Revolutionizing Fraud Detection

Artificial intelligence – commonly known as AI – means a form of technology with multiple uses. As a result, it has become extremely valuable to a number of businesses across

AI innovation

Companies Leading AI Innovation in 2023

Artificial intelligence (AI) has been transforming industries and revolutionizing business operations. AI’s potential to enhance efficiency and productivity has become crucial to many businesses. As we move into 2023, several

data fivetran pricing

Fivetran Pricing Explained

One of the biggest trends of the 21st century is the massive surge in analytics. Analytics is the process of utilizing data to drive future decision-making. With so much of

kubernetes logging

Kubernetes Logging: What You Need to Know

Kubernetes from Google is one of the most popular open-source and free container management solutions made to make managing and deploying applications easier. It has a solid architecture that makes

ransomware cyber attack

Why Is Ransomware Such a Major Threat?

One of the most significant cyber threats faced by modern organizations is a ransomware attack. Ransomware attacks have grown in both sophistication and frequency over the past few years, forcing

data dictionary

Tools You Need to Make a Data Dictionary

Data dictionaries are crucial for organizations of all sizes that deal with large amounts of data. they are centralized repositories of all the data in organizations, including metadata such as