C++ strtok() – C++ Standard Library

The strtok() function in C++ returns the next token in a C-string (null terminated byte string).

“Tokens” are smaller chunks of the string that are separated by a specified character, called the delimiting character.

This function is defined in the cstring header file.

Example

#include <cstring>
#include <iostream>
using namespace std;

int main() {
  char quote[] = "Remember me when you look at the moon!";

// break the string when it encounters empty space char* word = strtok(quote, " ");

cout << word; return 0; } // Output: Remember

strtok() Syntax

The syntax of strtok() is:

strtok(char* str, const char* delim);

In simple terms,

  • str – the string from which we want to get the tokens
  • delim – the delimiting character i.e. the character that separates the tokens

strtok() Parameters

  • str – pointer to the null-terminated byte string (C-string) to tokenize
  • delim – pointer to the null-terminated byte string that contains the separators

strtok() Return Value

The strtok() function returns:

  • the pointer to the next token if there is any
  • NULL value if no more tokens are found

strtok() Prototype

The prototype of the strtok() function as defined in the cstring header file is:

char* strtok(char* str, const char* delim);

Multiple Calls of strtok()

strtok() can be called multiple times to obtain tokens from the same string. There are two cases:

Case 1: str is not NULL

  • This is the first call to strtok() for that string.
  • The function searches for the first character that is not contained in delim.
  • If no such character is found, the string does not contain any token. So a null pointer is returned.
  • If such a character is found, then from there on the function searches for a character that is present in delim.
    • If no separator is found, str has only one token.
    • If a separator is found, it is replaced by '\0' and the pointer to the following character is stored in a static location for subsequent invocations.
  • Finally, the function returns the pointer to the beginning of the token.

Case 2: str is NULL

  • This call is considered as a subsequent call to strtok() with str.
  • The function continues from where it left in previous invocation.

Example 1: C++ strtok()

#include <cstring>
#include <iostream>
using namespace std;

int main() {
  char quote[] = "Remember me when you look at the moon!";

// break the string when it encounters empty space // str = quote, delim = " " char* word = strtok(quote, " ");

cout << "token1 = " << word << endl;

// get the next token i.e. word before second empty space // NULL indicates we are using the same pointer we used previously i.e. quote word = strtok(NULL, " ");

cout << "token2 = " << word;

// get the third token word = strtok(NULL, " ");

cout << "token3 = " << word; return 0; }

Output

token1 = Remember
token2 = me
token3 = when

In this program,

  • We have tokenized the quote C-string with an empty space " " as the delimiting character delim. This separates quote into tokens the every time strtok() encounters a space " ".
  • The first time we call the function, we need to pass the quote string to specify that it is the source string parameter src.
    char* word = strtok(quote, " ");
  • All subsequent calls to the strtok() function take NULL as the src parameter. This argument instructs the compiler to use the previously-used pointer to C-string (i.e. quote) as the source src.
    word = strtok(NULL, " ");

Example 2: Print All Tokens in a String

#include <cstring>
#include <iostream>
using namespace std;

int main() {
  char str[] = "Remember me when you look at the moon!";
  char delim[] = " ";

  cout << "The tokens are:" << endl;

  // tokenize str in accordance with delim
  char *token = strtok(str,delim);

  // loop until strtok() returns NULL
  while (token)  {

    // print token
   cout << token << endl;
   
    // take subsequent tokens
    token = strtok(NULL,delim);
  }

  return 0;
}

Output

The tokens are:
Remember
me
when
you
look
at
the
moon!