How strtok () breaks a string into tokens in C?

Question

Please explain to me how the strtok() function works. The manual says that it breaks the string into tokens. I cannot understand from the manual what it actually does.

I added a clock to str and *pch to test it when the first while loop happened, the contents of str was only "this". How was the result shown below printed on the screen?

 /* strtok example */ #include <stdio.h> #include <string.h> int main () { char str[] ="- This, a sample string."; char * pch; printf ("Splitting string \"%s\" into tokens:\n",str); pch = strtok (str," ,.-"); while (pch != NULL) { printf ("%s\n",pch); pch = strtok (NULL, " ,.-"); } return 0; } 

Output:

 Splitting string "- This, a sample string."  into tokens:
 This
 a
 sample
 string

Answer 1

strtok() splits the string into tokens. those. starting from any of the delimiters until the next, this will be your only token. In your case, the initial token will be from "-" and ends with the next space "". Then the next token starts with "" and ends with ",". Here you get "This" as the output. In the same way, the rest of the line is split into tokens from space to space and, finally, the last token ends with "."

Answer 2

strtok execution function works as follows

The first time you call strtok, you provide the string you want tokenize

 char s[] = "this is a string"; 

in the above string space seems like a good separator between words, so it allows you to use this:

 char* p = strtok(s, " "); 

now a search is made for 's' until a whitespace character is found, the first token ('this') is returned and p points to this token (string)

to get the next token and continue with the same string, NULL is passed as the first argument, since strtok supports a static pointer to your previous passed string:

 p = strtok(NULL," "); 

p now points to 'is'

etc., until more spaces are found, then the last line will be returned as the last token 'string'.

more conveniently, you could write this like this, instead, to print all the tokens:

 for (char *p = strtok(s," "); p != NULL; p = strtok(NULL, " ")) { puts(p); } 

EDIT:

If you want to keep the return values ​​from strtok , you need to copy the token to another buffer, for example. strdup(p); since the original string (indicated by a static pointer inside strtok ) changes between iterations to return a token.

Answer 3

strtok maintains a static internal reference pointing to the next available token in the string; if you pass it a NULL pointer, it will work from this internal link.

For this reason, strtok not repetitive; as soon as you give it a new pointer, this old internal link will be knocked down.

Answer 4

strtok does not change the parameter ( str ) itself. It stores this pointer (in a local static variable). He can then change what this parameter indicates in subsequent calls without returning the parameter. (And he can advance this pointer, which he has kept, but he must carry out his operations.)

On the POSIX strtok page:

This function uses static storage to track the current line position between calls.

There is a thread-safe option ( strtok_r ) that does not perform this type of magic.

Answer 5

The first time you call the tokenize string on strtok . And then, to get the following tokens, you simply pass NULL this function if it returns a non- NULL pointer.

The strtok writes the string that you first specified when you called it. (Which is really dangerous for multithreaded applications)

Answer 6

strtok will tokenize the string, i.e. convert it to a series of substrings.

This is done by looking for delimiters separating these tokens (or substrings). And you specify the delimiters. In your case, you want "or", "or". or '-' is a delimiter.

The programming model for extracting these tokens is that you pass strtok your main line and a set of delimiters. Then you call it again, and each time strtok returns the next token that it finds. Until it reaches the end of the main line, when it returns zero. Another rule is that you only pass a string for the first time, and NULL for subsequent times. This is the way to tell strtok if you are starting a new tokenization session using a new line, or you are extracting tokens from a previous tokenization session. Note that strtok remembers its state for the tokenization session. And for this reason, it is not reentrant or thread safe (you should use strtok_r instead). Another thing to know is that it actually modifies the original string. He writes '\ 0' for the delimiters he finds.

One way to invoke strtok, succintly, is as follows:

 char str[] = "this, is the string - I want to parse"; char delim[] = " ,-"; char* token; for (token = strtok(str, delim); token; token = strtok(NULL, delim)) { printf("token=%s\n", token); } 

Result:

 this is the string I want to parse 

Answer 7

strtok changes its input string. It puts null characters ('\ 0') in it so that it returns bits of the original string as tokens. Actually strtok does not allocate memory. You can understand this better if you draw a line as a sequence of boxes.

Answer 8

To understand how strtok() works, you first need to know what a static variable is . This link explains it pretty well ....

The key to the strtok() operation is to save the location of the last separator between secret calls (therefore, strtok() continues parsing the original string itself, which is passed to it when it is called using the null pointer in successive calls).

Take a look at my own implementation of strtok() called zStrtok() , which has poor functionality than the one provided by strtok()

 char *zStrtok(char *str, const char *delim) { static char *static_str=0; /* var to store last address */ int index=0, strlength=0; /* integers for indexes */ int found = 0; /* check if delim is found */ /* delimiter cannot be NULL * if no more char left, return NULL as well */ if (delim==0 || (str == 0 && static_str == 0)) return 0; if (str == 0) str = static_str; /* get length of string */ while(str[strlength]) strlength++; /* find the first occurance of delim */ for (index=0;index<strlength;index++) if (str[index]==delim[0]) { found=1; break; } /* if delim is not contained in str, return str */ if (!found) { static_str = 0; return str; } /* check for consecutive delimiters *if first char is delim, return delim */ if (str[0]==delim[0]) { static_str = (str + 1); return (char *)delim; } /* terminate the string * this assignmetn requires char[], so str has to * be char[] rather than *char */ str[index] = '\0'; /* save the rest of the string */ if ((str + index + 1)!=0) static_str = (str + index + 1); else static_str = 0; return str; } 

And here is a usage example

  Example Usage char str[] = "A,B,,,C"; printf("1 %s\n",zStrtok(s,",")); printf("2 %s\n",zStrtok(NULL,",")); printf("3 %s\n",zStrtok(NULL,",")); printf("4 %s\n",zStrtok(NULL,",")); printf("5 %s\n",zStrtok(NULL,",")); printf("6 %s\n",zStrtok(NULL,",")); Example Output 1 A 2 B 3 , 4 , 5 C 6 (null) 

The code from the string processing library that I support on Github is called zString. Look at the code or even contribute :) https://github.com/fnoyanisi/zString

Answer 9

strtok replaces the characters in the second argument with the NULL character, and the NULL character is also the end of the string.

http://www.cplusplus.com/reference/clibrary/cstring/strtok/

Answer 10

Here is my implementation that uses a hash table for the delimiter, which means that O (n) instead of O (n ^ 2) (here is the code link) :

 #include<stdio.h> #include<stdlib.h> #include<string.h> #define DICT_LEN 256 int *create_delim_dict(char *delim) { int *d = (int*)malloc(sizeof(int)*DICT_LEN); memset((void*)d, 0, sizeof(int)*DICT_LEN); int i; for(i=0; i< strlen(delim); i++) { d[delim[i]] = 1; } return d; } char *my_strtok(char *str, char *delim) { static char *last, *to_free; int *deli_dict = create_delim_dict(delim); if(!deli_dict) { /*this check if we allocate and fail the second time with entering this function */ if(to_free) { free(to_free); } return NULL; } if(str) { last = (char*)malloc(strlen(str)+1); if(!last) { free(deli_dict); return NULL; } to_free = last; strcpy(last, str); } while(deli_dict[*last] && *last != '\0') { last++; } str = last; if(*last == '\0') { free(deli_dict); free(to_free); deli_dict = NULL; to_free = NULL; return NULL; } while (*last != '\0' && !deli_dict[*last]) { last++; } *last = '\0'; last++; free(deli_dict); return str; } int main() { char * str = "- This, a sample string."; char *del = " ,.-"; char *s = my_strtok(str, del); while(s) { printf("%s\n", s); s = my_strtok(NULL, del); } return 0; } 

Answer 11

strtok () stores the pointer in a static variable where you last stopped, so when you call it the second time, when we pass null, strtok () gets the pointer from the static variable.

If you specify the same string name, it starts from the beginning again.

Furthermore strtok () is destructive, i.e. It makes changes to the orignal line. so make sure you always have a copy of orignal.

Another problem with using strtok () is that when storing the address in static variables in multi-threaded programming, calling strtok () more than once causes an error. To do this, use strtok_r ().

Answer 12

This is how I implemented strtok, not so good, but after working 2 hours it finally worked. It supports multiple delimiters.

 #include "stdafx.h" #include <iostream> using namespace std; char* mystrtok(char str[],char filter[]) { if(filter == NULL) { return str; } static char *ptr = str; static int flag = 0; if(flag == 1) { return NULL; } char* ptrReturn = ptr; for(int j = 0; ptr != '\0'; j++) { for(int i=0 ; filter[i] != '\0' ; i++) { if(ptr[j] == '\0') { flag = 1; return ptrReturn; } if( ptr[j] == filter[i]) { ptr[j] = '\0'; ptr+=j+1; return ptrReturn; } } } return NULL; } int _tmain(int argc, _TCHAR* argv[]) { char str[200] = "This,is my,string.test"; char *ppt = mystrtok(str,", ."); while(ppt != NULL ) { cout<< ppt << endl; ppt = mystrtok(NULL,", ."); } return 0; } 

Answer 13

For those who still don't understand this strtok() , take a look at this #include int main () {char s [] = "Hello, my name is? Matthew! Hey."; char * p; for (char * p = strtok (s, ",?!."); p! = NULL; p = strtok (NULL, ",?!.")) {puts (p); } return 0; } & mode = edit & origin = opt-frontend.js & py = c & rawInputLstJSON = [] rel = "nofollow noreferrer"> pythontutor example , this is a great tool for visualizing C code (or C ++, Python ...).

If the link was broken, insert:

 #include <stdio.h> #include <string.h> int main() { char s[] = "Hello, my name is? Matthew! Hey."; char* p; for (char *p = strtok(s," ,?!."); p != NULL; p = strtok(NULL, " ,?!.")) { puts(p); } return 0; } 

Loans are sent to Anders K.