Proper Use of DriveApp.continueFileIterator (continuationToken)

Question

I wrote a script to iterate a large number of files in a Google Drive folder. Due to the processing that I do in these files, it exceeds the maximum execution time. Naturally, I wrote in a script to use DriveApp.continueFileIterator (continueationToken): the token is stored in the project properties, and when the script starts, it checks if there is a token if it creates a FileIterator from the token if it does not start again.

What I found, even if restarting the script with the continuation of the token, it still starts from the beginning of the iteration, trying to process the same files that lose time for subsequent executions. Did I miss something vital, like in a team or method, to start from where it left off? Should I update the continuation token at different stages while while (contents.hasNext ())?

Here, the sample code has been reduced to give you an idea:

function listFilesInFolder() { var id= '0fOlDeRiDg'; var scriptProperties = PropertiesService.getScriptProperties(); var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN'); var lastExecution = scriptProperties.getProperty('LAST_EXECUTION'); if (continuationToken == null) { // first time execution, get all files from drive folder var folder = DriveApp.getFolderById(id); var contents = folder.getFiles(); // get the token and store it in a project property var continuationToken = contents.getContinuationToken(); scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', continuationToken); } else { // we continue to import from where we left var contents = DriveApp.continueFileIterator(continuationToken); } var file; var fileID; var name; var dateCreated; while(contents.hasNext()) { file = contents.next(); fileID = file.getId(); name = file.getName(); dateCreated = file.getDateCreated(); if(dateCreated > lastExecution) { processFiles(fileID); } } // Finished processing files so delete continuation token scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN'); var currentExecution = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd HH:mm:ss"); scriptProperties.setProperty('LAST_EXECUTION',currentExecution); }; 

Answer 1

As Jonathan said, you are not comparing dates correctly. But this is not the main problem with your script and what you requested.

The basic concept that you are wrong about is that the continuation token cannot be saved before you make your loop. When you receive a token, it is saved where you were at that moment, if you continue to repeat it later, it is not saved, and you will repeat these steps later, like you.

To get the marker later, you cannot let your script terminate with an error. You need to measure how many files you can process in 5 minutes, and stop your script manually before that so that you can save the token.

Here is the correct way:

 function listFilesInFolder() { var MAX_FILES = 20; //use a safe value, don't be greedy var id = 'folder-id'; var scriptProperties = PropertiesService.getScriptProperties(); var lastExecution = scriptProperties.getProperty('LAST_EXECUTION'); if( lastExecution === null ) lastExecution = ''; var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN'); var iterator = continuationToken == null ? DriveApp.getFolderById(id).getFiles() : DriveApp.continueFileIterator(continuationToken); try { for( var i = 0; i < MAX_FILES && iterator.hasNext(); ++i ) { var file = iterator.next(); var dateCreated = formatDate(file.getDateCreated()); if(dateCreated > lastExecution) processFile(file); } } catch(err) { Logger.log(err); } if( iterator.hasNext() ) { scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', iterator.getContinuationToken()); } else { // Finished processing files so delete continuation token scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN'); scriptProperties.setProperty('LAST_EXECUTION', formatDate(new Date())); } } function formatDate(date) { return Utilities.formatDate(date, "GMT", "yyyy-MM-dd HH:mm:ss"); } function processFile(file) { var id = file.getId(); var name = file.getName(); //your processing... Logger.log(name); } 

In any case, it is possible that the file is created between your runs, and you will not get it at the continuation of the iteration. Then, saving the run time after the last run, you can skip it the next time you run it. I do not know your use case if it is acceptable to end up processing some files or skipping some. If you can’t have a single situation at all, then the only solution I see is to save the identifiers of all the files that you have already processed. You may need to store them in a disk file because the PropertyService may be too small for too many identifiers.

Answer 2

your date comparison will not work the way you do.

 var currentExecution = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd HH:mm:ss"); 

saves "2014-04-18 08:32:01" , while the date of the file.getDateCreated() file will return a Date object comparing them with < or > will always return false.

therefore, I suggest that you save time as a timestamp (because you cannot store Date objects), and then compare this with the timestamp of the created file date.

 // stored time stamp var lastExecution = scriptProperties.getProperty('LAST_EXECUTION'); … dateCreated = file.getDateCreated().getTime(); … var currentExecution = new Date().getTime(); scriptProperties.setProperty('LAST_EXECUTION',currentExecution); 

This comparison will work as you expect.