Using FFMPEG: how to perform scene change detection? with a temporary code?

Question

Based on this article, it seems that FFMPEG can be used to detect scene changes in the video: http://www.luckydinosaur.com/u/ffmpeg-scene-change-detector

Now I have a video that displays the text of the book, and when the text is spoken (word or sentence), it is highlighted. Something like this audiobook: https://youtu.be/lA7L6ZNVKjc

I need to know the timestamp when the text is highlighted (therefore, scene change), this will allow me to add timestamps on my YouTube video, so it’s easier for listeners to navigate the audiobook.

What magic command line will do this?

Many thanks!

Answer 1

Combining the scene filter (to detect scene changes) and the showinfo filter should achieve what you want:

ffmpeg -i input.flv \ -filter:v "select='gt(scene,0.4)',showinfo" \ -f null \ - 2> ffout 

This command retrieves all frames that differ from the previous frame by more than ( gt ) 0.4 (on a scale of 0 to 1 ). For these frames, information is printed ( showinfo ) as follows

 [Parsed_showinfo_1 @ 0x2d85e60] n: 0 pts:2537204 pts_time:2.5372 pos: 2998114 fmt:rgb24 sar:1/1 s:1920x1200 i:P iskey:1 type:I checksum:5616582E plane_checksum:[5616582E] 

Now you only need to extract the timestamp. I think you are interested in pts_time . You can do it like this:

 grep showinfo ffout | grep pts_time:[0-9.]* -o | grep [0-9.]* -o > timestamps 

This will give you a list of all timestamps:

 2.5372 4.37799 6.65301 8.09344 

For this approach to work, you must have a version of FFmpeg that implements scene detection. In addition, you must choose the appropriate value for the threshold ( 0.4 in the first command). You can try to find the optimal threshold by extracting frames for different thresholds (and then checking frames manually) like this

 ffmpeg -i input.flv \ -filter:v "select='gt(scene,0.1)',showinfo" \ -vsync 0 frames/%05d.jpg 

Just to clarify: grep [0-9.]* Does not exclude integers, as indicated in another answer. It matches any sequence of characters consisting of numbers and periods, but it also does not correspond to numbers, such as "4.4.4". However, ffmpeg should not output such poorly-formed timestamps.

Answer 2

I don't have an answer to post a comment on the above answer, but I wanted to indicate that grep sent by both @ckoehn and @keypulsations will only capture timestamps that are floating point. To capture both floating point timestamps and integers, use the following regular expression

 grep showinfo ffout | grep pts_time:[0-9.]* -o | grep -E '[0-9]+(?:\.[0-9]*)?' -o > timestamps 

Answer 3

I tried to answer @ckoehn and it worked until it stopped working, the asterisk in the last grep caused problems. To avoid this, I recommend using double quotes in grep clauses, such as:

 grep showinfo ffout | grep pts_time:[0-9.]* -o | grep "[0-9.]*" -o > timestamps