You would add something like this to the top of the syntax file html.vim:
if getline (1) = ~? '<! DOCTYPE html>'
let b: html5 = 1
else
let b: html5 = 0
endif
And then in the syntax file you can use if b:html5to check if html5 is used for the current buffer.
if b: html5
"new html 5 tags
syn keyword htmlTagName contained video canvas
endif