Extract sequential html snippets with

Question

I need to clear html which has the following look:

<div id='content'>
    <h3>Headline1</h3>
    <div>Text1</div>
    <div>Text2</div>
    <div>Text3</div>
    <h3>Headline2</h3>
    <div>Text4</div>
    <div>Text5</div>
    <h3>Headline3</h3>
    <div>Text6</div>
    <div>... and so on ...</div>
</div>

I need to get the contents between the headers as separate fragments. So, from one heading to the next. Unfortunately, there is no container tag for the desired ranges.

I tried the fragment selector {[:h3] [:h3]}, but somehow it only returns those h3 tags, with no tags in between:   (({:tag :h3, :attrs nil, :content ("Headline1")}) ({:tag :h3, :attrs nil, :content ("Headline2")}) ({:tag :h3, :attrs nil, :content ("Headline3")}))

What works is this {[[:h3 (html/nth-of-type 1)]] [[:h3 (html/nth-of-type 2)]]}. This gives me all the html between the first and second h3 tag. However, this does not give me all the pieces I need with one selector.

Can this be done at all, or should I resort to regular expression?

Thank!

Answer 1

div.content, .

, , :

(defn separate*
  "Produces a sequence of (parent child*)*, coll must start with a parent"
  [child? coll]
  (lazy-seq
   (when-let [s (seq coll)]
     (let [run (cons (first s)
                     (take-while child? (next s)))]
       (cons run (separate* child? (drop (count run) s)))))))

partition-by, :

(partition-by keyword? [:foo 1 2 3 :bar :baz 4 5])
;; => ((:foo) (1 2 3) (:bar :baz) (4 5))

(separate* (compliment keyword?) [:foo 1 2 3 :bar :baz 4 5])
;; => ((:foo 1 2 3) (:bar) (:baz 4 5))

, :

(defn separate
  [parent? coll]
  (when-let [s (seq coll)]
    (if (parent? (first coll))
      (separate* (complement parent?) coll)
      (let [child? (complement parent?)
            run (take-while child? s)]
        (cons (cons nil run)
              (separate* child? (drop (count run) s)))))))

(separate keyword? [1 2 :foo 3 4])
;; => ((nil 1 2) (:foo 3 4))

:

(def x [{:tag :h3 :content "1"}
        {:tag :div :content "A"}
        {:tag :div :content "B"}
        {:tag :h3 :content "2"}
        {:tag :div :content "C"}
        {:tag :div :content "D"}])

(def sections (separate #(= :h3 (:tag %)) x))
=> (({:content "1", :tag :h3}
     {:content "A", :tag :div
     {:content "B", :tag :div})
    ({:content "2", :tag :h3}
     {:content "C", :tag :div}
     {:content "D", :tag :div}))

h3:

(map rest sections)
=> (({:content "A", :tag :div} {:content "B", :tag :div})
    ({:content "C", :tag :div} {:content "D", :tag :div}))