Geek Answers Handbook

How to create a hierarchy of wikipedia categories?

I am trying to build a treegraph of wikipedia articles and its categories. What do I need to do?

From this site ( http://dumps.wikimedia.org/enwiki/latest/ ), I downloaded:

  • enwiki-last-page.sql.gz
  • enwiki-last-categorylinks.sql.gz
  • enwiki-20141106-category.sql.gz

I tried to follow the answer here ( Wikipedia's Category Hierarchy of Dumps ), but it doesn't seem like the category links have the same layout (without the pageId column).

What is the correct way to build a hierarchy?

Bonus question: how can I determine which of the 35M pages in enwiki-latest-page.sql.gz are articles (presumably about 5 million according to wikipedia statistics).

thank

+4
source share
2 answers

Yes, it turns out that this fooobar.com/questions/605311 / ... was right. He referenced the correct datasets, but I was too dense to understand how to link them together.

Thanks to @svick for going through individual steps in private chat.

For the benefit of others, I have described in detail the relationship between data sets and the exact steps to cross the graph in my blog, which is a summary of our private chat.

Analysis Wikipedia Pages Hierarchy

+3
source

I met the same issue for Japanese Wikipedia.

I solved this problem as follows:

  • get sql for category, link categories, page and import to my mysql server.
  • run the following command. You can get the subcategory "学問".
    MariaDB [wikipedia]> select page.page_title from categorylinks join page on page.page_id = categorylinks.cl_from join category on categorylinks.cl_to = category.cat_title where categorylinks.cl_type = 'subcat' and category.cat_title like '学問';
+-----------------------------------+
| page_title                        |
+-----------------------------------+
| 学問の分野                        |
| 科学                              |
| 学問スタブ                        |
| 架空の思想・学問                  |
| 学者                              |
| 学術出版                          |
| 学術称号                          |
| 学術団体                          |
| 学生                              |
| 学派                              |
| 学問の賞                          |
| 研究                              |
| 高等教育                          |
| 知識                              |
| 問題                              |
| ルネサンス・ユマニスム            |
+-----------------------------------+
16 rows in set (0.00 sec)
+1

More articles:


All Articles