library(tidyverse)
library(rvest)Website parsing with rvest
What is rvest?
rvest is a very useful R package to parse HTML files of web pages. Usually information is not directly provided in Excel files but embedded in one or more pages in a web site.
Main sources
rvestmain development page: https://rvest.tidyverse.org/- Tutorial on rvest, httr and RSelenium
rvest Usage Examples
Since rvest is part of the tidyverse package we can easily use pipes with it.
In the first example, we will extract tables from BKM’s (Interbank Card Center) sectoral development reports (see the website). Let’s start with getting the web page.
the_url <- "https://bkm.com.tr/secilen-aya-ait-sektorel-gelisim/?filter_year=2020&filter_month=1&List=Listele"
html_obj <- read_html(the_url)When we check the object itself, we see it is a bunch of html code. We are on a good path.
html_objWe can examine the html structure by using the powerful html_structure() function. Since it is a bit of verbose see for yourself.
html_obj %>% html_structure()We can simply extract the table using html_table(). If you get an error about inconsistent fields, use parameter fill=TRUE.
[[1]]
# A tibble: 2 × 3
X1 X2 X3
<chr> <chr> <chr>
1 SEÇİLEN AYA AİT SEKTÖREL GELİŞİM Seçilen Aya Ait Sektörel Gelişim Seçilen Aya…
2 Seçilen Aya Ait Sektörel Gelişim <NA> <NA>
[[2]]
# A tibble: 1 × 1
X1
<chr>
1 Seçilen Aya Ait Sektörel Gelişim
[[3]]
# A tibble: 1 × 1
X1
<chr>
1 "Yıl:\n 2023202220212020201920182017201620152014Ay:\n …
[[4]]
# A tibble: 29 × 5
X1 X2 X3 X4 X5
<chr> <chr> <chr> <chr> <chr>
1 İşyeri Grubu İşlem Adedi İşle… "İşl… "İşl…
2 İşyeri Grubu İşlem Adedi(Kredi K… İşle… "İşl… "İşl…
3 ARABA KİRALAMA 341.271 70.9… "212… "23,…
4 ARAÇ KİRALAMA-SATIŞ/SERVİS/YEDEK PARÇA 3.497.791 969.… "2.6… "217…
5 BENZİN VE YAKIT İSTASYONLARI 27.488.661 11.6… "5.6… "1.0…
6 BIREYSEL EMEKLILIK 2.078.609 2.043 "771… "1,0…
7 ÇEŞİTLİ GIDA 33.777.181 21.5… "5.2… "1.1…
8 DOĞRUDAN PAZARLAMA 627.712 60.3… "728… "11,…
9 EĞİTİM / KIRTASİYE / OFİS MALZEMELERİ 6.274.208 3.56… "2.0… "337…
10 ELEKTRİK-ELEKTRONİK EŞYA, BİLGİSAYAR 8.809.733 2.60… "4.1… "522…
# ℹ 19 more rowshtml_obj %>% html_table(fill = TRUE)Now we are getting somewhere on the fourth item but it is not up to our quality standard. Let’s deploy dplyr functions to make it better.
html_df <- read_html(the_url) %>%
html_table(fill = TRUE) %>%
`[[`(4)
html_df %>%
# Since we do not have too many columns let's rename them manually
# number (num) or value (val) of transactions (txn)
# by credit card (cc) or debit card (dc)
rename(category = 1, num_txn_cc = 2, num_txn_dc = 3, val_txn_cc = 4, val_txn_dc = 5) %>%
# remove the first two rows because they are actually titles
slice(-(1:2)) %>%
# then convert every numeric value by using parse_number function from readr
mutate(
across(
-category,
~ readr::parse_number(.,
locale = readr::locale(decimal_mark = ",", grouping_mark = ".")
)
)
)# A tibble: 27 × 5
category num_txn_cc num_txn_dc val_txn_cc val_txn_dc
<chr> <dbl> <dbl> <dbl> <dbl>
1 ARABA KİRALAMA 341271 70947 212. 23.2
2 ARAÇ KİRALAMA-SATIŞ/SERVİS/YEDEK… 3497791 969202 2676. 217.
3 BENZİN VE YAKIT İSTASYONLARI 27488661 11642138 5608. 1060.
4 BIREYSEL EMEKLILIK 2078609 2043 772. 1.03
5 ÇEŞİTLİ GIDA 33777181 21569718 5284. 1124.
6 DOĞRUDAN PAZARLAMA 627712 60381 728. 11.2
7 EĞİTİM / KIRTASİYE / OFİS MALZEM… 6274208 3565787 2044. 337.
8 ELEKTRİK-ELEKTRONİK EŞYA, BİLGİS… 8809733 2609689 4185. 523.
9 GİYİM VE AKSESUAR 26750805 12487879 5670. 1957.
10 HAVAYOLLARI 2013912 589812 2033. 755.
# ℹ 17 more rowsIn the second example we will harvest the links from Istanbul’s Şehir Hatları (ferry line) domestic trips.
html_obj <- read_html("https://sehirhatlari.istanbul/en/timetables/domestic-trips")Let’s get all the links using “a” nodes and “href” attributes. We are looking for “domestic trips” links.
links_vec <- html_obj %>%
html_nodes("a") %>%
html_attr("href")
links_vec [1] "/en/corporate/privacy-and-cookie-policy-744"
[2] "javascript:__doPostBack('ctl00$Cookie$btnCerezler','')"
[3] NA
[4] "/tr"
[5] "/en"
[6] "#"
[7] "/en/corporate/sehir-hatlari-616"
[8] "/en/corporate/presidents-message-720"
[9] "/en/corporate/general-manager-721"
[10] "/en/corporate/vision-and-values-217"
[11] "/en/corporate/management-systems-policy-364"
[12] "#"
[13] "/en/timetables/domestic-trips"
[14] "/en/timetables/ferry-vehicle"
[15] "/en/timetables/bosphorus-tours"
[16] "/en/price-list"
[17] "/uploads/pdf/Sefer-Haritasi-Line-Map-2022.pdf"
[18] "#"
[19] "/en/information/disabled-guests-186"
[20] "/en/information/corporate-film-618"
[21] "/en/frequently-asked-questions"
[22] "#"
[23] "/en/corporate/rental-services-208"
[24] "/en/corporate/shipyard-services-617"
[25] "/en/corporate/advertising-areas-209"
[26] "/en/corporate/filming-210"
[27] "#"
[28] "/en/contact"
[29] "/en/applies"
[30] "https://twitter.com/sehir_hatlari"
[31] "https://www.facebook.com/sehirhatlari"
[32] "https://www.instagram.com/sehir_hatlari/"
[33] "https://www.youtube.com/user/sehirhatlari1"
[34] NA
[35] "/tr"
[36] "#"
[37] "#"
[38] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines-26"
[39] "/en/timetables/domestic-trips/bosphorus-lines-52"
[40] "/en/timetables/domestic-trips/adalar-princes-islands-lines-176"
[41] "/en"
[42] "/en/timetables"
[43] "#ichatlartarifeleri"
[44] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines-26"
[45] "/en/timetables/domestic-trips/bosphorus-lines-52"
[46] "/en/timetables/domestic-trips/adalar-princes-islands-lines-176"
[47] "/en/timetables/ferry-vehicle"
[48] "/en/timetables/bosphorus-tours"
[49] "#ucrettarifeleri"
[50] "/en/price-list/ferry-lines-79"
[51] "/en/price-list/bosphorus-tours-78"
[52] "/en/price-list/ferry-vehicle-line-77"
[53] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-kabatas-766"
[54] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-karakoy-eminonu-163"
[55] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/uskudar-karakoy-eminonu-164"
[56] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-besiktas-165"
[57] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/halic-hatti-37"
[58] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-karakoy-besiktas-768"
[59] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/bostanci-karakoy-kabatas-166"
[60] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/asiyan-anadolu-hisari-kucuksu-circuit-line-2014"
[61] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/uskudar-asiyan-2015"
[62] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-kasimpasa-fener-haskoy-sutluce-eyupsultan-line-2017"
[63] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/besiktas-kabatas-karakoy-kasimpasa-sutluce-eyupsultan-line-2019"
[64] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/cengelkoy-kabatas-line-2021"
[65] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/bostanci-moda-kadikoy-kabatas-line-2024"
[66] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/avcilar-bakirkoy-yenikapi-kadikoy-line-2027"
[67] "/en/timetables/domestic-trips/inner-istanbul-ferry-lines/avcilar-bostanci-line-2028"
[68] "/en/timetables/domestic-trips/bosphorus-lines/to-bosphorus-from-bosphorus-167"
[69] "/en/timetables/domestic-trips/bosphorus-lines/anadolu-kavagi-rumeli-kavagi-sariyer-168"
[70] "/en/timetables/domestic-trips/bosphorus-lines/kucuksu-besiktas-kabatas-169"
[71] "/en/timetables/domestic-trips/bosphorus-lines/cengelkoy-istinye-170"
[72] "/en/timetables/domestic-trips/bosphorus-lines/kadikoy-sariyer-171"
[73] "/en/timetables/domestic-trips/bosphorus-lines/anadolu-kavagi-uskudar-172"
[74] "/en/timetables/domestic-trips/bosphorus-lines/ortakoy-uskudar-kadikoy-173"
[75] "/en/timetables/domestic-trips/bosphorus-lines/rumeli-kavagi-eminonu-174"
[76] "/en/timetables/domestic-trips/bosphorus-lines/kucuksu-istinye-175"
[77] "/en/timetables/domestic-trips/bosphorus-lines/bebek-emirgan-595"
[78] "/en/timetables/domestic-trips/bosphorus-lines/beykoz-sariyer-767"
[79] "/en/timetables/domestic-trips/bosphorus-lines/ortakoy-besiktas-eminonu-2078"
[80] "/en/timetables/domestic-trips/adalar-princes-islands-lines/kabatas-adalar-177"
[81] "/en/timetables/domestic-trips/adalar-princes-islands-lines/adalar-besiktas-769"
[82] "/en/timetables/domestic-trips/adalar-princes-islands-lines/bostanci-adalar-circuit-line-770"
[83] "/en/timetables/domestic-trips/adalar-princes-islands-lines/maltepe-buyukada-heybeliada-burgazada-kinaliada-lines-2020"
[84] "/uploads/dokuman/46-2/tarife.pdf"
[85] "https://www.ibb.istanbul"
[86] "tel:+153"
[87] "http://ibb.tv/yayin"
[88] "/en/corporate/sehir-hatlari-616"
[89] "/en/corporate/presidents-message-720"
[90] "/en/corporate/general-manager-721"
[91] "/en/corporate/vision-and-values-217"
[92] "/en/corporate/management-systems-policy-364"
[93] "/en/ferries"
[94] "/en/piers"
[95] "/en/journeys/domestic-trips"
[96] "/en/journeys/bosphorus-tours"
[97] "/en/journeys/ferry-vehicle"
[98] "/en/price-list"
[99] "/en/information/disabled-guests-186"
[100] "en/information/corporate-film-618"
[101] "/en/frequently-asked-questions"
[102] "/en/corporate/filming-210"
[103] "/en/corporate/advertising-areas-209"
[104] "/en/corporate/rental-services-208"
[105] "/en/corporate/shipyard-services-617"
[106] "/en/contact"
[107] "/en/applies"
[108] "http://www.sehirhatlari.com.tr/sanaltur/index.htm"
[109] "https://itunes.apple.com/tr/app/%C5%9Fehir-hatlar%C4%B1/id783673371?l=tr&mt=8"
[110] "https://play.google.com/store/apps/details?id=com.spexco.flexcoder2.sehirhatlari.activities&hl=tr"
[111] "https://www.facebook.com/sehirhatlari"
[112] "https://www.instagram.com/sehir_hatlari/"
[113] "https://twitter.com/sehir_hatlari"
[114] "https://www.youtube.com/user/sehirhatlari1"
[115] "https://www.trios.com.tr" With a simple regex and adding the root url we can get all the relevant links from the web page.
domestic_trips_links <- paste0("https://sehirhatlari.istanbul", links_vec[grepl("/domestic-trips/", links_vec)])
domestic_trips_links [1] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines-26"
[2] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines-52"
[3] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines-176"
[4] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines-26"
[5] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines-52"
[6] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines-176"
[7] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-kabatas-766"
[8] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-karakoy-eminonu-163"
[9] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/uskudar-karakoy-eminonu-164"
[10] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-besiktas-165"
[11] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/halic-hatti-37"
[12] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-karakoy-besiktas-768"
[13] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/bostanci-karakoy-kabatas-166"
[14] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/asiyan-anadolu-hisari-kucuksu-circuit-line-2014"
[15] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/uskudar-asiyan-2015"
[16] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-kasimpasa-fener-haskoy-sutluce-eyupsultan-line-2017"
[17] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/besiktas-kabatas-karakoy-kasimpasa-sutluce-eyupsultan-line-2019"
[18] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/cengelkoy-kabatas-line-2021"
[19] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/bostanci-moda-kadikoy-kabatas-line-2024"
[20] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/avcilar-bakirkoy-yenikapi-kadikoy-line-2027"
[21] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/avcilar-bostanci-line-2028"
[22] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/to-bosphorus-from-bosphorus-167"
[23] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/anadolu-kavagi-rumeli-kavagi-sariyer-168"
[24] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/kucuksu-besiktas-kabatas-169"
[25] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/cengelkoy-istinye-170"
[26] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/kadikoy-sariyer-171"
[27] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/anadolu-kavagi-uskudar-172"
[28] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/ortakoy-uskudar-kadikoy-173"
[29] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/rumeli-kavagi-eminonu-174"
[30] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/kucuksu-istinye-175"
[31] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/bebek-emirgan-595"
[32] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/beykoz-sariyer-767"
[33] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/ortakoy-besiktas-eminonu-2078"
[34] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/kabatas-adalar-177"
[35] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/adalar-besiktas-769"
[36] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/bostanci-adalar-circuit-line-770"
[37] "https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/maltepe-buyukada-heybeliada-burgazada-kinaliada-lines-2020" You can click all the links below.
paste0("+ ", domestic_trips_links, " <br>", collapse = " ")[1] “+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines-26
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines-52
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines-176
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines-26
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines-52
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines-176
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-kabatas-766
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-karakoy-eminonu-163
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/uskudar-karakoy-eminonu-164
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-besiktas-165
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/halic-hatti-37
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-karakoy-besiktas-768
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/bostanci-karakoy-kabatas-166
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/asiyan-anadolu-hisari-kucuksu-circuit-line-2014
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/uskudar-asiyan-2015
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/kadikoy-kasimpasa-fener-haskoy-sutluce-eyupsultan-line-2017
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/besiktas-kabatas-karakoy-kasimpasa-sutluce-eyupsultan-line-2019
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/cengelkoy-kabatas-line-2021
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/bostanci-moda-kadikoy-kabatas-line-2024
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/avcilar-bakirkoy-yenikapi-kadikoy-line-2027
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/inner-istanbul-ferry-lines/avcilar-bostanci-line-2028
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/to-bosphorus-from-bosphorus-167
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/anadolu-kavagi-rumeli-kavagi-sariyer-168
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/kucuksu-besiktas-kabatas-169
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/cengelkoy-istinye-170
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/kadikoy-sariyer-171
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/anadolu-kavagi-uskudar-172
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/ortakoy-uskudar-kadikoy-173
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/rumeli-kavagi-eminonu-174
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/kucuksu-istinye-175
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/bebek-emirgan-595
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/beykoz-sariyer-767
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/bosphorus-lines/ortakoy-besiktas-eminonu-2078
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/kabatas-adalar-177
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/adalar-besiktas-769
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/bostanci-adalar-circuit-line-770
+ https://sehirhatlari.istanbul/en/timetables/domestic-trips/adalar-princes-islands-lines/maltepe-buyukada-heybeliada-burgazada-kinaliada-lines-2020
”
Exercises
- Collect multiple periods from BKM page and create an analysis.
- Collect all timetables from Şehir Hatları domestic lines and create a timetable Shiny app.