스크래핑할 요소의 타게팅 - 태그

In [1]:

from IPython.core.interactiveshell import InteractiveShell 
InteractiveShell.ast_node_interactivity = "all"

2-3. 원하는 요소 가져오기 I - 책 이름 모으기¶

• 연습만이 살길! 예시 사이트를 통해 스크래핑을 진행해봅시다.

Target: Mock Book Data¶

다음 사이트에 있는 책들의 이름 정보를 스크래핑 해봅시다 : http://books.toscrape.com/catalogue/category/books/travel_2/index.html

해당 웹 페이지는 임의의 책 정보가 담긴 웹 사이트입니다.

In [3]:

# 스크래핑에 필요한 라이브러리를 불러와봅시다.

import requests # 요청, 응답을 위해
from bs4 import BeautifulSoup # 만들어진 DOM의 파싱을 위해

In [4]:

# 예시 사이트에 요청을 진행하고, 응답을 바탕으로 BeautifulSoup 객체를 만들어봅시다.

res = requests.get("http://books.toscrape.com/catalogue/category/books/travel_2/index.html")
soup = BeautifulSoup(res.text, "html.parser")

In [7]:

# <h3> 태그에 해당하는 요소를 하나 찾아봅시다

soup.find("h3")
print(soup.find("h3").prettify())

Out[7]:

<h3><a href="../../../its-only-the-himalayas_981/index.html" title="It's Only the Himalayas">It's Only the Himalayas</a></h3>

<h3>
 <a href="../../../its-only-the-himalayas_981/index.html" title="It's Only the Himalayas">
  It's Only the Himalayas
 </a>
</h3>

In [10]:

# <h3> 태그에 해당하는 요소를 모두 찾아봅시다

h3_results = soup.find_all("h3")
print(len(h3_results))
print()

for op in outputs:
    print(op)
    print()

11

<h3><a href="../../../its-only-the-himalayas_981/index.html" title="It's Only the Himalayas">It's Only the Himalayas</a></h3>

<h3><a href="../../../full-moon-over-noahs-ark-an-odyssey-to-mount-ararat-and-beyond_811/index.html" title="Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond">Full Moon over Noah’s ...</a></h3>

<h3><a href="../../../see-america-a-celebration-of-our-national-parks-treasured-sites_732/index.html" title="See America: A Celebration of Our National Parks &amp; Treasured Sites">See America: A Celebration ...</a></h3>

<h3><a href="../../../vagabonding-an-uncommon-guide-to-the-art-of-long-term-world-travel_552/index.html" title="Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel">Vagabonding: An Uncommon Guide ...</a></h3>

<h3><a href="../../../under-the-tuscan-sun_504/index.html" title="Under the Tuscan Sun">Under the Tuscan Sun</a></h3>

<h3><a href="../../../a-summer-in-europe_458/index.html" title="A Summer In Europe">A Summer In Europe</a></h3>

<h3><a href="../../../the-great-railway-bazaar_446/index.html" title="The Great Railway Bazaar">The Great Railway Bazaar</a></h3>

<h3><a href="../../../a-year-in-provence-provence-1_421/index.html" title="A Year in Provence (Provence #1)">A Year in Provence ...</a></h3>

<h3><a href="../../../the-road-to-little-dribbling-adventures-of-an-american-in-britain-notes-from-a-small-island-2_277/index.html" title="The Road to Little Dribbling: Adventures of an American in Britain (Notes From a Small Island #2)">The Road to Little ...</a></h3>

<h3><a href="../../../neither-here-nor-there-travels-in-europe_198/index.html" title="Neither Here nor There: Travels in Europe">Neither Here nor There: ...</a></h3>

<h3><a href="../../../1000-places-to-see-before-you-die_1/index.html" title="1,000 Places to See Before You Die">1,000 Places to See ...</a></h3>

찾아온 데이터들은 모두 객체이므로, 저희가 익숙한 방식대로 데이터를 추출할 수 있습니다.

가져온 객체에서 책 제목 뽑아보기¶

먼저 h3 태그 한개만 살펴보자¶

In [12]:

# book_list에서 우리가 원하는 제목(title)만 추출해봅시다.
# 먼저 h3 태그 한개만 가져와서 살펴보자

book = soup.find("h3")
book

Out[12]:

<h3><a href="../../../its-only-the-himalayas_981/index.html" title="It's Only the Himalayas">It's Only the Himalayas</a></h3>

In [14]:

# <h3>태그 바로 아래에 있는 <a>태그를 접근하는 간단한 방법
book.a
# 그 <a>태그의 텍스트에 접근
book.a.text

Out[14]:

<a href="../../../its-only-the-himalayas_981/index.html" title="It's Only the Himalayas">It's Only the Himalayas</a>

Out[14]:

"It's Only the Himalayas"

모든 h3 태그들에 접근해 책 제목을 가져오는 방법¶

반복문을 사용¶

In [16]:

h3_results = soup.find_all("h3")

for title in h3_results:
    print(title.a.text)

It's Only the Himalayas
Full Moon over Noah’s ...
See America: A Celebration ...
Vagabonding: An Uncommon Guide ...
Under the Tuscan Sun
A Summer In Europe
The Great Railway Bazaar
A Year in Provence ...
The Road to Little ...
Neither Here nor There: ...
1,000 Places to See ...

중간에 잘리지 않은 완전한 형태의 책제목 얻기¶

결과물 중에 Full Moon over Noahâ s ... 처럼 말줄임표로 끝나는 온전치 않은 결과물들이 있는데 개발자 도구로 검사를 진행해보면 .text가 아닌 a 태그의 attribute인 title로 접근해야 잘리지 않은 완전한 책제목을 얻을 수 있다.

어떤 태그의 속성(attribute)에 접근하는 방법은 딕셔너리와 같다.¶

In [17]:

for title in h3_results:
    print(title.a["title"])

It's Only the Himalayas
Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond
See America: A Celebration of Our National Parks & Treasured Sites
Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel
Under the Tuscan Sun
A Summer In Europe
The Great Railway Bazaar
A Year in Provence (Provence #1)
The Road to Little Dribbling: Adventures of an American in Britain (Notes From a Small Island #2)
Neither Here nor There: Travels in Europe
1,000 Places to See Before You Die