3rd week of Web development course

Olivia·2022년 5월 11일

beautifulsoup python

Web development course

목록 보기

3/4

The goals

🤓Quick review of last week

1. Understand the basic python syntax

2. Crawling pages

3. Control mongoDB through pymongo

🤔Practice-scraping music website

Quick review of last week

<script>
        $(document).ready(function(){
        listing();
        });

        function listing() {
            $('#cards-box').empty()
         $.ajax({
             type: "GET",
             url: "http://spartacodingclub.shop/web/api/movie",
             data: {},
             success: function(response) {
                 let rows = response['movies']
                 for (let i = 0; i < rows.length; i++) {
                     let title = rows[i]['title']
                     let desc = rows[i]['desc']
                     let image = rows[i]['image']
                     let star = rows[i]['star']
                     let comment = rows[i]['comment']

                     let star_image = '⭐'.repeat(star)
                    
                     let temp_html = `
                     <div class="col">
                         <div class="card h-100">
                             <img src="${image}"
                                 class="card-img-top" alt="...">
                             <div class="card-body">
                                 <h5 class="card-title">${title}</h5>
                                 <p class="card-text">${desc}</p>
                                 <p>${star_image}</p>
                                 <p class="mycomment">${comment}</p>
                             </div>
                         </div>
                     </div>
                     `
                     $('#cards-box').append(temp_html);
                 }
             }
         })
        }

        function open_box() {
            $('#post-box').show();
        } 
        function close_box() {
            $('#post-box').hide();
        }
    </script>

Since last week I have learned what API is and how to use it, this time used movie API to get movie data and used jQuery to show the only what I need among all the API data.

What is Python?

Python is a clear and powerful object-oriented programming language, comparable to Perl, Ruby, Scheme, or Java.
Python combines remarkable power with very clear syntax. It has modules, classes, exceptions, very high level dynamic data types, and dynamic typing. There are interfaces to many system calls and libraries, as well as to various windowing systems. New built-in modules are easily written in C or C++ (or other languages, depending on the chosen implementation). Python is also usable as an extension language for applications written in other languages that need easy-to-use scripting or automation interfaces.

❗️Indentation is really important in Python!

Understand the basic syntax of Python

Python package

We need to use library which is already made by other people to crawl with Python. We call it as "Python package".
Just imagine a situation that your project leader wants to change a version of project you are working on, then do we need to start all over again on new version?
Nope! definitely not! so we need "Virtual Environment" which we usually call as "venv". It contains python packages.

Installed package 'request' to extract selected data from API.

Filtered!

Try Beautiful Soup!

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.

This is the basic usage of Beautiful Soup. We can pull data out simply and quickly. In this case, I used IMDB movie ranking page to scrape data.

Database

In computing, a database is an organized collection of data stored and accessed electronically. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage.
We use a database to find data conveniently!

Connected to Mongo DB and uploaded data (above picture)

And then tried to extract data like above picture.

We can also update data directly by writing code below,

db.users.update_one({'name':'Bob'},{'$set':{'age':19}})

also delete data with this code.

db.users.delete_one({'name':'Bob'})

Try out with some quizzes

# Bring the rating of movie '가버나움(Capernaum)'
movie = db.movies.find_one({'title':'가버나움'})
print(movie['star'])

# Bring a movie has the same rating with Capernaum
star = movie['star']
all_movies = list(db.movies.find({'star': star},{'_id': False}))
for m in all_movies:
    print(m['title'])

# Change the rating of Capernaum
db.movies.update_one({'title':'가버나움'},{'$set':{'star':'0'}})

🤔Practice-scraping music website

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
data = requests.get('https://www.genie.co.kr/chart/genre?ditc=D&ymd=20220510&genrecode=M0200',headers=headers)

soup = BeautifulSoup(data.text, 'html.parser')

songs = soup.select('#body-content > div.newest-list > div > table > tbody > tr')

for song in songs:
    rank = song.select_one('td.number').text[0:2].strip()
    title = song.select_one('td.info > a.title.ellipsis').text.strip()
    artist = song.select_one('td.info > a.artist.ellipsis').text
    print(rank, title, artist)

And then if we run this code, the result is printed out like this: