Web Scraping: How to use Python selenium to extract data from HTML table

Image from Edureka

This short tutorial will help you to extract data from HTML table and create a structured data file in the end.

If you want to know how to use the code. You may watch it here

https://www.youtube.com/watch?v=FJbqSqTGBcA

1- Open the Web Page

# Import Library
from selenium import webdriver
import pandas as pd
# Open Browser
driver = webdriver.Chrome(executable_path='D:\chromedriver.exe')
# Get the  URL
url = 'give url'
driver.get(url)
driver.maximize_window()

2(a)- Extract all data from the HTML table

# Read and Convert Web table into data frame
webtable_df = pd.read_html(driver.find_element_by_xpath("//table[@id='dtTbl']").get_attribute('outerHTML'))[0]

3- Write() to CSV file

# Write() to CSV file
webtable_df.to_csv('file2.csv')

I hope this will help to scrape data from web pages using Selenium python. I have used xpath to locate the table on the web page and converted into a DataFrame with a single line code. This way it was easy to write all the information into a file..