We see a lot of tables on the web page. If you want to get the data inside or convert it into other formats,
You need to get the form and organize it.
In Python, there are many ways to get web tables. The following are some commonly used methods and libraries:
1. Use Pandas' read_html
The Pandas library provides a very convenient function read_html, which automatically recognizes tables in HTML and converts them into DataFrame objects.
import pandas as pd
# Read from URLdfs = pd.read_html('/some_page_with_tables.html')
# Read from a filedfs = pd.read_html('path_to_your_file.html')
# Access the first DataFramedf = dfs[0]
This method is very simple to obtain tables and is also very convenient to analyze data. It is a common method to directly obtain web tables.
2. Use BeautifulSoup and pandas
If you need more fine-grained control, you can use BeautifulSoup to parse HTML, then manually extract the table data and convert it to a DataFrame of pandas.
from bs4 import BeautifulSoup
import pandas as pd
# Assume html_doc is your HTML contenthtml_doc = """
<table>
<tr>
<th>Column1</th>
<th>Column2</th>
</tr>
<tr>
<td>Value1</td>
<td>Value2</td>
</tr>
</table>
"""
# Use BeautifulSoup to parse HTMLsoup = BeautifulSoup(html_doc, '')
# Extract formtable = ('table')
# Extract the table headerheaders = [ for th in table.find_all('th')]
# Extract table datarows = []
for tr in table.find_all('tr')[1:]: # Skip the header cells = [ for td in tr.find_all('td')]
(cells)
# Create DataFramedf = (rows, columns=headers)
This method mainly involves traversing various parts of the table and saving it. Such methods can be made in detail and adjustments, such as filtering out some unnecessary content.
3. Use the lxml library
lxml is a powerful XML and HTML parsing library that provides XPath support and can be used to extract complex HTML structures.
from lxml import html
# Assume html_doc is your HTML contenthtml_doc = """
<table>
<tr>
<th>Column1</th>
<th>Column2</th>
</tr>
<tr>
<td>Value1</td>
<td>Value2</td>
</tr>
</table>
"""
# parse HTMLtree = (html_doc)
# Extract table data using XPathrows = ('//tr')
# Extract the table headerheaders = [header.text_content() for header in rows[0].xpath('.//th')]
# Extract table datadata = []
for row in rows[1:]:
cells = [cell.text_content() for cell in ('.//td')]
(cells)
# Create DataFramedf = (data, columns=headers)
4. Using Scrapy Framework
Scrapy is an application framework for crawling websites and extracting structured data from pages. It provides a complete set of tools that can be used to handle complex crawler tasks.
import scrapy
class MySpider():
name = 'my_spider'
start_urls = ['/some_page_with_tables.html']
def parse(self, response):
for table in ('table'):
for row in ('tr'):
columns = ('td::text').getall()
yield {
'Column1': columns[0],
'Column2': columns[1],
}
5. Use Selenium's find_element to get it
The specific method is as follows
Detailed method:
1. Import the necessary libraries
First, make sure you have the Selenium library installed and have downloaded the corresponding WebDriver.
from selenium import webdriver from import By
2. Create a WebDriver instance
Create a WebDriver instance, here taking Chrome as an example.
driver = ()
3. Open the landing page
Use the get method to open a web page containing the table.
("/some_page_with_tables.html")
4. Positioning table elements
Use the find_element method to locate the table element.
table = driver.find_element(By.TAG_NAME, 'table')
5. Print the table contents
Method 1: Use get_attribute('outerHTML')
This method can directly obtain the HTML code of the entire table and print it out.
print(table.get_attribute('outerHTML'))
Method 2: Iterate over table rows and cells
If you want to process table data in more detail, you can iterate through each row and cell of the table and then print the contents of each cell.
rows = table.find_elements(By.TAG_NAME, 'tr')
for row in rows:
cells = row.find_elements(By.TAG_NAME, 'td')
cell_texts = [ for cell in cells]
print(cell_texts)
This method prints out the cell text of each row and displays it in a list.
6. Close the browser
Don't forget to close your browser after you're done.
()
Complete code example
from selenium import webdriver
from import By
# Create a WebDriver instancedriver = ()
# Open the landing page("/some_page_with_tables.html")
# Position table elementstable = driver.find_element(By.TAG_NAME, 'table')
# Method 1: Print the HTML of the entire tableprint(table.get_attribute('outerHTML'))
# Method 2: Iterate over and print every row and cell content of the tablerows = table.find_elements(By.TAG_NAME, 'tr')
for row in rows:
cells = row.find_elements(By.TAG_NAME, 'td')
cell_texts = [ for cell in cells]
print(cell_texts)
# Close the browser()
These methods have their own advantages and disadvantages, and you can choose the most suitable method according to your specific needs and the complexity of the project.
For simple table extraction, pd.read_html is usually the fastest way.
For situations where more complex processing is required, BeautifulSoup and lxml and selenium provide more flexibility. Scrapy is suitable for large-scale crawler projects.
This is the article about the various methods of obtaining web forms in Python. For more related contents of python, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!