SoFunction
Updated on 2025-04-11

Python uses TensorFlow to read and batch CSV files

1. Experimental purpose

This experiment aims to use the TensorFlow library to read CSV files and process and present their data in batches. Through this experiment, we hope to master the use of TensorFlow and how to parse CSV data and batch process it.

2. Experimental environment

  • programming language：Python
  • Main library：TensorFlow、os
  • operating system：Windows
  • Experimental data: The CSV file located in C:\Users\30597\Desktop\sye\, containing three columns of data: Name, Age and Occupation.

3. Experimental steps

1. Import the necessary libraries

import tensorflow as tf
import os

ImporttensorflowThe library is used for data processing.osThe library is used for file path verification.

2. Define CSV read function

def csv_reader(file_path, batch_size=2):
    # 1. Create Dataset and skip the header    dataset = (file_path).skip(1)
 
    # 2. Define CSV parsing function    def parse_line(line):
        record_defaults = [
            (["Unknown"], ),  # Name column            ([0], tf.int32),  #Age column            (["Unknown"], )  # Occupation column        ]
        fields = .decode_csv(line, record_defaults)
        return fields
 
    # 3. Application parsing and batch processing    dataset = (parse_line)
    dataset = (batch_size, drop_remainder=False)
    return dataset
  • Create Dataset and skip the header: Use   to read every line of the CSV file and skip the header via skip(1).
  • Define CSV parsing functions: The parse_line function uses .decode_csv to parse data for each row and specify the default value for each column.
  • Application parsing and batch processing: Use the map method to apply the parsing function to each data row, and then use the batch method to divide the data into batches of the specified size.

3. Main program logic

if __name__ == "__main__":
    # Specify the specific file path    csv_path = r"C:\Users\30597\Desktop\sye\"
 
    # Verify the existence of the file    if not (csv_path):
        raise FileNotFoundError(f"CSVfile not found：{csv_path}")
 
    # Create a dataset    dataset = csv_reader(csv_path, batch_size=2)
 
    # Iterate the data batch    for batch_num, (names, ages, occupations) in enumerate(dataset):
        print(f"\nbatch {batch_num + 1}:")
 
        # Decode the byte string into a normal string        names_str = [('utf-8') for name in ()]
        occupations_str = [('utf-8') for occ in ()]
 
        print("Name:", names_str)
        print("age:", ().tolist())
        print("Profession:", occupations_str)
  • Specify file path and verify existence: Use the function to check whether the CSV file exists. If it does not exist, a FileNotFoundError exception will be thrown.
  • Create a dataset: Call the csv_reader function to create a data set.
  • Iterative data batches: Iterate through each batch of the dataset, decode the byte string into a normal string, and print the name, age, and occupation information of each batch.

4. Experimental results

The experiment successfully reads the specified CSV file and processes and presents the data in batches. Each batch contains two records showing name, age, and occupation information respectively. If there are missing values ​​in the CSV file, the default values ​​will be filled.

5. Experimental summary and reflection

advantage

  • Modules using TensorFlow: This module provides efficient data processing and iteration functions, which can easily handle large-scale data sets.
  • Data parsing and batch processing: By defining parsing functions and using map and batch methods, automatic data analysis and batch processing are realized, improving the readability and maintainability of the code.
  • File path verification: Perform path verification before reading the file to avoid runtime errors caused by the non-existence of the file.

Deficiencies and improvement directions

  • Error handling: The current code only handles the situation where the file does not exist, and does not handle exceptions such as CSV file format errors, data type mismatch, etc. More exception handling logic can be added to improve the robustness of the code.
  • Code scalability: The number of columns and default values ​​of the CSV file are hardcoded in the code. If the structure of the CSV file changes, the code needs to be modified manually. You can consider passing column information and default values ​​as parameters to the csv_reader function to improve the scalability of the code.
  • Performance optimization: For large-scale datasets, current batch processing methods may lead to excessive memory usage. You can consider using the prefetch method for data prefetch to improve the performance of data processing.

Overall, this experiment successfully implemented the reading and batch processing of CSV files by using TensorFlow, laying the foundation for subsequent data processing and analysis.

The above is the detailed content of python using TensorFlow to read and batch CSV files. For more information about python TensorFlow to read CSV, please follow my other related articles!