Introduction to JSONL (JSON Lines)
JSONL (JSON Lines, also known as newline-delimited JSON) is a lightweight data serialization format consisting of a series of independent JSON objects, each line with a valid JSON object separated by a line break (\n).
JSONL is a "lightweight" variant of traditional JSON. Through the design of "one JSON object per line", it solves the memory and efficiency problems during large-scale data processing, and has become an ideal choice for log analysis, big data processing, streaming computing and other scenarios.
Avoid performance overhead when arrays wrap large objects in traditional JSON (such as time and memory consumption for parsing large arrays).
Each line can be verified and processed independently, and errors in one line do not affect other lines (while an error in an element in an array in traditional JSON will cause the entire file to be parsed for failure).
Each line is a legal JSON and can be processed by any JSON parser, just read it line by line.
Core features
1. Simple structure
Each JSON object takes one line, and there are no peripheral array brackets (such as [ and ]) or comma separators in the file.
Example:
{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}
{"name": "Charlie", "age": 35}
2. Streaming friendly
It does not require loading the entire file into memory at one time, and can read and process data line by line, making it suitable for processing super-large data sets (such as GB and TB-level files).
It supports real-time processing (such as log streams, data streams), and parses while generating and having extremely low memory usage.
3. Independent objects
Each row is a complete and independent JSON object, with no dependencies between rows, which is convenient for parallel processing (such as distributed computing and multi-threaded parsing).
Comparison with JSON
| characteristic | JSON | JSONL |
|---|---|---|
| structure | Single object/array (need to be wrapped by {} or []) | Multiple rows of independent objects (one {} per row) |
| Memory usage | Need to load as a whole, large files can easily lead to insufficient memory | Processing line by line, stable memory usage |
| Random access | Supported (by index or key) | Not supported (read in line order) |
| Applicable scenarios | Small-scale data, API interaction, configuration files | Large-scale data, logs, batch processing, streaming data |
| Analysis method | Requires a complete parser (such as JSON library) | Can be read line by line without complex parsing logic |
Application scenarios
1. Big data processing
Store machine learning data sets (such as sample branching, easy to train distributedly).
Process log files (such as web server logs, application monitoring logs, and real-time streaming analysis).
2. Batch data transmission
In the ETL (Extract-Transform-Load) process, it supports incremental processing as a data exchange format between different systems.
Database export/import (such as exporting table data into JSONL by row, for easier subsequent analysis).
3. Parallel computing
Each row of data can be processed independently, suitable for distributed frameworks such as MapReduce and Spark to improve processing efficiency.
4. Simple log format
Instead of CSV and other formats, it supports complex data structures (such as nested objects and arrays), while maintaining readability.
Generate/parse JSONL
Generate: Write JSON strings by line, each line ending with \n (be careful to avoid using non-standard newlines).
Analysis: Read the file line by line and call the JSON parser for each line of string.
Python example (generate JSONL)
import json
data = [
{"name": "Alice", "age": 30},
{"name": "Bob", "age": 25},
{"name": "Charlie", "age": 35}
]
with open("", "w") as f:
for item in data:
((item) + "\n")
Python example (parse JSONL)
import json
with open("", "r") as f:
for line in f:
item = (())
print(item["name"]) # Process data line by lineComplete code
import os
import json
import argparse
from PIL import Image
class ImageConverter:
def __init__(self, src_folder, dest_folder, output_text="a dog", target_size=1024,
output_format="JPEG", quality=95, recursive=False,
prefix="", suffix="", metadata_file=""):
"""
Initialize the image converter
:param src_folder: Source image folder path
:param dest_folder: Destination output folder path
:param output_text: Generate the content of the text field in JSON
:param target_size: The target size of the output image (default is 1024)
:param output_format: output image format (default is JPEG)
:param quality: Output image quality (0-100, only available for certain formats)
:param recursive: Whether to process subfolders recursively
:param prefix: output filename prefix
:param suffix: output filename suffix
:param metadata_file: metadata file name
"""
self.src_folder = src_folder
self.dest_folder = dest_folder
self.output_text = output_text
self.target_size = target_size
self.output_format = output_format
= quality
= recursive
= prefix
= suffix
self.metadata_file = metadata_file
self.image_list = []
def setup_directories(self):
"""Make sure the source folder exists, create the target folder (if it does not exist)"""
if not (self.src_folder):
raise FileNotFoundError(f"Source folder {self.src_folder} Does not exist")
(self.dest_folder, exist_ok=True)
print(f"The target folder is ready: {self.dest_folder}")
def collect_images(self):
"""Collect all supported image files in the source folder"""
supported_formats = ('.png', '.jpg', '.jpeg', '.bmp', '.gif', '.tiff')
self.image_list = []
if :
for root, _, files in (self.src_folder):
for f in files:
if ().endswith(supported_formats):
relative_path = (root, self.src_folder)
target_subfolder = (self.dest_folder, relative_path)
(target_subfolder, exist_ok=True)
self.image_list.append(((root, f), target_subfolder))
else:
files = [f for f in (self.src_folder) if ().endswith(supported_formats)]
() # Sort by file name self.image_list = [((self.src_folder, f), self.dest_folder) for f in files]
print(f"turn up {len(self.image_list)} Image")
def resize_and_pad_image(self, img):
"""
Scaling the image in an equal ratio and putting it into a white background square of the specified size
:param img: PIL image object
:return: Processed new image
"""
original_width, original_height =
ratio = min(self.target_size / original_width, self.target_size / original_height)
new_size = (int(original_width * ratio), int(original_height * ratio))
resized_img = (new_size, )
# Create a white background image padded_img = ("RGB", (self.target_size, self.target_size), (255, 255, 255))
# Center paste position = ((self.target_size - new_size[0]) // 2, (self.target_size - new_size[1]) // 2)
padded_img.paste(resized_img, position)
return padded_img
def convert_and_rename_images(self):
"""Convert image format and rename it while scaling and filling"""
results = []
for idx, (src_path, dest_subfolder) in enumerate(self.image_list):
filename = (src_path)
base_name, _ = (filename)
# Build a new file name new_filename = f"{}{idx:02d}{}.{self.output_format.lower()}"
dest_path = (dest_subfolder, new_filename)
# Calculate the relative path (relative to dest_folder) relative_path = (dest_path, self.dest_folder)
try:
with (src_path) as img:
# Convert to RGB mode (if not) if not in ('RGB', 'RGBA'):
img = ('RGB')
processed_img = self.resize_and_pad_image(img)
# Save parameters according to the output format save_args = {}
if self.output_format.lower() == 'jpeg':
save_args['quality'] =
save_args['optimize'] = True
elif self.output_format.lower() == 'png' and == 'RGBA':
save_args['format'] = 'PNG'
processed_img.save(dest_path, **save_args)
({"text": self.output_text, "file_name": relative_path})
print(f"Saved: {relative_path}")
except Exception as e:
print(f"deal with {filename} An error occurred while: {e}")
({"text": self.output_text, "file_name": relative_path, "error": str(e)})
return results
def generate_jsonl(self, results):
"""Create a file, one JSON object per line, text before"""
jsonl_path = (self.dest_folder, self.metadata_file)
with open(jsonl_path, "w", encoding="utf-8") as f:
for item in results:
(f"{(item, ensure_ascii=False)}\n")
print(f"JSONL File generated: {jsonl_path}")
def run(self):
"""Execute the entire process"""
self.setup_directories()
self.collect_images()
results = self.convert_and_rename_images()
self.generate_jsonl(results)
def main():
parser = (description='Image Converter - Processing images and generating JSONL metadata')
# Required parameters parser.add_argument('--src', required=True, help='Source Image Folder Path')
parser.add_argument('--dest', required=True, help='Target output folder path')
# Optional parameters parser.add_argument('--text', default="a dog", help='The content of text field in JSON')
parser.add_argument('--size', type=int, default=1024, help='Target size of output image')
parser.add_argument('--format', default="JPEG", choices=["JPEG", "PNG", "WEBP"], help='Output image format')
parser.add_argument('--quality', type=int, default=95, help='Output image quality (0-100, only for certain formats)')
parser.add_argument('--recursive', action='store_true', help='Recursively process subfolders')
parser.add_argument('--prefix', default="", help='Output filename prefix')
parser.add_argument('--suffix', default="", help='Output filename suffix')
parser.add_argument('--metadata', default="", help='Metadata file name')
args = parser.parse_args()
# Create and run the converter converter = ImageConverter(
src_folder=,
dest_folder=,
output_text=,
target_size=,
output_format=,
quality=,
recursive=,
prefix=,
suffix=,
metadata_file=
)
()
if __name__ == "__main__":
main()Run the script directly through the command line
python image_converter.py --src path/to/source --dest path/to/destination --text "a dog" --size 512 --recursive
Or call in the code:
# Sample callconverter = ImageConverter(
src_folder="path/to/source/images",
dest_folder="path/to/destination",
output_text="a dog",
target_size=1024,
output_format="PNG",
recursive=True
)
()
result
{"text": "a dog", "file_name": ""}
{"text": "a dog", "file_name": ""}
Python processes images and generates JSONL metadata files - Flexible text version
Complete code
import os
import json
import argparse
from PIL import Image
from xpinyin import Pinyin
class ImageConverter:
def __init__(self, src_folder, dest_folder, target_size=1024,
output_format="JPEG", quality=95, recursive=False,
prefix="", suffix="", metadata_file=""):
"""
Initialize the image converter
:param src_folder: Source image folder path
:param dest_folder: Destination output folder path
:param target_size: The target size of the output image (default is 1024)
:param output_format: output image format (default is JPEG)
:param quality: Output image quality (0-100, only available for certain formats)
:param recursive: Whether to process subfolders recursively
:param prefix: output filename prefix
:param suffix: output filename suffix
:param metadata_file: metadata file name
"""
self.src_folder = src_folder
self.dest_folder = dest_folder
self.target_size = target_size
self.output_format = output_format
= quality
= recursive
= prefix
= suffix
self.metadata_file = metadata_file
self.image_list = []
self.pinyin_converter = Pinyin()
def setup_directories(self):
"""Make sure the source folder exists, create the target folder (if it does not exist)"""
if not (self.src_folder):
raise FileNotFoundError(f"Source folder {self.src_folder} Does not exist")
(self.dest_folder, exist_ok=True)
print(f"The target folder is ready: {self.dest_folder}")
def collect_images(self):
"""Collect all supported image files in the source folder"""
supported_formats = ('.png', '.jpg', '.jpeg', '.bmp', '.gif', '.tiff')
self.image_list = []
if :
for root, _, files in (self.src_folder):
for f in files:
if ().endswith(supported_formats):
relative_path = (root, self.src_folder)
target_subfolder = (self.dest_folder, relative_path)
(target_subfolder, exist_ok=True)
self.image_list.append(((root, f), target_subfolder, f))
else:
files = [f for f in (self.src_folder) if ().endswith(supported_formats)]
() # Sort by file name # Fixed here: Add for loop traversal file for f in files:
self.image_list.append(((self.src_folder, f), self.dest_folder, f))
print(f"turn up {len(self.image_list)} Image")
def filename_to_pinyin(self, filename):
"""Convert the Chinese part of the file name to a tone pinyin"""
base_name, ext = (filename)
chinese_chars = ''.join([c for c in base_name if '\u4e00' <= c <= '\u9fff'])
if not chinese_chars:
return "" # Return empty when there are no Chinese characters (can be adjusted according to requirements)
pinyin_list = []
for char in chinese_chars:
py = self.pinyin_converter.get_pinyin(char, tone_marks='numbers')
# Make sure the tone exists, and the default is 1 if not any(() for c in py):
py += '1'
pinyin_list.append(()) # Convert to lowercase
return '_'.join(pinyin_list)
def resize_and_pad_image(self, img):
"""
Scaling the image in an equal ratio and putting it into a white background square of the specified size
:param img: PIL image object
:return: Processed new image
"""
original_width, original_height =
ratio = min(self.target_size / original_width, self.target_size / original_height)
new_size = (int(original_width * ratio), int(original_height * ratio))
resized_img = (new_size, )
# Create a white background image padded_img = ("RGB", (self.target_size, self.target_size), (255, 255, 255))
# Center paste position = ((self.target_size - new_size[0]) // 2, (self.target_size - new_size[1]) // 2)
padded_img.paste(resized_img, position)
return padded_img
def convert_and_rename_images(self):
"""Convert image format and rename it while scaling and filling"""
results = []
for idx, (src_path, dest_subfolder, filename) in enumerate(self.image_list):
base_name, ext = (filename)
pinyin_text = self.filename_to_pinyin(base_name)
# Build a new file name new_filename = f"{}{idx:02d}{}.{self.output_format.lower()}"
dest_path = (dest_subfolder, new_filename)
# Calculate the relative path (relative to dest_folder) relative_path = (dest_path, self.dest_folder)
try:
with (src_path) as img:
# Convert to RGB mode (if not) if not in ('RGB', 'RGBA'):
img = ('RGB')
processed_img = self.resize_and_pad_image(img)
# Save parameters according to the output format save_args = {}
if self.output_format.lower() == 'jpeg':
save_args['quality'] =
save_args['optimize'] = True
elif self.output_format.lower() == 'png' and == 'RGBA':
save_args['format'] = 'PNG'
processed_img.save(dest_path, **save_args)
({
"text": pinyin_text, # Use the converted pinyin as text "file_name": relative_path
})
print(f"Saved: {relative_path} (text={pinyin_text})")
except Exception as e:
print(f"deal with {filename} An error occurred while: {e}")
({
"text": pinyin_text,
"file_name": relative_path,
"error": str(e)
})
return results
def generate_jsonl(self, results):
"""Create a file, one JSON object per line"""
jsonl_path = (self.dest_folder, self.metadata_file)
with open(jsonl_path, "w", encoding="utf-8") as f:
for item in results:
(f"{(item, ensure_ascii=False)}\n")
print(f"JSONL File generated: {jsonl_path}")
def run(self):
"""Execute the entire process"""
self.setup_directories()
self.collect_images()
results = self.convert_and_rename_images()
self.generate_jsonl(results)
def main():
parser = (description='Image Converter - Processing images and generating JSONL metadata')
# Required parameters parser.add_argument('--src', required=True, help='Source Image Folder Path')
parser.add_argument('--dest', required=True, help='Target output folder path')
# Optional parameters parser.add_argument('--size', type=int, default=1024, help='Target size of output image')
parser.add_argument('--format', default="JPEG", choices=["JPEG", "PNG", "WEBP"], help='Output image format')
parser.add_argument('--quality', type=int, default=95, help='Output image quality (0-100, only for certain formats)')
parser.add_argument('--recursive', action='store_true', help='Recursively process subfolders')
parser.add_argument('--prefix', default="", help='Output filename prefix')
parser.add_argument('--suffix', default="", help='Output filename suffix')
parser.add_argument('--metadata', default="", help='Metadata file name')
args = parser.parse_args()
# Create and run the converter converter = ImageConverter(
src_folder=,
dest_folder=,
target_size=,
output_format=,
quality=,
recursive=,
prefix=,
suffix=,
metadata_file=
)
()
if __name__ == "__main__":
main()
Instead of using fixed output_text, the Chinese part is extracted from the image file name and converted to pinyin.
Added the pinyin conversion function filename_to_pinyin method specifically handles file names, only Chinese characters are retained and converted to tone pinyin (such as Tian.jpg→tian1)
How to use
python --src ./images --dest ./processed_images --size 512 --format PNG
This is the article about Python processing images and generating JSONL metadata files. For more related Python processing images, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!