Linux system debugging ltrace tool usage and debugging process

1. Definition and function of ltrace

ltrace is a debugging tool used to track dynamic library functions in Linux environments. It can capture the names, parameters and return values of shared library functions called when the application is running.

Its core functions include:

Analysis of the interaction details of the program and dynamic link library
Problem of exceptions in positioning library function call
Statistics the time and frequency of function calls

The difference from strace

tool	Tracking objects	Application scenarios	Hierarchical relationship
ltrace	User state library function call	Dynamic library interaction analysis	Application Layer
strace	Kernel state system calls	System resource access monitoring	Operating system layer

2. Working principle of ltrace

The intercept code is injected through a dynamic linker (LD_PRELOAD) and the following process is implemented in combination with ptrace system calls.

1. Hijack the process's PLT/GOT table

‌PLT/GOT table function‌：

In a dynamically linked program, the function call passes ‌Process Link Table (PLT)andGlobal Offset Table (GOT)accomplish.
PLT is responsible for jumping to the actual function address stored in the GOT, and the GOT is populated with the real function address by the dynamic linker when the program is run.

‌Hijacking mechanism‌：

Modify the GOT entry of the objective function to point to a custom monitoring function.
For example,putsReplace the GOT address of the function with a custom functionmy_puts, thus implementing call redirection.

2. Redirect function calls to monitoring module

‌LD_PRELOAD Hijacking‌：

useLD_PRELOADPreload the custom dynamic library for environment variables, defining symbols with the same name as the objective function (such asputs）。
When the program is running, the dynamic linker preferentially loads the function implementation in this library, overwriting the original function.

‌Function redirection implementation‌：

Pass in custom librarydlsymGet the original function address and insert monitoring logic into the custom function.

For example:

// Custom dynamic library code ()#define _GNU_SOURCE
#include &lt;&gt;
#include &lt;&gt;

// Define the original function pointertypedef int (*orig_puts_type)(const char*);

int puts(const char* str) {
    // Get the original function address    orig_puts_type orig_puts = (orig_puts_type)dlsym(RTLD_NEXT, "puts");
    // Monitoring logic: Print parameters    printf("[monitor] Call puts(\"%s\")\n", str);
    // Call the original function and return the result    return orig_puts(str);
}

3. Record function entry parameters and return results

‌Parameter capture‌：

In a custom function, access the function parameters directly through the parameter list.
For example, the aboveputsin the functionstrparameter.

‌Return value record‌：

After calling the original function, save the return value and choose to log to log or output in real time.
For example:

int puts(const char* str) {
    orig_puts_type orig_puts = (orig_puts_type)dlsym(...);
    int ret = orig_puts(str);
    printf("[Monitor] Return value is %d\n", ret);
    return ret;
}

4. Real-time output call information to the terminal

‌Terminal output mechanism‌

Use directly in custom functionsprintfOr file operation function outputs monitoring information to the terminal or log file.

‌Process control combined with ptrace‌

passptraceThe system call is attached to the target process, suspends its execution and injects monitoring code.
For example, attach and load custom libraries when the process starts to ensure that the hijacking takes effect.

5. Example: Monitor puts function calls

‌Compile custom libraries:

gcc -shared -fPIC -o   -ldl

‌Run the target program and inject monitoring:

LD_PRELOAD=./ ./target_program

Output effect:

[Monitoring] Call puts("Hello")
Hello
[Monitor] Return value is 6

3. Installation and use

1. Installation method

# Debian/Ubuntu
sudo apt-get install ltrace

# RHEL/CentOS
sudo yum install ltrace

2. Basic usage

‌ltrace‌ Used to track process callsDynamic library functions‌ (such as libc, glibc, etc.), captures function entry parameters, return values and call order, and the syntax is as follows:

ltrace [Options] &lt;Executable file&gt; [Program parameters]

Example

# Basic trackingltrace ./your_program

# Track the specified processltrace -p &lt;PID&gt;

# Output to fileltrace -o  ./server

4. Detailed explanation of functions

1. Tracking library function calls

1.1 Specific function tracking

# Monitor memory related functionsltrace -e "malloc+free" ./memory_test

# Output examplemalloc(1024)         = 0x14762a0
free(0x14762a0)      = &lt;void&gt;

1.2 Full-quantity function tracking

ltrace ./network_tool  # All library function calls are displayed by default

2. Output format analysis

2.1 Typical output contains three core parts

fopen("", "r")       = 0x7f8a5c00b8a0  # Function name + parameter → Return valuestrlen("Hello")                = 5               # string length calculationgettimeofday(0x7ffd8943f370, NULL) = 0           # Time fetch call

2.2 Performance Analysis

ltrace -c ./algorithm

# Output example% time     seconds  usecs/call     calls  function
 35.21    0.004235         105        40   malloc
 28.17    0.003387          84        40   free
 20.04    0.002410          60        40   strlen

3. Debugging dynamic link library issues

# Check SSL library callsltrace -e "SSL_*" ./https_client

# Typical problem: SSL_shutdown not calledSSL_new(0x7f1344000ac0)          = 0x152c300
SSL_connect(0x152c300)           = -1

Safety Analysis

# Monitor file operationsltrace -e "fopen+fclose" ./uploader

#Suspicious behavior examplesfopen("/etc/passwd", "r")        = 0x173da20

4. Learn library function usage

ltrace ./encryption_tool | grep AES_
AES_set_encrypt_key("secret", 128, 0x7ffc52a3fb10) = 0
AES_cbc_encrypt(0x7ffc52a3fb90, 0x173da20, 64, ...) = <void>

5. Performance optimization

ltrace -cS ./image_processor | sort -nrk 1
# Results show 80% Time is consumed in libjpeg of jpeg_write_scanlines()

5. Common options and filtering techniques

‌Options‌	effect	‌Example‌
-e <function name>	Filter specific functions (regularity supports)	-e 'mem*' Match memcpy/memset
-c	Statistics the number of function calls and time consumption	ltrace -c ./app
-o <file>	Output to log file	-o
-f	Tracking child processes (multi-process programs)	-f ./multiprocess_app
-t	Display timestamp (-tt microsecond accuracy)	-tt for performance analysis

‌Combination usage example‌:

ltrace -f -e 'f*' -tt -o  ./multithread_app  # Track allfFunctions at the beginning，Record child processes and timestamps

5. Error scenario diagnosis

FAQ	‌ltrace performance‌	‌Solution‌
Dynamic library not found	dlopen("", 1) = 0	Check LD_LIBRARY_PATH or Install Library
Function parameter type error	printf(0x55a1a2e2e260, 0x7f, 0x2a)	Verify the matching of format strings and parameters
Double memory release	free(0x55a1a2e2e260) = <void> appears multiple times	Check the free call logic in the code
File handle leak	fopen("", "w") No corresponding fclose	Ensure resource release

‌Error log example‌:

fopen("", "r")                     = 0  # Return the NULL pointer, actually check errnoprintf("%s", 0x55a1a2e2e260)                  = -1 # Failure due to mismatch of parameter types

6. Debugging skills

1. Track running processes

ltrace -p 1234 -e fprintf  # Attached toPID 1234And monitorfprintfCall

2. Filter third-party library functions

ltrace -e '*' ./https_client  # trackOpenSSLAll functions of the library

3. Signal and multithreaded support

ltrace -f -i ./multithread_server  # Track multithreading and display instruction pointers

7. Key tracking categories and examples

1. Memory management functions

Debugging scenario: Detect memory leaks (mallocUnmatchedfree）

ltrace -e malloc,free ./memory_app

#####################################################################################################malloc(1024)            = 0x55a1a2e2e260  # Allocate 1KB of memoryfree(0x55a1a2e2e260)    = &lt;void&gt;          # Free memory

2. String operation function

Error analysis:strcpyTrigger buffer overflow (destination address space is insufficient)

ltrace -e strcpy,strlen ./string_processor

#########################################
strlen("Hello")                              = 5
strcpy(0x7ffd4a3d4e60, "World")              = 0x7ffd4a3d4e60  # Copy string

3. File I/O functions

fopen/fread/fclose‌Traces standard file flow operations.

Performance optimization: high frequency and small sizefreadPrompt to increase the buffer area

ltrace -e fopen,fread,fclose ./file_reader

######### Example output####################################################
fopen("", "rb")                      = 0x55a1a2e2e290  # Open the filefread(0x7ffd4a3d4e60, 1, 4096, 0x55a1a2e2e290) = 1024           # Read 1024 bytesfclose(0x55a1a2e2e290)                       = 0               # Close the file

4. Mathematical library functions

ltrace -c -e pow,sqrt ./math_solver          # -c Statistics the number of calls and time consumption
#########################################% time   seconds  usecs/call   calls   function
------ ----------- ----------- ------ ------------
 68.2    0.420105     4201        100    pow
 31.8    0.196200     1962        100    sqrt

5. Network communication functions

ltrace -e gethostbyname,connect ./network_client

#######################################################################
gethostbyname("")                 = 0x55a1a2e2e350  #DNS resolutionconnect(3, {sa_family=AF_INET, sin_port=htons(80)...}, 16) = 0  # TCPconnect

6. Hotspot Function Analysis

Optimization direction: high frequencymallocPrompt to introduce memory pool

ltrace -c ./image_processor  # Output function call statistics table

#########################################
% time   seconds  usecs/call   calls   function
------ ----------- ----------- ------ ------------
 45.3    1.20210      1202      1000    malloc
 30.1    0.80105       801      1000    free
 24.6    0.65300       653      1000    memcpy

7. Time-consuming function positioning

Combined-TShows the time taken for a single call

ltrace -T ./encryption_tool

###############################################
AES_encrypt(0x7ffd4a3d4e60, 0x7ffd4a3d4f60, 0x55a1a2e2e290) = &lt;void&gt; &lt;3.142000&gt;  # Time-consuming for a single encryption3.1Second

Summarize

The above is personal experience. I hope you can give you a reference and I hope you can support me more.