At present, I develop HTTP services and use the beego framework, which is much more convenient.
However, sometimes, there are still some special scenes.
For example: Filter logs.
This should be a typical stream, and the data volume is also moderate, so there will be no one. For this, we use some heavy frameworks.
This logic can be described intuitively
Other components generate log
||
\ /
My components, business processing
||
\ /
User, http client
In this scenario, there are several special points:
1. It is difficult to collect data using string or byte array
2. On the data source side, data generation is constantly
3. Data buffering, if it occupies too much memory, it may cause the service to crash.
Normally, we prepare the data, then call the Beego framework method to send the data to the client, and ignore it.
And what should we do if we need to write data to the client multiple times according to the processing situation?
First, for this simple streaming data, golang provides a structure.
pipeReader, pipeWriter := ()
This is the prototype of this method
func Pipe() (*PipeReader, *PipeWriter)
It returns a pair of Readers and Writers that are closely linked. Their "life cycle" is the same.
Any data written to the Writer flows directly into the Reader. This is very similar to "pipeline |" in the Linux command line.
Let's start a goroutine to receive log data
go func () {
for {
var log []byte
//log =
(log)
//break;
}
()
}
In main logic, process logs
defer ()
rr := ((pipeReader))
for {
line, err := ('\n')
if == err {
break
}
........
}
Finally, output to the client
var out []byte (out) ()
Summarize:
iopipe directly connects to the log output, the buffer is very small.
The processed results are output directly to the http client.
Especially the second point, it is very important. When I was processing this logic, I found that the server crashed several times. Later, I realized that if the beego controller buffers the processed data, it may still occupy a large amount of memory.
The above is personal experience. I hope you can give you a reference and I hope you can support me more. If there are any mistakes or no complete considerations, I would like to give you advice.