Performance optimization points

[liujinyang-highgo]

Hi，
I am currently trying to overcome some system bottlenecks. When I use BenchmarkSQL for stress testing, I capture flame graphs using perf. Below are some points I have identified. Could you please help me check if they are feasible? Or are there any other optimization points?

1. Enarge read buffer
   currently, READBUFSZ is 1024, can we enlarge it to 2048, then can reduce the times to read data from fd.

2. use epoll() instead of select()
   in code branch read_packets_and_process() ->select(), from perf flame graphs ,select() cpu costs reached 4.37%.

3. Relevant processing of pending messages for extended query

   (1)read_kind_from_backend()->pool_pending_message_head_message() : cpu cost--3.27%
   main cost is copy_pending_messgae().
   currently, msg was changed only by：
   pool_pending_message_query_context_dest_set(msg, msg->query_context);
   When copy_pending_messgae(), msg->query_context same as query_context of first msg from 'session_context->pending_messages'.

   so, I think there is no need copy msg, we can return the first msg from 'session_context->pending_messages' directly.
   that will save msg copy cost and some msg free cost.

   (2)read_kind_from_backend()->pool_pending_message_query_context_dest_set() : cpu cost--4%
   this function, the main cost is the loop to set where_to_send[], since the type of node_ids[i] and where_to_send[i] are same, we can use memcpy() instead of this loop
   （3）read_kind_from_backend->pool_pending_message_pull_out() : cpu cost--0.55%
   main cost is copy_pending_message().
   no need copy msg, only return first msg of 'session_context->pending_messages'.
   only one place need the kind and name of this msg in function CloseComplete， so we can add two out parameters for pool_pending_message_pull_out() ：
   pool_pending_message_pull_out(char * kind, char * name);
   this can save msg copy and free cost.

4. pool_flush()

   I summaried pool_flush() cpu cost for main code branch.

process backend msg:
ProcessBackendResponse->CommandComplete->forward_command_complete->forward_packet_to_frontend->pool_flush --cpu cost:5.48%
ProcessBackendResponse->ReadyForQuery->pool_flush cpu cost：--3.45%

process frontend msg:
ProcessFrontendResponse->Bind->pool_extended_send_and_wait->send_extended_protocol_message->pool_flush --cpu cost:0.45%
ProcessFrontendResponse->Execute->pool_extended_send_and_wait->pool_flush --cpu cost:1.17%
ProcessFrontendResponse->Execute->pool_extended_send_and_wait->send_extended_protocol_message->pool_flush --cpu cost:2.36%
ProcessFrontendResponse->SimpleForwardToBackend->pool_flush --cpu cost:5.47%

check msg from frontend and backend:
read_packets_and_process->ProcessFrontendResponse->Bind->pool_extended_send_and_wait->send_extended_protocol_message->pool_flush --cpu cost:2.73%

at present, I have no idea about this part, I am not sure the impaction if use pool_write() in stead of pool_write_and _flush().
waiting for your advice, thanks a lot.     

flame graphs attachment:

all_system_flame_1-pgpool.zip