Performance Tuning Issue #22924
-
|
Hi, I have sort of an odd setup, 1 node group of 3 nodes that is setup to receive data largely from aws_s3. This group then send its via vector sink to vector source on another vector node group of 2, all nodes are c6a.2xlarge. The node group that is receiving data is avg around 30K events/s each and CPU is pegged 100% all the time and disk buffer is constantly at around 12GB on each node. BUT the nodes recieiving and doing most of the processing are barely breathing hard like 0-20% CPU load. The bottle neck seems to be on sending data fast enough but Im not certain. Any thoughts on how to better set this up or changes to configs? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Update this was due to s3 ingets of large files >100MB. updated below settings in the source which brought everything down along with worked with the team to reduce file size to a more manageable size. sqs:
client_concurrency: 4
poll_secs: 20
max_number_of_messages: 1 |
Beta Was this translation helpful? Give feedback.
Update this was due to s3 ingets of large files >100MB. updated below settings in the source which brought everything down along with worked with the team to reduce file size to a more manageable size.