Is this scenario a valid stress test?
I have a server application that handles client requests differently.
I want to know how many users can be served with minimum latency, so I made a small stress test application that simulates user requests; at the same time, another application keeps track of memory / CPU usage.
The stress test tool creates a stream every second where each stream represents a user. If a stress test cannot create a new thread due to lack of resources, it launches a new instance of the stress testing tool.
The problem is that each thread writes to the file the delay for each request and the current number of threads, so this causes I / O problems, since after a couple of minutes you have many threads that need to be written to disk this behavior will not exist in real scenarios as the client only requests data.
How can I overcome this problem as I want to measure the maximum latency for each user?
PS:
Some answers say to run on different machines to account for network latency, this will be my last stress test, I am currently doing this test on the same server to see how many users are supported with minimum latency.
a source to share
It is not entirely clear if this is a network application or not. If it's online, you can simply stress test by stealing every desktop over the weekend to stress test it. This may be the easiest way to scale your test if it's just a few custom tests.
However, it looks like there could be some simple improvements. If that means a lengthy stress test, instead of creating a new thread for each request, you can create a thread pool to work with (or even easier to use a thread pool that will scale automatically). So you would define a benchmark to say 2000 users and run 2000 threads that clog the server. Each thread will essentially be in a loop that executes the test and repeats.
Another unclear element is whether all streams you are trying to use are the same file. One way to make this less of a bottleneck is to keep the information in memory until the program exits. Or spin a stream of letters, which is responsible for writing a file, and all other streams provide information to it. If the IO is restored, you will be recording a stream that will be kept in memory until IO is available and your worker threads can continue to clog the server on average. Just keep in mind that due to related thread synchronization this may not scale, so you might want to buffer some writes in the worker thread and only synchronize with the file write thread once every 100 requests. I don't think it will be a big problem as it doesn't seem likethat you are tracking more than response time.
Edit: Based on a comment In this case, I would like to try and use a single thread to control you I / O operations. All of you worker threads instead of writing to a file, create an object with any details and pass it to the queue to be written to the file. To reduce blocking / unlocking, use a queue on the worker thread and also sync them as often. Make sure you block when you are communicating on a thread. Also, I might watch the memory usage as this will allow anything that needs to be built in memory. If that still forces you to block io I would look at writing less, or perhaps tweaking or adding a faster hard drive.
a source to share
If you are interested in the maximum latency for each user, why not just collect this in a thread, and when the test stops, all threads will write our maximum latency. You can also do statistics by calculating min / max / variance and number of threads / users. You also don't have to update the output screen. If you are afraid of data loss, write data to disk often.
Threads are suboptimal, performing this test for a client / server application. With only a limited number of cores, only a very few of the threads actually run in parallel, but get their time limits. It's much better, and gives you some network latency data to run your program on multiple clients. The server software can - if possible - use this hardware as it would in the final configuration, where clients would be running on a LAN or WAN.
Obviously, you will have a mixed environment, as you cannot have many client machines as you simulate users, but scenarios such as concurrent calls from independent hardware will show up in such a stresstest since the calls are not quasi-serialized at time intervals.
a source to share