In a distributed system where communication between services are happening at all time and at the same time network issues or hiccups occur frequently. A well designed service consumer should able to handle the network connection or timeout issues in a graceful manner. This issue is magnified to multiple folds if a service client is time bound and expects responses to come back in milliseconds. So what are the general guidelines or best practices for dealing with the issues outlined above?
Here is a short list of possible solutions. One must carefully pick the right solution for one’s specific application needs.
- Exponential backoff – more info
- Denied access gate with background pinging thread
Regardless of which solution is chosen, what is a good way to simulate the network issues. This is where the “iptable” command comes in. “iptable” command is generally used by network administrators to administer the tables of IP packet filter rules. The rule that is useful for our purpose is “DROP”, which drops the packets on the floor. To simulate a network connection, we can setup a filter rule for a specific host such that all packets that are supposed to go this host will be dropped on the floor.
In short here is the command to set up such filter rule:
sudo iptables -A OUTPUT -p tcp -d <remote host ip> –dport <remote port> -j DROP
When testing is done, make sure to remove the packet filter rule with the following command:
sudo iptables -D OUTPUT -p tcp -d <remote host ip> –dport <remote port> -j DROP
Now we know how to simulate network connection issue and this should help in testing the connection issue error handling code.
