Load balancing difficulties

yherrebaut · Posted: Wed Mar 31, 2010 12:13 pm **Post subject: Load balancing difficulties**

I have configured two FirstSpot servers in load balancing setup, but I'm having some difficulties.

The servers themselves start perfectly fine in load balancing setup, but the clients have connectivity issues: sometimes they don't seem able to load websites, or the loading takes extremely long.

When I checked the IP configuration of my clients, they seem to be getting two default gateways (the IP adresses of both FirstSpot servers), which is (according to me) the source of the connectivity issues. I've tried giving a client a static IP configuration with only one default gateway (the IP address of the primary server) and then everything worked as it should. When I tried a static IP configuration with the IP address of the secondary server as default gateway, the client couldn't ping any destination past the default gateway. However when I tried both FirstSpot servers separately in single mode, they both performed as expected.

Currently I'm using the FirstSpot DHCP servers, but in the future I'm probably going to use a separate DHCP server, which should normally give only one default gateway to the clients.

As far as I understand, it seems that each client should balance its load between the two servers, and not that the clients are distributed between the servers (since every client is redirected to the portal site of the primary server). If I'm going to use a separate DHCP server, which will give only one default gateway, all traffic will go through this one gateway and there's really no point in configuring load balancing. A solution I can think of is to place a gateway server between the access points and the FirstSpot servers, but then again there's no point in configuring FirstSpot load balancing.

So I'm wondering if there is a solution to this multiple default gateway issue, or am I doing something wrong here?

yherrebaut · Posted: Wed Mar 31, 2010 2:00 pm **Post subject:**

I want to add to my previous post that this problem isn't caused by a misconfiguration of my secondary FirstSpot server. I've switched primary and secondary server, and the problem stays exactly the same: the server that is set with highest priority (doesn't matter which machine it is) works fine, but the server that is set with lower priority (again, it doesn't matter which machine this is) refuses to act as gateway. So every time a request is sent to the second gateway, it fails.

alan · **Forum facilitator** Joined: 26 Sep 2003 Posts: 4435

FirstSpot Load Balancing works by "partitioning" client PCs to different 2 different FirstSpot servers. So it is normal that the clients get 2 default gateways (in 2 different sequence).

To move forward, please start FirstSpot, and then, email us at firstspot@patronsoft.com both your firstspot_gw_srv_[timestamp].log and firstspot_HB_[timestamp].log in both FirstSpot machine.

Also, try to capture the problematic client "ipconfig/all" and "tracert patronsoft.com" output.
_________________
~ Patronsoft Limited ~

yherrebaut · Posted: Thu Apr 01, 2010 11:21 am **Post subject:**

I've sent you an email with the requested documents.
Hopefully you can discover something usefull.

alan · **Forum facilitator** Joined: 26 Sep 2003 Posts: 4435

The tracert looks okay. The client can access the Internet through 10.20.10.1 only (but not 10.20.10.3) after login. This is by design.

Again, the Load Balancing occurs for the additional client that gets gateway setting 10.20.10.3/10.20.10.1 from DHCP server. Note that you shouldn't set IP statically and should obtain IP from FirstSpot DHCP server automatically.
_________________
~ Patronsoft Limited ~

yherrebaut · Posted: Thu Apr 01, 2010 1:42 pm **Post subject:**

The problem is that my Windows clients send requests to both the first AND the second gateway, and you say that the second one is disabled by design, so this explains the source of the problem. But isn't there a way to make sure that clients send data only to the first gateway, and only start to send data to the second gateway when the first becomes unavailable?

yherrebaut · Posted: Fri Apr 02, 2010 9:21 am **Post subject:**

I've discovered that both gateways have the same metric, so if you want Windows to only use one gateway (the 10.20.10.1 in my case), that gateway should get a lower metric than the 10.20.10.3, is this possible via FirstSpot?

alan · **Forum facilitator** Joined: 26 Sep 2003 Posts: 4435

Since we are still in holiday, I will try to address your issue on Wednesday.

Thank you for your understanding.
_________________
~ Patronsoft Limited ~

alan · **Forum facilitator** Joined: 26 Sep 2003 Posts: 4435

Our understanding is that if the 2 metrics are the same, Windows will follow the sequence of the gateway.

Again, FirstSpot DHCP server should hand out gateway 10.20.10.1/10.20.10.3 for some clients while hand out 10.20.10.3/10.20.10.1 for other clients. That is how the Load Balancing occurs.
_________________
~ Patronsoft Limited ~

yherrebaut · Posted: Wed Apr 07, 2010 5:02 pm **Post subject:**

According to my experience, Windows (7) does not. When I use both servers separately, they both work perfectly fine, while when I use them in load balancing setup, my clients get timeouts -roughly- half of the time. If my client only used the first gateway in he sequence, it shouldn't have these problems. So logically, the only thing I can think of is that the client is using both gateways.

alan · **Forum facilitator** Joined: 26 Sep 2003 Posts: 4435

We improve our Load Balancing behavior a bit. Please make sure you are using the latest v6.0.6, and then download the below patch:

http://patronsoft.com/firstspot/download/patch/607/load_balancing.zip

Here are the instructions:
1) stop FirstSpot
2) unzip the zip file
3) backup the original EXE \FirstSpot\dispatcher\FirstSpotCore.exe
4) copy FirstSpotCore.exe to the above directory
5) backup the original driver \windows\system32\drivers\firsthop.sys
6) copy firsthop.sys to the above directory
7) reboot Windows
_________________
~ Patronsoft Limited ~

yherrebaut · Posted: Fri Apr 23, 2010 1:36 pm **Post subject:**

I just finished some tests with your new patch, and it dramatically improved the performance of my load balancing setup. Now, practically all sites load immediately, so that's very good news.

In case I encounter any more problems, I'll post here again, but for now my problem is solved, thanks.

yherrebaut · Posted: Wed Apr 28, 2010 8:24 am **Post subject:**

After some more extensive tests, the load balancing component seems to be working fine. Users are divided equally over the two servers and use only their first default gateway.

However, I still discovered some problems when I tested the failover component of the load balancing function. I've created a scenario in which two FirstSpot servers are running in load balancing mode. I am logged onto server1, and I stop the dispatcher service. server1 correctly identifies this as a problem and shuts itself down, as expected. After that, I am being redirected to the portal site and need to relogin, which works fine. However, after I log in again, the browsing is terrible. I used wireshark to track the problem and it seems that traffic is still being sent to the broken gateway (server 1).

In a real-life situation with real users, they would probably log out, disconnect from the network, and reconnect. I also did this, but to my surprise the FirstSpot DHCP server still assigned me the broken gateway as first default gateway. I used Wireshark again to track the traffic flow, and it seems that some traffic is directed to the broken gateway, while other traffic is directed to the working gateway, which explains why I have trouble browsing the web. But this is actually strage for two reasons: 1) The broken gateway still appears first when I issue the ipconfig command, so why is some traffic redirected to the second default gateway? I expected that all traffic would be sent to the broken gateway. And 2) In one of your previous posts you said that the second default gateway is blocked, but I think that this is not the case because some websites succeed to (most of the time partially) load, and because the first gateway is broken, the data must be flowing through the second one.

alan · **Forum facilitator** Joined: 26 Sep 2003 Posts: 4435

We further improve the Load Balancing feature based on your feedback.

Please download the below patch:
http://patronsoft.com/firstspot/download/patch/607/load_balancing2.zip

You can apply the patch using the same steps as above. Please give us feedback on whether this fixes your problem.
_________________
~ Patronsoft Limited ~

yherrebaut · Posted: Mon May 03, 2010 12:39 pm **Post subject:**

I've did some extensive tests in the meantime, and I'm afraid that my problem isn't fully fixed yet.

I've tested the patch in the following scenario's:
1) DHCP Service crash
When I'm browsing the web via server1 (high priority) and I simulate a DHCP server crash by stopping it through Computer Management, the server goes down as expected. After I re-login, I can continue browsing the web. I've also run Wireshark in the background, and all TCP connections are directed towards the working gateway after I logged in again.

2) Dispatcher Service crash
Identical to the first scenario, I'm browsing the web via server1 and I stop the dispatcher service. Server1 goes down, and I need to re-login. This works eventually (but it seems to take a little longer than in scenario 1). After that, I have the same problem as before patch 2: most sites won't load or only partially load. Wireshark reveals the same: TCP connections are yet again sent to both gateways.

3) Manual stop of FirstSpot
I'm browsing the web via server1. I manually stop server1 through the Configuration Manager. Like in scenario 2, I can re-login eventually, but after that the TCP connections are also divided between both gateways, and as a result it's impossible to properly browse the web.

I've tested this thoroughly and on both my servers, and it seems that only when the DHCP service fails, FirstSpot does its failover correctly and all traffic a client sends and receives goes through the working gateway. In all other scenario's, it seems that traffic is directed to both gateways when one server fails. In all other failover cases, traffic is sent towards both gateways.

I hope this feedback helps.