This project is read-only.

Doesn't seem to work over mobile connection - wifi only?

Sep 23, 2013 at 9:59 AM
Hi, I've posted a possible issue at https://socketio4net.codeplex.com/workitem/16

We've been using SocketIO4Net for a few months now over WiFi and it's been working just great. However, mobile testing has shown that the client doesn't seem to work over a mobile internet connection.

If anyone can provide any pointers as to where the problem might lie, it would be greatly appreciated!

Many thanks,
Nick Talbot

Magic Hat Solutions Ltd
Sep 23, 2013 at 2:04 PM
Edited Sep 23, 2013 at 2:06 PM
Hmm, I'm not at my development rig right now, but a bit more digging has revealed a probable bug in the dependency SuperSocket ClientEngine project, somehere in the various versions of a .ConnectAsync call, e.g. https://clientengine.codeplex.com/SourceControl/latest#Common/ConnectAsyncExtension.Net.cs

There are various calls to ConnectAsync that do not check the return value of Socket.ConnectAsync, and assume the call is always asynchronous, and does not handle the possible synchronous return value.

This is also tied in with possible complication of using dns-resolvable addresses instead of pre-resolved IP addresses in a mobile environment causing some sort of timeout.

I'll look further into it today and report back.

Nick Talbot
Coordinator
Sep 23, 2013 at 2:36 PM
You had me at Xamarin!

It sounds like you might have a bead on the issue - so I'll wait to hear back on your progress...

Not trying to pass the buck, but as you're probably already aware, Socketio4net simply leverages WebSocket4Net for websocket connectivity/communication. I mention this because as I wouldn't be able to make direct changes for the async / sync side you've digging into. I'm happy to work with you and kerryjiang to figure things out and update/release with any fixes just the same.

Please update when you can...

Jim
Sep 23, 2013 at 2:37 PM
Also .. TcpClientSession.ProcessConnect() seems to += add a .Completed handler halfway down a method very possibly after the event has already fired, see https://clientengine.codeplex.com/SourceControl/latest#Core/TcpClientSession.cs

More digging required ...
Sep 23, 2013 at 2:41 PM
Hi JStott, yep I'll get back a bit later today once I've had a chance to debug again under wifi and mobile environments.

As you say, SocketIO4Net is built on a stack, so the problem is most likely in WebSocket4Net or SuperSocket.ClientEngine. SocketIO4Net has been working great in WiFi testing, there's obviously a wrinkle somewhere regarding mobile TCP.

Up until now, we'd been treating SocketIO4Net and its dependents as a black box - but digging into the stack looks the way to go now, I'll report any progress.
Sep 23, 2013 at 9:42 PM
Edited Sep 23, 2013 at 10:00 PM
Ok, I've got to the bottom of why this is happening, and the solution isn't a simple one.

There's also nothing wrong with the SocketIO4Net / WebSocket / SuperSocket stack and source base - for my particular application anyway. (The code path taken for Xamarin MonoAndroid projects in the stack uses private implementations of .ConnectAsync() which are checked for synchronous execution. However, other uses of the stack should probably be altered to check for the result of .ConnectAsync() calls).

The problem I'm experiencing is being caused by a dark secret of mobile internet connections this testing has revealed - Mobile Internet connections explicitly do NOT support Websocket-based Socket.IO connections.

Debugging inside https://websocket4net.codeplex.com/SourceControl/latest#WebSocket4Net/WebSocket.cs in OnDataReceived() and looking at the commandInfo.Text property, I checked the response received by the stack from the server.

When running on a WiFi connection, the first text received when initiating a Socket.IO connection is:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: JrINHtDHm+/3Vsju75+kN54pKkU=
This is then correctly followed by more raw Socket.IO text.

Ok, next I debugged while on a Mobile Phone Internet Connection (Orange/EE 3G UK). The code was unchanged and made the same calls to the same internet-based Socket.IO server. The text received was:
HTTP/1.1 503 Service Unavailable
Server: WebProxy/5.2
Date: Mon, 23 Sep 2013 17:39:14 GMT
Content-Length: 0
Connection: keep-alive
The 'WebProxy' and 503 give the game away - Socket.IO is not supported by my (and presumably) many other Mobile Internet providers.

Remember that in the case of 'normal' Javascript/Socket.IO usage based on the /socket.io/socket.io javascript provided by a NodeJS server, a Browser-hosted Socket.IO client will gracefully degrade to another Socket.IO transport.

To double check this assertion, I added some console debug to a internet-available website of ours, based on http://stackoverflow.com/questions/15028706/browser-websocket-fallback-and-support along the lines of:
console.log(socketNamespace.socket.transport.name);
Running a browser with a debug console on a laptop first connected to WiFi and connecting to the website produced:
websocket
Then disconnecting the laptop from WiFi, and using the 'Personal Hotspot' WiFi of my mobile phone to use the Mobile Internet on the laptop and connecting to the website confirmed the theory:
xhr-polling
So - even for full Internet Browsers (e.g. Chrome, Safari), when accessing existing NodeJS/Socket.IO based internet websites - on a WiFi connection websockets are used, but when on mobile internet connection, the fallback transport xhr-polling is used instead.

Now the problem is that SocketIO4Net currently only supports the Websocket transport - see http://socketio4net.codeplex.com/discussions/449091 - the fallback Socket.IO transports are not supported.

This effectively means that in its current state, SocketIO4Net cannot be used to access NodeJS/Socket.IO services from a Mobile Internet connection.


Where do we go from here? The obvious solution is to provide client support for a fallback Socket.IO transport within SocketIO4Net, e.g. for xhr-polling.

Looking at http://showmetheco.de/articles/2011/8/socket-io-for-backend-developers.html (XHR Polling section) it appears that this would involve a long GET to listen for the server result, and client commands sent with POSTs. This could be achieved with e.g. WebClient.DownloadString() and WebClient.UploadString().

Does anyone have any more pointers about implementing the spec of a XHR-POLLING based Socket.IO C# client, and e.g. the recommended timeouts before switching to the fallback transport. How hard does everyone think it would be?

Many thanks,
Nick Talbot
Coordinator
Sep 23, 2013 at 11:22 PM
Indeed, from your sample, the websocket connection is being prevented (blocked). This can be an issue even with wifi within some corporate/education firewalls & proxies.

Have you tried SSL or other ports? I'm not all that familiar with being on Orange - but this (somewhat dated) SO link FlappySocks indicates that it has worked over port 1442, and looks like SSL/443

Some simple sites for a quick check (via browser) that should also give you a feedback for ssl/ports
websocket.org
websockettest

As far as supporting a fallback to something like xhr-polling, I haven't come across any library that supports long polling in a C# client. The closest thing I've seen would be to review the SignalR code and see what the transport stuff looks like.
Sep 24, 2013 at 10:24 AM
Edited Sep 24, 2013 at 11:02 AM
Hi Jim, thanks for those links, especially http://websocketstest.com/ and the SignalR reference, very useful.

Looking at the websocketstest.com results in different environments, I can see the difficulties that websockets face.

My own WiFi: Websockets over Port 80 available
A Corporate Client internet connection: Websockets not available at all
UK Orange/EE 3G: Websockets not available on port 80, but are available on port 443 (over https/wss)

This confirms your other link, in that Websockets are sometimes available on a mobile internet connection over HTTPS.

However, considering that we're writing a Mobile Phone App, we need to make sure that the solution used will work on any and all mobile internet connections, regardless of the mobile provider being used, we can't rely on the secure version possibly working, or the vagaries and whims of different mobile internet providers. In my own personal case, moving to a secure version of a server would work for me, but I can't be sure that would work on all internet connections. The example corporate client WiFi connection I also tested with shows that sometimes even WiFi/Hard-wired connections don't support Websockets.

The only sensible solution that can guarantee comms over any connection is to support fallback xhr-polling.

I note that the ClientR stack provides a C# client that implements long-polling xhr-polling, and on the face of it, it should be quite easy to write something similar that slots into SocketIO4Net.

We're working to a .NET 4.5 stack and already making heavy use of await-based asynchronous Task calls. The HttpClient class provides await-able GetAsync() and PostAsync() methods, so following the recommendations in http://showmetheco.de/articles/2011/8/socket-io-for-backend-developers.html shouldn't be too hard.

There is also more information relevant to writing a SocketIO client at https://github.com/LearnBoost/socket.io-spec

I'm going to have a crack at crowbar-ing an xhr-polling style fallback transport into SocketIO4Net for when websockets are not available. Looking through the existing stack, I may have to access/duplicate some of the code in we WebSocket4Net project, as that contains some of the transport decoding code, we'll see.

I'll let you know how I get on!

Thanks,
Nick
Coordinator
Sep 24, 2013 at 1:52 PM
Glad that helped.

Please let me know how things progress, if you run into any issues you can contact me here or direct as you see fit.

Jim
Sep 24, 2013 at 2:28 PM
Edited Sep 24, 2013 at 2:30 PM
jstott wrote:
Glad that helped.

Please let me know how things progress, if you run into any issues you can contact me here or direct as you see fit.

Jim
Will do Jim, I think I can get a quick-and-dirty implementation running in the next few days.

Digging in already, referring to http://en.wikipedia.org/wiki/WebSocket I can see that https://github.com/kerryjiang/WebSocket4Net/blob/master/WebSocket4Net/Protocol/DraftHybi10Processor.cs sends the server the required
GET /mychat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
via a hand-built HTTP-compatible payload over a raw socket in DraftHybi10Processor.SendHandshake()

According to http://showmetheco.de/articles/2011/8/socket-io-for-backend-developers.html (XHR Polling section) I need to write a slottable-in replacement that simply makes an await-able HttpClient.GetAsync() call to generate the required long-polling xhr-polling instantiation request instead (with appropriate Sec ID and t=timestamp_ticks):
GET /socket.io/1/xhr-polling/15725592722050266739/?t=1312200210467 HTTP/1.1
Host: 127.0.0.1:5000
Connection: Keep-Alive
This is in addition to POSTed messages/commands/heartbeats.

As I have all the SocketIO4Net/WebSocket4Net/SuperSocket projects & source code in my solution, I intend to abstract the Methods/Events/Properties of WebSocket that are used by SocketIO4Net.Client into an IWebSocket Interface, and change the this.wsClient reference in https://socketio4net.codeplex.com/SourceControl/latest#SocketIO/Client.cs from WebSocket to IWebSocket.

If I can then create an XhrPolling class that implements all the new IWebSocket interface appropriately, we should be in business.

Will keep you posted,

Thanks,
Nick
Sep 25, 2013 at 1:24 AM
Ok, good news! I have made a first stab at implementing an XhrPolling plug-in for SocketIO4Net, and I am very happy to report that it seems to work very well. I have also tested the replacement over a mobile phone internet connection, and it really does seem to work as required.

It was actually much simpler to implement than I anticipated (in part due to .NET 4.5 async/await support), and hopefully shows a possible pattern of fallback transport support for SocketIO4Net.

Ok, first I added an interface to the supporting WebSocket4Net project, ITransport
using System;
using SuperSocket.ClientEngine;

namespace WebSocket4Net
{
    public interface ITransport
    {
        WebSocketState State { get; }

        bool EnableAutoSendPing { get; set; }

        event EventHandler Opened;
        event EventHandler Closed;
        event EventHandler<ErrorEventArgs> Error;
        event EventHandler<MessageReceivedEventArgs> MessageReceived;

        void Open ();
        void Close ();
        void Send (string message);
    }
}

All of the above is already implemented by WebSocket, so the only alteration to Websocket.cs need is:
public partial class WebSocket : ITransport
Next, in SocketIO4Net's Client.cs, the reference wsClient is changed from a WebSocket to an ITransport:
protected ITransport wsClient;
Then for now (until I implement a fallback strategy tomorrow), the line in Client.cs assigning a new WebSocket to wsClient is changed to:
this.wsClient = new XhrPolling (uri, this.HandShake.SID);
Finally, the implementation of the new XhrPolling class:
using System;
using System.Linq;
using WebSocket4Net;
using System.Net.Http;
using SuperSocket.ClientEngine;

namespace SocketIOClient
{
    public class XhrPolling : ITransport
    {
        private readonly HttpClient _http = new HttpClient ();
        private Uri _uri;

        public event EventHandler Opened;
        public event EventHandler Closed;
        public event EventHandler<ErrorEventArgs> Error;
        public event EventHandler<MessageReceivedEventArgs> MessageReceived;

        public XhrPolling(Uri baseUri, string sID)
        {
            _uri = new Uri (baseUri, string.Format ("socket.io/1/xhr-polling/{0}/", sID));
        }

        private Uri Uri
        {
            get { return new UriBuilder (_uri) { Query = string.Format("t={0}", Environment.TickCount) }.Uri; }
        }

        private void OnError(Exception ex)
        {
            if (Error != null) { Error (this, new ErrorEventArgs (ex)); }
        }

        public WebSocketState State { get; private set; }
        public bool EnableAutoSendPing { get; set; }

        public async void Open()
        {
            try
            {
                State = WebSocketState.Connecting;

                var pollTask = _http.GetStringAsync (Uri);

                State = WebSocketState.Open;

                if (Opened != null) { Opened (this, EventArgs.Empty); }

                while (pollTask != null)
                {
                    var received = await pollTask;

                    if (MessageReceived != null)
                    {
                        var frames = received.Split ('\ufffd').Skip (1).Where ((s, i) => i % 2 != 0).ToList ();

                        if (frames.Count > 0)
                        {
                            frames.ForEach (frame => MessageReceived (this, new MessageReceivedEventArgs (frame)));
                        }
                        else
                        {
                            MessageReceived (this, new MessageReceivedEventArgs (received));
                        }
                    }

                    pollTask = State != WebSocketState.Open ? null : _http.GetStringAsync (Uri); // State is changed by Close()
                }
            }
            catch (HttpRequestException ex)
            {
                OnError (ex);
            }
        }

        public async void Send (string message)
        {
            try
            {
                (await _http.PostAsync (Uri, new StringContent (message))).EnsureSuccessStatusCode ();
            }
            catch (HttpRequestException ex)
            {
                OnError (ex);
            }
        }

        public async void Close ()
        {
            State = WebSocketState.Closing;

            try
            {
                (await _http.PostAsync (new UriBuilder (_uri) { Query = "disconnect" }.Uri, null)).EnsureSuccessStatusCode ();
            }
            catch (HttpRequestException ex)
            {
                OnError (ex);
            }

            State = WebSocketState.Closed;

            if (Closed != null) { Closed (this, EventArgs.Empty); }
        }
    }
}
The new XhrPolling class follows (I think) the expected behavour that SocketIO4Net expects, and is handles the Sending and Receiving of Socket.IO data asynchronously on a single thread in a non-blocking fashion, thanks to .NET 4.5's async/await support.

Initial testing has shown that this seems to work pretty well - our mobile phone app is working just as it did on WiFi, but now works over a Mobile Internet Connection using xhr-polling.

Any comment and suggestions are welcome, there are a few points to tidy up yet:
  • I'm not convinced disconnect is handled properly or optimally yet.
  • I haven't checked if any extra work is required for a heartbeat - I need to run a longer test and see if SocketIO4Net is taking care of this
  • EnableAutoSendPing is unused and ignored
  • The SocketIO4Net Client is currently hardwired to use XhrPolling now
On the last point - I still need to implement a fallback mechanism whereby the client attempts to use WebSockets first, and if that fails then to try again using XhrPolling. I need to look into what the trigger should be - if WebSockets fails because of a mobile internet connection, I think the Opened event isn't fired, and Error maybe is - I need to check this tomorrow.

Hope this helps,

Regards,
Nick Talbot
Magic Hat Solutions Ltd
Coordinator
Sep 25, 2013 at 3:59 AM
That looks so much simpler to implement than I anticipated as well - nice going!

I'll dig a little further in the morning - but the only item beyond what you've already mentioned is the that stuck out is the ErrorEventArgs for the Error event, coming from SuperSocket.ClientEngine vs SocketIOClient - that's minor to clean up.

I think the a graceful fallback setup is the next host spot as you've identified.

As for the SocketIO4Net library itself - do you mind if this is all consumed as part of the overall library? My initial thought would be for two build versions - continue to support .net v4.0, but allow those able to upgrade to .net v4.5 the immediate advantage of xhrpolling support?

Again - very nice work. Excited to play with this more!

Jim
Sep 25, 2013 at 9:30 AM
Edited Sep 25, 2013 at 9:31 AM
Hi Jim,

I'll work on the Websocket to Polling fallback code a bit later today (and also check heartbeat handling), but other than that, the new code satisfies our needs for Socket.IO over a mobile internet connection.

Please feel free to incorporate the above code into SocketIO4Net in any way you see fit - without the SocketIO4Net/WebSocket4Net/SuperSocket stack we wouldn't have been able to progress our own app development as quickly as we have. It's nice to be able to give something useful back to the project!

I've performed some more mobile testing this morning while on the move, and the polling code seems to be working well.

Please also feel free to add your own comments/changes, and post about them or pm me.

Many thanks,
Nick
Sep 25, 2013 at 8:08 PM
Ok, here are my final changes:

I've beefed up XhrPolling.Open() to report an error when receiving a non-OK/200 result from the server:
        public async void Open()
        {
            try
            {
                State = WebSocketState.Connecting;

                var pollTask = _http.GetAsync (Uri);

                State = WebSocketState.Open;

                if (Opened != null) { Opened (this, EventArgs.Empty); }

                while (pollTask != null)
                {
                    var response = await pollTask;

                    response.EnsureSuccessStatusCode (); // Checks for OK 200 result

                    if (MessageReceived != null)
                    {
                        var received = await response.Content.ReadAsStringAsync ();

                        var frames = received.Split ('\ufffd').Skip (1).Where ((s, i) => i % 2 != 0).ToList ();

                        if (frames.Count > 0)
                        {
                            frames.ForEach (frame => MessageReceived (this, new MessageReceivedEventArgs (frame)));
                        }
                        else
                        {
                            MessageReceived (this, new MessageReceivedEventArgs (received));
                        }
                    }

                    pollTask = State != WebSocketState.Open ? null : _http.GetAsync (Uri); // State is changed by Close()
                }
            }
            catch (HttpRequestException ex)
            {
                OnError (ex);
            }
        }
I also made the internal _uri member readonly as it is only set in the constructor (just code tidying):

        private readonly Uri _uri;
        private readonly HttpClient _http = new HttpClient ();
I've also implemented an extremely rudimentary fallback from Websockets to XhrPolling in Client.cs. This has been done very roughly, and specifically for the case that interests me, i.e. having internet connectivity on a mobile network that blocks websocket requests. In this case, there is a response from the network 'webproxy', but its not the response that the WebSocket code expects, so it closes the connection.

In this particular case then, the Close event is fired, even through the Opened event has not been fired yet. This is how I determine to fallback from Websocket if it is currently in use.

The code then - I moved all the wsClient wiring and unwiring of events into helper methods:
        private void WireUpEvents()
        {
            this.wsClient.Opened += this.wsClient_OpenEvent;
            this.wsClient.MessageReceived += this.wsClient_MessageReceived;
            this.wsClient.Error += this.wsClient_Error;
            this.wsClient.Closed += wsClient_Closed;
        }

        private void UnWireEvents()
        {
            this.wsClient.Closed -= this.wsClient_Closed;
            this.wsClient.MessageReceived -= wsClient_MessageReceived;
            this.wsClient.Error -= wsClient_Error;
            this.wsClient.Opened -= this.wsClient_OpenEvent;
        }
Then I altered the existing code that uses this code, so in Client.Connect(), the code after the check for valid handshake becomes (with the WebSocket instantiation code re-instated):
                        else
                        {
                            string wsScheme = (uri.Scheme == Uri.UriSchemeHttps ? "wss" : "ws");
                            this.wsClient = new WebSocket(
                                string.Format("{0}://{1}:{2}/socket.io/1/websocket/{3}", wsScheme, uri.Host, uri.Port, this.HandShake.SID),
                                string.Empty,
                                this.socketVersion) { EnableAutoSendPing = false }; // #4 tkiley: Websocket4net client library initiates a websocket heartbeat, causes delivery problems

                            WireUpEvents();

                            this.wsClient.Open();
                        }
and Client.closeWebSocketClient() becomes:
        protected void closeWebSocketClient()
        {
            if (this.wsClient != null)
            {
                UnWireEvents ();

                if (this.wsClient.State == WebSocketState.Connecting || this.wsClient.State == WebSocketState.Open)
                {
                    try { this.wsClient.Close(); }
                    catch { Trace.WriteLine("exception raised trying to close websocket: can safely ignore, socket is being closed"); }
                }
                this.wsClient = null;
            }
        }
And finally to the fallback code, which uses the newly extracted helpers, in Client.wsClient_Closed()
        private void wsClient_Closed(object sender, EventArgs e)
        {
            if (this.wsClient is WebSocket && this.socketHeartBeatTimer == null) // Open Event never happenned, fallback to Xhr Polling
            {
                UnWireEvents ();

                this.wsClient = new XhrPolling (uri, this.HandShake.SID);

                WireUpEvents ();

                this.wsClient.Open ();
            }
            else if (this.retryConnectionCount < this.RetryConnectionAttempts   )
            {
                this.ConnectionOpenEvent.Reset();
                this.ReConnect();
            }
            else
            {
                this.Close();
                this.OnSocketConnectionClosedEvent(this, EventArgs.Empty);
            }
        }
I have simply added an initial check in the close handler to see if the wsClient is a WebSocket, and is closing even though it never 'opened' - determined by socketHeartBeatTimer still being null when it is normally set to a Timer instance in wsClient_OpenEvent.

The code then unwires the Websocket events, then creates an XhrPolling instance, and wires that up instead. Finally the asynchronous Open() method is called, which will await a http.GetAsync() call, return, and unwind the wsClient_Closed() call stack, while the XhrPolling receive code remains active.

It's not the prettiest, and your mileage may vary, but it's a start - I've tested the code, and I get a Websocket connection if available and allowed by the internet connection, or a XhrPolling connection otherwise.

Hope this helps,

Regards,
Nick
Coordinator
Sep 25, 2013 at 9:16 PM
Outstanding Nick!

I can't thank you enough for taking the time to work all of this out, it's fantastic.
Thank your for helping to make this whole OSS thing work, and making it worth the effort.

I'll consume all of this, and will most likely transfer the source over to GitHub, primarily to re-organize for the release assemblies and clean a few things up.
Will update you on that progress...

Well done sir, and many thanks!

Jim