Dealing with missing messages in JavaScript when using BOSH
We recently entered a private beta of our flagship product and had a small launch event. Unfortunately the meeting place was a terrible wireless connection and packets were dropping left and right causing havoc without a system, basically it couldn't work at all! Luckily, we managed to switch to another network and save the demo. This highlighted what I knew was already a problem, but did not appreciate how much of a problem it could be. Our system relies heavily on BOSH and has a fairly large JavaScript codebase that works pretty well on good networking right now. However, we need it to work well under bad network conditions.
Because of the way XMPP, the fire and forget system, works, it is not easy to tell if the message you sent or should have received was actually sent or received. For example, we have an offer system, one user will send an offer to another through BOSH. When this message is received by the server, it publishes a message for the users offers_sent PEP node and a similar message to the receiving users offers_received PEP node. While the sending user can tell if their proposal was sent (relatively) easily, if the notification to the receiving user is never received, the user will never know they missed the message.
A little about JavaScript customization, it has 4 main layers:
- StropheJS
- MVC framework for low-level tasks and building on top
- The application layer containing the logic of the application, controller models, and so on, as well as the browser cache of the model data.
- The UI layer that receives events and publishes events to and from the application layer.
One way to deal with the missing messages problem is to periodically check the PEP nodes for new data that the browser is not aware of. If a new message was found, the browser cache would be invalid and all new data would be requested from the server. I'm not sure if this is the best way to go and it also doesn't cover all situations. We certainly don't want to get into a situation where we send messages to confirm that the previous message was received at this destination, as this will double the network traffic.
As the number of live websites grows daily, this is a problem that other developers may have faced, it would be interesting to see how others have solved it. As far as I can see, there are two situations in which messages go missing:
- On bad connections, messages are not sent or received due to dropped packets.
- When navigating between pages, the message is received by the browser, but not fully processed and stored in the local cache before the page is unloaded. Or, a message is added to the submit queue, but never sent until the page is unloaded.
I suspect the hardest problem to solve will be number 2. Any thoughts on this would be much appreciated.
a source to share
There is no good solution for this, however there is a workable solution.
BOSH sessions remain valid only for a specified amount of time (60 seconds, by default, in most implementations). At the end of the session, the fake c2s connection is closed and the user has to login again.
As long as the session is valid, messages should not be lost or out of order. The only potential loss is within the sixty second window allowing HTTP to reopen the connection, and as mentioned, if that window is closed, then a new session must be created. If a new HTTP request is created in this window, then nothing will be lost or out of order.
I would suggest that since you are using PEP as your store, you have a hook on the client whenever a session is created, you get items from your PEP nodes to initialize the client side cache (see section 6.5 of XEP-0060 ) ...
Messages can be lost if they were successfully received by your BOSH client, but the web page is closed or reloaded before they can be successfully processed. However, for other conditions, you no longer see data loss, just additional lag during startup due to item lookups.
a source to share
You should use request and response confirmations to resolve this issue: http://xmpp.org/extensions/xep-0124.html#acks The author of strophe.js mentioned that it will add support for this in the future.
a source to share