SleepBot Synchronization

by Edison Wang

Since we started planning for a web platform for SleepBot, the biggest problem I had was how to efficiently synchronize the data people already have on their phone with the central database.

The simple part: When a user sign in on their phone, all the existing data on the phone are transmitted to the server. The server would first get all the data the user has that already on the server, match the ones that needs to be updated (base on their last modified time) and insert the ones that are missing. The ones that are missing or updated on the device will be sent back. The device would then verify the data, and if there are no error, mark the last updated time.

Now there are a few different things that can happen:

User punched in on the website.
User edited/updated sleep data on the website.
User punched in on the phone.
User edited/updated sleep data on the website.

For Android 2.2 or above devices that have C2DM enabled, we can use push notifications:

For case 1 and 2, SleepBot will queue all the updates on the server and send updates to the devices via push notifications.
For case 1, I call it a state change, in which the user has to see immediately on their phone, so when a device receive a notification on case 1, it will try to synchronize immediately, if possible. If any error happens, the device will mark itself as "needs to be updated". When the user opens SleepBot, the device will try to contact with the server again for updates. In these exchanges, the last update time will be used to only exchange the modified entries.
For case 2, SleepBot will only mark the device as "needs to be updated", this way in case there are more updates, only one update will happen until user open SleepBot again on their phones.

For Androids that does not support push notification (Kindle Fire, Android 2.1 or lower) and iOS, SleepBot will try to synchronize every time it is opened.

Regardless of the device, for case 3, SleepBot will synchronize immediately.

Regardless of the device, for case 4, SleepBot will synchronize upon close.

Note that on the server side, whenever a device try to synchronize, it will propagate the changes to all the devices that are connected to the account except the device that requested the synchronization.

At a glance, this is what I came up with. It should cover all the cases.....not sure if this is the most efficient way of handling it though.

by Edison Wang

There are just too many posts aout Node.js these days that lies on the extremes so I want to put my 2c in the pod as well.

My first hand experience with Node.js was when I did my summer internship at Intuit last year. I did an innovation project with it to introduce the language to other developers there in the office. I ABSOLUTELY loved it. It was very low on memory usage and very easy to code and interact with other web services and it IS fast against apache for simple tasks. But as the code base grows it gets increasingly hard to manage and the chain of callbacks gets messy even if you use awesome libraries such as Step.

Citing from the article from hacker news, "Callback spaghetti is about the last pattern with which you'd ever want to write anything", Also, being a very young language it lacks "dependency injection and proper modularization are actually good things that help you maintain code over time. JavaScript has very little support for any of those nice things: it doesn't even have namespaces" as said in the above link. Node.js, however, in my opinion very fast and low on memory. The place I work for also has done some benchmark and see nearly tenfold increase on output. Companies like Linkedin and Amazon also has made public results of their experience with Node.js that has been very positive.

Personally, I think Node.js is awesome to develop real time applications that does not require long blocks (process api calls) and it is not ready to use as the primary language yet. You still have to fall back to the more mature languages.

Happy New Year! SleepBot, and my new web project for fun.

by Edison Wang

It's been almost a year since I have posted anything here.

2011 was a great year for me.

SleepBot now has over 500,000 downloads and is one of the top rated apps in the Android Market. Jane and I, the SleepBot Team, has just released the web platform to our registered beta testers and friends and families. (https://beta.mysleepbot.com, though to use the App with it you will have to use a special version that supports Sync, and we are not releasing this to the public because we need to monitor the traffic requirements for scale and fix bugs)

It has been a great learning experience building this site. The site is built on Codeigniter PHP framework. It's a great framework: lightweight, powerful, and yet simple. It allowed us to focus on building the real features. However, as the code base gets large, refactor code becomes painful, since PHP is interpreted. Maybe we will switch to Facebook's HipPop, but at this point, we just want to focus on getting a usable product out. There were a lot design decisions to make, and a lot of them were focused on how to do as much as possible on the client side (since we are bootstrapping the costs).

60% of the time ended up building the front-end (JQuery based, though I tend to do things from scratch). This is also something I learned from working at Squarespace(awesome company!).

-Your html body should be with as little in between the tags as possible. -Your back end's job is to spit out a usable JSON so the front-end can parse it into HTML. -So yes, your back-end is mostly a router for the pages and an API server. -Use Ajax to prevent reloading the whole page as much as possible. -CSS is parsed using less.js. -Most environment variables are cached in Apache's process memory.

And this is not possible if it was not pre-2011. Browser supports are much better these days, and clients are usually powerful enough to render the contents without noticeable lags.

The user experience is great on the site, every post, every error message, and every page is carefully designed to maximize UX.

What took 30% of the time was making Sync working between the central server and various devices. There were so many cases to consider, and many different places for code injections. It is the most important part of the site.

For SleepBot, we Android's push notification to either request the client to sync immediately (such as when there's a state change that needs to be reflected) or update when the user opens the app next time. It works beautifully on the few devices that we have tested (2.2 and above). I'm not sure if we are going to ever support non-Google Android devices or just use Urbanairship (It's Helium reports works, but we might just build our own and make it open source...I don't believe this is not something developers should pay for).

One thing that I don't like the most in the process is also be the sysadmin of the project. Setting up servers took too much time from development time.

I did end up teach Jane CSS in the process though. Jane did all the graphic and styling for the site and she did a great job. :) Teaching her about Bash/SSH/Git/basic networking principles....and how she has to git add * and eventually understood it made me very proud of us as a Team!

Oh. and I started on a new project for fun: Web as a Shell (https://github.com/wzsddtc2/Web-as-a-Shell), this is the first project that I have as public repository on GitHub. :) ---In short, it will allow you to execute commands such as : facebook post "Happy New Year" in command line. And the best thing is...it lives inside your browser and requires no server to run. The app is fully built on HTML5, will support basic file IO and have user permission model. Oh, and any one will be able to add any new "Handler" to handle any new commands with a line of code! Detail in my next post. :)