Nugget Post: Golang: RPC server/client implementation

As part of my efforts to learn golang, I attempted to write a simple RPC server / client implementation. Between gobs (see my previous post) and the RPC implementation, golang makes it very easy to build a distributed system where servers and client share not only data via gobs, but also methods and functions via RPC.

It turns out RPC isn’t difficult at all, but to me at least, it required a slight mindshift. Because of my prevalent use of REST APIs, I went into this thinking the methods defined on the server could accept arbitrary arguments, then the method itself would use those arguments to build new objects, increase counters, and do whatever other work is needed. This is not so. The most valuable nugget I learnt during this experiment is listed up top on the golang RPC documentation page:

Only methods that satisfy these criteria will be made available for remote access; other methods will be ignored:
- the method is exported.
- the method has two arguments, both exported (or builtin) types.
- the method's second argument is a pointer.
- the method has return type error.
In effect, the method must look schematically like
func (t *T) MethodName(argType T1, replyType *T2) error

 

First of all, RPC works by “exporting” methods that are defined on types. Note that the methods need to be exported – that means must start with a capital letter people!

Second, every method defined can have only two arguments. The second argument is reserved for the reply that is sent back to the client, which means the “input” to the method must be contained within the first argument. This in turn means that the first argument is nearly always a struct, enabling you to pass multiple input values encapsulated within a single variable. Unless, of course, you define a method which only needs a single input, like in the snippet below (see full code on github page):

// create a function named "PersonGetter"
func (p *Person) PersonGetter(id int, per *Person) error {
	*per = PersonMap[id]
	return nil
}

In the above, we are fetching an element from a map, determined by a single input, so the first argument need not be a struct. Also note that in the snippet above, the function returns type error, as per the specs laid out above.

Since golang enforces a certain signature for the RPC exported methods,  it makes you as a developer be more structured. If one peruses the code from the various examples, you will note that while the method definitions reside solely on the server, the data structs and types  that describe the input and reply are mostly shared on both the client and server, and this helps to keep a certain amount on continuity between the client and server.

View my full code on githubhttps://github.com/dvas0004/learning/tree/master/golang/myown/rpc

References:

About these ads

Lessons Learned: Setting up TVHeadend for live TV streaming in Malta

I recently had the opportunity of setting up a system which captures free-to-air channels and streams them over IP for use by TV set-top boxes such as RaspBMC (or plain old XBMC). We decided on using the excellent TVHeadend. The system was made up of two components:

  • A TVHeadend server. This was simply a desktop PC that had ubuntu installed on it. Then, tvheadend was installed from packages using the following guide: https://tvheadend.org/projects/tvheadend/wiki/AptRepository
  • A TVHeadend client. We used the TVheadend client plugin that comes with RaspBMC / XBMC

There were a couple f peculiarities in our setup which i’d like to list here for my future reference.

  • We bought a TBS6285 DVB tuner card, which while it worked in linux, needed driver compilation from source (never ideal because as soon as you update the kernel, you need to recompile). The instructions that worked best for me are here: http://linuxtv.org/wiki/index.php/TBS6285
  • Malta is not listed as one of the DVB locations in TVHeadend. Using Italy’s IT_ALL works just fine
    Selection_011
  • There is lots of advice on the ‘net about the proper DVB adapter configuration. The below worked for me. Of special note is the selection of “close device handle when idle“. The TBS6285 has a habit of not properly closing it’s device descriptor handle, so after some time (approx 24 hours), attempting to access a TV stream would result in the TVHeadend client showing “no input detected“. After some digging around in the logs, I found that in such TVheadend complains of not having the proper file/device descriptor, leading me to enable this option – and solving the issue

Selection_012

A huge shoutout and congrats go to the developers and maintainers of TVheadend, XBMC, raspBMC and TVHeadend clients, without which this would definitely not have been possible

Below is a video of the setup working:


References:

Nugget Post: Golang : Gobs and exported identifiers

This one had me scratching my head for a while… while messing around with “gobs” – binary representation of data that can be sent over the network between two golang programs – I ran into the problem of “exported fields”. I had followed the post on the golang blog regarding gobs, with some modifications. I had initially defined the struct to be transferred as a gob like so:

type Person struct {
	name     string
	surname  string
	age      int64
	employed bool
}

However this didn’t go down well and the receiving end started seeing:

{ 0, false}

Which basically means an empty struct of type Person (sidenote: when golang is not passed values for fields in a struct, it inserts “defaults”, these being empty for strings, 0 for integers, and false for boolean – hence explaining the output I was receiving}. The problem was not obvious to me (though it was in hindsight, as you can see in my comments in the code of my golang learning repo). The problem was to do with “exported identifiers“. In golang, one way of making sure that identifiers (or fields of a struct) remain private (in java parlance) is to give them a variable name beginning with a small letter. Public or exported (visible to all packages and functions) identifiers must start with a capital letter. So changing the above Person struct to:

type Person struct {
	Name     string
	Surname  string
	Age      int64
	Employed bool
}

resolved my issue in a hurry. Once I knew what to look for, in all fairness, if I had RTFM properly i’d have seen the following in the gobs blogpost:

// Only exported fields are encoded and decoded

References:

Android GPS positioning without Google Maps

Don’t get me wrong, google maps are awesome! And along with maps, so is google earth. In fact, my initial dabbling into GPS tracking used google earth to display GPS co-ordinates on a map. You can see a quick example here:

 

But I also got to wondering, if it would be (easier?) possible to use a third party mapping service while achieving a look and feel which is similar to google maps. Some research quickly turned up leafletJS, and it does look like it hits the nail on the head. I decided to use OpenStreetMap as a backend. The solution was going to be web based, but since I wanted to integrate GPS positioning from my phone, it had to be a hybrid app. Good… a perfect opportunity to learn about android webviews….

The final (demo) project can be found on GitHub, so i’ll gloss over the details of getting GPS co-ordinates (there are plenty of tutorials on that online), and instead focus more on the webview part. First of all, I created a simple index page in the android assets folder, and within that same folder I placed all the necessary CSS, Javascript, and images that leafletJS and I required. The only really interesting part there is vehicleScript.js, where apart from initializing the map as per leafletJS documenation, I added my own little function:


function updateLocation(lat,lng,time){
map.panTo(new L.LatLng(lat,lng));
var marker = L.marker([lat, lng]).addTo(map);
marker.bindPopup("Your Location at Time:"+time).openPopup();
}

You can see the function accepts three values, the latitude, longitude, and time. These three values would be provided by the native android app, which i’ll explain now…

 

In the andoid main activity, you can see in lines 99-101 where I added the webview to the layout:

 

mapView = (WebView) findViewById(R.id.mapView);
mapView.getSettings().setJavaScriptEnabled(true);
mapView.loadUrl("file:///android_asset/index.html");

 

Make sure not to forget that setJavaScriptEnabled(true), else javascript will be disabled in the webview. Android does this by default for security reasons. Also note the android way of referencing the asset folder, which is file:///android_asset. The next part comes in lines 72-74, where you can see exactly how the native android app sends the latitude, longitude, and time to the webview. I register a broadcast receiver to hook into the intents broadcast by the GPS service, and we then pass the received values to the webview:

if (mapView != null){
mapView.loadUrl("javascript:updateLocation("+lat+","+lng+",\""+time.format("%d-%m-%Y %H:%M:%S")+"\")");
}

So any url that is prefixed with the javascript: will actually execute within the webview as javascript. The end result looks something like so:

The end result using leafletJS and webviews

The end result using leafletJS and webviews

Resources:

Android Troubleshooting: App Crash on multiple orientation change

More of a note to my future self when developing android than anything else, since this is quite a specific corner case, but who knows, maybe it helps someone else out there :)

 

Problem

An app I wrote seemed to be crashing randomly when a user did a orientation change (i.e. changed from landscape to portrait mode, or vice versa). On closed look, this happened when the user changed the android handset orientation multiple times in quick succession, by turning the handset 90degrees and back about 4 times in a row. This caused the app to show the dreaded “Unfortunately the app has stopped responding”….

Cause

A null pointer exception of course…. Specifically, I was firing off an async task as soon as the activity launches, and such an async task was trying to access an android list layout. It turns out, on every orientation change, android recreates the activity. This means re-drawing the layout. So on every orientation change, an async task was launched. With the user changing the orientation multiple times in quick succession, the layout was still being drawn by the time the first async task completed it’s work and tried to display it’s results on screen, resulting in the null pointer exception

Solution

I surrounded the code in the async task which accesses the layout, and upon a null exception, cancel the async task using something like:

     catch (Exception e){
               cancel(true);
               if (isCancelled()){ 
                        return;
               }
     }


Make sure to cancel the async task before any server calls are done. It seems like a waste of computing resources and network bandwidth to have the async task complete the server call and then just cancelling it, only to have subsequent async calls go through the same process.

 

Note, my research led me across the “retained fragment”, which is explained oh-so-well here:

http://www.androiddesignpatterns.com/2013/04/retaining-objects-across-config-changes.html

Basically:

 “retained” means that the fragment will not be destroyed on configuration changes. That is, the Fragment will beretained even if the configuration change causes the underlying Activity to be destroyed. [1]

And resulted in only the addition of the following to my fragment:

@Override
 public void onCreate(Bundle savedInstanceState) {
          super.onCreate(savedInstanceState);

          // Retain this fragment across configuration changes.
          setRetainInstance(true);
 }

 

However, that did not solve the issue for me, and I still required the isCancelled trick….

[1] http://stackoverflow.com/questions/11182180/understanding-fragments-setretaininstanceboolean

Lessons learned: migrating Ruby on Rails app to a private server

I needed to migrate a ruby on rails app from amazon web services to a privately owned hosting service. AWS has it’s own method of building and provisioning rails app (basically by using their in house tool called “beanstalk“), so provisioning and migrating the data to a normal, private server meant I took some quick notes I spent the better part of the morning learning the hard way.

First off, making a backup is just common sense, especially of the database (mysql in this particular case). Once you are sure you have all the data (usually this is just ruby app files and sql dump), you’re ready to go:

  • I followed the guide on installing the ruby on rails environment along with nginx + passenger here: https://www.digitalocean.com/community/articles/how-to-install-rails-and-nginx-with-passenger-on-ubuntu. This was quite painless and proceed according to plan. Only one note:
    • Make sure you select the version of ruby that is best for your application. In the above link, they use v1.9.3, which may or may not be adequate for you
  • Copy over your rails app files to a suitable location. Do the same for your SQL dump
  • Navigate into the rails app directory (eg cd /var/www/your-rails-app) and run the command bundle install. This will install all the necessary gems your application requires. Make sure you have all dependencies install before hand (like mysql-server, client, and development libraries) via apt-get install
  • If, like me, you are migrating from AWS to a private server, you probably need to change your production database connection string in database.yml to reflect your private server’s database settings
  • Restore your SQL file into the new database

Till here, all well and good. But, on attempting to open my app, it seemed that the server was simply responding with 404 for all the static assets in the rails app. Having followed the above guide to the letter, and make sure that the permissions were correct, I was stumped. This is what solved the issue for me:

  • In the production.rb environment config file, I noticed a line similar to the below, which essentially assumes that the webserver itself (nginx in this case) is going to serve the static files:

config.serve_static_assets = false

  • To make sure that the nginx server really was serving up static files, I added the following to the  file:

server {
listen 80;
server_name example.com;
root /var/www/xxxxxx/public;
passenger_enabled on;
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|html|txt|manifest|eot|svg|ttf|woff)$ {
root /var/www/pms/public;
gzip_static on;
expires max;
add_header Cache-Control public;
}
}

  • On testing, I now got a 403 (unauthorized) error, which almost always is a filesystem permission error. This was quickly solved by making sure the files in the public dir
    • had an owner which matched the user under which nginx was running
    • had both read and execute permissions.