# Pleasant Programmer

# GraphServer

Link: http://graphserver.github.io/graphserver/

One other routing webapp I saw was GraphServer. It’s actually more of a general purpose Graph library which supports GTFS and OSM data than an actual dedicated routing software like OpenTripPlanner. It’s also based off python and C instead of Java, so it feels a lot less heavy.

The instructions on the website are already pretty good. There are just some minor errors with it. Where it says `gs_gtfsdb_build`, you should actually use `gs_gtfsdb_compile`. Also, when running `gs_osmdb_compile` you might need to use `-t` for tolerant in case you follow the instructions on chopping up the original OSM data.

A nice suggestion from the GraphServer instructions was to crop the OSM data to minimize the graph size. This is actually quite helpful if you downloaded the entire Philippine OSM dump. It reduced the original 900MB file to 135MB which was a lot more workable. I did hit a problem with their instructions though. The linked version of osmosis is an old one, which doesn’t support 64-bit ids. The latest version of Osmosis easily did the job though.

The actual routing though, was not exactly good. I only tried one route which should normally take 1-2 transfers, it suggested a route which involved 4+ transfers. It also didn’t provide any alternate routes aside from that one. I’m not sure if it’s a limitation of the provided routeserver, but I didn’t bother checking if it supported parameters which might provide better routes.

I think graphserver could be useful, but it seems more involved than say OpenTripPlanner. There do seem to be people who use graphserver for their routing apps, but for the bounds of the contest, or just as a side project, it might require too much effort.

Posted: 2013-07-23 | Comments

# Transit Wand

Link: http://transitwand.com

Overall, this was the simplest of the open-source transit tools to actually get up and running. There’s already a deployed instance of the server, and you can easily download the phone app via the Play Store. Even running the server by yourself didn’t have any of the hiccups I had with GTFS Editor.

The phone app is actually quite simple. It allows you to capture a trip, which will record your GPS coordinates as you ride public transit. It also allows you to mark points of the trip where you stop and also how long the stop took. Lastly, it allows you to record embarking and disembarking passengers which is potentially useful for ridership data.

After doing a capture session, you can review the data on the phone. It will plot out the route on a map, with markers for the stops. You then either delete the data if it looks wrong, or you can upload it to the Transit Wand server. Uploading involves registering an account, but it’s free and you don’t even actually need to put in a username or anything. It simply registers the phone’s IMEI on the server and gives you a 6-digit identifier.

You can then use the 6-digit identifier to view the data on Transit Wand’s server, which is good since uploading any data automatically deletes it from the phone. There really isn’t much else you can do with it though. It just allows you to view the data, and export it as a Shapefile.

As is, this is purely a data collection client-server app. Barring looking at the database, there is no way to get a list of phones which have collected data. Only the person who initiated the data collection knows the 6-digit code to view their data. There’s also no way to extract the ridership information from the server yet. This isn’t to say that the data won’t eventually go public though.

An interesting thing you can do with the Transit Wand data is import it into GTFS Editor to make a new route. You don’t even have to manually download and upload the data. Just type in your 6-digit identifier and it will give you a list of routes you’ve captured via Transit Wand. This is wonderful as you get all the stop data, as well as the shape of the route.

I imagine these two tools were how the DOTC came up with all the GTFS data we have now. What I don’t understand is why the shape data isn’t present. Importing from Transit Wand already gets you shape data. There are even facilities to edit the shape within the editor if clean up is necessary. The only problem I saw was the fact that you can’t easily move stops, you have to input coordinates to change the position.

It might also be possible that when the DOTC was still collecting the data, the route collection or editing features weren’t present yet. That would just be lame and depressing though.

Overall, Transit Wand does what it’s supposed to do. You collect data, and then upload it to a server. There is a lot of room for improvement though. It would be nice to have a better API that allows access to more of the data. Building in analysis tools for the ridership data might also be a welcome thing. I imagine it would also be great if you could encourage people to use the app and upload their own trips.

Posted: 2013-07-15 | Comments

# Fare Data

As part of the data released by the DOTC, we also have the fare matrix for aircon buses, ordinary buses and jeeps. All as wonderful images. The data is also actually available from the LTFRB website. Generally, the fare scheme is represented as “pay X pesos for the first Y kilometers, pay Z for every succeeding kilometer.” Instead of a table, we can simply represent this as a formula instead,

    base_fare + (distance - initial) * per_km

The relevant values for the three services are:

It isn’t as simple as that though. Fares are also rounded to the nearest 25 centavos. So we’d need to round them off correctly. This can be achieved by doing,

    round(calculated_fare * 4.0)/4.0

There’s also the discounted fare for students, senior citizens and persons with disability. They get 20% off the fare (prior to rounding) and the resulting fare is rounded off as well.

Doing just this, we actually do get the same results as the fare matrices in the image for the most part. There are some discrepancies with the discounted jeep fares. I’ve tried to resolve it by tweaking around with the formulas, but it really doesn’t make sense in any way. I presume these were manually adjusted for one reason or another.

Here’s a script that generates CSVs of all the three fare matrices. If you’re too lazy to run it, here are links to the aircon bus, ordinary bus and jeep fare matrices.

### GTFS compatibility

As is, the provided GTFS data does not have any fare data. I imagine this is because the existing spec doesn’t have good support for distance-based fares like we have in the Philippines. Judging from the fare examples, the only reasonable way we could implement distance-based fares is following example 6. This would involve setting a fare for each possible pair of stops based on the distance between them. This isn’t exactly ideal. In fact, the people originally working on the DOTC project have voiced issues and made proposals for having distance-based fares included into GTFS.

Apparently, public transit fares are a really complicated thing. You have fares based on distance, number of stops passed through, and transfers which may or may not cost extra. Not only that, you might have discounted fares, or first-class vs economy fares. The community will want to get it right before it’s formally included in the spec. You can see the current state of the consolidated GTFS fare proposal here.

Even in it’s proposal form though, we might have hope of being able to see these being used. There’s currently a pull request for supporting the distance-based fare scheme into the OneBusAway libraries. The libraries actually used by GTFS Editor and OpenTripPlanner for working with GTFS data.

### Remaining Problems

Given all that, it would probably still be a long way before this allows us to make a really good routing app. We still don’t have shape data, so the distance estimates would really be rough estimates at best. There’s no support for rounding to the nearest centavo. I realize that’s just nitpicking, but if we want something truly polished, even that has to be taken care of.

We also don’t know if the jeeps or buses strictly follow the distance-based scheme. After all, if you can get on and off anywhere, you can’t really measure distance that exactly. I assume they generally work off the notion of “zones” than actual distance travelled. In that sense, they work more similarly to the LRT which has fares based on how many stops you pass. For jeeps and buses, your fare is probably based more on how many “zones” you pass through.

### Conclusion

Philip, a co-worker of mine at By Implication, had suggested that we might want to use a different model than what the GTFS proposes. I have to agree with him. At this point, the GTFS doesn’t really fit with our system. But I do think that open data and standards are great. In fact, I applaud the developers who made proposals for the fare system, as those are great first steps towards making the GTFS a more universal standard.

Side note: I’d also actually really like to hear about the DOTC developers' experience with the project. It would be nice if they had a devblog.

Posted: 2013-07-13 | Comments

# GTFS Editor

Link: https://github.com/conveyal/gtfs-editor

TL;DR they really meant under development

When I first saw the source of GTFS Editor, I was ecstatic. They used Play framework!!! Not only that, they’re targeting PostgreSQL as the main database. Those are our favorite tools for building webapps at By Implication. I was a bit sad though, when I saw it was on the 1.x release of Play though. I did have some experience with that release, but not as much compared to 2.x.

Getting it to actually run though, wasn’t very pleasant. The initial setup was easy enough. Get Play 1.2.5, install Postgres with PostGIS, clone the repo and create backing database in Postgres. Some minor additional steps you need are to create the PostGIS extension on the database. The schema is automatically generated and applied by Play so that should be all that’s necessary. Wonderful. Then, run play, open a browser, go to http://localhost:9000, compilation error. Fantastic.

If you don’t want to go through the technical details, you can just jump to the conclusion.

## Let’s Debug!

I’ll be splitting the next section up into 2 parts. In the first pass, I’ll talk about what I did to just get the app to run but I won’t try hard to fix any bugs. This generally is what I do when I try to get apps to run. I’ll also be dropping enough information so that you can actually figure out what the real problem is. In the second pass, I’ll explain what the problems were and how I fixed them.

### First Pass

A thing to note about Play (and one of the reasons it’s a lovely Java framework) is that you don’t need to do manual compilation. Just edit some source files, refresh your browser and it will automatically do the compilation for you. One less argument for using PHP. It even shows you (in the browser!) the source and which line of code caused the compilation error. So that’s what I saw, `Error: type Check already defined`

    @Retention(RetentionPolicy.RUNTIME)
    @Target({ElementType.METHOD, ElementType.TYPE})
    public @interface Check { // error here

        String[] value();
    }

You also know that typical behavior among programmers where your program doesn’t compile, but you keep trying to compile it anyway hoping that it will magically just work. That’s what I did, and it actually ran. I couldn’t really just let this pass, so I decided to try deleting `Check.java`. I got another compilation error, `Error: type Secure already defined`

    public class Secure extends Controller { // error here

        @Before(unless={"login", "authenticate", "logout"})

        static void checkAccess() throws Throwable {

At that point, I just decided to just debug it later. It works by just forcing it anyway. So I put `Check.java` back in and proceeded to just refresh until it compiled and ran.

The next problem is a sort of common thing most webapp developers have to solve one way or another. How do you set up the initial admin account? Phrased a different way, how do I login to this thing? The first thing I tried was just add a user into the `account` table directly. One problem though was how to set the password correctly. Plaintext obviously wouldn’t work.

Another note regarding Play 1.x, it provides the secure module which handles logins and keeping state, you simply need to implement the method `boolean authenticate(String username, String password)`. It leaves the actual process of verifying the login to the programmer. This can be exploited by just making the method return `true` and then any login would work. No need to actually set the password. Excellent.

And we’re logged in, just in time to encounter a runtime exception. This also works much like compilation errors in Play. It shows a page with the error and the relevant source lines. Now we get, `IndexOutOfBoundsException occured : Index: 0, Size: 0`

    if(session.get("agencyId") == null) {

        Agency agency = agencies.get(0); // error here

        session.put("agencyId", agency.id);
        session.put("agencyName", agency.name);

Apparently, we need to have an agency. That’s generally simple enough. You just manually insert an agency into the `agency` table. After that’s done, we finally have a view of the actual application. It’s very Bootstrap-y, but that’s just fine. The workflow though, is not perfectly intuitive, but I’ll talk about that some other day.

That’s not the end of it though, we still have to fix these bugs. The developer obviously didn’t have to put up with this when they were working, so what happened? Also, the log is showing some weird things,

    ~        _            _
    ~  _ __ | | __ _ _  _| |
    ~ | '_ \| |/ _' | || |_|
    ~ |  __/|_|\____|\__ (_)
    ~ |_|            |__/
    ~
    ~ play! 1.2.5, http://www.playframework.org
    ~
    ~ Ctrl+C to stop
    ~
    CompilerOracle: exclude jregex/Pretokenizer.next
    Listening for transport dt_socket at address: 8000
    23:32:14,943 INFO  ~ Starting /Users/thomas/Workspace/maps/gtfs-editor
    23:32:14,948 WARN  ~ Declaring modules in application.conf is deprecated. Use dependencies.yml instead (module.secure)
    23:32:14,948 INFO  ~ Module secure is available (/Users/thomas/.root/opt/play-1.2.5/modules/secure)
    23:32:15,830 WARN  ~ You're running Play! in DEV mode
    23:32:15,952 INFO  ~ Listening for HTTP on port 9000 (Waiting a first request to start) ...
    23:32:28,792 ERROR ~

    @6f02fa9dd
    Internal Server Error (500) for request GET /

    Compilation error (In /app/controllers/Check.java around line 10)
    The file /app/controllers/Check.java could not be compiled. Error raised is : The type Check is already defined

    play.exceptions.CompilationException: The type Check is already defined
    	at play.classloading.ApplicationCompiler$2.acceptResult(ApplicationCompiler.java:246)
    	at org.eclipse.jdt.internal.compiler.Compiler.handleInternalException(Compiler.java:672)
    	at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:516)
    	at play.classloading.ApplicationCompiler.compile(ApplicationCompiler.java:282)
    	at play.classloading.ApplicationClassloader.getAllClasses(ApplicationClassloader.java:426)
    	at play.Play.start(Play.java:516)
    	at play.Play.detectChanges(Play.java:630)
    	at play.Invoker$Invocation.init(Invoker.java:198)
    	at Invocation.HTTP Request(Play!)
    23:32:31,551 INFO  ~ Connected to jdbc:postgresql://127.0.0.1/gtfs_editor
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/Users/thomas/Workspace/maps/gtfs-editor/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/Users/thomas/.root/opt/play-1.2.5/framework/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    23:32:32,490 INFO  ~ Initializing HBSpatialExtension
    23:32:32,492 INFO  ~ Attempting to load Hibernate Spatial Provider org.hibernatespatial.postgis.DialectProvider
    23:32:32,494 INFO  ~ Checking for default configuration file.
    23:32:32,496 INFO  ~ No configuration file hibernate-spatial.cfg.xml on the classpath.
    23:32:34,077 INFO  ~ Application 'gtfs-editor' is now started !
    23:32:34,151 INFO  ~ Bootstrapping Database...
    23:32:34,297 DEBUG ~ select count(*) as col_0_0_ from Agency agency0_ limit ?
    play.exceptions.UnexpectedException: Unexpected Error
    	at play.vfs.VirtualFile.contentAsString(VirtualFile.java:180)
    	at play.templates.TemplateLoader.load(TemplateLoader.java:78)
    	at play.test.Fixtures.loadModels(Fixtures.java:174)
    	at jobs.BootstrapDatabase.doJob(BootstrapDatabase.java:57)
    	at play.jobs.Job.doJobWithResult(Job.java:50)
    	at play.jobs.Job.call(Job.java:146)
    	at play.jobs.Job.run(Job.java:132)
    	at play.jobs.JobsPlugin.afterApplicationStart(JobsPlugin.java:116)
    	at play.plugins.PluginCollection.afterApplicationStart(PluginCollection.java:531)
    	at play.Play.start(Play.java:547)
    	at play.Play.detectChanges(Play.java:630)
    	at play.Invoker$Invocation.init(Invoker.java:198)
    	at play.server.PlayHandler$NettyInvocation.init(PlayHandler.java:189)
    	at play.Invoker$Invocation.run(Invoker.java:276)
    	at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:229)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    	at java.lang.Thread.run(Thread.java:680)
    Caused by: play.exceptions.UnexpectedException: Unexpected Error
    	at play.vfs.VirtualFile.inputstream(VirtualFile.java:111)
    	at play.vfs.VirtualFile.contentAsString(VirtualFile.java:178)
    	... 22 more
    Caused by: java.io.FileNotFoundException: /Users/thomas/.root/opt/play-1.2.5/modules/docviewer/app/initial-agencies-data.yml (No such file or directory)
    	at java.io.FileInputStream.open(Native Method)
    	at java.io.FileInputStream.<init>(FileInputStream.java:120)
    	at play.vfs.VirtualFile.inputstream(VirtualFile.java:109)
    	... 23 more
    23:32:34,316 ERROR ~ java.lang.RuntimeException: Cannot load fixture initial-agencies-data.yml: Unexpected Error
    23:32:40,989 DEBUG ~ select account0_.id as id15_, account0_.active as active15_, account0_.admin as admin15_, account0_.agency_id as agency9_15_, account0_.email as email15_, account0_.lastLogin as lastLogin15_, account0_.password as password15_, account0_.passwordChangeToken as password7_15_, account0_.username as username15_ from Account account0_ where account0_.username=? limit ?
    23:32:40,994 DEBUG ~ select count(*) as col_0_0_ from Account account0_ limit ?
    23:32:40,999 DEBUG ~ select nextval ('hibernate_sequence')
    23:32:41,051 DEBUG ~ insert into Account (active, admin, agency_id, email, lastLogin, password, passwordChangeToken, username, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
    23:32:41,061 DEBUG ~ select agency0_.id as id24_, agency0_.color as color24_, agency0_.defaultLat as defaultLat24_, agency0_.defaultLon as defaultLon24_, agency0_.defaultRouteType_id as default12_24_, agency0_.gtfsAgencyId as gtfsAgen5_24_, agency0_.lang as lang24_, agency0_.name as name24_, agency0_.phone as phone24_, agency0_.systemMap as systemMap24_, agency0_.timezone as timezone24_, agency0_.url as url24_ from Agency agency0_ order by agency0_.name
    23:32:41,175 ERROR ~

    @6f02fa9dg
    Internal Server Error (500) for request GET /

    Execution exception (In /app/controllers/Application.java around line 57)
    IndexOutOfBoundsException occured : Index: 0, Size: 0

    play.exceptions.JavaExecutionException: Index: 0, Size: 0
    	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:237)
    	at Invocation.HTTP Request(Play!)
    Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    	at java.util.ArrayList.RangeCheck(ArrayList.java:547)
    	at java.util.ArrayList.get(ArrayList.java:322)
    	at controllers.Application.initSession(Application.java:57)
    	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:510)
    	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:484)
    	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:479)
    	at play.mvc.ActionInvoker.handleBefores(ActionInvoker.java:328)
    	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:142)
    	... 1 more

After `23:32:34` is when I get the login page. `23:32:40` is after I’ve logged in.

### Second Pass

So how did you do? First, the error that `type Check already defined` usually does mean that `Check` was already defined elsewhere. Looking in the app folder though, there was nothing of the sort. It’s the only one there that was `Check.java`. But remember the secure module? Modules work by providing source files and Play just compiles them all together. Bingo, `Check.java`. Doing a diff shows nothing was changed. So the solution really was just simply delete `Check.java` and also `Secure.java`. No more compilation errors!

The next question is, how do you get the initial user? There actually is some code that looks like it creates the default admin user,

    if(Security.isConnected()) {
        ...
        Account account = Account.find("username = ?", Security.connected()).first();
        ...
        if(account == null && Account.count() == 0) {
            account = new Account("admin", "admin", "admin@test.com", true, null);
            account.save();
        }
        ...
    }

You can actually see this in action at `23:32:41,051` in the log. So what’s wrong with all of this? The account creation happened after I’ve already logged in. In fact, `Security.isConnected()` checks whether the user is already logged in or not. How does this even make sense?

Lastly, we have the problem of the agencies. Just by looking at the log, you can safely say we’re missing a file called `initial-agencies-data.yml`. Ok, apparently it’s a fixture like you would use for testing. It’s easy enough to infer what the file’s contents should be. We just copy it over from the GTFS data.

But then where do you put the file? If you look at the log, it says `/Users/thomas/.root/opt/play-1.2.5/modules/docviewer/app/initial-agencies-data.yml` but that doesn’t look right. That’s in the Play distribution directory, probably not somewhere something app-specific should go into. Well, a fixture is used for testing, so maybe the `test/` directory? No, that doesn’t work either since we’re not running a test.

What I ended up doing was just looking at the sources for `Fixtures.load`. If you follow the stack trace, you end up finding `Play.javaPath` which sort of works like PATH for Fixtures and some other things. So where can we put the file? `app/` and `conf/`. And with that, we’re done.

GTFS Editor is very much in development. Just getting it to run was problematic. There also seem to be a lot of missing issues judging from the Github Issues page. If you want to try it out for yourself, I suggest you clone my branch as I’ve fixed the issues discussed earlier. The default login is `admin:admin`.

Even after getting it to run, it’s still not quite usable. Not in the UX sense, but you really can’t do much with it. There is no way to import the GTFS data into the webapp. There is something like import from TransitWand but even that is unclear to me. And even if we do get that running as well, we still don’t have any data we can play around with. We would need database dumps from the already running tools for these to be of any use right now.

Posted: 2013-07-10 | Comments

# One Bus (or maybe Jeep) Away

Link: http://onebusaway.org/

TL;DR no routing; useless in Philippines

OneBusAway is a transit information app. It provides data on what bus stops are near you, which buses pass by. You can also get schedules and the route of a particular bus given the number. It can also provide realtime updates like how many minutes until the next bus arrives. It does not, however, provide routing. There is no support for providing directions to get from point A to point B.

It’s comparable to what you get in some bus stops abroad. You’d get a vicinity map and a list of buses passing through the stop. You might also get the times when the next buses will pass. It’s useful for locals who already know how to get around, and want to avoid waiting for the bus. But it’s not particularly good for people who want to know how to get around the city.

OneBusAway is quite comprehensive in its platform support though. There is a webapp, apps for iOS, Android and Windows Phone, as well as SMS and Voice support. This would all be nice but we don’t have the necessary infrastructure yet in the Philippines. We don’t have bus or jeepney stops. We also wouldn’t have realtime data to make the app particularly useful.

You can try it out for yourself by following their Quickstart Guide. One caveat is you will have to add `-P tripEntriesFactory.throwExceptionOnInvalidStopToShapeMappingException=false` when building the bundle. This has to do with the OneBusAway having difficulty matching the stops to the shape data.

Here’s some screenshots of the app with the Philippine data. Notice how you only see the stops but there isn’t a line for the route. This is a problem with our GTFS data. Also, at some points it’s hard to tell where the jeep is going to pass since there isn’t any indication of order either. This is more of a OneBusAway problem. It usually expects there to be shape data available.

It doesn’t really handle too many routes passing through a stop. The list just overflows past the bubble. You can still actually read it by panning the map. It’s just a bit weird though.

If you also noticed, there are usually 2 of each route. This is how the jeepney data was modeled as jeep routes might be different going one way and going back. This isn’t the case for all jeeps though, so it might also be an implementation issue with the GTFS editor.

Posted: 2013-07-09 | Comments