blog/output/rss.xml

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Pleasant Programmer</title>
    <link>https://pleasantprogrammer.com/</link>
    <description>Recent content on Pleasant Programmer</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Sun, 23 Jan 2022 15:06:47 +0900</lastBuildDate>
    
	<atom:link href="https://pleasantprogrammer.com/rss.xml" rel="self" type="application/rss+xml" />
    
    
    <item>
      <title>SSH Access to Git Repository in Docker</title>
      <link>https://pleasantprogrammer.com/posts/ssh-git-repo-docker.html</link>
      <pubDate>Sun, 23 Jan 2022 15:06:47 +0900</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/ssh-git-repo-docker.html</guid>
      <description>With the likes of Gogs and Gitea, self-hosting a personal git service has become quite common. It&amp;rsquo;s also not unlikely that the software is run via docker but this brings a problem with regards to SSH access.
Ideally, the git service container just exposes port 22 but this would conflict with the host&amp;rsquo;s own SSH service. One solution would be to just use different ports for the git SSH and host SSH and that&amp;rsquo;s perfectly fine.</description>
      <content:encoded>&lt;p&gt;With the likes of &lt;a href=&#34;https://gogs.io/&#34;&gt;Gogs&lt;/a&gt; and &lt;a href=&#34;https://gitea.io&#34;&gt;Gitea&lt;/a&gt;,
self-hosting a personal git service has become quite common. It&amp;rsquo;s also not
unlikely that the software is run via docker but this brings a problem with
regards to SSH access.&lt;/p&gt;
&lt;p&gt;Ideally, the git service container just exposes port 22 but this would conflict
with the host&amp;rsquo;s own SSH service. One solution would be to just use different
ports for the git SSH and host SSH and that&amp;rsquo;s perfectly fine. But we can also
just have the host SSH service forward the request to the git service itself
using the &lt;code&gt;command&lt;/code&gt; option in &lt;code&gt;authorized_keys&lt;/code&gt;. And as we&amp;rsquo;ll find out later,
the git service itself is using this functionality.&lt;/p&gt;
&lt;h2 id=&#34;how-git-over-ssh-works&#34;&gt;How git over SSH works&lt;/h2&gt;
&lt;p&gt;When you do a &lt;code&gt;git push&lt;/code&gt;, what actually happens is it runs &lt;code&gt;git-send-pack&lt;/code&gt; which
runs &lt;code&gt;git-receive-pack &amp;lt;directory&amp;gt;&lt;/code&gt; on the remote using SSH. The actual
communication then just simply happens via stdin/stdout. Conversely, doing a
&lt;code&gt;git pull&lt;/code&gt; just runs &lt;code&gt;git-fetch-pack&lt;/code&gt; and the somewhat confusingly named
&lt;code&gt;git-upload-pack&lt;/code&gt; on the remote.&lt;/p&gt;
&lt;p&gt;So far so good, but did you notice that when cloning via SSH, the remote is
typically &lt;code&gt;git@github.com:org/repo&lt;/code&gt;? If everyone SSHs in as the &lt;code&gt;git&lt;/code&gt; user, how
does the git service know which user is which? And how does it prevent users
from accessing each other&amp;rsquo;s repositories?&lt;/p&gt;
&lt;p&gt;One typically thinks of &lt;code&gt;authorized_keys&lt;/code&gt; as just a list of allowed SSH keys but
it can do &lt;a href=&#34;https://linux.die.net/man/8/sshd&#34;&gt;much more than that&lt;/a&gt;. Of particular
interest is the &lt;code&gt;command&lt;/code&gt; directive which gets run instead of the user supplied
command. The original command is passed in as an environment variable
&lt;code&gt;SSH_ORIGINAL_COMMAND&lt;/code&gt; which can be used to check if we allow it to be run or
not.&lt;/p&gt;
&lt;p&gt;So with an &lt;code&gt;authorized_keys&lt;/code&gt; file like so:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;command=&amp;#34;verify-user user-1&amp;#34; ssh-rsa ...
command=&amp;#34;verify-user user-1&amp;#34; ssh-ed25519 ...
command=&amp;#34;verify-user user-2&amp;#34; ssh-rsa ...
command=&amp;#34;verify-user user-3&amp;#34; ssh-rsa ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Each key is tied to a particular user by virtue of &lt;code&gt;command&lt;/code&gt;. And &lt;code&gt;verify-user&lt;/code&gt;
can check the &lt;code&gt;SSH_ORIGINAL_COMMAND&lt;/code&gt; if the particular user is allowed access to
the particular repository. You can also implement additional restrictions like
pull-only or push-only permissions with this setup. This is how both Gogs and
Gitea work.&lt;/p&gt;
&lt;h2 id=&#34;forwarding-git-ssh-to-docker&#34;&gt;Forwarding git SSH to Docker&lt;/h2&gt;
&lt;p&gt;When running Gogs on the host, it&amp;rsquo;s typically run as the &lt;code&gt;git&lt;/code&gt; user and when an
SSH key is added or removed, it simply rewrites &lt;code&gt;~git/.ssh/authorized_keys&lt;/code&gt;.
Thus it just works with the host SSH service without problems. When running
inside docker, one thing we can do is bind mount the host&amp;rsquo;s &lt;code&gt;~git/.ssh&lt;/code&gt; folder
into the docker container so that the host can authorize the SSH connections.&lt;/p&gt;
&lt;p&gt;The problem lies with the &lt;code&gt;command&lt;/code&gt; which only exists inside the docker
container itself. So any user trying to connect can authenticate successfully
but will get a &lt;code&gt;command not found&lt;/code&gt; error. For the Gogs docker image, the command
looks like &lt;code&gt;/app/gogs/gogs serv key-1&lt;/code&gt;. So we can just make &lt;code&gt;/app/gogs/gogs&lt;/code&gt;
available on the host and forward the command to the docker container.&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://docs.gitea.io/en-us/install-with-docker/#ssh-container-passthrough&#34;&gt;Most&lt;/a&gt;
&lt;a href=&#34;http://www.ateijelo.com/blog/2016/07/09/share-port-22-between-docker-gogs-ssh-and-local-system&#34;&gt;instructions&lt;/a&gt;
I&amp;rsquo;ve seen with regards to this involves using &lt;code&gt;ssh&lt;/code&gt; to connect to the internal
docker SSH service but this just seems overly complicated to me. If you
remember, at it&amp;rsquo;s core git really only communicates over stdin/stdout and SSH is
just a means to get that.&lt;/p&gt;
&lt;p&gt;If all we need is for our shim &lt;code&gt;/app/gogs/gogs&lt;/code&gt; to be able to run a command
inside the docker container with stdin/stdout attached, then we can actually
just do that with &lt;code&gt;docker exec&lt;/code&gt;. So it can be something like this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-shell&#34; data-lang=&#34;shell&#34;&gt;&lt;span style=&#34;color:#0f0;font-weight:bold&#34;&gt;#!/usr/bin/env bash
&lt;/span&gt;&lt;span style=&#34;color:#0f0;font-weight:bold&#34;&gt;&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;# Requires the following in sudoers&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;# git ALL=(ALL) NOPASSWD: /app/gogs/gogs&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;# Defaults:git env_keep=SSH_ORIGINAL_COMMAND&lt;/span&gt;

GOGS_CONTAINER=git-gogs-1

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt; [[ $EUID -ne &lt;span style=&#34;color:#ff0;font-weight:bold&#34;&gt;0&lt;/span&gt; ]]; &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;then&lt;/span&gt;
  &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;exec&lt;/span&gt; sudo &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$0&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$@&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;fi&lt;/span&gt;

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt; [ &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$1&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt; != &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;serv&amp;#34;&lt;/span&gt; ]; &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;then&lt;/span&gt;
  &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;exit&lt;/span&gt; &lt;span style=&#34;color:#ff0;font-weight:bold&#34;&gt;1&lt;/span&gt;
&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;fi&lt;/span&gt;

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;exec&lt;/span&gt; docker &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;exec&lt;/span&gt; -i -u git -e &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;SSH_ORIGINAL_COMMAND=&lt;/span&gt;$SSH_ORIGINAL_COMMAND&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$GOGS_CONTAINER&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt; /app/gogs/gogs &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$@&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;So for git SSH access to Gogs running in docker, the necessary steps here are:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Have a &lt;code&gt;git&lt;/code&gt; user on the host&lt;/li&gt;
&lt;li&gt;Bind mount &lt;code&gt;~git/.ssh&lt;/code&gt; to &lt;code&gt;/data/git/.ssh&lt;/code&gt; in the Gogs container&lt;/li&gt;
&lt;li&gt;Add the shim script to &lt;code&gt;/app/gogs/gogs&lt;/code&gt; (make sure it&amp;rsquo;s owned by root and
is chmod-ed &lt;code&gt;0755&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Add the listed sudoers rules&lt;/li&gt;
&lt;/ol&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Android Multisim Pre-5.1</title>
      <link>https://pleasantprogrammer.com/posts/android-multisim.html</link>
      <pubDate>Sun, 12 Sep 2021 19:37:22 +0900</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/android-multisim.html</guid>
      <description>NOTE if you&amp;rsquo;re just looking for a library to use, there&amp;rsquo;s MultiSim. I&amp;rsquo;ve never used this so I can&amp;rsquo;t guarantee anything about it. It also only supports SIM information and not SMS.
Phones that can take multiple SIM cards are quite popular in the Philippines. The two major telecoms would have unlimited SMS packages for messages within their networks. It was quite common to have a SIM for each telco and use the appropriate one depending on who you were sending to.</description>
      <content:encoded>&lt;p&gt;&lt;strong&gt;NOTE&lt;/strong&gt; if you&amp;rsquo;re just looking for a library to use, there&amp;rsquo;s &lt;a href=&#34;https://github.com/UseHover/MultiSim&#34;&gt;MultiSim&lt;/a&gt;. I&amp;rsquo;ve never used this so I can&amp;rsquo;t guarantee anything about it. It also only supports SIM information and not SMS.&lt;/p&gt;
&lt;p&gt;Phones that can take multiple SIM cards are quite popular in the Philippines. The two major telecoms would have unlimited SMS packages for messages within their networks. It was quite common to have a SIM for each telco and use the appropriate one depending on who you were sending to.&lt;/p&gt;
&lt;p&gt;Android&amp;rsquo;s API only officially supported multiple SIM cards in 5.1 (API level 22) but Android phones with dual-SIM (and even triple-SIM) capabilities were already available at least as far back as 2.3 (API level 10) when I first needed to support it. Since there was no official API for this, the manufacturers just invented their own and of course each one implemented it in a different way.&lt;/p&gt;
&lt;h2 id=&#34;mediatek&#34;&gt;Mediatek&lt;/h2&gt;
&lt;p&gt;The first phone we started working on was a &lt;a href=&#34;https://www.gsmarena.com/lenovo_a60-4711.php&#34;&gt;Lenovo A60&lt;/a&gt; which used a Mediatek SOC. We somehow got a library from the manufacturer that let us use the dual-SIM functionality, but it was quite a pain to get working as there was limited documentation and we were quite new to Android development at the time.&lt;/p&gt;
&lt;p&gt;When we disassembled the library that they gave us, we noticed that the names they used for the additional functions were quite interesting. They were all the &lt;code&gt;TelephonyManager&lt;/code&gt; and &lt;code&gt;SmsManager&lt;/code&gt; methods with a &lt;code&gt;Gemini&lt;/code&gt; suffix and they would take an additional &lt;code&gt;int&lt;/code&gt; parameter in addition to the original.&lt;/p&gt;
&lt;p&gt;It turned out that these were available on the standard &lt;code&gt;TelephonyManager&lt;/code&gt; instance and could be accessed via reflection. The &lt;code&gt;SmsManager&lt;/code&gt; was a bit trickier but we ended up figuring out that there was a &lt;code&gt;android.telephony.gemini.GeminiSmsManager&lt;/code&gt; class that had the functionality.&lt;/p&gt;
&lt;p&gt;In a different phone with a Mediatek SOC, this got renamed to &lt;code&gt;com.mediatek.telephony.gemini.SmsManager&lt;/code&gt; for some reason and dropped the &lt;code&gt;Gemini&lt;/code&gt; suffix only for the &lt;code&gt;SmsManager&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&#34;intel&#34;&gt;Intel&lt;/h2&gt;
&lt;p&gt;It was also around this time that Intel started making SOCs for smartphones. We had an &lt;a href=&#34;https://www.gsmarena.com/asus_fonepad_7_%282014%29-6394.php&#34;&gt;ASUS Fonepad 7&lt;/a&gt;. Unlike with the Mediatek device, we didn&amp;rsquo;t have a library to use here and had to use reflection to find the hidden classes / methods.&lt;/p&gt;
&lt;p&gt;What we found was that instead of having a single instance with every method taking a &lt;code&gt;sim&lt;/code&gt; parameter, they instead had separate instances of &lt;code&gt;TelephonyManager&lt;/code&gt; and &lt;code&gt;SmsManager&lt;/code&gt; for each SIM. You would call &lt;code&gt;TelephonyManager.get2ndTm()&lt;/code&gt; and &lt;code&gt;SmsManager.get2ndSmsManager()&lt;/code&gt; to have access to the 2nd SIM.&lt;/p&gt;
&lt;h2 id=&#34;qualcomm&#34;&gt;Qualcomm&lt;/h2&gt;
&lt;p&gt;The last phone I looked at was a &lt;a href=&#34;https://www.gsmarena.com/motorola_moto_g_dual_sim-5978.php&#34;&gt;dual-SIM Moto G&lt;/a&gt;. What&amp;rsquo;s interesting about this one is that the API completely changed in the upgrade from 4.4 to 5.0.&lt;/p&gt;
&lt;p&gt;On Android 4.4, the API was pretty close to the Mediatek one. You had a single instance that could dispatch to other SIMs by having an extra parameter on all the methods. These were in &lt;code&gt;android.telephony.MSimTelephonyManager&lt;/code&gt; and &lt;code&gt;android.telephony.MSimSmsManager&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;On Android 5.0, the API was a weird mix of all the above and also the introduction of &lt;code&gt;android.telephony.SubscriptionManager&lt;/code&gt; which was quite close but not exactly the same as what ended up in the official API. Instead of &lt;code&gt;getActiveSubscriptionInfoList&lt;/code&gt; there was &lt;code&gt;getActiveSubIdList&lt;/code&gt; which only returned &lt;code&gt;long[]&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For the information that would normally exist in &lt;code&gt;SubscriptionInfo&lt;/code&gt;, you had to query the main &lt;code&gt;TelephonyManager&lt;/code&gt; instance which had methods with an extra &lt;code&gt;long&lt;/code&gt; parameter for the subscription id. The &lt;code&gt;SmsManager&lt;/code&gt; was simpler with just &lt;code&gt;getSmsManagerForSubscriber&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;With Android 5.1, I assume they just switched to using the official API so this phone would have gone through 3 different multi-SIM APIs over the course of it&amp;rsquo;s life.&lt;/p&gt;
&lt;h2 id=&#34;epilogue&#34;&gt;Epilogue&lt;/h2&gt;
&lt;p&gt;Around the release of Android 5.1, we stopped work on the app so I never actually got to use the official API myself ironically. We also never really got a big deployment so while I saw quite the variety of multi-SIM implementations, that&amp;rsquo;s probably not all that&amp;rsquo;s been out in the wild.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>ISP Issues</title>
      <link>https://pleasantprogrammer.com/posts/isp-issues.html</link>
      <pubDate>Thu, 16 Aug 2018 23:03:05 +0900</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/isp-issues.html</guid>
      <description>At the first office I worked at, we had 2 different ISPs. This was supposed to be for reliability, as one was fast but spotty, and the other was slow but reliable. Since they weren&amp;rsquo;t too expensive, we just went and got both.
We have monitoring setup to watch our office IPs from the outside so we could see how often the connection goes down. The interesting thing we found was that the fast and spotty connection had perfect uptime.</description>
      <content:encoded>&lt;p&gt;At the first office I worked at, we had 2 different ISPs. This was supposed to be for reliability, as one was fast but spotty, and the other was slow but reliable. Since they weren&amp;rsquo;t &lt;em&gt;too&lt;/em&gt; expensive, we just went and got both.&lt;/p&gt;
&lt;p&gt;We have monitoring setup to watch our office IPs from the outside so we could see how often the connection goes down. The interesting thing we found was that the fast and spotty connection had perfect uptime. Even when there was clearly no internet from the office, it was still &amp;ldquo;up&amp;rdquo; according to our monitoring.&lt;/p&gt;
&lt;p&gt;So we tried pinging our office IP using the other connection and to our surprise it was indeed up. There was even a webserver running on it (we only have VPN exposed). Apparently, it was someone elses CCTV admin page. We could actually see a hallway with people walking by sometimes!&lt;/p&gt;
&lt;p&gt;Apparently someone else had our IP address and nothing good comes from an IP conflict. This was completely baffling as our internet line was supposed to be a &amp;ldquo;business line&amp;rdquo; and that came with a static IP address. So the only scenarios where this could happen is, the ISP mistakenly gave the same IP to 2 different lines or the ISP allows some clients to freely set their own IP.&lt;/p&gt;
&lt;p&gt;We complained to the ISP and eventually got it resolved. They just gave us an entirely new IP address, but they never explained what went wrong. We already had quite a negative opinion of that particular ISP though, and they somehow managed to outdo themselves.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Projects</title>
      <link>https://pleasantprogrammer.com/pages/projects.html</link>
      <pubDate>Sat, 10 Feb 2018 16:47:55 +0900</pubDate>
      
      <guid>https://pleasantprogrammer.com/pages/projects.html</guid>
      <description>I&amp;rsquo;ve made a lot of things in my free time mostly to try out new web technologies. I&amp;rsquo;ve also made some non-web things as well, but those aren&amp;rsquo;t as easy to show off.
Datalinks The Datalinks is a recreation of the in-game manual of Sid Meier&amp;rsquo;s Alpha Centauri. I really like the game, particularly how well the setting and atmosphere was done. The voice quotes whenever you discover a new technology were what really made the game for me and I&amp;rsquo;ve included those into the site.</description>
      <content:encoded>&lt;p&gt;I&amp;rsquo;ve made a lot of things in my free time mostly to try out new web
technologies. I&amp;rsquo;ve also made some non-web things as well, but those aren&amp;rsquo;t as
easy to show off.&lt;/p&gt;
&lt;h2 id=&#34;datalinks&#34;&gt;Datalinks&lt;/h2&gt;
&lt;p&gt;The &lt;a href=&#34;https://datalinks.pleasantprogrammer.com&#34;&gt;Datalinks&lt;/a&gt; is a recreation of the
in-game manual of Sid Meier&amp;rsquo;s Alpha Centauri. I really like the game,
particularly how well the setting and atmosphere was done. The voice quotes
whenever you discover a new technology were what really made the game for me and
I&amp;rsquo;ve included those into the site. I also tried my very best to capture the
original look and feel from the game using HTML and CSS.&lt;/p&gt;
&lt;h2 id=&#34;audventure&#34;&gt;Audventure&lt;/h2&gt;
&lt;p&gt;&lt;a href=&#34;https://audventure.pleasantprogrammer.com&#34;&gt;Audventure&lt;/a&gt; is a clone of the
Gameboy Advance game, bit generations: Soundvoyager. The game really got my
interest as the main mechanic involves navigating via sound. The original game
was comprised of 5 different minigames, but I&amp;rsquo;ve only really implemented one of
them.&lt;/p&gt;
&lt;h2 id=&#34;gtfshtml&#34;&gt;GTFS.html&lt;/h2&gt;
&lt;p&gt;&lt;a href=&#34;https://gtfs.pleasantprogrammer.com&#34;&gt;GTFS.html&lt;/a&gt; is a viewer for transit data in
the GTFS format. I work a lot with GTFS for Sakay.ph and it helps quite a lot to
be able to visualize the GTFS quickly instead of looking at the raw CSV files or
actually loading it into the routing engine and testing manually.&lt;/p&gt;
&lt;h2 id=&#34;typingfreaks&#34;&gt;Typingfreaks&lt;/h2&gt;
&lt;p&gt;&lt;a href=&#34;https://typingfreaks.pleasantprogrammer.com&#34;&gt;Typingfreaks&lt;/a&gt; is a clone of an old
Japanese flash game called TypingMania. It&amp;rsquo;s a speed typing game where you have
to type lyrics (mainly Japanese) in time with the song as it&amp;rsquo;s playing.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Audventure</title>
      <link>https://pleasantprogrammer.com/posts/audventure.html</link>
      <pubDate>Sun, 19 Nov 2017 22:00:00 +0800</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/audventure.html</guid>
      <description>Sometime around 2013 I wrote a clone of the GBA game bit Generations SoundVoyager called audventure. SoundVoyager is actually a collection of mini-games where sound is the main focus. You can actually play the game blind, and at some point, that&amp;rsquo;s pretty much what happens.
sound catcher The signature mini-game in SoundVoyager is sound catcher. In the mini-game, you can only move left and right at the bottom of the stage, while a &amp;ldquo;sound&amp;rdquo; falls from the top.</description>
      <content:encoded>&lt;p&gt;Sometime around 2013 I wrote a clone of the GBA game &lt;a href=&#34;https://www.nintendo.co.jp/n08/bit_g/&#34;&gt;bit Generations
SoundVoyager&lt;/a&gt; called
&lt;a href=&#34;https://audventure.pleasantprogrammer.com&#34;&gt;audventure&lt;/a&gt;. SoundVoyager is
actually a collection of mini-games where sound is the main focus. You can
actually play the game blind, and at some point, that&amp;rsquo;s pretty much what
happens.&lt;/p&gt;
&lt;h2 id=&#34;sound-catcher&#34;&gt;sound catcher&lt;/h2&gt;
&lt;p&gt;The signature mini-game in SoundVoyager is sound catcher. In the mini-game, you
can only move left and right at the bottom of the stage, while a &amp;ldquo;sound&amp;rdquo; falls
from the top. Your goal is to catch the sound which is signified by a green dot.
When you catch it, the sound or beat becomes part of the BGM and a new dot
appears with a different sound.&lt;/p&gt;
&lt;p&gt;You can of course use your eyes and move accordingly, but if you put on
earphones, you can actually hear where the dot is, either on your left or right,
with it getting louder as it gets close to you. As you collect more sounds, the
dot gets more and more transparent. Eventually (and this is where it gets fun),
you won&amp;rsquo;t be able to see the sounds anymore and will have to rely mostly on your
ears.&lt;/p&gt;
&lt;p&gt;You can see what the original game looks like in &lt;a href=&#34;https://www.youtube.com/watch?v=C12WRgfIOC8&#34;&gt;this
video&lt;/a&gt; or you can play it under
&lt;em&gt;sound safari&lt;/em&gt; in &lt;a href=&#34;https://audventure.pleasantprogrammer.com&#34;&gt;audventure&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&#34;webaudio-vs-flash&#34;&gt;WebAudio vs Flash&lt;/h2&gt;
&lt;p&gt;At the time I wrote audventure, only Chrome supported WebAudio. Also, the API
looked (and still looks) quite complicated. Flash on the other hand, was
starting to die, but still well-supported so I went with that. For the most
part, it worked okay though Chrome actually had timing issues when playing
sounds. Now, it doesn&amp;rsquo;t work in any browser. I tried to debug the issues but
ultimately ended up just rewriting it to use WebAudio instead.&lt;/p&gt;
&lt;p&gt;For the game, I needed to simulate the source of the sound in 2D/3D space. Flash
only really gives you stereo panning and volume control. With some maths, we can
actually get an acceptable solution. Less importantly, I needed to be able to
get frequency data of the currently playing &amp;ldquo;sound&amp;rdquo; to pulse the background. For
this, I actually had to implement the feature in the Flash library I was using.&lt;/p&gt;
&lt;p&gt;With WebAudio, spatial audio is already built-in and you can simply give it the
coordinates of the sounds and the listener. There are some other options to
tweak, but for the most part, no complex math is needed. Getting frequency data
for a sound is also actually built-in and didn&amp;rsquo;t take too long to integrate.&lt;/p&gt;
&lt;p&gt;Overall, I was impressed by how much you can do with WebAudio out-of-the-box. I
kind of understand why it&amp;rsquo;s complicated, but there&amp;rsquo;s some simple functionality
that I wish was included. For example, there is no API to pause and then resume
playing an audio buffer. You have to manually save the elapsed time and play
from there.&lt;/p&gt;
&lt;h2 id=&#34;other-mini-games&#34;&gt;Other mini-games&lt;/h2&gt;
&lt;p&gt;So far I&amp;rsquo;ve only actually implemented the sound catcher mini-game. There are
around 4 different categories with slight variations in between.&lt;/p&gt;
&lt;h3 id=&#34;sound-catcher--sound-slalom&#34;&gt;sound catcher / sound slalom&lt;/h3&gt;
&lt;p&gt;I&amp;rsquo;ve explained sound catcher a while ago; sound slalom is a minor variation on
that. Instead of waiting for the &amp;ldquo;sound&amp;rdquo; to reach you, you now have to guide
yourself in between 2 &amp;ldquo;poles&amp;rdquo; of sound, as in &lt;a href=&#34;https://en.wikipedia.org/wiki/Slalom_skiing&#34;&gt;slalom
skiing&lt;/a&gt;. But this time, you can
also accelerate forward. The goal is to finish the course before the time runs
out.&lt;/p&gt;
&lt;h3 id=&#34;sound-drive--sound-chase&#34;&gt;sound drive / sound chase&lt;/h3&gt;
&lt;p&gt;In sound drive, you&amp;rsquo;re driving against the flow on a 5 lane road. You have to
avoid oncoming cars, trucks and animals until you reach the end. You&amp;rsquo;re allowed
to change lanes and accelerate, and the game tracks your best time. Sound chase
is pretty much the same, except you&amp;rsquo;re trying to catch up to a &amp;ldquo;sound&amp;rdquo;.&lt;/p&gt;
&lt;h3 id=&#34;sound-cannon&#34;&gt;sound cannon&lt;/h3&gt;
&lt;p&gt;In sound cannon, you&amp;rsquo;re immobile but can rotate within a 180 degree angle. Your
goal is too shoot down &amp;ldquo;sounds&amp;rdquo; which are heading your way. If a sound reaches
you, it&amp;rsquo;s game over. You win when you kill all the sounds.&lt;/p&gt;
&lt;h3 id=&#34;sound-picker--sound-cock&#34;&gt;sound picker / sound cock&lt;/h3&gt;
&lt;p&gt;In sound picker, you can move in a giant square field where various sounds are
scattered around. Your goal is to pick up all the sounds within the time limit.
Sound cock is similar, except the sounds are chickens and you have to chase them
around.&lt;/p&gt;
&lt;h2 id=&#34;source-code&#34;&gt;Source Code&lt;/h2&gt;
&lt;p&gt;If you want to see the source code, you can check it out
&lt;a href=&#34;https://git.pleasantprogrammer.com/games/audventure&#34;&gt;here&lt;/a&gt;. The sound files
aren&amp;rsquo;t in the repo though, since I&amp;rsquo;m not quite sure about the licensing. If you
want to contribute music or sound effects, I&amp;rsquo;d gladly appreciate it.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>OpenPrepPad</title>
      <link>https://pleasantprogrammer.com/posts/openpreppad.html</link>
      <pubDate>Sun, 15 Jan 2017 19:31:00 +0800</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/openpreppad.html</guid>
      <description>Smart electronics and IoT (Internet of Things) are all the rage these days. You have a lot of companies sprout up trying to make the next big thing, which also leads to a lot of failures big and small. Pebble, the maker of my smartwatch, got bought out by Fitbit recently. This left watch owners without any official support, but thankfully, community members stepped up to continue maintaining it.
Another casualty of the IoT boom was the Orange Chef Prep Pad.</description>
      <content:encoded>&lt;p&gt;Smart electronics and IoT (Internet of Things) are all the rage these days. You
have a lot of companies sprout up trying to make the next big thing, which also
leads to a lot of failures big and small. Pebble, the maker of my smartwatch,
got bought out by Fitbit recently. This left watch owners without any official
support, but thankfully, community members &lt;a href=&#34;http://rebble.io/&#34;&gt;stepped up&lt;/a&gt; to continue
maintaining it.&lt;/p&gt;
&lt;p&gt;Another casualty of the IoT boom was the &lt;a href=&#34;http://theorangechef.com/&#34;&gt;Orange Chef&lt;/a&gt; &lt;a href=&#34;https://www.amazon.com/Orange-Chef-Smart-Scale-Silver/dp/B00KFW8L90&#34;&gt;Prep Pad&lt;/a&gt;. It&amp;rsquo;s a
bluetooth connected weighing scale to make it easy to track your calories and
carb/fat/protein intake. My dad bought it last year only to find out that the
app was incredibly buggy. The search function doesn&amp;rsquo;t work which makes the whole
thing practically useless. I also found out later that you can&amp;rsquo;t even download
the app to use the scale anymore.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; I just found out as I was writing this post that it &lt;em&gt;may&lt;/em&gt; get supported
by &lt;a href=&#34;http://www.prnewswire.com/news-releases/perfect-company-acquires-orange-chefs-prep-pad-related-ip-continues-momentum-in-the-connected-kitchen-300383178.html&#34;&gt;another company&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;So the app is useless, but at least you can use it as a scale, right?&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;https://pleasantprogrammer.com/galleries/openpreppad/preppad.jpg&#34; alt=&#34;Prep Pad&#34;&gt;&lt;/p&gt;
&lt;p&gt;Nope. The device has no display whatsoever. The only controls on it are the
on/off button and a green LED that isn&amp;rsquo;t even that useful at telling you whether
it&amp;rsquo;s on or not. At this point, it&amp;rsquo;s just a giant paperweight.&lt;/p&gt;
&lt;h2 id=&#34;reverse-engineering&#34;&gt;Reverse Engineering&lt;/h2&gt;
&lt;p&gt;Since I essentially had nothing to lose, I tried poking at the thing to figure
out how it works. I didn&amp;rsquo;t really have experience with bluetooth besides trying
to get my bluetooth mouse connected on Linux. The main thing I used then was
&lt;code&gt;bluetoothctl&lt;/code&gt; which is essentially a CLI for managing bluetooth devices so I
started there.&lt;/p&gt;
&lt;p&gt;I started up &lt;code&gt;bluetoothctl&lt;/code&gt; and turned on the Prep Pad. And it showed up!&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;[bluetooth]# power on
[CHG] Controller ... Class: 0x00010c
Changing power on succeeded
[CHG] Controller ... Powered: yes
[bluetooth]# scan on
Discovery started
[CHG] Device 1C:BA:8C:21:7C:BB RSSI: -51
[CHG] Device 1C:BA:8C:21:7C:BB Name: CHSLEEV_00
[CHG] Device 1C:BA:8C:21:7C:BB Alias: CHSLEEV_00
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;I then connected to it, which was surprisingly easy.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;[bluetooth]# connect 1C:BA:8C:21:7C:BB
Attempting to connect to 1C:BA:8C:21:7C:BB
[CHG] Device 1C:BA:8C:21:7C:BB Connected: yes
[CHG] Device 1C:BA:8C:21:7C:BB Name: CH BTScale_00
[CHG] Device 1C:BA:8C:21:7C:BB Alias: CH BTScale_00
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Now normally, when you turn the device on, the green light flashes occasionally.
Once I connected to it, the green light stayed on permanently. Clearly, I was
making progress. A lot of services were also discovered but I had no idea what
those things were at that point.&lt;/p&gt;
&lt;p&gt;After a lot of poking around, I could check the general device information. You
could get the hardware, software and firmware version. There&amp;rsquo;s also the device
serial number which was nowhere on the actual physical device.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;[CHSLEEV_00]# select-attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017
[CH BTScale_00:/service0010/char0017]# attribute-info
Characteristic - Firmware Revision String
	UUID: 00002a26-0000-1000-8000-00805f9b34fb
	Service: /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010
	Value: 0x31
	Value: 0x2e
	Value: 0x31
	Value: 0x33
	Value: 0x41
	Value: 0x00
	Flags: read
[CH BTScale_00:/service0010/char0017]# read
Attempting to read /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017 Value: 0x31
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017 Value: 0x2e
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017 Value: 0x31
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017 Value: 0x33
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017 Value: 0x41
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0010/char0017 Value: 0x00
  31 2e 31 33 41 00                                1.13A.
[CH BTScale_00:/service0010/char0017]#
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;There was also a service which contained Accel Enable, Accel Range, Accel
X-Coordinate, Accel Y-Coordinate, and Accel Z-Coordinate. I guess it stands for
accelerometer, which is probably what it uses to weigh things.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;[CHSLEEV_00]# select-attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026
[CH BTScale_00:/service0023/char0024/desc0026]# read
Attempting to read /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x41
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x63
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x63
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x65
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x6c
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x20
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x45
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x6e
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x61
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x62
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x6c
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024/desc0026 Value: 0x65
  41 63 63 65 6c 20 45 6e 61 62 6c 65              Accel Enable
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;I couldn&amp;rsquo;t read from any of the Accel Coordinates. It kept saying permission
denied. I could however, notify on them. But that didn&amp;rsquo;t yield anything as well.
What I &lt;em&gt;could&lt;/em&gt; read was Accel Enable, which was set to 00. I guess that means it
was off. After writing 01 to Accel Enable, I found I could get values out of
Accel X-Coordinate! Also, the green LED which was permanently on turned off.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;[CHSLEEV_00]# select-attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024
[CH BTScale_00:/service0023/char0024]# write 01
Attempting to write /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char0024
[CH BTScale_00:/service0023/char0024]# select-attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a
[CH BTScale_00:/service0023/char002a]# notify on
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Notifying: yes
Notify started
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x5b
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0xa3
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x02
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x55
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0xa3
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x02
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x59
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0xa3
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x02
[CHG] Attribute /org/bluez/hci0/dev_1C_BA_8C_21_7C_BB/service0023/char002a Value: 0x00
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;I tried pressing the scale down a few times, and the values changed accordingly.
Now, I just had to figure out how to convert the values into grams. It looked
like the values were 32-bit integers sent as 4 bytes. In the above example it
would be &lt;code&gt;0x0002a35b&lt;/code&gt;, &lt;code&gt;0x0002a355&lt;/code&gt;, &lt;code&gt;0x0002a359&lt;/code&gt; or 172891, 172855, 172899. The
values also decrease as you exert more effort on the scale. So assuming you take
the initial value as &lt;em&gt;tare&lt;/em&gt;, you simply subtract any succeeding value from that
&lt;em&gt;tare&lt;/em&gt; and you get the &amp;ldquo;weight&amp;rdquo;.&lt;/p&gt;
&lt;p&gt;The values I got didn&amp;rsquo;t seem to be in grams though. After weighing some things
on an actual scale and comparing the values I got, I found I can just divide the
values by 14 and get something in grams. That 14 is entirely a magic number
though and I have no idea whether other Prep Pad&amp;rsquo;s would have the same constant.&lt;/p&gt;
&lt;h2 id=&#34;openpreppad&#34;&gt;OpenPrepPad&lt;/h2&gt;
&lt;p&gt;With all that figured out, I went ahead and made a &lt;a href=&#34;https://github.com/thatsmydoing/openpreppad&#34;&gt;simple CLI application&lt;/a&gt;
to interface with the Prep Pad. Ironically, node was the simplest thing I found
that had &lt;a href=&#34;https://github.com/sandeepmistry/noble&#34;&gt;nice bluetooth library support&lt;/a&gt; so that&amp;rsquo;s what I wrote it in. I
also added most of the technical details in the README for that as well.&lt;/p&gt;
&lt;p&gt;While this is all well and cool, I doubt the intersection of Linux users and
people who &lt;!-- raw HTML omitted --&gt;got ripped off&lt;!-- raw HTML omitted --&gt; bought the Prep Pad is anyone besides me. In
light of that, I&amp;rsquo;m in the process of making a React Native version of the app,
but that&amp;rsquo;s still a work in progress. Who knows, if the new owners of Prep Pad
are good, I might not even need to finish it.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Haproxy Charset</title>
      <link>https://pleasantprogrammer.com/posts/haproxy-charset.html</link>
      <pubDate>Fri, 24 Jun 2016 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/haproxy-charset.html</guid>
      <description>A common problem we encounter is for things like ñ not showing up correctly. This actually caused some issues in the recent Philippine elections, but this isn&amp;rsquo;t about hash codes or anything like that.
By default, we use UTF-8 for text storage and rendering. A problem is that browsers don&amp;rsquo;t assume UTF-8 as the default and you need to have either a &amp;lt;meta charset=&amp;quot;utf-8&amp;quot; /&amp;gt; in the HTML or Content-Type: text/html; charset=utf-8 in the headers.</description>
      <content:encoded>&lt;p&gt;A common problem we encounter is for things like &lt;em&gt;ñ&lt;/em&gt; not showing up correctly. This actually caused &lt;a href=&#34;http://www.rappler.com/nation/politics/elections/2016/132894-human-error-hash-election-results-code-mismatch&#34;&gt;some issues&lt;/a&gt; in the recent Philippine elections, but this isn&amp;rsquo;t about hash codes or anything like that.&lt;/p&gt;
&lt;p&gt;By default, we use UTF-8 for text storage and rendering. A problem is that browsers don&amp;rsquo;t assume UTF-8 as the default and you need to have either a &lt;code&gt;&amp;lt;meta charset=&amp;quot;utf-8&amp;quot; /&amp;gt;&lt;/code&gt; in the HTML or &lt;code&gt;Content-Type: text/html; charset=utf-8&lt;/code&gt; in the headers. A few of our services don&amp;rsquo;t set the &lt;code&gt;Content-Type&lt;/code&gt; with the &lt;code&gt;charset=utf-8&lt;/code&gt; part so you&amp;rsquo;d get piñata instead of piñata.&lt;/p&gt;
&lt;p&gt;Being lazy, we usually just correct this at the reverse proxy side. It&amp;rsquo;s trivial to do in nginx. You just need to add &lt;code&gt;charset utf-8;&lt;/code&gt; to your configuration and you&amp;rsquo;re good. For haproxy though, I couldn&amp;rsquo;t readily find a solution for it and had to go through the docs to see what I could do.&lt;/p&gt;
&lt;p&gt;After a bit of experimenting, I had success with this:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;# set content-type to utf-8 if not already
acl has_charset hdr_sub(content-type) -i charset=
rspirep (Content-Type.*) \1;\ charset=utf-8 unless has_charset
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This is probably not the best way to do it. Arguably, we should just fix our services to have the correct &lt;code&gt;Content-Type&lt;/code&gt; in the first place, but I can do that some other time.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Cloudflare Shenanigans</title>
      <link>https://pleasantprogrammer.com/posts/cloudflare-shenanigans.html</link>
      <pubDate>Fri, 25 Dec 2015 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/cloudflare-shenanigans.html</guid>
      <description>An old client of ours managed to convince a telco to zero-rate the data for their app. In order to whitelist it though, we needed to use plain HTTP for domain whitelisting. For HTTPS, they can only whitelist by IP address. Like any good developer, we were using HTTPS. Also, like any good developer, we put our server behind Cloudflare.
Now the problem is that Cloudflare can put you behind any IP they own, which is a huge range.</description>
      <content:encoded>&lt;p&gt;An old client of ours managed to convince a telco to zero-rate the data for their app. In order to whitelist it though, we needed to use plain HTTP for domain whitelisting. For HTTPS, they can only whitelist by IP address. Like any good developer, we were using HTTPS. Also, like any good developer, we put our server behind Cloudflare.&lt;/p&gt;
&lt;p&gt;Now the problem is that Cloudflare can put you behind &lt;a href=&#34;https://www.cloudflare.com/ips/&#34;&gt;any IP they own&lt;/a&gt;, which is a huge range. There&amp;rsquo;s no guarantee that the IP we have now is going to be the same later on. So we did the reasonable thing and asked them to whitelist all of the Cloudflare IPs. And the telco agreed! We were in total disbelief when that happened. But hey, if life gives you free internet, you take it.&lt;/p&gt;
&lt;p&gt;We never actually empirically tested whether other sites hosted on Cloudflare were also actually zero-rated. But I like to think that we saved a lot of people on their data costs from browsing Reddit and 4chan. But alas, good things must come to an end.&lt;/p&gt;
&lt;p&gt;A few months after we started beta testing the app, Cloudflare added more IPs to their range. Unfortunately, our server got moved to those new IPs which were not whitelisted yet. Apparently, the telco whitelisting process was incredibly convoluted and time consuming. Our client didn&amp;rsquo;t want to bother asking them to whitelist more IPs. We also tried asking Cloudflare to move us back to the original IP range, but they could only do that if we were in their enterprise tier. We couldn&amp;rsquo;t really afford that, so we looked for other options.&lt;/p&gt;
&lt;p&gt;Since Cloudflare was essentially just a giant reverse proxy, theoretically there should be no distinction between one IP address from another. The specific IP we get is probably just for load balancing. So we tried accessing the IPs in the range directly and just setting the Host header and it worked! But we get SSL errors because the IP itself doesn&amp;rsquo;t have its own certificate.&lt;/p&gt;
&lt;p&gt;After more testing, we figured out that you could actually use any Cloudflare backed domain so long as we properly set the Host header. We just needed to find one still in the old range. Coincidentally, 4chan.org was. Which led to this wonderful commit&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-diff&#34; data-lang=&#34;diff&#34;&gt;commit 123456789abcdef
Author: ~~~~~~
Date:   ~~~~~~

    4chan hack

&lt;span style=&#34;font-weight:bold&#34;&gt;diff --git a/src/com/client/common/Util.java b/src/com/client/common/Util.java
&lt;/span&gt;&lt;span style=&#34;font-weight:bold&#34;&gt;&lt;/span&gt;--- a/src/com/client/common/Util.java
+++ b/src/com/client/common/Util.java
&lt;span style=&#34;font-weight:bold&#34;&gt;@@ -210,7 +210,8 @@ public class Util {
&lt;/span&gt;&lt;span style=&#34;font-weight:bold&#34;&gt;&lt;/span&gt;        }

        public static String getServerAddress(Context context) {
-               String address = &amp;#34;https://backend.client.com&amp;#34;;
+               // String address = &amp;#34;https://backend.client.com&amp;#34;;
+               String address = &amp;#34;https://4chan.org&amp;#34;;
                if(!isDebug(context)) return address;
                try {
&lt;span style=&#34;font-weight:bold&#34;&gt;diff --git a/src/com/client/common/logging/APIClient.java b/src/com/client/common/logging/APIClient.java
&lt;/span&gt;&lt;span style=&#34;font-weight:bold&#34;&gt;&lt;/span&gt;--- a/src/com/client/common/logging/APIClient.java
+++ b/src/com/client/common/logging/APIClient.java
&lt;span style=&#34;font-weight:bold&#34;&gt;@@ -101,6 +101,7 @@ public class APIClient {
&lt;/span&gt;&lt;span style=&#34;font-weight:bold&#34;&gt;&lt;/span&gt;        private HttpResponse postInternal(String url, List&amp;lt;NameValuePair&amp;gt; data, boolean forRegistration) throws ClientProtocolException, IOException {
                HttpPost request = new HttpPost(Util.getServerAddress(mContext)+&amp;#34;/api/&amp;#34;+url);
                request.setHeader(&amp;#34;X-API-VERSION&amp;#34;, apiVersion);
+               request.setHeader(&amp;#34;Host&amp;#34;, &amp;#34;backend.client.com&amp;#34;);

                if(data == null) {
                        data = new ArrayList&amp;lt;NameValuePair&amp;gt;();
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Eventually, we did decide to just abandon Cloudflare for the server. We probably weren&amp;rsquo;t going to be the target of a DDOS or anything. This also allowed us to do more secure things like pinning the server certificate in the application itself. Clearly, this is what we should have just done in the first place, but at the time we just wanted a stopgap solution.&lt;/p&gt;
&lt;p&gt;I just still find it funny we were making people&amp;rsquo;s phones go to 4chan.org everyday for more than a year.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>TiddlyWiki in the Sky (or TiddlyWeb for TW5)</title>
      <link>https://pleasantprogrammer.com/posts/tiddlywiki-in-the-sky-or-tiddlyweb-for-tw5.html</link>
      <pubDate>Thu, 24 Dec 2015 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/tiddlywiki-in-the-sky-or-tiddlyweb-for-tw5.html</guid>
      <description>I&amp;rsquo;ve always liked TiddlyWiki. Back when it first came out, it was really amazing. A wiki all in one file, that worked in the browser. It didn&amp;rsquo;t need a backend, it would just save itself as an all new HTML file with all your posts inside. I&amp;rsquo;ve used it a lot over the years, as a personal wiki/journal and a class notebook. I even had a blog with it at one point using one of the server-side forks.</description>
      <content:encoded>&lt;p&gt;I&amp;rsquo;ve always liked &lt;a href=&#34;http://tiddlywiki.com&#34;&gt;TiddlyWiki&lt;/a&gt;. Back when it first came out, it was really amazing. A wiki all in one file, that worked in the browser. It didn&amp;rsquo;t need a backend, it would just save itself as an all new HTML file with all your posts inside. I&amp;rsquo;ve used it a lot over the years, as a personal wiki/journal and a class notebook. I even had a blog with it at one point using one of the server-side forks.&lt;/p&gt;
&lt;p&gt;Now, there&amp;rsquo;s TiddlyWiki5 which is a rewrite of the original TiddlyWiki that looks a whole lot snazzier, and I assume has better architecture overall. It also has experimental support for all the server-side platforms (particularly TiddlyWeb) that have cropped up.&lt;/p&gt;
&lt;p&gt;If you&amp;rsquo;re just looking for a simple server setup for TiddlyWiki5, it has native support for that on its own. There&amp;rsquo;s plenty of documentation on the site. But if you&amp;rsquo;re looking for more advanced features (like storing your posts in git or a database), then you&amp;rsquo;ll need to use it with TiddlyWeb. The problem is that most of the documentation for TiddlyWeb still refers to the old TiddlyWiki.&lt;/p&gt;
&lt;p&gt;To support TiddlyWiki5, we&amp;rsquo;ll need a version of the wiki which has the TiddlyWeb plugin already installed and configured. After that, some tweaking is necessary to get TiddlyWeb to provide what the wiki requires.&lt;/p&gt;
&lt;h2 id=&#34;setting-up-tiddlywiki&#34;&gt;Setting Up TiddlyWiki&lt;/h2&gt;
&lt;p&gt;TiddlyWiki5 provides a command line tool via &lt;code&gt;npm&lt;/code&gt; that allows building custom versions of the wiki. In fact, it comes with templates, called &amp;ldquo;editions&amp;rdquo;, that we can use for our setup. Assuming you already have it installed, create the wiki using&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;tiddlywiki mywiki --init tw5tank          &lt;span style=&#34;color:#007f7f&#34;&gt;# create wiki from template&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This creates a wiki intended for use with &lt;a href=&#34;https://tank.peermore.com/&#34;&gt;Tank&lt;/a&gt;, which is built on top of TiddlyWeb. From here, you should look in &lt;code&gt;mywiki/tiddlers/system&lt;/code&gt; which contain the entries for &lt;code&gt;SiteTitle&lt;/code&gt;, &lt;code&gt;SiteSubtitle&lt;/code&gt;, &lt;code&gt;DefaultTiddlers&lt;/code&gt;, and &lt;code&gt;tiddlyweb-host&lt;/code&gt;. The first 3 should be configured however you want. These are necessary because they&amp;rsquo;re needed before the wiki can load them from the server. &lt;code&gt;tiddlyweb-host&lt;/code&gt; contains the location of the TiddlyWeb server, this should be &lt;code&gt;http://localhost:8080/&lt;/code&gt; if you&amp;rsquo;re just testing locally. With everything configured, you can build the new wiki by running&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;tiddlywiki mywiki --build
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This will output the wiki to &lt;code&gt;mywiki/output/tw5tank.html&lt;/code&gt;. You can now serve it using your favorite local webserver, like &lt;code&gt;python -m http.server&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&#34;setting-up-tiddlyweb&#34;&gt;Setting Up TiddlyWeb&lt;/h2&gt;
&lt;p&gt;The TiddlyWeb tutorial recommends using &lt;code&gt;tiddlywebwiki&lt;/code&gt; which has all the plugins setup for a nice wiki instance for the old TiddlyWiki. It has a lot of features that aren&amp;rsquo;t really needed, so we won&amp;rsquo;t go with that. So first, we&amp;rsquo;ll need to install TiddlyWeb and any plugins we might want to use.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;pip install tiddlyweb tiddlywebplugins.status tiddlywebplugins.cherrypy tiddlywebplugins.cors
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Next, we&amp;rsquo;ll need the tiddlyweb configuration in &lt;code&gt;tiddlywebconfig.py&lt;/code&gt;&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;color:#007f7f&#34;&gt;# A basic configuration.&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;# `pydoc tiddlyweb.config` for details on configuration items.&lt;/span&gt;

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;import&lt;/span&gt; tiddlywebplugins.status

config = {
    &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;system_plugins&amp;#39;&lt;/span&gt;: [&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;tiddlywebplugins.status&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;tiddlywebplugins.cors&amp;#39;&lt;/span&gt;],
    &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;secret&amp;#39;&lt;/span&gt;: &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;36c98d6d14618c79f0ed2d49cd1b9e272d8d4bd0&amp;#39;&lt;/span&gt;,
    &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;wsgi_server&amp;#39;&lt;/span&gt;: &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;tiddlywebplugins.cherrypy&amp;#39;&lt;/span&gt;,
    &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;cors.enable_non_simple&amp;#39;&lt;/span&gt;: &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;True&lt;/span&gt;
}

original_gather_data = tiddlywebplugins.status._gather_data

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;def&lt;/span&gt; _status_gather_data(environ):
    data = original_gather_data(environ)
    data[&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;space&amp;#39;&lt;/span&gt;] = {&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;recipe&amp;#39;&lt;/span&gt;: &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;default&amp;#39;&lt;/span&gt;}
    &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;return&lt;/span&gt; data

tiddlywebplugins.status._gather_data = _status_gather_data
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The tweaks involved are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;using the status plugin which the wiki requires&lt;/li&gt;
&lt;li&gt;monkeypatching the status plugin for the wiki to use the correct &amp;ldquo;recipe&amp;rdquo;&lt;/li&gt;
&lt;li&gt;using cherrypy server instead of the buggy default one&lt;/li&gt;
&lt;li&gt;using cors since we&amp;rsquo;re not hosting the wiki itself on the same server&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;With that, we just need to create the store that will hold our data&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;twanager recipe default &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;lt;&amp;lt;EOF
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;desc: standard TiddlyWebWiki environment
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;policy: {&amp;#34;read&amp;#34;: [], &amp;#34;create&amp;#34;: [], &amp;#34;manage&amp;#34;: [&amp;#34;R:ADMIN&amp;#34;], &amp;#34;accept&amp;#34;: [], &amp;#34;write&amp;#34;: [&amp;#34;R:ADMIN&amp;#34;], &amp;#34;owner&amp;#34;: &amp;#34;administrator&amp;#34;, &amp;#34;delete&amp;#34;: [&amp;#34;R:ADMIN&amp;#34;]}
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;/bags/default/tiddlers
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;EOF&lt;/span&gt;

twanager bag default &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;lt;&amp;lt;EOF
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;{&amp;#34;policy&amp;#34;: {&amp;#34;read&amp;#34;: [], &amp;#34;create&amp;#34;: [], &amp;#34;manage&amp;#34;: [&amp;#34;R:ADMIN&amp;#34;], &amp;#34;accept&amp;#34;: [], &amp;#34;write&amp;#34;: [], &amp;#34;owner&amp;#34;: &amp;#34;administrator&amp;#34;, &amp;#34;delete&amp;#34;: []}}
&lt;/span&gt;&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;EOF&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Finally, we can start the TiddlyWeb server&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;twanager server
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;putting-it-all-together&#34;&gt;Putting it all together&lt;/h2&gt;
&lt;p&gt;Once you have the TiddlyWeb server running, you can just go to wherever you&amp;rsquo;re hosting the wiki html and it should work. You can try creating some posts, and the check mark on the sidebar should be red for a while and then turn black. Once that&amp;rsquo;s done it&amp;rsquo;s saved. You can refresh your browser and your posts should still be there.&lt;/p&gt;
&lt;p&gt;At this point, you can start customizing your TiddlyWeb instance, by changing your store to something like a database, or adding authorization. You can also tweak the server setup so you won&amp;rsquo;t need CORS anymore.&lt;/p&gt;
&lt;p&gt;TiddlyWiki5 is still relatively new. I hope that eventually, support for server-side and the plugin ecosystem grows to be as great as the old TiddlyWiki.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Is My Terminal Window Active?</title>
      <link>https://pleasantprogrammer.com/posts/is-my-terminal-window-active.html</link>
      <pubDate>Sun, 07 Jun 2015 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/is-my-terminal-window-active.html</guid>
      <description>I&amp;rsquo;ve been working in OSX for almost 3 years now, but I recently switched back to Linux because of all the problems people encountered with Yosemite. There are some things I missed from OSX though. One of which is zsh-notify. It&amp;rsquo;s a zsh plugin that alerts you if your long-running task is complete, and whether it failed or not.
It&amp;rsquo;s pretty convenient when you&amp;rsquo;re compiling something and then go on to browse reddit while waiting.</description>
      <content:encoded>&lt;p&gt;I&amp;rsquo;ve been working in OSX for almost 3 years now, but I recently switched back to Linux because of all the problems people encountered with Yosemite. There are some things I missed from OSX though. One of which is &lt;a href=&#34;https://github.com/marzocchi/zsh-notify&#34;&gt;zsh-notify&lt;/a&gt;. It&amp;rsquo;s a zsh plugin that alerts you if your long-running task is complete, and whether it failed or not.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s pretty convenient when you&amp;rsquo;re compiling something and then go on to browse reddit while waiting. Usually, I spend too much time just reading and forget about the compilation entirely. With the plugin, I get the notification and maybe go back to work.&lt;/p&gt;
&lt;p&gt;One nice feature it has is that if you&amp;rsquo;re currently looking at the terminal window of the job that just finished, it won&amp;rsquo;t notify you. It only notifies on windows that aren&amp;rsquo;t currently in focus. To do this, it has to actually talk to Terminal.app or iTerm2 to see if the window and tab are active.&lt;/p&gt;
&lt;p&gt;This is alright in OSX since those 2 are the generally most used terminal emulators. On Linux though, everyone has their own favorite terminal. Given that, I figured I could probably rely on talking to X to see if the window is active instead of each single terminal emulator. X can&amp;rsquo;t tell if the tab is active though, but I don&amp;rsquo;t use tabs in my current setup so it should still be good.&lt;/p&gt;
&lt;h2 id=&#34;xdotool&#34;&gt;xdotool&lt;/h2&gt;
&lt;p&gt;&lt;a href=&#34;http://superuser.com/questions/382616/detecting-currently-active-window&#34;&gt;Preliminary research&lt;/a&gt; reveals that we can easily get what the active window is with xdotool. &lt;code&gt;xdotool getactivewindow&lt;/code&gt; gives us the X window id of the active one. Now all we need is a way to get the window id of the terminal we&amp;rsquo;re in.&lt;/p&gt;
&lt;h2 id=&#34;first-attempt-windowid&#34;&gt;First Attempt: $WINDOWID&lt;/h2&gt;
&lt;p&gt;Apparently, xterm and similar terminal emulators define an environment variable called &lt;code&gt;$WINDOWID&lt;/code&gt; with the window id of the terminal. Obviously, this is too good to be true. In xterm and konsole the &lt;code&gt;$WINDOWID&lt;/code&gt; was correct, but in VTE-based terminal emulators, &lt;code&gt;$WINDOWID&lt;/code&gt; had the wrong value. In terminology, it didn&amp;rsquo;t define &lt;code&gt;$WINDOWID&lt;/code&gt; altogether. So &lt;code&gt;$WINDOWID&lt;/code&gt; wasn&amp;rsquo;t going to work.&lt;/p&gt;
&lt;h2 id=&#34;second-attempt-xdotool-search-magic&#34;&gt;Second Attempt: xdotool search $MAGIC&lt;/h2&gt;
&lt;p&gt;My second idea was that you can use zsh to change the window title to a magic number and then just check if the active window is the same one as the window with the magic number. This sort of worked for most terminals, except konsole which does whatever it wants with the window title. There&amp;rsquo;s also the problem of some zsh configs automatically settings the window title to the current command.&lt;/p&gt;
&lt;p&gt;In hindsight, I could probably have just done &lt;code&gt;xdotool search --name xdotool&lt;/code&gt; since in most cases, when you run the search, zsh or konsole will set the window name to the current command. Maybe that&amp;rsquo;s another option I can explore some day.&lt;/p&gt;
&lt;h2 id=&#34;third-attempt-ppid&#34;&gt;Third Attempt: $PPID&lt;/h2&gt;
&lt;p&gt;My third idea was another environment variable called &lt;code&gt;$PPID&lt;/code&gt;, which is the process id of the parent of the shell. As it happens, the parent is the window containing the zsh instance. This is actually pretty consistent across most terminals. The only problem was if you launched zsh from another shell since your new zsh&amp;rsquo;s parent will now be another zsh instance instead of an X window.&lt;/p&gt;
&lt;p&gt;At first glance, launching zsh within zsh doesn&amp;rsquo;t seem like something most people would do, but this is what happens when you run screen or tmux. To work around this, we can actually just save the original &lt;code&gt;$PPID&lt;/code&gt; in a different variable and use that instead.&lt;/p&gt;
&lt;p&gt;Now that we have the PID of the window from zsh, we can once again use xdotool to get the PID of the current active window with &lt;code&gt;xdotool getactivewindow getwindowpid&lt;/code&gt;. We just simply compare that with our &lt;code&gt;$PPID&lt;/code&gt; and we can tell if we&amp;rsquo;re in an active window or not. Overall, this approach worked surprisingly well so that&amp;rsquo;s the final solution I went with.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Removing PLDTMyDSLBiz from the ZyXEL P-2612HNU</title>
      <link>https://pleasantprogrammer.com/posts/removing-pldtmydslbiz-from-the-zyxel-p-2612hnu.html</link>
      <pubDate>Wed, 27 Nov 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/removing-pldtmydslbiz-from-the-zyxel-p-2612hnu.html</guid>
      <description>I&amp;rsquo;ve always thought that people were just too lazy to change their SSIDs when I see &amp;ldquo;PLDTMyDSLBizCafeJapan&amp;rdquo;. It became apparent when we got our own PLDT line that it was because the bundled router/modem does not allow you to remove the prefix.
This is not the kind of thing you expect as a business customer. Even for home customers, I feel it&amp;rsquo;s still a bit dishonest. I&amp;rsquo;d be fine if it was just the default SSID, but forcing people to have it as part of their SSID is like advertising that your company (I mean PLDT) is a douche.</description>
      <content:encoded>&lt;p&gt;I&amp;rsquo;ve always thought that people were just too lazy to change their SSIDs when I see &amp;ldquo;PLDTMyDSLBizCafeJapan&amp;rdquo;. It became apparent when we got our own PLDT line that it was because the bundled router/modem &lt;em&gt;does not&lt;/em&gt; allow you to remove the prefix.&lt;/p&gt;
&lt;p&gt;This is not the kind of thing you expect as a business customer. Even for home customers, I feel it&amp;rsquo;s still a bit dishonest. I&amp;rsquo;d be fine if it was just the default SSID, but forcing people to have it as part of their SSID is like advertising that your company (I mean PLDT) is a douche.&lt;/p&gt;
&lt;p&gt;Of course, we couldn&amp;rsquo;t just leave the SSID prefix there, so we tried a number of things to get rid of it. There are articles for removing it from the &lt;a href=&#34;http://www.phandroidinternet.com/2013/06/how-to-remove-on-wifi-name-or-ssid-on.html&#34;&gt;Prolink H5004N&lt;/a&gt; or the &lt;a href=&#34;http://www.symbianize.com/showthread.php?t=730091&#34;&gt;ZyXEL P-660HN-T1A&lt;/a&gt; but not for the one we got which was the ZyXEL P-2612HNU-F1F.&lt;/p&gt;
&lt;p&gt;We did still try the firebug/inspector tricks, but it seems that there is a server-side check that adds in the &amp;ldquo;PLDTMyDSLBiz&amp;rdquo;. We tried a number of things, but the one that ultimately worked (and we had a good laugh about) was to backup the configuration, edit the dumped file and restore it.&lt;/p&gt;
&lt;p&gt;The backup is actually just an XML file. You can search for SSID and change the parameter there. It&amp;rsquo;s a bit annoying because the router has to restart after restoring the configuration, but it works!&lt;/p&gt;
&lt;p&gt;A minor note, the router doesn&amp;rsquo;t seem to support SSIDs with a comma (,) well. It just gets everything before the comma as the SSID for some reason.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Console Keymap Switching</title>
      <link>https://pleasantprogrammer.com/posts/console-keymap-switching.html</link>
      <pubDate>Tue, 29 Oct 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/console-keymap-switching.html</guid>
      <description>At the office, we have some people who use DVORAK. Normally, this isn&amp;rsquo;t a problem. To each his own after all. It does become a bit problematic though, when we&amp;rsquo;re dealing with the servers around the office.
We normally leave the servers on QWERTY. After all, most people start off as QWERTY typists and migrate to something else. That said, it&amp;rsquo;s apparently difficult to stay fluent in both. People tend to forget how to type in QWERTY once they learn DVORAK or something else.</description>
      <content:encoded>&lt;p&gt;At the office, we have some people who use DVORAK. Normally, this isn&amp;rsquo;t a problem. To each his own after all. It does become a bit problematic though, when we&amp;rsquo;re dealing with the servers around the office.&lt;/p&gt;
&lt;p&gt;We normally leave the servers on QWERTY. After all, most people start off as QWERTY typists and migrate to something else. That said, it&amp;rsquo;s apparently difficult to stay fluent in both. People tend to forget how to type in QWERTY once they learn DVORAK or something else. While it is true that they can just look a the keyboard while typing, my coworkers would prefer it to just be in DVORAK.&lt;/p&gt;
&lt;p&gt;For the console, they&amp;rsquo;d typically do &lt;code&gt;sudo loadkeys dvorak&lt;/code&gt; after logging in. The problem with this is, after they logout, the keymapping is still on DVORAK. This has been quite annoying for a few times since I can&amp;rsquo;t even login to change the keymap. What I wanted was something like you get in the graphical login screens where you can pick your keymap before logging in. Apparently, there isn&amp;rsquo;t a readily available thing for the console.&lt;/p&gt;
&lt;p&gt;I googled around for solutions and came across &lt;a href=&#34;http://superuser.com/questions/548234/how-can-i-easily-toggle-between-dvorak-and-qwerty-keyboard-layouts-from-a-linux&#34;&gt;a nice idea&lt;/a&gt;. You could alias &lt;code&gt;asdf&lt;/code&gt; to load the DVORAK mapping and &lt;code&gt;aoeu&lt;/code&gt; (the equivalent to asdf in DVORAK) to load the QWERTY mapping. This actually makes sense since you don&amp;rsquo;t really have to know where the letters are. The only problem is, you once again have to be logged in to change the key mappings.&lt;/p&gt;
&lt;p&gt;After some further searching, I found &lt;a href=&#34;http://unix.stackexchange.com/questions/2884/toggle-between-dvorak-and-qwerty&#34;&gt;something close to what I wanted&lt;/a&gt;. Apparently, Alt+Up sends a KeyboardSignal keycode to the init process, which can act on that. It also works anywhere, even before being logged in. For SysVinit systems, you can just add a line to your inittab for a command to be run when Alt+Up is pressed.&lt;/p&gt;
&lt;p&gt;In the office, however, we generally use Arch Linux which uses SystemD. But apparently, it also has a mechanism of accepting the Alt+Up press. It runs the kbrequest target whenever it gets the keypress. &lt;code&gt;kbrequest.target&lt;/code&gt; is normally aliased to run the rescue service though, so you have to manually create the file in &lt;code&gt;/etc/systemd/system/kbrequest.target&lt;/code&gt; and fill it with a description:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-ini&#34; data-lang=&#34;ini&#34;&gt;&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;[Unit]&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;Description&lt;/span&gt;=&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;kbrequest target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;We can then add a service to be run whenever the target is called. Something like &lt;code&gt;/etc/systemd/system/keymap-switch.service&lt;/code&gt;:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-ini&#34; data-lang=&#34;ini&#34;&gt;&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;[Unit]&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;Description&lt;/span&gt;=&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;Keymap Switch Service&lt;/span&gt;

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;[Service]&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;Type&lt;/span&gt;=&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;oneshot&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;ExecStart&lt;/span&gt;=&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;/usr/local/bin/keymap-switch&lt;/span&gt;

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;[Install]&lt;/span&gt;
&lt;span style=&#34;color:#007f7f&#34;&gt;WantedBy&lt;/span&gt;=&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;kbrequest.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;After enabling said service, we only need the actual keymap switcher, &lt;code&gt;/usr/local/bin/keymap-switch&lt;/code&gt;. The StackOverflow answer provides different ways of detecting the current keymap so we know which one to switch to. Since we&amp;rsquo;re using SystemD, we can use that instead for managing which keymap we&amp;rsquo;re actually using. It stores the current settings inside &lt;code&gt;/etc/vconsole.conf&lt;/code&gt;. We can also then switch keymaps by using &lt;code&gt;localectl set-keymap&lt;/code&gt;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;color:#0f0;font-weight:bold&#34;&gt;#!/bin/sh
&lt;/span&gt;&lt;span style=&#34;color:#0f0;font-weight:bold&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;source&lt;/span&gt; /etc/vconsole.conf

&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt; [ &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$TERM&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt; = &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;dumb&amp;#34;&lt;/span&gt; ]; &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;then&lt;/span&gt;
  &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt; [ &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt;$KEYMAP&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;&lt;/span&gt; = &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;dvorak&amp;#34;&lt;/span&gt; ]; &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;then&lt;/span&gt;
    localectl set-keymap us
  &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;else&lt;/span&gt;
    localectl set-keymap dvorak
  &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;fi&lt;/span&gt;
&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;After putting it all together, it works! We can switch keymaps on the fly by simply pressing Alt+Up.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Geocoding Services</title>
      <link>https://pleasantprogrammer.com/posts/geocoding-services.html</link>
      <pubDate>Wed, 25 Sep 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/geocoding-services.html</guid>
      <description>A key component for any routing service is being able to do geocoding. Most people who are looking for routes most probably don&amp;rsquo;t know exactly where their start and end points are on the map. Even then, manually looking for a location on a map is a time-consuming task.
The gold standard for doing geocoding right now is Google Maps. It&amp;rsquo;s hard to find a better location search experience. If they actually provided routing for jeeps here in the Philippines, I imagine there wouldn&amp;rsquo;t be that much you could do for the competition.</description>
      <content:encoded>&lt;p&gt;A key component for any routing service is being able to do geocoding. Most people who are looking for routes most probably don&amp;rsquo;t know exactly where their start and end points are on the map. Even then, manually looking for a location on a map is a time-consuming task.&lt;/p&gt;
&lt;p&gt;The gold standard for doing geocoding right now is Google Maps. It&amp;rsquo;s hard to find a better location search experience. If they actually provided routing for jeeps here in the Philippines, I imagine there wouldn&amp;rsquo;t be &lt;em&gt;that&lt;/em&gt; much you could do for the competition.&lt;/p&gt;
&lt;p&gt;When the competition started though, I took it as a challenge to avoid Google Maps as much as possible. I wanted to see how much is currently possible with other options such as OpenStreetMap. In fact, OSM does have a geocoding service called &lt;a href=&#34;http://nominatim.openstreetmap.org&#34;&gt;Nominatim&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Sadly, for a mapping app, what you want to do is not simply just geocoding. With geocoding, you take an address and turn it into coordinates. When you want to search for a place in a mapping app, you take part of an address, infer the rest of it, and give the user options to choose from.&lt;/p&gt;
&lt;p&gt;Given a typical mapping app, you might type in &amp;ldquo;ateneo&amp;rdquo; and expect it to give you Ateneo de Manila University. With typical geocoding services like Nominatim or even Google&amp;rsquo;s &lt;a href=&#34;https://developers.google.com/maps/documentation/javascript/geocoding&#34;&gt;geocoding API&lt;/a&gt;, you probably won&amp;rsquo;t get any result for this. What you want to use is the &lt;a href=&#34;https://developers.google.com/maps/documentation/javascript/places&#34;&gt;Places API&lt;/a&gt; which provides an autocomplete search box. Using it, when you type in &amp;ldquo;ateneo&amp;rdquo;, it automatically suggests in the dropdown, &amp;ldquo;Ateneo de Manila University&amp;rdquo;.&lt;/p&gt;
&lt;p&gt;A downside to using the Places API is that it&amp;rsquo;s against the terms of service to use it with something that isn&amp;rsquo;t Google Maps, which means no OpenStreetMap. If there were more time, writing your own autocompletion engine using OpenStreetMap&amp;rsquo;s data will probably be a better long term solution.&lt;/p&gt;
&lt;p&gt;For now, since the competition&amp;rsquo;s deadline is just a few days away, I&amp;rsquo;ll be using Google Maps.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Jeep and Bus Schedules</title>
      <link>https://pleasantprogrammer.com/posts/jeep-and-bus-schedules.html</link>
      <pubDate>Sun, 28 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/jeep-and-bus-schedules.html</guid>
      <description>Wouldn&amp;rsquo;t it be wonderful if there were no buses or jeepneys in the Philippines over the weekends? It would truly be a cyclist&amp;rsquo;s paradise. Imagine biking along EDSA, normally that would be a death sentence, but according to the GTFS data, you shouldn&amp;rsquo;t worry. I can assure you, it&amp;rsquo;s still a death sentence.
The GTFS spec defines 2 ways of statically specifying trip schedules. You can define the exact times that a service will arrive at a stop.</description>
      <content:encoded>&lt;p&gt;Wouldn&amp;rsquo;t it be wonderful if there were no buses or jeepneys in the Philippines over the weekends? It would truly be a cyclist&amp;rsquo;s paradise. Imagine biking along EDSA, normally that would be a death sentence, but according to the GTFS data, you shouldn&amp;rsquo;t worry. I can assure you, it&amp;rsquo;s still a death sentence.&lt;/p&gt;
&lt;p&gt;The GTFS spec defines 2 ways of statically specifying trip schedules. You can define the exact times that a service will arrive at a stop. You can also specify between what times the service is active and how often a new bus or jeep leaves the first stop. You also define which days those rules apply. You could say every MWF, the bus operates from 9:00AM to 9:00PM and every TTH, the bus services from 3:00AM to 11:00PM.&lt;/p&gt;
&lt;p&gt;This should be sufficient in theory, but real world conditions like traffic or the weather could throw the schedules off. To solve this, there&amp;rsquo;s another spec, GTFS-realtime. This allows transit agencies to push temporary schedule updates and service announcements.&lt;/p&gt;
&lt;p&gt;Like much everything else about the Philippine transit system, there aren&amp;rsquo;t really any &amp;ldquo;schedules&amp;rdquo; to speak of. It&amp;rsquo;s generally whenever the buses or jeeps feel like it. So we have no static schedules. We don&amp;rsquo;t have a central agency or the tracking technology to make it feasible to push updates via GTFS-RT.&lt;/p&gt;
&lt;p&gt;Ideally, we shouldn&amp;rsquo;t bother inputting the schedule information into GTFS. Only the route data is really important for jeeps and buses. However, the schedule information is required in the GTFS, and routing apps wouldn&amp;rsquo;t work without it. So we have to add a reasonable trip schedule for jeeps and buses.&lt;/p&gt;
&lt;p&gt;The current GTFS data does define these trip schedules. We assume that jeeps and buses operate between 6:00AM and 11:00PM and a new jeep passes by every 10 minutes. Also, jeeps and buses are defined to only operate on weekdays.&lt;/p&gt;
&lt;p&gt;While there might be jeeps who change routes or don&amp;rsquo;t operate on weekends, I&amp;rsquo;m pretty sure that jeeps and buses run on weekends. We&amp;rsquo;ll have to fix it ourselves temporarily since there&amp;rsquo;s no central GTFS feed yet.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;color:#007f7f&#34;&gt;# 724594 seems to be the service id used by jeeps and buses&lt;/span&gt;
sed -i .bak &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;/^724594/ s/0,0/1,1/&amp;#39;&lt;/span&gt; calendar.txt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Another thing we could do is to adjust the time between buses, although the improvement is arguable. With the current 10 minutes between jeeps, it might provide some routes a significant advantage just because the timing is right. So you might get differing route suggestions depending on what time you planned the route. This makes sense when you&amp;rsquo;re sure what the times are, so you can minimize the wait, but with jeeps, you never really know how long the wait will actually be.&lt;/p&gt;
&lt;p&gt;If we set the frequency to one minute, it &lt;em&gt;might&lt;/em&gt; give better routes by eliminating the timing issue. Or not, it&amp;rsquo;s kind of hard to tell.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;color:#007f7f&#34;&gt;# jeep and bus route ids tend to start with 72&lt;/span&gt;
sed -i .bak &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#39;/^72/ s/,600/,60/&amp;#39;&lt;/span&gt; frequencies.txt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Overall, the problems we&amp;rsquo;re having is a symptom of the mismatch between our transit system and the GTFS. It would be great if our transit system gets better and we don&amp;rsquo;t need to do hackish things for it to fit the GTFS, but that&amp;rsquo;s still a dream. For now, all we can really do is fit a triangle into a square hole.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Highways in OTP</title>
      <link>https://pleasantprogrammer.com/posts/highways-in-otp.html</link>
      <pubDate>Wed, 24 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/highways-in-otp.html</guid>
      <description>One of the weird things that happens with OTP is sometimes it gives absurdly roundabout routes. Here is OTP&amp;rsquo;s suggested route for walking from UP to Ateneo:
This is just so hilariously wrong. It&amp;rsquo;s much simpler to just walk along Katipunan Avenue.
OTP couldn&amp;rsquo;t possibly be that dumb though, so there must be something we&amp;rsquo;re doing wrong. If you notice, Katipunan Avenue is colored red compared to the other streets. OTP seems to be avoiding any path that goes along Katipunan Avenue.</description>
      <content:encoded>&lt;p&gt;One of the weird things that happens with OTP is sometimes it gives absurdly roundabout routes. Here is OTP&amp;rsquo;s suggested route for walking from UP to Ateneo:&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;../galleries/transit/otproundabout.png&#34; alt=&#34;Roundabout route from UP to Ateneo&#34;&gt;&lt;/p&gt;
&lt;p&gt;This is just so hilariously wrong. It&amp;rsquo;s much simpler to just walk along Katipunan Avenue.&lt;/p&gt;
&lt;p&gt;OTP couldn&amp;rsquo;t possibly be that dumb though, so there must be something we&amp;rsquo;re doing wrong. If you notice, Katipunan Avenue is colored red compared to the other streets. OTP seems to be avoiding any path that goes along Katipunan Avenue. The problem might have something to do with the &amp;ldquo;road type&amp;rdquo; designated to Katipunan.&lt;/p&gt;
&lt;p&gt;Apparently, by default OTP will consider roads of type &lt;code&gt;trunk&lt;/code&gt; to be non-walkable and non-bikable. This is documented in the &lt;a href=&#34;http://wiki.openstreetmap.org/wiki/OpenTripPlanner&#34;&gt;OpenStreetMap wiki&lt;/a&gt; and the &lt;a href=&#34;https://github.com/openplans/OpenTripPlanner/wiki/GraphBuilder#permissions-and-bicycle-safety&#34;&gt;OTP wiki&lt;/a&gt; as well. There are actually multiple ways to go about this then. The first solution that came to mind was to just edit the original OSM XML file.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;sed -i .bak s/trunk/primary/g manila.osm
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;And rebuild the graph. It doesn&amp;rsquo;t really matter much because the OSM data isn&amp;rsquo;t used to render the maps. It&amp;rsquo;s just used to build the routing data. This is actually what I did for &lt;a href=&#34;http://maps.pleasantprogrammer.com&#34;&gt;maps.pleasantprogrammer.com&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s also possible to set the default way properties in OTP. Instead of disallowing walking and biking on &lt;code&gt;highway=trunk&lt;/code&gt; we could allow that. This is not much better than the &lt;code&gt;sed&lt;/code&gt; solution though. It&amp;rsquo;s better since you keep the weighting done by OTP, but you&amp;rsquo;re still saying that all trunks are walkable which might not be the case.&lt;/p&gt;
&lt;p&gt;The most correct way to actually fix this is to go through each of the trunks and specifying &lt;code&gt;foot=yes&lt;/code&gt; and &lt;code&gt;bicycle=yes&lt;/code&gt; for those trunks that are actually walkable. You could either do this locally with the dumped data, or contribute it directly to OSM. I&amp;rsquo;m not sure on the particulars with updating OSM though.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Elevation Data in OTP</title>
      <link>https://pleasantprogrammer.com/posts/elevation-data-in-otp.html</link>
      <pubDate>Tue, 23 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/elevation-data-in-otp.html</guid>
      <description>One thing I hadn&amp;rsquo;t tested out last time was OTP&amp;rsquo;s support for elevation data. It makes use of this by showing the elevation you have to traverse while walking along the suggested route. It can also take it into account when suggesting bike routes.
The 5 minute tutorial actually discusses the elevation data briefly, but a more in-depth thing you can look at is the GraphBuilder documentation. It suggests using the ASTER dataset which is free but requires registration.</description>
      <content:encoded>&lt;p&gt;&lt;img src=&#34;../galleries/transit/otpelevation.png&#34; alt=&#34;OpenTripPlanner showing elevation data&#34;&gt;&lt;/p&gt;
&lt;p&gt;One thing I hadn&amp;rsquo;t tested out last time was OTP&amp;rsquo;s support for elevation data. It makes use of this by showing the elevation you have to traverse while walking along the suggested route. It can also take it into account when suggesting bike routes.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&#34;https://github.com/openplans/OpenTripPlanner/wiki/FiveMinutes&#34;&gt;5 minute tutorial&lt;/a&gt; actually discusses the elevation data briefly, but a more in-depth thing you can look at is the &lt;a href=&#34;https://github.com/openplans/OpenTripPlanner/wiki/GraphBuilder#elevation-data&#34;&gt;GraphBuilder documentation&lt;/a&gt;. It suggests using the ASTER dataset which is free but requires registration. I just opted to use the SRTM data available from the &lt;a href=&#34;http://www.philgis.org/freegisdata.htm&#34;&gt;PhilGIS website&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;I don&amp;rsquo;t know about the ASTER dataset, but the PhilGIS data was in the ERDAS img format. OTP only supports GeoTIFF so there was a need to convert it beforehand. You can use &lt;a href=&#34;http://www.gdal.org/&#34;&gt;GDAL&lt;/a&gt; for this. You&amp;rsquo;d just then run,&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;gdal_translate srtm41_90m_phl.img phil.tiff
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Afterwards, it&amp;rsquo;s just a matter of following the OTP instructions on using a local elevation dataset. The process actually doubled the size of the generated Graph.obj so it might not be ideal if you&amp;rsquo;re running on limited RAM.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;ve actually hosted a &lt;a href=&#34;http://maps.pleasantprogrammer.com&#34;&gt;working example&lt;/a&gt;. It&amp;rsquo;s pretty much at the limits of the RAM so it might be slow and unreliable, but you can test it out just for fun. Please don&amp;rsquo;t abuse it though.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>GraphServer</title>
      <link>https://pleasantprogrammer.com/posts/graphserver.html</link>
      <pubDate>Tue, 23 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/graphserver.html</guid>
      <description>Link: http://graphserver.github.io/graphserver/
One other routing webapp I saw was GraphServer. It&amp;rsquo;s actually more of a general purpose Graph library which supports GTFS and OSM data than an actual dedicated routing software like OpenTripPlanner. It&amp;rsquo;s also based off python and C instead of Java, so it feels a lot less heavy.
The instructions on the website are already pretty good. There are just some minor errors with it. Where it says gs_gtfsdb_build, you should actually use gs_gtfsdb_compile.</description>
      <content:encoded>&lt;p&gt;Link: &lt;a href=&#34;http://graphserver.github.io/graphserver/&#34;&gt;http://graphserver.github.io/graphserver/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;One other routing webapp I saw was GraphServer. It&amp;rsquo;s actually more of a general purpose Graph library which supports GTFS and OSM data than an actual dedicated routing software like OpenTripPlanner. It&amp;rsquo;s also based off python and C instead of Java, so it feels a lot less heavy.&lt;/p&gt;
&lt;p&gt;The instructions on the website are already pretty good. There are just some minor errors with it. Where it says &lt;code&gt;gs_gtfsdb_build&lt;/code&gt;, you should actually use &lt;code&gt;gs_gtfsdb_compile&lt;/code&gt;. Also, when running &lt;code&gt;gs_osmdb_compile&lt;/code&gt; you might need to use &lt;code&gt;-t&lt;/code&gt; for tolerant in case you follow the instructions on chopping up the original OSM data.&lt;/p&gt;
&lt;p&gt;A nice suggestion from the GraphServer instructions was to crop the OSM data to minimize the graph size. This is actually quite helpful if you downloaded the entire Philippine OSM dump. It reduced the original 900MB file to 135MB which was a lot more workable. I did hit a problem with their instructions though. The linked version of osmosis is an old one, which doesn&amp;rsquo;t support 64-bit ids. The &lt;a href=&#34;http://wiki.openstreetmap.org/wiki/Osmosis&#34;&gt;latest version of Osmosis&lt;/a&gt; easily did the job though.&lt;/p&gt;
&lt;p&gt;The actual routing though, was not exactly good. I only tried one route which should normally take 1-2 transfers, it suggested a route which involved 4+ transfers. It also didn&amp;rsquo;t provide any alternate routes aside from that one. I&amp;rsquo;m not sure if it&amp;rsquo;s a limitation of the provided routeserver, but I didn&amp;rsquo;t bother checking if it supported parameters which might provide better routes.&lt;/p&gt;
&lt;p&gt;I think graphserver could be useful, but it seems more involved than say OpenTripPlanner. There do seem to be people who use graphserver for their routing apps, but for the bounds of the contest, or just as a side project, it might require too much effort.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Transit Wand</title>
      <link>https://pleasantprogrammer.com/posts/transit-wand.html</link>
      <pubDate>Mon, 15 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/transit-wand.html</guid>
      <description>Link: http://transitwand.com
Overall, this was the simplest of the open-source transit tools to actually get up and running. There&amp;rsquo;s already a deployed instance of the server, and you can easily download the phone app via the Play Store. Even running the server by yourself didn&amp;rsquo;t have any of the hiccups I had with GTFS Editor.
The phone app is actually quite simple. It allows you to capture a trip, which will record your GPS coordinates as you ride public transit.</description>
      <content:encoded>&lt;p&gt;Link: &lt;a href=&#34;https://play.google.com/store/apps/details?id=com.conveyal.transitwand&#34;&gt;http://transitwand.com&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Overall, this was the simplest of the &lt;a href=&#34;http://philippine-transit.hackathome.com/use-this-code/&#34;&gt;open-source transit tools&lt;/a&gt; to actually get up and running. There&amp;rsquo;s already a deployed instance of the server, and you can easily download the phone app via the &lt;a href=&#34;https://play.google.com/store/apps/details?id=com.conveyal.transitwand&#34;&gt;Play Store&lt;/a&gt;. Even running the server by yourself didn&amp;rsquo;t have any of the hiccups I had with GTFS Editor.&lt;/p&gt;
&lt;p&gt;The phone app is actually quite simple. It allows you to capture a trip, which will record your GPS coordinates as you ride public transit. It also allows you to mark points of the trip where you stop and also how long the stop took. Lastly, it allows you to record embarking and disembarking passengers which is potentially useful for ridership data.&lt;/p&gt;
&lt;p&gt;After doing a capture session, you can review the data on the phone. It will plot out the route on a map, with markers for the stops. You then either delete the data if it looks wrong, or you can upload it to the Transit Wand server. Uploading involves registering an account, but it&amp;rsquo;s free and you don&amp;rsquo;t even actually need to put in a username or anything. It simply registers the phone&amp;rsquo;s IMEI on the server and gives you a 6-digit identifier.&lt;/p&gt;
&lt;p&gt;You can then use the 6-digit identifier to view the data on Transit Wand&amp;rsquo;s server, which is good since uploading any data automatically deletes it from the phone. There really isn&amp;rsquo;t much else you can do with it though. It just allows you to view the data, and export it as a &lt;a href=&#34;https://en.wikipedia.org/wiki/Shapefile&#34;&gt;Shapefile&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;As is, this is purely a data collection client-server app. Barring looking at the database, there is no way to get a list of phones which have collected data. Only the person who initiated the data collection knows the 6-digit code to view their data. There&amp;rsquo;s also no way to extract the ridership information from the server yet. This isn&amp;rsquo;t to say that the data won&amp;rsquo;t eventually go public though.&lt;/p&gt;
&lt;p&gt;An interesting thing you &lt;em&gt;can&lt;/em&gt; do with the Transit Wand data is import it into GTFS Editor to make a new route. You don&amp;rsquo;t even have to manually download and upload the data. Just type in your 6-digit identifier and it will give you a list of routes you&amp;rsquo;ve captured via Transit Wand. This is wonderful as you get all the stop data, as well as the shape of the route.&lt;/p&gt;
&lt;p&gt;I imagine these two tools were how the DOTC came up with all the GTFS data we have now. What I don&amp;rsquo;t understand is why the shape data isn&amp;rsquo;t present. Importing from Transit Wand already gets you shape data. There are even facilities to edit the shape within the editor if clean up is necessary. The only problem I saw was the fact that you can&amp;rsquo;t easily move stops, you have to input coordinates to change the position.&lt;/p&gt;
&lt;p&gt;It &lt;em&gt;might&lt;/em&gt; also be possible that when the DOTC was still collecting the data, the route collection or editing features weren&amp;rsquo;t present yet. That would just be lame and depressing though.&lt;/p&gt;
&lt;p&gt;Overall, Transit Wand does what it&amp;rsquo;s supposed to do. You collect data, and then upload it to a server. There is a lot of room for improvement though. It would be nice to have a better API that allows access to more of the data. Building in analysis tools for the ridership data might also be a welcome thing. I imagine it would also be great if you could encourage people to use the app and upload their own trips.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Fare Data</title>
      <link>https://pleasantprogrammer.com/posts/fare-data.html</link>
      <pubDate>Sat, 13 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/fare-data.html</guid>
      <description>As part of the data released by the DOTC, we also have the fare matrix for aircon buses, ordinary buses and jeeps. All as wonderful images. The data is also actually available from the LTFRB website. Generally, the fare scheme is represented as &amp;ldquo;pay X pesos for the first Y kilometers, pay Z for every succeeding kilometer.&amp;rdquo; Instead of a table, we can simply represent this as a formula instead,</description>
      <content:encoded>&lt;p&gt;As part of the data released by the DOTC, we also have the &lt;a href=&#34;http://philippine-transit.hackathome.com/dataset-philippines-transit-information-service-gtfs/&#34;&gt;fare matrix&lt;/a&gt; for aircon buses, ordinary buses and jeeps. All as wonderful images. The data is also actually available from the &lt;a href=&#34;http://ltfrb.gov.ph/main/farerates&#34;&gt;LTFRB website&lt;/a&gt;. Generally, the fare scheme is represented as &amp;ldquo;pay &lt;em&gt;X&lt;/em&gt; pesos for the first &lt;em&gt;Y&lt;/em&gt; kilometers, pay &lt;em&gt;Z&lt;/em&gt; for every succeeding kilometer.&amp;rdquo; Instead of a table, we can simply represent this as a formula instead,&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;base_fare + (distance - initial) * per_km
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The relevant values for the three services are:&lt;/p&gt;
&lt;!-- raw HTML omitted --&gt;
&lt;p&gt;It isn&amp;rsquo;t as simple as that though. Fares are also rounded to the nearest 25 centavos. So we&amp;rsquo;d need to round them off correctly. This can be achieved by doing,&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;round(calculated_fare * 4.0)/4.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;There&amp;rsquo;s also the discounted fare for students, senior citizens and persons with disability. They get 20% off the fare (prior to rounding) and the resulting fare is rounded off as well.&lt;/p&gt;
&lt;p&gt;Doing just this, we actually do get the same results as the fare matrices in the image for the most part. There are some discrepancies with the discounted jeep fares. I&amp;rsquo;ve tried to resolve it by tweaking around with the formulas, but it really doesn&amp;rsquo;t make sense in any way. I presume these were manually adjusted for one reason or another.&lt;/p&gt;
&lt;p&gt;Here&amp;rsquo;s a &lt;a href=&#34;../uploads/farematrix.rb&#34;&gt;script&lt;/a&gt; that generates CSVs of all the three fare matrices. If you&amp;rsquo;re too lazy to run it, here are links to the &lt;a href=&#34;../uploads/pub_aircon.csv&#34;&gt;aircon bus&lt;/a&gt;, &lt;a href=&#34;../uploads/pub_ordinary.csv&#34;&gt;ordinary bus&lt;/a&gt; and &lt;a href=&#34;../uploads/puj.csv&#34;&gt;jeep&lt;/a&gt; fare matrices.&lt;/p&gt;
&lt;h3 id=&#34;gtfs-compatibility&#34;&gt;GTFS compatibility&lt;/h3&gt;
&lt;p&gt;As is, the provided GTFS data does not have any fare data. I imagine this is because the existing spec doesn&amp;rsquo;t have good support for distance-based fares like we have in the Philippines. Judging from the &lt;a href=&#34;https://code.google.com/p/googletransitdatafeed/wiki/FareExamples&#34;&gt;fare examples&lt;/a&gt;, the only reasonable way we could implement distance-based fares is following example 6. This would involve setting a fare for each possible pair of stops based on the distance between them. This isn&amp;rsquo;t exactly ideal. In fact, the people originally working on the DOTC project have voiced &lt;a href=&#34;https://groups.google.com/forum/#!topic/gtfs-fare-wg/V63xRSnQJGw&#34;&gt;issues&lt;/a&gt; and made &lt;a href=&#34;https://groups.google.com/forum/#!msg/gtfs-changes/uybrAokZ9Cg/rqlzXdMypUgJ&#34;&gt;proposals&lt;/a&gt; for having distance-based fares included into GTFS.&lt;/p&gt;
&lt;p&gt;Apparently, public transit fares are a really complicated thing. You have fares based on distance, number of stops passed through, and transfers which may or may not cost extra. Not only that, you might have discounted fares, or first-class vs economy fares. The community will want to get it right before it&amp;rsquo;s formally included in the spec. You can see the current state of the consolidated &lt;a href=&#34;https://docs.google.com/document/d/1mK3--o5g4-3cCXaqmch92U63JTwChh0L2VCmcDViIlM/edit&#34;&gt;GTFS fare proposal here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Even in it&amp;rsquo;s proposal form though, we might have hope of being able to see these being used. There&amp;rsquo;s currently a &lt;a href=&#34;https://github.com/OneBusAway/onebusaway-gtfs-modules/pull/30&#34;&gt;pull request&lt;/a&gt; for supporting the distance-based fare scheme into the OneBusAway libraries. The libraries actually used by GTFS Editor and OpenTripPlanner for working with GTFS data.&lt;/p&gt;
&lt;h3 id=&#34;remaining-problems&#34;&gt;Remaining Problems&lt;/h3&gt;
&lt;p&gt;Given all that, it would probably still be a long way before this allows us to make a really good routing app. We still don&amp;rsquo;t have shape data, so the distance estimates would really be rough estimates at best. There&amp;rsquo;s no support for rounding to the nearest centavo. I realize that&amp;rsquo;s just nitpicking, but if we want something truly polished, even that has to be taken care of.&lt;/p&gt;
&lt;p&gt;We also don&amp;rsquo;t know if the jeeps or buses strictly follow the distance-based scheme. After all, if you can get on and off anywhere, you can&amp;rsquo;t really measure distance that exactly. I assume they generally work off the notion of &amp;ldquo;zones&amp;rdquo; than actual distance travelled. In that sense, they work more similarly to the LRT which has fares based on how many stops you pass. For jeeps and buses, your fare is probably based more on how many &amp;ldquo;zones&amp;rdquo; you pass through.&lt;/p&gt;
&lt;h3 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h3&gt;
&lt;p&gt;Philip, a co-worker of mine at By Implication, had suggested that we might want to use a different model than what the GTFS proposes. I have to agree with him. At this point, the GTFS doesn&amp;rsquo;t really fit with our system. But I do think that open data and standards are great. In fact, I applaud the developers who made proposals for the fare system, as those are great first steps towards making the GTFS a more universal standard.&lt;/p&gt;
&lt;p&gt;Side note: I&amp;rsquo;d also actually really like to hear about the DOTC developers&#39; experience with the project. It would be nice if they had a devblog.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>GTFS Editor</title>
      <link>https://pleasantprogrammer.com/posts/gtfs-editor.html</link>
      <pubDate>Wed, 10 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/gtfs-editor.html</guid>
      <description>Link: https://github.com/conveyal/gtfs-editor
TL;DR they really meant under development
When I first saw the source of GTFS Editor, I was ecstatic. They used Play framework!!! Not only that, they&amp;rsquo;re targeting PostgreSQL as the main database. Those are our favorite tools for building webapps at By Implication. I was a bit sad though, when I saw it was on the 1.x release of Play though. I did have some experience with that release, but not as much compared to 2.</description>
      <content:encoded>&lt;p&gt;Link: &lt;a href=&#34;https://github.com/conveyal/gtfs-editor&#34;&gt;https://github.com/conveyal/gtfs-editor&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; they really meant under development&lt;/p&gt;
&lt;p&gt;When I first saw the source of GTFS Editor, I was ecstatic. They used &lt;a href=&#34;http://playframework.com/&#34;&gt;Play framework&lt;/a&gt;!!! Not only that, they&amp;rsquo;re targeting PostgreSQL as the main database. Those are our favorite tools for building webapps at By Implication. I was a bit sad though, when I saw it was on the 1.x release of Play though. I did have some experience with that release, but not as much compared to 2.x.&lt;/p&gt;
&lt;p&gt;Getting it to actually run though, wasn&amp;rsquo;t very pleasant. The initial setup was easy enough. Get &lt;a href=&#34;http://www.playframework.com/download&#34;&gt;Play 1.2.5&lt;/a&gt;, install Postgres with PostGIS, clone the repo and create backing database in Postgres. Some minor additional steps you need are to create the PostGIS extension on the database. The schema is automatically generated and applied by Play so that should be all that&amp;rsquo;s necessary. Wonderful. Then, run play, open a browser, go to &lt;a href=&#34;http://localhost:9000&#34;&gt;http://localhost:9000&lt;/a&gt;, compilation error. Fantastic.&lt;/p&gt;
&lt;p&gt;If you don&amp;rsquo;t want to go through the technical details, you can just jump to the &lt;a href=&#34;#conclusion&#34;&gt;conclusion&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&#34;lets-debug&#34;&gt;Let&amp;rsquo;s Debug!&lt;/h2&gt;
&lt;p&gt;I&amp;rsquo;ll be splitting the next section up into 2 parts. In the first pass, I&amp;rsquo;ll talk about what I did to just get the app to run but I won&amp;rsquo;t try hard to fix any bugs. This generally is what I do when I try to get apps to run. I&amp;rsquo;ll also be dropping enough information so that you can actually figure out what the real problem is. In the second pass, I&amp;rsquo;ll explain what the problems were and how I fixed them.&lt;/p&gt;
&lt;h3 id=&#34;first-pass&#34;&gt;First Pass&lt;/h3&gt;
&lt;p&gt;A thing to note about Play (and one of the reasons it&amp;rsquo;s a lovely Java framework) is that you don&amp;rsquo;t need to do manual compilation. Just edit some source files, refresh your browser and it will automatically do the compilation for you. One less argument for using PHP. It even shows you (in the browser!) the source and which line of code caused the compilation error. So that&amp;rsquo;s what I saw, &lt;code&gt;Error: type Check already defined&lt;/code&gt;&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-java&#34; data-lang=&#34;java&#34;&gt;@Retention(RetentionPolicy.&lt;span style=&#34;color:#007f7f&#34;&gt;RUNTIME&lt;/span&gt;)
@Target({ElementType.&lt;span style=&#34;color:#007f7f&#34;&gt;METHOD&lt;/span&gt;, ElementType.&lt;span style=&#34;color:#007f7f&#34;&gt;TYPE&lt;/span&gt;})
&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;public&lt;/span&gt; @interface Check { &lt;span style=&#34;color:#007f7f&#34;&gt;// error here
&lt;/span&gt;&lt;span style=&#34;color:#007f7f&#34;&gt;&lt;/span&gt;
    String[] value();
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;You also know that typical behavior among programmers where your program doesn&amp;rsquo;t compile, but you keep trying to compile it anyway hoping that it will magically just work. That&amp;rsquo;s what I did, and it actually ran. I couldn&amp;rsquo;t really just let this pass, so I decided to try deleting &lt;code&gt;Check.java&lt;/code&gt;. I got another compilation error, &lt;code&gt;Error: type Secure already defined&lt;/code&gt;&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-java&#34; data-lang=&#34;java&#34;&gt;&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;public&lt;/span&gt; &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;class&lt;/span&gt; Secure &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;extends&lt;/span&gt; Controller { &lt;span style=&#34;color:#007f7f&#34;&gt;// error here
&lt;/span&gt;&lt;span style=&#34;color:#007f7f&#34;&gt;&lt;/span&gt;
    @Before(unless={&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;login&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;authenticate&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;logout&amp;#34;&lt;/span&gt;})

    &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;static&lt;/span&gt; &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;void&lt;/span&gt; checkAccess() &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;throws&lt;/span&gt; Throwable {
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;At that point, I just decided to just debug it later. It works by just forcing it anyway. So I put &lt;code&gt;Check.java&lt;/code&gt; back in and proceeded to just refresh until it compiled and ran.&lt;/p&gt;
&lt;p&gt;The next problem is a sort of common thing most webapp developers have to solve one way or another. How do you set up the initial admin account? Phrased a different way, how do I login to this thing? The first thing I tried was just add a user into the &lt;code&gt;account&lt;/code&gt; table directly. One problem though was how to set the password correctly. Plaintext obviously wouldn&amp;rsquo;t work.&lt;/p&gt;
&lt;p&gt;Another note regarding Play 1.x, it provides the &lt;a href=&#34;http://www.playframework.com/documentation/1.2.5/secure&#34;&gt;secure module&lt;/a&gt; which handles logins and keeping state, you simply need to implement the method &lt;code&gt;boolean authenticate(String username, String password)&lt;/code&gt;. It leaves the actual process of verifying the login to the programmer. This can be exploited by just making the method return &lt;code&gt;true&lt;/code&gt; and then any login would work. No need to actually set the password. Excellent.&lt;/p&gt;
&lt;p&gt;And we&amp;rsquo;re logged in, just in time to encounter a runtime exception. This also works much like compilation errors in Play. It shows a page with the error and the relevant source lines. Now we get, &lt;code&gt;IndexOutOfBoundsException occured : Index: 0, Size: 0&lt;/code&gt;&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-java&#34; data-lang=&#34;java&#34;&gt;&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt;(session.&lt;span style=&#34;color:#007f7f&#34;&gt;get&lt;/span&gt;(&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;agencyId&amp;#34;&lt;/span&gt;) == &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;null&lt;/span&gt;) {

    Agency agency = agencies.&lt;span style=&#34;color:#007f7f&#34;&gt;get&lt;/span&gt;(0); &lt;span style=&#34;color:#007f7f&#34;&gt;// error here
&lt;/span&gt;&lt;span style=&#34;color:#007f7f&#34;&gt;&lt;/span&gt;
    session.&lt;span style=&#34;color:#007f7f&#34;&gt;put&lt;/span&gt;(&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;agencyId&amp;#34;&lt;/span&gt;, agency.&lt;span style=&#34;color:#007f7f&#34;&gt;id&lt;/span&gt;);
    session.&lt;span style=&#34;color:#007f7f&#34;&gt;put&lt;/span&gt;(&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;agencyName&amp;#34;&lt;/span&gt;, agency.&lt;span style=&#34;color:#007f7f&#34;&gt;name&lt;/span&gt;);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Apparently, we need to have an agency. That&amp;rsquo;s generally simple enough. You just manually insert an agency into the &lt;code&gt;agency&lt;/code&gt; table. After that&amp;rsquo;s done, we finally have a view of the actual application. It&amp;rsquo;s very Bootstrap-y, but that&amp;rsquo;s just fine. The workflow though, is not perfectly intuitive, but I&amp;rsquo;ll talk about that some other day.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s not the end of it though, we still have to fix these bugs. The developer obviously didn&amp;rsquo;t have to put up with this when they were working, so what happened? Also, the log is showing some weird things,&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;~        _            _
~  _ __ | | __ _ _  _| |
~ | &amp;#39;_ \| |/ _&amp;#39; | || |_|
~ |  __/|_|\____|\__ (_)
~ |_|            |__/
~
~ play! 1.2.5, http://www.playframework.org
~
~ Ctrl+C to stop
~
CompilerOracle: exclude jregex/Pretokenizer.next
Listening for transport dt_socket at address: 8000
23:32:14,943 INFO  ~ Starting /Users/thomas/Workspace/maps/gtfs-editor
23:32:14,948 WARN  ~ Declaring modules in application.conf is deprecated. Use dependencies.yml instead (module.secure)
23:32:14,948 INFO  ~ Module secure is available (/Users/thomas/.root/opt/play-1.2.5/modules/secure)
23:32:15,830 WARN  ~ You&amp;#39;re running Play! in DEV mode
23:32:15,952 INFO  ~ Listening for HTTP on port 9000 (Waiting a first request to start) ...
23:32:28,792 ERROR ~

@6f02fa9dd
Internal Server Error (500) for request GET /

Compilation error (In /app/controllers/Check.java around line 10)
The file /app/controllers/Check.java could not be compiled. Error raised is : The type Check is already defined

play.exceptions.CompilationException: The type Check is already defined
	at play.classloading.ApplicationCompiler$2.acceptResult(ApplicationCompiler.java:246)
	at org.eclipse.jdt.internal.compiler.Compiler.handleInternalException(Compiler.java:672)
	at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:516)
	at play.classloading.ApplicationCompiler.compile(ApplicationCompiler.java:282)
	at play.classloading.ApplicationClassloader.getAllClasses(ApplicationClassloader.java:426)
	at play.Play.start(Play.java:516)
	at play.Play.detectChanges(Play.java:630)
	at play.Invoker$Invocation.init(Invoker.java:198)
	at Invocation.HTTP Request(Play!)
23:32:31,551 INFO  ~ Connected to jdbc:postgresql://127.0.0.1/gtfs_editor
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/thomas/Workspace/maps/gtfs-editor/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/thomas/.root/opt/play-1.2.5/framework/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
23:32:32,490 INFO  ~ Initializing HBSpatialExtension
23:32:32,492 INFO  ~ Attempting to load Hibernate Spatial Provider org.hibernatespatial.postgis.DialectProvider
23:32:32,494 INFO  ~ Checking for default configuration file.
23:32:32,496 INFO  ~ No configuration file hibernate-spatial.cfg.xml on the classpath.
23:32:34,077 INFO  ~ Application &amp;#39;gtfs-editor&amp;#39; is now started !
23:32:34,151 INFO  ~ Bootstrapping Database...
23:32:34,297 DEBUG ~ select count(*) as col_0_0_ from Agency agency0_ limit ?
play.exceptions.UnexpectedException: Unexpected Error
	at play.vfs.VirtualFile.contentAsString(VirtualFile.java:180)
	at play.templates.TemplateLoader.load(TemplateLoader.java:78)
	at play.test.Fixtures.loadModels(Fixtures.java:174)
	at jobs.BootstrapDatabase.doJob(BootstrapDatabase.java:57)
	at play.jobs.Job.doJobWithResult(Job.java:50)
	at play.jobs.Job.call(Job.java:146)
	at play.jobs.Job.run(Job.java:132)
	at play.jobs.JobsPlugin.afterApplicationStart(JobsPlugin.java:116)
	at play.plugins.PluginCollection.afterApplicationStart(PluginCollection.java:531)
	at play.Play.start(Play.java:547)
	at play.Play.detectChanges(Play.java:630)
	at play.Invoker$Invocation.init(Invoker.java:198)
	at play.server.PlayHandler$NettyInvocation.init(PlayHandler.java:189)
	at play.Invoker$Invocation.run(Invoker.java:276)
	at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:229)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:680)
Caused by: play.exceptions.UnexpectedException: Unexpected Error
	at play.vfs.VirtualFile.inputstream(VirtualFile.java:111)
	at play.vfs.VirtualFile.contentAsString(VirtualFile.java:178)
	... 22 more
Caused by: java.io.FileNotFoundException: /Users/thomas/.root/opt/play-1.2.5/modules/docviewer/app/initial-agencies-data.yml (No such file or directory)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.&amp;lt;init&amp;gt;(FileInputStream.java:120)
	at play.vfs.VirtualFile.inputstream(VirtualFile.java:109)
	... 23 more
23:32:34,316 ERROR ~ java.lang.RuntimeException: Cannot load fixture initial-agencies-data.yml: Unexpected Error
23:32:40,989 DEBUG ~ select account0_.id as id15_, account0_.active as active15_, account0_.admin as admin15_, account0_.agency_id as agency9_15_, account0_.email as email15_, account0_.lastLogin as lastLogin15_, account0_.password as password15_, account0_.passwordChangeToken as password7_15_, account0_.username as username15_ from Account account0_ where account0_.username=? limit ?
23:32:40,994 DEBUG ~ select count(*) as col_0_0_ from Account account0_ limit ?
23:32:40,999 DEBUG ~ select nextval (&amp;#39;hibernate_sequence&amp;#39;)
23:32:41,051 DEBUG ~ insert into Account (active, admin, agency_id, email, lastLogin, password, passwordChangeToken, username, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
23:32:41,061 DEBUG ~ select agency0_.id as id24_, agency0_.color as color24_, agency0_.defaultLat as defaultLat24_, agency0_.defaultLon as defaultLon24_, agency0_.defaultRouteType_id as default12_24_, agency0_.gtfsAgencyId as gtfsAgen5_24_, agency0_.lang as lang24_, agency0_.name as name24_, agency0_.phone as phone24_, agency0_.systemMap as systemMap24_, agency0_.timezone as timezone24_, agency0_.url as url24_ from Agency agency0_ order by agency0_.name
23:32:41,175 ERROR ~

@6f02fa9dg
Internal Server Error (500) for request GET /

Execution exception (In /app/controllers/Application.java around line 57)
IndexOutOfBoundsException occured : Index: 0, Size: 0

play.exceptions.JavaExecutionException: Index: 0, Size: 0
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:237)
	at Invocation.HTTP Request(Play!)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.RangeCheck(ArrayList.java:547)
	at java.util.ArrayList.get(ArrayList.java:322)
	at controllers.Application.initSession(Application.java:57)
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:510)
	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:484)
	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:479)
	at play.mvc.ActionInvoker.handleBefores(ActionInvoker.java:328)
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:142)
	... 1 more
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;After &lt;code&gt;23:32:34&lt;/code&gt; is when I get the login page. &lt;code&gt;23:32:40&lt;/code&gt; is after I&amp;rsquo;ve logged in.&lt;/p&gt;
&lt;h3 id=&#34;second-pass&#34;&gt;Second Pass&lt;/h3&gt;
&lt;p&gt;So how did you do? First, the error that &lt;code&gt;type Check already defined&lt;/code&gt; usually does mean that &lt;code&gt;Check&lt;/code&gt; was already defined elsewhere. Looking in the app folder though, there was nothing of the sort. It&amp;rsquo;s the only one there that was &lt;code&gt;Check.java&lt;/code&gt;. But remember the secure module? Modules work by providing source files and Play just compiles them all together. Bingo, &lt;code&gt;Check.java&lt;/code&gt;. Doing a diff shows nothing was changed. So the solution really was just simply delete &lt;code&gt;Check.java&lt;/code&gt; and also &lt;code&gt;Secure.java&lt;/code&gt;. No more compilation errors!&lt;/p&gt;
&lt;p&gt;The next question is, how do you get the initial user? There actually is some code that looks like it creates the default admin user,&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4&#34;&gt;&lt;code class=&#34;language-java&#34; data-lang=&#34;java&#34;&gt;&lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt;(Security.&lt;span style=&#34;color:#007f7f&#34;&gt;isConnected&lt;/span&gt;()) {
    ...
    Account account = Account.&lt;span style=&#34;color:#007f7f&#34;&gt;find&lt;/span&gt;(&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;username = ?&amp;#34;&lt;/span&gt;, Security.&lt;span style=&#34;color:#007f7f&#34;&gt;connected&lt;/span&gt;()).&lt;span style=&#34;color:#007f7f&#34;&gt;first&lt;/span&gt;();
    ...
    &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;if&lt;/span&gt;(account == &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;null&lt;/span&gt; &amp;amp;&amp;amp; Account.&lt;span style=&#34;color:#007f7f&#34;&gt;count&lt;/span&gt;() == 0) {
        account = &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;new&lt;/span&gt; Account(&lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;admin&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;admin&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#0ff;font-weight:bold&#34;&gt;&amp;#34;admin@test.com&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;true&lt;/span&gt;, &lt;span style=&#34;color:#fff;font-weight:bold&#34;&gt;null&lt;/span&gt;);
        account.&lt;span style=&#34;color:#007f7f&#34;&gt;save&lt;/span&gt;();
    }
    ...
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;You can actually see this in action at &lt;code&gt;23:32:41,051&lt;/code&gt; in the log. So what&amp;rsquo;s wrong with all of this? The account creation happened after I&amp;rsquo;ve already logged in. In fact, &lt;code&gt;Security.isConnected()&lt;/code&gt; checks whether the user is already logged in or not. How does this even make sense?&lt;/p&gt;
&lt;p&gt;Lastly, we have the problem of the agencies. Just by looking at the log, you can safely say we&amp;rsquo;re missing a file called &lt;code&gt;initial-agencies-data.yml&lt;/code&gt;. Ok, apparently it&amp;rsquo;s a &lt;a href=&#34;http://www.playframework.com/documentation/1.2.5/test#fixtures&#34;&gt;fixture&lt;/a&gt; like you would use for testing. It&amp;rsquo;s easy enough to infer what the file&amp;rsquo;s contents should be. We just copy it over from the GTFS data.&lt;/p&gt;
&lt;p&gt;But then where do you put the file? If you look at the log, it says &lt;code&gt;/Users/thomas/.root/opt/play-1.2.5/modules/docviewer/app/initial-agencies-data.yml&lt;/code&gt; but that doesn&amp;rsquo;t look right. That&amp;rsquo;s in the Play distribution directory, probably not somewhere something app-specific should go into. Well, a fixture is used for testing, so maybe the &lt;code&gt;test/&lt;/code&gt; directory? No, that doesn&amp;rsquo;t work either since we&amp;rsquo;re not running a test.&lt;/p&gt;
&lt;p&gt;What I ended up doing was just looking at the sources for &lt;code&gt;Fixtures.load&lt;/code&gt;. If you follow the stack trace, you end up finding &lt;code&gt;Play.javaPath&lt;/code&gt; which sort of works like PATH for Fixtures and some other things. So where can we put the file? &lt;code&gt;app/&lt;/code&gt; and &lt;code&gt;conf/&lt;/code&gt;. And with that, we&amp;rsquo;re done.&lt;/p&gt;
&lt;!-- raw HTML omitted --&gt;
&lt;p&gt;GTFS Editor is very much in development. Just getting it to run was problematic. There also seem to be a lot of missing issues judging from the Github Issues page. If you want to try it out for yourself, I suggest you clone &lt;a href=&#34;https://github.com/thatsmydoing/gtfs-editor&#34;&gt;my branch&lt;/a&gt; as I&amp;rsquo;ve fixed the issues discussed earlier. The default login is &lt;code&gt;admin:admin&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Even after getting it to run, it&amp;rsquo;s still not quite usable. Not in the UX sense, but you really can&amp;rsquo;t do much with it. There is no way to import the GTFS data into the webapp. There is something like import from TransitWand but even that is unclear to me. And even if we do get that running as well, we still don&amp;rsquo;t have any data we can play around with. We would need database dumps from the already running tools for these to be of any use right now.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>One Bus (or maybe Jeep) Away</title>
      <link>https://pleasantprogrammer.com/posts/one-bus-or-maybe-jeep-away.html</link>
      <pubDate>Tue, 09 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/one-bus-or-maybe-jeep-away.html</guid>
      <description>Link: http://onebusaway.org/
TL;DR no routing; useless in Philippines
OneBusAway is a transit information app. It provides data on what bus stops are near you, which buses pass by. You can also get schedules and the route of a particular bus given the number. It can also provide realtime updates like how many minutes until the next bus arrives. It does not, however, provide routing. There is no support for providing directions to get from point A to point B.</description>
      <content:encoded>&lt;p&gt;Link: &lt;a href=&#34;http://onebusaway.org/&#34;&gt;http://onebusaway.org/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; no routing; useless in Philippines&lt;/p&gt;
&lt;p&gt;OneBusAway is a transit information app. It provides data on what bus stops are near you, which buses pass by. You can also get schedules and the route of a particular bus given the number. It can also provide realtime updates like how many minutes until the next bus arrives. It does not, however, provide routing. There is no support for providing directions to get from point A to point B.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s comparable to what you get in some bus stops abroad. You&amp;rsquo;d get a vicinity map and a list of buses passing through the stop. You might also get the times when the next buses will pass. It&amp;rsquo;s useful for locals who already know how to get around, and want to avoid waiting for the bus. But it&amp;rsquo;s not particularly good for people who want to know how to get around the city.&lt;/p&gt;
&lt;p&gt;OneBusAway is quite comprehensive in its platform support though. There is a webapp, apps for iOS, Android and Windows Phone, as well as SMS and Voice support. This would all be nice but we don&amp;rsquo;t have the necessary infrastructure yet in the Philippines. We don&amp;rsquo;t have bus or jeepney stops. We also wouldn&amp;rsquo;t have realtime data to make the app particularly useful.&lt;/p&gt;
&lt;p&gt;You can try it out for yourself by following their &lt;a href=&#34;https://github.com/OneBusAway/onebusaway-application-modules/wiki/OneBusAway-Quickstart-Guide&#34;&gt;Quickstart Guide&lt;/a&gt;. One caveat is you will have to add &lt;code&gt;-P tripEntriesFactory.throwExceptionOnInvalidStopToShapeMappingException=false&lt;/code&gt; when building the bundle. This has to do with the OneBusAway having difficulty matching the &lt;a href=&#34;https://github.com/OneBusAway/onebusaway-application-modules/wiki/Stop-to-Shape-Matching&#34;&gt;stops to the shape data&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Here&amp;rsquo;s some screenshots of the app with the Philippine data. Notice how you only see the stops but there isn&amp;rsquo;t a line for the route. This is a problem with our GTFS data. Also, at some points it&amp;rsquo;s hard to tell where the jeep is going to pass since there isn&amp;rsquo;t any indication of order either. This is more of a OneBusAway problem. It usually expects there to be shape data available.&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;../galleries/transit/onebusaway1.png&#34;&gt;&lt;img src=&#34;../galleries/transit/onebusaway1.png&#34; alt=&#34;OneBusAway&#34; title=&#34;All the stops along Katipunan Avenue are named Katipunan Avenue.&#34;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;It doesn&amp;rsquo;t really handle too many routes passing through a stop. The list just overflows past the bubble. You can still actually read it by panning the map. It&amp;rsquo;s just a bit weird though.&lt;/p&gt;
&lt;p&gt;If you also noticed, there are usually 2 of each route. This is how the jeepney data was modeled as jeep routes might be different going one way and going back. This isn&amp;rsquo;t the case for all jeeps though, so it might also be an implementation issue with the GTFS editor.&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;../galleries/transit/onebusaway2.png&#34;&gt;&lt;img src=&#34;../galleries/transit/onebusaway2.png&#34; alt=&#34;OneBusAway&#34;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;../galleries/transit/onebusaway3.png&#34;&gt;&lt;img src=&#34;../galleries/transit/onebusaway3.png&#34; alt=&#34;OneBusAway&#34;&gt;&lt;/a&gt;&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Open Trip Planner</title>
      <link>https://pleasantprogrammer.com/posts/open-trip-planner.html</link>
      <pubDate>Tue, 09 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/open-trip-planner.html</guid>
      <description>Link: http://www.opentripplanner.org
TL;DR routes pretty well; data might cause weird issues
OpenTripPlanner, as the name implies, is a routing app. Given point A and point B, it can provide possible routes by taking transit, riding a bike, or a mix of both. You can also specify options on how much walking you&amp;rsquo;re willing to do or if you prefer fewer transfers over trip time. It could be a good competitor to the transit directions of Google Maps.</description>
      <content:encoded>&lt;p&gt;Link: &lt;a href=&#34;http://www.opentripplanner.org&#34;&gt;http://www.opentripplanner.org&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; routes pretty well; data might cause weird issues&lt;/p&gt;
&lt;p&gt;OpenTripPlanner, as the name implies, is a routing app. Given point A and point B, it can provide possible routes by taking transit, riding a bike, or a mix of both. You can also specify options on how much walking you&amp;rsquo;re willing to do or if you prefer fewer transfers over trip time. It could be a good competitor to the transit directions of Google Maps.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s actually in general use by the &lt;a href=&#34;http://maps.trimet.org/&#34;&gt;Trimet&lt;/a&gt;, Portland&amp;rsquo;s public transit system. I think a good reason why they deployed OpenTripPlanner is that Portland is a very bike friendly area. OpenTripPlanner&amp;rsquo;s support for multi-modal (bike + transit) routing is one thing that even Google Maps doesn&amp;rsquo;t have. This actually just screams &lt;a href=&#34;http://philippine-transit.hackathome.com/prizes/&#34;&gt;Inclusive Technology Award&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&#34;setup&#34;&gt;Setup&lt;/h3&gt;
&lt;p&gt;Getting OpenTripPlanner up and running involves a bit more downloading than OneBusAway. I&amp;rsquo;d suggest going through the &lt;a href=&#34;https://github.com/openplans/OpenTripPlanner/wiki/FiveMinutes&#34;&gt;5-minute introduction&lt;/a&gt; if you want to actually work with the Philippine data. You will also need to download the &lt;a href=&#34;http://download.geofabrik.de/asia/philippines.html&#34;&gt;Philippine data dump&lt;/a&gt; from OSM. You will want the &lt;code&gt;osm.bz2&lt;/code&gt; one (WARNING: 900MB unzipped).&lt;/p&gt;
&lt;p&gt;Once you get the webapp running, you&amp;rsquo;ll notice the map tiles won&amp;rsquo;t load correctly. This is because the default tileset used is from Mapbox which doesn&amp;rsquo;t provide publicly free tilesets anymore. To actually see things on the map, you should click the + on the upper-right of the map and change the base layer to something like Open Street Map or OSM Mapquest. This has been fixed in their latest sources.&lt;/p&gt;
&lt;h3 id=&#34;issues&#34;&gt;Issues&lt;/h3&gt;
&lt;p&gt;The next thing you&amp;rsquo;ll notice is that you can&amp;rsquo;t search for a place. You can only just pick points on the map and route between those. I&amp;rsquo;m still not exactly sure if it&amp;rsquo;s supposed to have it, since the Trimet one has it. But even then adding it by using the Google Maps or MapQuest APIs shouldn&amp;rsquo;t be too difficult.&lt;/p&gt;
&lt;p&gt;Another missing thing compared to the Trimet planner is being able to look at the routes akin to OneBusAway. And even then, Trimet&amp;rsquo;s implementation isn&amp;rsquo;t as good for exploring as OneBusAway is. Seeing the schedule or seeing which routes pass through a stop are left to an external site to do.&lt;/p&gt;
&lt;p&gt;There have been some weird issues with the routing though. A common occurence is the steps are somewhat disconnected (see image below). The left half shows OpenTripPlanner, it shows that you get off at a &amp;ldquo;stop&amp;rdquo; in EDSA and you should magically teleport to Arguilla street and start walking. To be fair, Google Maps (right half) shows that there is a street from the &amp;ldquo;stop&amp;rdquo; to Arguilla.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;../galleries/transit/otp1.png&#34; alt=&#34;OpenTripPlanner disconnected route&#34;&gt;&lt;/p&gt;
&lt;p&gt;I can&amp;rsquo;t say this isn&amp;rsquo;t a bug. After all, OpenTripPlanner couldn&amp;rsquo;t have magically known there really was a street there. If it did, it wouldn&amp;rsquo;t just teleport you to the corner. I&amp;rsquo;m more inclined to think though that this is a result of not having any shape data. As a result, OpenTripPlanner might try to assume the stop could mean places within a certain radius of where it was defined.&lt;/p&gt;
&lt;p&gt;Another weird issue that pops up sometimes is where it tells you to ride a jeep past the stop you want and then after a while, get off and ride a jeep back to your stop. A more general issue is that sometimes it won&amp;rsquo;t give you the best route because it thinks you have to loop around to get to where you want to be. I don&amp;rsquo;t really know how common this issue pops up though. It&amp;rsquo;s highly dependent on where you put the marker. If you just move the marker down a little bit, it actually does give the correct route.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;../galleries/transit/otp2.png&#34; alt=&#34;OpenTripPlanner loopy route&#34;&gt;&lt;/p&gt;
&lt;p&gt;Much like the earlier problem, I can&amp;rsquo;t tell if this is a bug or it&amp;rsquo;s a result of the bad data. Once again though, I think the issue is more of bad data. If you look at the plotted UP-Katipunan route, the stops aren&amp;rsquo;t even on the road. This probably makes it more difficult for OpenTripPlanner to actually tell if the stop and road are connected.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;../galleries/transit/upkatipunan.jpg&#34; alt=&#34;UP Katipunan Route&#34;&gt;&lt;/p&gt;
&lt;p&gt;From what they said during the launch, most of the route data was collected by getting a person to ride a jeep with a smartphone. That would explain why the coordinates aren&amp;rsquo;t that exact. Even then, it would have been nice if they at least cleaned up the data by moving the stops to the road. They would have had to go over them to name the stops anyway.&lt;/p&gt;
&lt;h3 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h3&gt;
&lt;p&gt;Overall though, I really like OpenTripPlanner. It handles most of the hard parts of the challenge. It provides a REST API for doing routing with the GTFS + OSM data. There&amp;rsquo;s also a lot of potential for additional open source work. A lot can be done to improve the default webapp. Adding a default location searcher would greatly improve usability. Adding in the route viewing features of OneBusAway would also be nice. Alternatively, you could even write your own client that just interfaces with the API.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Jeepney and Bus Routes</title>
      <link>https://pleasantprogrammer.com/posts/jeepney-and-bus-routes.html</link>
      <pubDate>Sun, 07 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/jeepney-and-bus-routes.html</guid>
      <description>In the last post, I talked about how we now have data about jeepney and bus routes in the Philippines. The data is actually in the GTFS format, which is the format the Google Maps consumes transit data. Apparently, the government will be submitting the GTFS data later this year. Transit directions for Metro Manila in Google Maps would be wonderful. That said, it definitely raises the bar for the app challenge people.</description>
      <content:encoded>&lt;p&gt;In the &lt;a href=&#34;philippine-transit-app-challenge.html&#34;&gt;last post&lt;/a&gt;, I talked about how we now have data about jeepney and bus routes in the Philippines. The data is actually in the &lt;a href=&#34;https://developers.google.com/transit/gtfs/&#34;&gt;GTFS format&lt;/a&gt;, which is the format the Google Maps consumes transit data. Apparently, the government will be submitting the GTFS data later this year. Transit directions for Metro Manila in Google Maps would be wonderful. That said, it definitely raises the bar for the app challenge people.&lt;/p&gt;
&lt;p&gt;In the last post, I mentioned the quality of the data isn&amp;rsquo;t quite good. Even before seeing the data, I was already a bit unsure of it. The key problem is how you model the routes. The GTFS format was inherently designed for more well developed and organized transit agencies which isn&amp;rsquo;t exactly what we have in the Philippines now.&lt;/p&gt;
&lt;p&gt;One potential problem is the nature of the jeeps and buses. GTFS routes are a collection of trips which are a sequence of stops. However, we don&amp;rsquo;t have jeepney stops, and even if we did they still just stop anywhere. There are also times where jeeps will take a shortcut if no passengers need to get dropped off along their normal route.&lt;/p&gt;
&lt;p&gt;From what I&amp;rsquo;ve seen of the data, they handled the first problem well enough. Stops are defined as where people typically get on the jeep or bus. This is good, but they didn&amp;rsquo;t define a shape for the routes. There is no information as to which exact roads they pass through. All we have to go by are the stops to show the route on a map.&lt;/p&gt;
&lt;p&gt;&lt;img src=&#34;http://i.imgur.com/NSVlryE.jpg&#34; alt=&#34;sample route&#34;&gt;&lt;/p&gt;
&lt;p&gt;The problem isn&amp;rsquo;t that bad though. The agencies could still add the shapes later on. Or maybe an app challenge participant could make an app around fixing the routes via crowd-sourcing or similar. The shape itself isn&amp;rsquo;t that important for a rudimentary directions app, but if we want better apps, we will need better data.&lt;/p&gt;
&lt;p&gt;There were also some minor issues with the data itself. Some of the files had extra columns. This normally isn&amp;rsquo;t an issue, but it caused problems for &lt;a href=&#34;https://github.com/harrisony/gtfs_SQL_importer&#34;&gt;GTFS SQL importer&lt;/a&gt;. There were also problems with matching the shape data with the stops when I tried it with &lt;a href=&#34;http://onebusaway.org&#34;&gt;OneBusAway&lt;/a&gt;. They could probably be &lt;a href=&#34;https://github.com/OneBusAway/onebusaway-application-modules/wiki/Stop-to-Shape-Matching&#34;&gt;fixed&lt;/a&gt; but that&amp;rsquo;s for another day.&lt;/p&gt;
</content:encoded>
    </item>
    
    <item>
      <title>Philippine Transit App Challenge</title>
      <link>https://pleasantprogrammer.com/posts/philippine-transit-app-challenge.html</link>
      <pubDate>Sun, 07 Jul 2013 00:00:00 +0000</pubDate>
      
      <guid>https://pleasantprogrammer.com/posts/philippine-transit-app-challenge.html</guid>
      <description>Last week, the DOTC launched the Philippine Transit App Challenge. It&amp;rsquo;s a competition to build something great using the newly available 1) jeepney/bus/rail routes and 2) traffic incident data in Metro Manila and Cebu.
I&amp;rsquo;m actually quite excited about this as it&amp;rsquo;s not everyday our government does wonderful things. Many people have been waiting for this kind of data to be available. Before, the only way to figure out which jeeps to ride to get from A to B is by asking other people.</description>
      <content:encoded>&lt;p&gt;Last week, the DOTC launched the &lt;a href=&#34;http://philippine-transit.hackathome.com&#34;&gt;Philippine Transit App Challenge&lt;/a&gt;. It&amp;rsquo;s a competition to build something great using the newly available 1) jeepney/bus/rail routes and 2) traffic incident data in Metro Manila and Cebu.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;m actually quite excited about this as it&amp;rsquo;s not everyday our government does wonderful things. Many people have been waiting for this kind of data to be available. Before, the only way to figure out which jeeps to ride to get from A to B is by asking other people. The website of the LTFRB used to just have a list of jeepney routes, but nothing else, no maps or list of stops.&lt;/p&gt;
&lt;p&gt;During the launch, they also presented how CITOM (the Cebu MMDA) is pilot-testing a &lt;a href=&#34;http://cebutraffic.org/&#34;&gt;traffic tracking system&lt;/a&gt;. What they did was supply Android phones to taxi drivers. The phones send GPS data which can then be aggregated to see what the average speeds along streets are. This also benefits the taxi company as it provides easy tracking of their taxis compared to their old telephone/radio with pen-and-paper process.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&#34;http://philippine-transit.hackathome.com/dataset-philippines-transit-information-service-gtfs/&#34;&gt;route data&lt;/a&gt; are already available, as well as the &lt;a href=&#34;http://philippine-transit.hackathome.com/dataset-citom-traffic-alert-platform/&#34;&gt;Cebu&lt;/a&gt; and &lt;a href=&#34;http://philippine-transit.hackathome.com/dataset-mmda-traffic-alert-platform/&#34;&gt;Metro Manila&lt;/a&gt; incident data. You do have to register to access them though. Right now, they&amp;rsquo;re just one-off dumps of the data but the various agencies have promised to provide consistently updated data. This will be provided via ASTI later in the year.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;ve just started looking at the route data and playing around with it. The quality could be better, but I&amp;rsquo;m glad we at least have something to work with.&lt;/p&gt;
</content:encoded>
    </item>
    
  </channel>
</rss>