blog/output/rss.xml

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="assets/xml/rss.xsl" media="all"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Pleasant Programmer</title><link>http://pleasantprogrammer.com/</link><description></description><atom:link type="application/rss+xml" rel="self" href="http://pleasantprogrammer.com/rss.xml"></atom:link><language>en</language><lastBuildDate>Fri, 25 Dec 2015 07:29:48 GMT</lastBuildDate><generator>https://getnikola.com/</generator><docs>http://blogs.law.harvard.edu/tech/rss</docs><item><title>Cloudflare Shenanigans</title><link>http://pleasantprogrammer.com/posts/cloudflare-shenanigans.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;An old client of ours managed to convince a telco to zero-rate the data for their app. In order to whitelist it though, we needed to use plain HTTP for domain whitelisting. For HTTPS, they can only whitelist by IP address. Like any good developer, we were using HTTPS. Also, like any good developer, we put our server behind Cloudflare.&lt;/p&gt;
&lt;p&gt;Now the problem is that Cloudflare can put you behind &lt;a href="https://www.cloudflare.com/ips/"&gt;any IP they own&lt;/a&gt;, which is a huge range. There's no guarantee that the IP we have now is going to be the same later on. So we did the reasonable thing and asked them to whitelist all of the Cloudflare IPs. And the telco agreed! We were in total disbelief when that happened. But hey, if life gives you free internet, you take it.&lt;/p&gt;
&lt;p&gt;We never actually empirically tested whether other sites hosted on Cloudflare were also actually zero-rated. But I like to think that we saved a lot of people on their data costs from browsing Reddit and 4chan. But alas, good things must come to an end.&lt;/p&gt;
&lt;p&gt;A few months after we started beta testing the app, Cloudflare added more IPs to their range. Unfortunately, our server got moved to those new IPs which were not whitelisted yet. Apparently, the telco whitelisting process was incredibly convoluted and time consuming. Our client didn't want to bother asking them to whitelist more IPs. We also tried asking Cloudflare to move us back to the original IP range, but they could only do that if we were in their enterprise tier. We couldn't really afford that, so we looked for other options.&lt;/p&gt;
&lt;p&gt;Since Cloudflare was essentially just a giant reverse proxy, theoretically there should be no distinction between one IP address from another. The specific IP we get is probably just for load balancing. So we tried accessing the IPs in the range directly and just setting the Host header and it worked! But we get SSL errors because the IP itself doesn't have its own certificate.&lt;/p&gt;
&lt;p&gt;After more testing, we figured out that you could actually use any Cloudflare backed domain so long as we properly set the Host header. We just needed to find one still in the old range. Coincidentally, 4chan.org was. Which led to this wonderful commit&lt;/p&gt;
&lt;pre class="code literal-block"&gt;commit 123456789abcdef
Author: ~~~~~~
Date:   ~~~~~~

    4chan hack

&lt;span class="gh"&gt;diff --git a/src/com/client/common/Util.java b/src/com/client/common/Util.java&lt;/span&gt;
&lt;span class="gd"&gt;--- a/src/com/client/common/Util.java&lt;/span&gt;
&lt;span class="gi"&gt;+++ b/src/com/client/common/Util.java&lt;/span&gt;
&lt;span class="gu"&gt;@@ -210,7 +210,8 @@ public class Util {&lt;/span&gt;
        }

        public static String getServerAddress(Context context) {
&lt;span class="gd"&gt;-               String address = "https://backend.client.com";&lt;/span&gt;
&lt;span class="gi"&gt;+               // String address = "https://backend.client.com";&lt;/span&gt;
&lt;span class="gi"&gt;+               String address = "https://4chan.org";&lt;/span&gt;
                if(!isDebug(context)) return address;
                try {
&lt;span class="gh"&gt;diff --git a/src/com/client/common/logging/APIClient.java b/src/com/client/common/logging/APIClient.java&lt;/span&gt;
&lt;span class="gd"&gt;--- a/src/com/client/common/logging/APIClient.java&lt;/span&gt;
&lt;span class="gi"&gt;+++ b/src/com/client/common/logging/APIClient.java&lt;/span&gt;
&lt;span class="gu"&gt;@@ -101,6 +101,7 @@ public class APIClient {&lt;/span&gt;
        private HttpResponse postInternal(String url, List&amp;lt;NameValuePair&amp;gt; data, boolean forRegistration) throws ClientProtocolException, IOException {
                HttpPost request = new HttpPost(Util.getServerAddress(mContext)+"/api/"+url);
                request.setHeader("X-API-VERSION", apiVersion);
&lt;span class="gi"&gt;+               request.setHeader("Host", "backend.client.com");&lt;/span&gt;

                if(data == null) {
                        data = new ArrayList&amp;lt;NameValuePair&amp;gt;();
&lt;/pre&gt;


&lt;p&gt;Eventually, we did decide to just abandon Cloudflare for the server. We probably weren't going to be the target of a DDOS or anything. This also allowed us to do more secure things like pinning the server certificate in the application itself. Clearly, this is what we should have just done in the first place, but at the time we just wanted a stopgap solution.&lt;/p&gt;
&lt;p&gt;I just still find it funny we were making people's phones go to 4chan.org everyday for more than a year.&lt;/p&gt;&lt;/div&gt;</description><category>cloudflare</category><category>sysadmin</category><guid>http://pleasantprogrammer.com/posts/cloudflare-shenanigans.html</guid><pubDate>Fri, 25 Dec 2015 06:13:26 GMT</pubDate></item><item><title>TiddlyWiki in the Sky (or TiddlyWeb for TW5)</title><link>http://pleasantprogrammer.com/posts/tiddlywiki-in-the-sky-or-tiddlyweb-for-tw5.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;I've always liked &lt;a href="http://tiddlywiki.com"&gt;TiddlyWiki&lt;/a&gt;. Back when it first came out, it was really amazing. A wiki all in one file, that worked in the browser. It didn't need a backend, it would just save itself as an all new HTML file with all your posts inside. I've used it a lot over the years, as a personal wiki/journal and a class notebook. I even had a blog with it at one point using one of the server-side forks.&lt;/p&gt;
&lt;p&gt;Now, there's TiddlyWiki5 which is a rewrite of the original TiddlyWiki that looks a whole lot snazzier, and I assume has better architecture overall. It also has experimental support for all the server-side platforms (particularly TiddlyWeb) that have cropped up.&lt;/p&gt;
&lt;p&gt;If you're just looking for a simple server setup for TiddlyWiki5, it has native support for that on its own. There's plenty of documentation on the site. But if you're looking for more advanced features (like storing your posts in git or a database), then you'll need to use it with TiddlyWeb. The problem is that most of the documentation for TiddlyWeb still refers to the old TiddlyWiki.&lt;/p&gt;
&lt;p&gt;To support TiddlyWiki5, we'll need a version of the wiki which has the TiddlyWeb plugin already installed and configured. After that, some tweaking is necessary to get TiddlyWeb to provide what the wiki requires.&lt;/p&gt;
&lt;h3&gt;Setting Up TiddlyWiki&lt;/h3&gt;
&lt;p&gt;TiddlyWiki5 provides a command line tool via &lt;code&gt;npm&lt;/code&gt; that allows building custom versions of the wiki. In fact, it comes with templates, called "editions", that we can use for our setup. Assuming you already have it installed, create the wiki using&lt;/p&gt;
&lt;pre class="code literal-block"&gt;tiddlywiki mywiki --init tw5tank          # create wiki from template
&lt;/pre&gt;


&lt;p&gt;This creates a wiki intended for use with &lt;a href="https://tank.peermore.com/"&gt;Tank&lt;/a&gt;, which is built on top of TiddlyWeb. From here, you should look in &lt;code&gt;mywiki/tiddlers/system&lt;/code&gt; which contain the entries for &lt;code&gt;SiteTitle&lt;/code&gt;, &lt;code&gt;SiteSubtitle&lt;/code&gt;, &lt;code&gt;DefaultTiddlers&lt;/code&gt;, and &lt;code&gt;tiddlyweb-host&lt;/code&gt;. The first 3 should be configured however you want. These are necessary because they're needed before the wiki can load them from the server. &lt;code&gt;tiddlyweb-host&lt;/code&gt; contains the location of the TiddlyWeb server, this should be &lt;code&gt;http://localhost:8080/&lt;/code&gt; if you're just testing locally. With everything configured, you can build the new wiki by running&lt;/p&gt;
&lt;pre class="code literal-block"&gt;tiddlywiki mywiki --build
&lt;/pre&gt;


&lt;p&gt;This will output the wiki to &lt;code&gt;mywiki/output/tw5tank.html&lt;/code&gt;. You can now serve it using your favorite local webserver, like &lt;code&gt;python -m http.server&lt;/code&gt;.&lt;/p&gt;
&lt;h3&gt;Setting Up TiddlyWeb&lt;/h3&gt;
&lt;p&gt;The TiddlyWeb tutorial recommends using &lt;code&gt;tiddlywebwiki&lt;/code&gt; which has all the plugins setup for a nice wiki instance for the old TiddlyWiki. It has a lot of features that aren't really needed, so we won't go with that. So first, we'll need to install TiddlyWeb and any plugins we might want to use.&lt;/p&gt;
&lt;pre class="code literal-block"&gt;pip install tiddlyweb tiddlywebplugins.status tiddlywebplugins.cherrypy tiddlywebplugins.cors
&lt;/pre&gt;


&lt;p&gt;Next, we'll need the tiddlyweb configuration in &lt;code&gt;tiddlywebconfig.py&lt;/code&gt;&lt;/p&gt;
&lt;pre class="code literal-block"&gt;&lt;span class="c"&gt;# A basic configuration.&lt;/span&gt;
&lt;span class="c"&gt;# `pydoc tiddlyweb.config` for details on configuration items.&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;tiddlywebplugins.status&lt;/span&gt;

&lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s"&gt;'system_plugins'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;'tiddlywebplugins.status'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;'tiddlywebplugins.cors'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="s"&gt;'secret'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;'36c98d6d14618c79f0ed2d49cd1b9e272d8d4bd0'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s"&gt;'wsgi_server'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;'tiddlywebplugins.cherrypy'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s"&gt;'cors.enable_non_simple'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;original_gather_data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tiddlywebplugins&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_gather_data&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_status_gather_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;original_gather_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;'space'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;'recipe'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;'default'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;

&lt;span class="n"&gt;tiddlywebplugins&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_gather_data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_status_gather_data&lt;/span&gt;
&lt;/pre&gt;


&lt;p&gt;The tweaks involved are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;using the status plugin which the wiki requires&lt;/li&gt;
&lt;li&gt;monkeypatching the status plugin for the wiki to use the correct "recipe"&lt;/li&gt;
&lt;li&gt;using cherrypy server instead of the buggy default one&lt;/li&gt;
&lt;li&gt;using cors since we're not hosting the wiki itself on the same server&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;With that, we just need to create the store that will hold our data&lt;/p&gt;
&lt;pre class="code literal-block"&gt;twanager recipe default &amp;lt;&amp;lt;EOF
desc: standard TiddlyWebWiki environment
policy: {"read": [], "create": [], "manage": ["R:ADMIN"], "accept": [], "write": ["R:ADMIN"], "owner": "administrator", "delete": ["R:ADMIN"]}

/bags/default/tiddlers
EOF

twanager bag default &amp;lt;&amp;lt;EOF
{"policy": {"read": [], "create": [], "manage": ["R:ADMIN"], "accept": [], "write": [], "owner": "administrator", "delete": []}}
EOF
&lt;/pre&gt;


&lt;p&gt;Finally, we can start the TiddlyWeb server&lt;/p&gt;
&lt;pre class="code literal-block"&gt;twanager server
&lt;/pre&gt;


&lt;h3&gt;Putting it all together&lt;/h3&gt;
&lt;p&gt;Once you have the TiddlyWeb server running, you can just go to wherever you're hosting the wiki html and it should work. You can try creating some posts, and the check mark on the sidebar should be red for a while and then turn black. Once that's done it's saved. You can refresh your browser and your posts should still be there.&lt;/p&gt;
&lt;p&gt;At this point, you can start customizing your TiddlyWeb instance, by changing your store to something like a database, or adding authorization. You can also tweak the server setup so you won't need CORS anymore.&lt;/p&gt;
&lt;p&gt;TiddlyWiki5 is still relatively new. I hope that eventually, support for server-side and the plugin ecosystem grows to be as great as the old TiddlyWiki.&lt;/p&gt;&lt;/div&gt;</description><category>sysadmin</category><category>tiddlywiki</category><guid>http://pleasantprogrammer.com/posts/tiddlywiki-in-the-sky-or-tiddlyweb-for-tw5.html</guid><pubDate>Thu, 24 Dec 2015 06:48:20 GMT</pubDate></item><item><title>Is My Terminal Window Active?</title><link>http://pleasantprogrammer.com/posts/is-my-terminal-window-active.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;I've been working in OSX for almost 3 years now, but I recently switched back to Linux because of all the problems people encountered with Yosemite. There are some things I missed from OSX though. One of which is &lt;a href="https://github.com/marzocchi/zsh-notify"&gt;zsh-notify&lt;/a&gt;. It's a zsh plugin that alerts you if your long-running task is complete, and whether it failed or not.&lt;/p&gt;
&lt;p&gt;It's pretty convenient when you're compiling something and then go on to browse reddit while waiting. Usually, I spend too much time just reading and forget about the compilation entirely. With the plugin, I get the notification and maybe go back to work.&lt;/p&gt;
&lt;p&gt;One nice feature it has is that if you're currently looking at the terminal window of the job that just finished, it won't notify you. It only notifies on windows that aren't currently in focus. To do this, it has to actually talk to Terminal.app or iTerm2 to see if the window and tab are active.&lt;/p&gt;
&lt;p&gt;This is alright in OSX since those 2 are the generally most used terminal emulators. On Linux though, everyone has their own favorite terminal. Given that, I figured I could probably rely on talking to X to see if the window is active instead of each single terminal emulator. X can't tell if the tab is active though, but I don't use tabs in my current setup so it should still be good.&lt;/p&gt;
&lt;h3&gt;xdotool&lt;/h3&gt;
&lt;p&gt;&lt;a href="http://superuser.com/questions/382616/detecting-currently-active-window"&gt;Preliminary research&lt;/a&gt; reveals that we can easily get what the active window is with xdotool. &lt;code&gt;xdotool getactivewindow&lt;/code&gt; gives us the X window id of the active one. Now all we need is a way to get the window id of the terminal we're in.&lt;/p&gt;
&lt;h3&gt;First Attempt: $WINDOWID&lt;/h3&gt;
&lt;p&gt;Apparently, xterm and similar terminal emulators define an environment variable called &lt;code&gt;$WINDOWID&lt;/code&gt; with the window id of the terminal. Obviously, this is too good to be true. In xterm and konsole the &lt;code&gt;$WINDOWID&lt;/code&gt; was correct, but in VTE-based terminal emulators, &lt;code&gt;$WINDOWID&lt;/code&gt; had the wrong value. In terminology, it didn't define &lt;code&gt;$WINDOWID&lt;/code&gt; altogether. So &lt;code&gt;$WINDOWID&lt;/code&gt; wasn't going to work.&lt;/p&gt;
&lt;h3&gt;Second Attempt: xdotool search $MAGIC&lt;/h3&gt;
&lt;p&gt;My second idea was that you can use zsh to change the window title to a magic number and then just check if the active window is the same one as the window with the magic number. This sort of worked for most terminals, except konsole which does whatever it wants with the window title. There's also the problem of some zsh configs automatically settings the window title to the current command.&lt;/p&gt;
&lt;p&gt;In hindsight, I could probably have just done &lt;code&gt;xdotool search --name xdotool&lt;/code&gt; since in most cases, when you run the search, zsh or konsole will set the window name to the current command. Maybe that's another option I can explore some day.&lt;/p&gt;
&lt;h3&gt;Third Attempt: $PPID&lt;/h3&gt;
&lt;p&gt;My third idea was another environment variable called &lt;code&gt;$PPID&lt;/code&gt;, which is the process id of the parent of the shell. As it happens, the parent is the window containing the zsh instance. This is actually pretty consistent across most terminals. The only problem was if you launched zsh from another shell since your new zsh's parent will now be another zsh instance instead of an X window.&lt;/p&gt;
&lt;p&gt;At first glance, launching zsh within zsh doesn't seem like something most people would do, but this is what happens when you run screen or tmux. To work around this, we can actually just save the original &lt;code&gt;$PPID&lt;/code&gt; in a different variable and use that instead.&lt;/p&gt;
&lt;p&gt;Now that we have the PID of the window from zsh, we can once again use xdotool to get the PID of the current active window with &lt;code&gt;xdotool getactivewindow getwindowpid&lt;/code&gt;. We just simply compare that with our &lt;code&gt;$PPID&lt;/code&gt; and we can tell if we're in an active window or not. Overall, this approach worked surprisingly well so that's the final solution I went with.&lt;/p&gt;&lt;/div&gt;</description><category>programming</category><guid>http://pleasantprogrammer.com/posts/is-my-terminal-window-active.html</guid><pubDate>Sun, 07 Jun 2015 08:20:45 GMT</pubDate></item><item><title>Removing PLDTMyDSLBiz from the ZyXEL P-2612HNU</title><link>http://pleasantprogrammer.com/posts/removing-pldtmydslbiz-from-the-zyxel-p-2612hnu.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;I've always thought that people were just too lazy to change their SSIDs when I see "PLDTMyDSLBizCafeJapan". It became apparent when we got our own PLDT line that it was because the bundled router/modem &lt;em&gt;does not&lt;/em&gt; allow you to remove the prefix.&lt;/p&gt;
&lt;p&gt;This is not the kind of thing you expect as a business customer. Even for home customers, I feel it's still a bit dishonest. I'd be fine if it was just the default SSID, but forcing people to have it as part of their SSID is like advertising that your company (I mean PLDT) is a douche.&lt;/p&gt;
&lt;p&gt;Of course, we couldn't just leave the SSID prefix there, so we tried a number of things to get rid of it. There are articles for removing it from the &lt;a href="http://www.phandroidinternet.com/2013/06/how-to-remove-on-wifi-name-or-ssid-on.html"&gt;Prolink H5004N&lt;/a&gt; or the &lt;a href="http://www.symbianize.com/showthread.php?t=730091"&gt;ZyXEL P-660HN-T1A&lt;/a&gt; but not for the one we got which was the ZyXEL P-2612HNU-F1F.&lt;/p&gt;
&lt;p&gt;We did still try the firebug/inspector tricks, but it seems that there is a server-side check that adds in the "PLDTMyDSLBiz". We tried a number of things, but the one that ultimately worked (and we had a good laugh about) was to backup the configuration, edit the dumped file and restore it.&lt;/p&gt;
&lt;p&gt;The backup is actually just an XML file. You can search for SSID and change the parameter there. It's a bit annoying because the router has to restart after restoring the configuration, but it works!&lt;/p&gt;
&lt;p&gt;A minor note, the router doesn't seem to support SSIDs with a comma (,) well. It just gets everything before the comma as the SSID for some reason.&lt;/p&gt;&lt;/div&gt;</description><category>sysadmin</category><guid>http://pleasantprogrammer.com/posts/removing-pldtmydslbiz-from-the-zyxel-p-2612hnu.html</guid><pubDate>Wed, 27 Nov 2013 02:12:31 GMT</pubDate></item><item><title>Console Keymap Switching</title><link>http://pleasantprogrammer.com/posts/console-keymap-switching.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;At the office, we have some people who use DVORAK. Normally, this isn't a problem. To each his own after all. It does become a bit problematic though, when we're dealing with the servers around the office.&lt;/p&gt;
&lt;p&gt;We normally leave the servers on QWERTY. After all, most people start off as QWERTY typists and migrate to something else. That said, it's apparently difficult to stay fluent in both. People tend to forget how to type in QWERTY once they learn DVORAK or something else. While it is true that they can just look a the keyboard while typing, my coworkers would prefer it to just be in DVORAK.&lt;/p&gt;
&lt;p&gt;For the console, they'd typically do &lt;code&gt;sudo loadkeys dvorak&lt;/code&gt; after logging in. The problem with this is, after they logout, the keymapping is still on DVORAK. This has been quite annoying for a few times since I can't even login to change the keymap. What I wanted was something like you get in the graphical login screens where you can pick your keymap before logging in. Apparently, there isn't a readily available thing for the console.&lt;/p&gt;
&lt;p&gt;I googled around for solutions and came across &lt;a href="http://superuser.com/questions/548234/how-can-i-easily-toggle-between-dvorak-and-qwerty-keyboard-layouts-from-a-linux"&gt;a nice idea&lt;/a&gt;. You could alias &lt;code&gt;asdf&lt;/code&gt; to load the DVORAK mapping and &lt;code&gt;aoeu&lt;/code&gt; (the equivalent to asdf in DVORAK) to load the QWERTY mapping. This actually makes sense since you don't really have to know where the letters are. The only problem is, you once again have to be logged in to change the key mappings.&lt;/p&gt;
&lt;p&gt;After some further searching, I found &lt;a href="http://unix.stackexchange.com/questions/2884/toggle-between-dvorak-and-qwerty"&gt;something close to what I wanted&lt;/a&gt;. Apparently, Alt+Up sends a KeyboardSignal keycode to the init process, which can act on that. It also works anywhere, even before being logged in. For SysVinit systems, you can just add a line to your inittab for a command to be run when Alt+Up is pressed.&lt;/p&gt;
&lt;p&gt;In the office, however, we generally use Arch Linux which uses SystemD. But apparently, it also has a mechanism of accepting the Alt+Up press. It runs the kbrequest target whenever it gets the keypress. &lt;code&gt;kbrequest.target&lt;/code&gt; is normally aliased to run the rescue service though, so you have to manually create the file in &lt;code&gt;/etc/systemd/system/kbrequest.target&lt;/code&gt; and fill it with a description:&lt;/p&gt;
&lt;pre class="code literal-block"&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;kbrequest target&lt;/span&gt;
&lt;/pre&gt;


&lt;p&gt;We can then add a service to be run whenever the target is called. Something like &lt;code&gt;/etc/systemd/system/keymap-switch.service&lt;/code&gt;:&lt;/p&gt;
&lt;pre class="code literal-block"&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Keymap Switch Service&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;oneshot&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/keymap-switch&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;kbrequest.target&lt;/span&gt;
&lt;/pre&gt;


&lt;p&gt;After enabling said service, we only need the actual keymap switcher, &lt;code&gt;/usr/local/bin/keymap-switch&lt;/code&gt;. The StackOverflow answer provides different ways of detecting the current keymap so we know which one to switch to. Since we're using SystemD, we can use that instead for managing which keymap we're actually using. It stores the current settings inside &lt;code&gt;/etc/vconsole.conf&lt;/code&gt;. We can also then switch keymaps by using &lt;code&gt;localectl set-keymap&lt;/code&gt;.&lt;/p&gt;
&lt;table class="codehilitetable"&gt;&lt;tr&gt;&lt;td class="linenos"&gt;&lt;div class="linenodiv"&gt;&lt;pre&gt; 1
 2
 3
 4
 5
 6
 7
 8
 9
10&lt;/pre&gt;&lt;/div&gt;&lt;/td&gt;&lt;td class="code"&gt;&lt;pre class="code literal-block"&gt;&lt;span class="c"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="nb"&gt;source&lt;/span&gt; /etc/vconsole.conf

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$TERM&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"dumb"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$KEYMAP&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"dvorak"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    localectl &lt;span class="nb"&gt;set&lt;/span&gt;-keymap us
  &lt;span class="k"&gt;else&lt;/span&gt;
    localectl &lt;span class="nb"&gt;set&lt;/span&gt;-keymap dvorak
  &lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/pre&gt;
&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;

&lt;p&gt;After putting it all together, it works! We can switch keymaps on the fly by simply pressing Alt+Up.&lt;/p&gt;&lt;/div&gt;</description><category>sysadmin</category><category>systemd</category><guid>http://pleasantprogrammer.com/posts/console-keymap-switching.html</guid><pubDate>Tue, 29 Oct 2013 12:02:06 GMT</pubDate></item><item><title>Geocoding Services</title><link>http://pleasantprogrammer.com/posts/geocoding-services.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;A key component for any routing service is being able to do geocoding. Most people who are looking for routes most probably don't know exactly where their start and end points are on the map. Even then, manually looking for a location on a map is a time-consuming task.&lt;/p&gt;
&lt;p&gt;The gold standard for doing geocoding right now is Google Maps. It's hard to find a better location search experience. If they actually provided routing for jeeps here in the Philippines, I imagine there wouldn't be &lt;em&gt;that&lt;/em&gt; much you could do for the competition.&lt;/p&gt;
&lt;p&gt;When the competition started though, I took it as a challenge to avoid Google Maps as much as possible. I wanted to see how much is currently possible with other options such as OpenStreetMap. In fact, OSM does have a geocoding service called &lt;a href="http://nominatim.openstreetmap.org"&gt;Nominatim&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Sadly, for a mapping app, what you want to do is not simply just geocoding. With geocoding, you take an address and turn it into coordinates. When you want to search for a place in a mapping app, you take part of an address, infer the rest of it, and give the user options to choose from.&lt;/p&gt;
&lt;p&gt;Given a typical mapping app, you might type in "ateneo" and expect it to give you Ateneo de Manila University. With typical geocoding services like Nominatim or even Google's &lt;a href="https://developers.google.com/maps/documentation/javascript/geocoding"&gt;geocoding API&lt;/a&gt;, you probably won't get any result for this. What you want to use is the &lt;a href="https://developers.google.com/maps/documentation/javascript/places"&gt;Places API&lt;/a&gt; which provides an autocomplete search box. Using it, when you type in "ateneo", it automatically suggests in the dropdown, "Ateneo de Manila University".&lt;/p&gt;
&lt;p&gt;A downside to using the Places API is that it's against the terms of service to use it with something that isn't Google Maps, which means no OpenStreetMap. If there were more time, writing your own autocompletion engine using OpenStreetMap's data will probably be a better long term solution.&lt;/p&gt;
&lt;p&gt;For now, since the competition's deadline is just a few days away, I'll be using Google Maps.&lt;/p&gt;&lt;/div&gt;</description><category>philippine-transit-app</category><category>programming</category><guid>http://pleasantprogrammer.com/posts/geocoding-services.html</guid><pubDate>Wed, 25 Sep 2013 04:26:59 GMT</pubDate></item><item><title>Jeep and Bus Schedules</title><link>http://pleasantprogrammer.com/posts/jeep-and-bus-schedules.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;Wouldn't it be wonderful if there were no buses or jeepneys in the Philippines over the weekends? It would truly be a cyclist's paradise. Imagine biking along EDSA, normally that would be a death sentence, but according to the GTFS data, you shouldn't worry. I can assure you, it's still a death sentence.&lt;/p&gt;
&lt;p&gt;The GTFS spec defines 2 ways of statically specifying trip schedules. You can define the exact times that a service will arrive at a stop. You can also specify between what times the service is active and how often a new bus or jeep leaves the first stop. You also define which days those rules apply. You could say every MWF, the bus operates from 9:00AM to 9:00PM and every TTH, the bus services from 3:00AM to 11:00PM.&lt;/p&gt;
&lt;p&gt;This should be sufficient in theory, but real world conditions like traffic or the weather could throw the schedules off. To solve this, there's another spec, GTFS-realtime. This allows transit agencies to push temporary schedule updates and service announcements.&lt;/p&gt;
&lt;p&gt;Like much everything else about the Philippine transit system, there aren't really any "schedules" to speak of. It's generally whenever the buses or jeeps feel like it. So we have no static schedules. We don't have a central agency or the tracking technology to make it feasible to push updates via GTFS-RT.&lt;/p&gt;
&lt;p&gt;Ideally, we shouldn't bother inputting the schedule information into GTFS. Only the route data is really important for jeeps and buses. However, the schedule information is required in the GTFS, and routing apps wouldn't work without it. So we have to add a reasonable trip schedule for jeeps and buses.&lt;/p&gt;
&lt;p&gt;The current GTFS data does define these trip schedules. We assume that jeeps and buses operate between 6:00AM and 11:00PM and a new jeep passes by every 10 minutes. Also, jeeps and buses are defined to only operate on weekdays.&lt;/p&gt;
&lt;p&gt;While there might be jeeps who change routes or don't operate on weekends, I'm pretty sure that jeeps and buses run on weekends. We'll have to fix it ourselves temporarily since there's no central GTFS feed yet.&lt;/p&gt;
&lt;pre class="code literal-block"&gt;&lt;span class="c"&gt;# 724594 seems to be the service id used by jeeps and buses&lt;/span&gt;
sed -i .bak &lt;span class="s1"&gt;'/^724594/ s/0,0/1,1/'&lt;/span&gt; calendar.txt
&lt;/pre&gt;


&lt;p&gt;Another thing we could do is to adjust the time between buses, although the improvement is arguable. With the current 10 minutes between jeeps, it might provide some routes a significant advantage just because the timing is right. So you might get differing route suggestions depending on what time you planned the route. This makes sense when you're sure what the times are, so you can minimize the wait, but with jeeps, you never really know how long the wait will actually be.&lt;/p&gt;
&lt;p&gt;If we set the frequency to one minute, it &lt;em&gt;might&lt;/em&gt; give better routes by eliminating the timing issue. Or not, it's kind of hard to tell.&lt;/p&gt;
&lt;pre class="code literal-block"&gt;&lt;span class="c"&gt;# jeep and bus route ids tend to start with 72&lt;/span&gt;
sed -i .bak &lt;span class="s1"&gt;'/^72/ s/,600/,60/'&lt;/span&gt; frequencies.txt
&lt;/pre&gt;


&lt;p&gt;Overall, the problems we're having is a symptom of the mismatch between our transit system and the GTFS. It would be great if our transit system gets better and we don't need to do hackish things for it to fit the GTFS, but that's still a dream. For now, all we can really do is fit a triangle into a square hole.&lt;/p&gt;&lt;/div&gt;</description><category>philippine-transit-app</category><category>programming</category><guid>http://pleasantprogrammer.com/posts/jeep-and-bus-schedules.html</guid><pubDate>Sun, 28 Jul 2013 08:26:31 GMT</pubDate></item><item><title>Highways in OTP</title><link>http://pleasantprogrammer.com/posts/highways-in-otp.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;One of the weird things that happens with OTP is sometimes it gives absurdly roundabout routes. Here is OTP's suggested route for walking from UP to Ateneo:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Roundabout route from UP to Ateneo" src="http://pleasantprogrammer.com/galleries/transit/otproundabout.png"&gt;&lt;/p&gt;
&lt;p&gt;This is just so hilariously wrong. It's much simpler to just walk along Katipunan Avenue.&lt;/p&gt;
&lt;p&gt;OTP couldn't possibly be that dumb though, so there must be something we're doing wrong. If you notice, Katipunan Avenue is colored red compared to the other streets. OTP seems to be avoiding any path that goes along Katipunan Avenue. The problem might have something to do with the "road type" designated to Katipunan.&lt;/p&gt;
&lt;p&gt;Apparently, by default OTP will consider roads of type &lt;code&gt;trunk&lt;/code&gt; to be non-walkable and non-bikable. This is documented in the &lt;a href="http://wiki.openstreetmap.org/wiki/OpenTripPlanner"&gt;OpenStreetMap wiki&lt;/a&gt; and the &lt;a href="https://github.com/openplans/OpenTripPlanner/wiki/GraphBuilder#permissions-and-bicycle-safety"&gt;OTP wiki&lt;/a&gt; as well. There are actually multiple ways to go about this then. The first solution that came to mind was to just edit the original OSM XML file.&lt;/p&gt;
&lt;pre class="code literal-block"&gt;sed -i .bak s/trunk/primary/g manila.osm
&lt;/pre&gt;


&lt;p&gt;And rebuild the graph. It doesn't really matter much because the OSM data isn't used to render the maps. It's just used to build the routing data. This is actually what I did for &lt;a href="http://maps.pleasantprogrammer.com"&gt;maps.pleasantprogrammer.com&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;It's also possible to set the default way properties in OTP. Instead of disallowing walking and biking on &lt;code&gt;highway=trunk&lt;/code&gt; we could allow that. This is not much better than the &lt;code&gt;sed&lt;/code&gt; solution though. It's better since you keep the weighting done by OTP, but you're still saying that all trunks are walkable which might not be the case.&lt;/p&gt;
&lt;p&gt;The most correct way to actually fix this is to go through each of the trunks and specifying &lt;code&gt;foot=yes&lt;/code&gt; and &lt;code&gt;bicycle=yes&lt;/code&gt; for those trunks that are actually walkable. You could either do this locally with the dumped data, or contribute it directly to OSM. I'm not sure on the particulars with updating OSM though.&lt;/p&gt;&lt;/div&gt;</description><category>philippine-transit-app</category><category>programming</category><guid>http://pleasantprogrammer.com/posts/highways-in-otp.html</guid><pubDate>Wed, 24 Jul 2013 15:15:57 GMT</pubDate></item><item><title>Elevation Data in OTP</title><link>http://pleasantprogrammer.com/posts/elevation-data-in-otp.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;&lt;img alt="OpenTripPlanner showing elevation data" src="http://pleasantprogrammer.com/galleries/transit/otpelevation.png"&gt;&lt;/p&gt;
&lt;p&gt;One thing I hadn't tested out last time was OTP's support for elevation data. It makes use of this by showing the elevation you have to traverse while walking along the suggested route. It can also take it into account when suggesting bike routes.&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://github.com/openplans/OpenTripPlanner/wiki/FiveMinutes"&gt;5 minute tutorial&lt;/a&gt; actually discusses the elevation data briefly, but a more in-depth thing you can look at is the &lt;a href="https://github.com/openplans/OpenTripPlanner/wiki/GraphBuilder#elevation-data"&gt;GraphBuilder documentation&lt;/a&gt;. It suggests using the ASTER dataset which is free but requires registration. I just opted to use the SRTM data available from the &lt;a href="http://www.philgis.org/freegisdata.htm"&gt;PhilGIS website&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;I don't know about the ASTER dataset, but the PhilGIS data was in the ERDAS img format. OTP only supports GeoTIFF so there was a need to convert it beforehand. You can use &lt;a href="http://www.gdal.org/"&gt;GDAL&lt;/a&gt; for this. You'd just then run,&lt;/p&gt;
&lt;pre class="code literal-block"&gt; gdal_translate srtm41_90m_phl.img phil.tiff
&lt;/pre&gt;


&lt;p&gt;Afterwards, it's just a matter of following the OTP instructions on using a local elevation dataset. The process actually doubled the size of the generated Graph.obj so it might not be ideal if you're running on limited RAM.&lt;/p&gt;
&lt;p&gt;I've actually hosted a &lt;a href="http://maps.pleasantprogrammer.com"&gt;working example&lt;/a&gt;. It's pretty much at the limits of the RAM so it might be slow and unreliable, but you can test it out just for fun. Please don't abuse it though.&lt;/p&gt;&lt;/div&gt;</description><category>philippine-transit-app</category><category>programming</category><guid>http://pleasantprogrammer.com/posts/elevation-data-in-otp.html</guid><pubDate>Tue, 23 Jul 2013 10:23:00 GMT</pubDate></item><item><title>GraphServer</title><link>http://pleasantprogrammer.com/posts/graphserver.html</link><dc:creator>Thomas Dy</dc:creator><description>&lt;div&gt;&lt;p&gt;Link: &lt;a href="http://graphserver.github.io/graphserver/"&gt;http://graphserver.github.io/graphserver/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;One other routing webapp I saw was GraphServer. It's actually more of a general purpose Graph library which supports GTFS and OSM data than an actual dedicated routing software like OpenTripPlanner. It's also based off python and C instead of Java, so it feels a lot less heavy.&lt;/p&gt;
&lt;p&gt;The instructions on the website are already pretty good. There are just some minor errors with it. Where it says &lt;code&gt;gs_gtfsdb_build&lt;/code&gt;, you should actually use &lt;code&gt;gs_gtfsdb_compile&lt;/code&gt;. Also, when running &lt;code&gt;gs_osmdb_compile&lt;/code&gt; you might need to use &lt;code&gt;-t&lt;/code&gt; for tolerant in case you follow the instructions on chopping up the original OSM data.&lt;/p&gt;
&lt;p&gt;A nice suggestion from the GraphServer instructions was to crop the OSM data to minimize the graph size. This is actually quite helpful if you downloaded the entire Philippine OSM dump. It reduced the original 900MB file to 135MB which was a lot more workable. I did hit a problem with their instructions though. The linked version of osmosis is an old one, which doesn't support 64-bit ids. The &lt;a href="http://wiki.openstreetmap.org/wiki/Osmosis"&gt;latest version of Osmosis&lt;/a&gt; easily did the job though.&lt;/p&gt;
&lt;p&gt;The actual routing though, was not exactly good. I only tried one route which should normally take 1-2 transfers, it suggested a route which involved 4+ transfers. It also didn't provide any alternate routes aside from that one. I'm not sure if it's a limitation of the provided routeserver, but I didn't bother checking if it supported parameters which might provide better routes.&lt;/p&gt;
&lt;p&gt;I think graphserver could be useful, but it seems more involved than say OpenTripPlanner. There do seem to be people who use graphserver for their routing apps, but for the bounds of the contest, or just as a side project, it might require too much effort.&lt;/p&gt;&lt;/div&gt;</description><category>philippine-transit-app</category><category>programming</category><guid>http://pleasantprogrammer.com/posts/graphserver.html</guid><pubDate>Tue, 23 Jul 2013 06:48:29 GMT</pubDate></item></channel></rss>