thatsmydoing
/
blog


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279
							<!DOCTYPE html>
<html lang="en-us">
<head>
	<meta charset="utf-8">
	<meta name="generator" content="Hugo 0.92.0" />
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<link rel="stylesheet" href="/assets/css/theme.css">
	<link rel="alternate" href="/rss.xml" type="application/rss+xml" title="Pleasant Programmer">
	<script type="text/javascript" src="//use.typekit.net/iwm5axp.js"></script>
	<script type="text/javascript">try{Typekit.load();}catch(e){}</script>
	<title>GTFS Editor - Pleasant Programmer</title>
</head>

<body>
	<header id="header" role="banner">
		<div id="thomas">
			<img src="/assets/img/thomas.gif" alt="DJ THOMAS IN DA HAUS">
			<img src="/assets/img/thomas.png" alt="Pleasant Programmer">
		</div>
		<h1 class="site-title"><a href="/">Pleasant Programmer</a></h1>
		<nav id="menu" role="navigation">
			<ul>
				<li><a href="/pages/projects.html">projects</a></li>
				<li><a href="/posts.html">archives</a></li>
				<li><a href="/tags.html">tags</a></li>
				<li><a href="/rss.xml">rss</a></li>
			</ul>
		</nav>
	</header>
	<div id="container">


<main id="content" role="main">
<article itemscope itemtype="http://schema.org/BlogPosting">
	<h1 class="p-name entry-title" itemprop="headline name">
		<a href="/posts/gtfs-editor.html">GTFS Editor</a></h1>
	<small>
		<span class="dateline">Posted: <time itemprop="datePublished" datetime="2013-07-10">2013-07-10</time></span>
		| More posts about
		
		<a class="tag p-category" href="/tags/philippine-transit-app.html" rel="tag">
			philippine-transit-app
		</a>
		
		<a class="tag p-category" href="/tags/programming.html" rel="tag">
			programming
		</a>
		
		<a class="tag p-category" href="/tags/lets-debug.html" rel="tag">
			lets-debug
		</a>
		
	</small>
	<div class="e-content entry-content" itemprop="entry-text">
		<p>Link: <a href="https://github.com/conveyal/gtfs-editor">https://github.com/conveyal/gtfs-editor</a></p>
<p><strong>TL;DR</strong> they really meant under development</p>
<p>When I first saw the source of GTFS Editor, I was ecstatic. They used <a href="http://playframework.com/">Play framework</a>!!! Not only that, they&rsquo;re targeting PostgreSQL as the main database. Those are our favorite tools for building webapps at By Implication. I was a bit sad though, when I saw it was on the 1.x release of Play though. I did have some experience with that release, but not as much compared to 2.x.</p>
<p>Getting it to actually run though, wasn&rsquo;t very pleasant. The initial setup was easy enough. Get <a href="http://www.playframework.com/download">Play 1.2.5</a>, install Postgres with PostGIS, clone the repo and create backing database in Postgres. Some minor additional steps you need are to create the PostGIS extension on the database. The schema is automatically generated and applied by Play so that should be all that&rsquo;s necessary. Wonderful. Then, run play, open a browser, go to <a href="http://localhost:9000">http://localhost:9000</a>, compilation error. Fantastic.</p>
<p>If you don&rsquo;t want to go through the technical details, you can just jump to the <a href="#conclusion">conclusion</a>.</p>
<h2 id="lets-debug">Let&rsquo;s Debug!</h2>
<p>I&rsquo;ll be splitting the next section up into 2 parts. In the first pass, I&rsquo;ll talk about what I did to just get the app to run but I won&rsquo;t try hard to fix any bugs. This generally is what I do when I try to get apps to run. I&rsquo;ll also be dropping enough information so that you can actually figure out what the real problem is. In the second pass, I&rsquo;ll explain what the problems were and how I fixed them.</p>
<h3 id="first-pass">First Pass</h3>
<p>A thing to note about Play (and one of the reasons it&rsquo;s a lovely Java framework) is that you don&rsquo;t need to do manual compilation. Just edit some source files, refresh your browser and it will automatically do the compilation for you. One less argument for using PHP. It even shows you (in the browser!) the source and which line of code caused the compilation error. So that&rsquo;s what I saw, <code>Error: type Check already defined</code></p>
<div class="highlight"><pre tabindex="0" style="color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">@Retention(RetentionPolicy.<span style="color:#007f7f">RUNTIME</span>)
@Target({ElementType.<span style="color:#007f7f">METHOD</span>, ElementType.<span style="color:#007f7f">TYPE</span>})
<span style="color:#fff;font-weight:bold">public</span> @interface Check { <span style="color:#007f7f">// error here
</span><span style="color:#007f7f"></span>
    String[] value();
}
</code></pre></div><p>You also know that typical behavior among programmers where your program doesn&rsquo;t compile, but you keep trying to compile it anyway hoping that it will magically just work. That&rsquo;s what I did, and it actually ran. I couldn&rsquo;t really just let this pass, so I decided to try deleting <code>Check.java</code>. I got another compilation error, <code>Error: type Secure already defined</code></p>
<div class="highlight"><pre tabindex="0" style="color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#fff;font-weight:bold">public</span> <span style="color:#fff;font-weight:bold">class</span> Secure <span style="color:#fff;font-weight:bold">extends</span> Controller { <span style="color:#007f7f">// error here
</span><span style="color:#007f7f"></span>
    @Before(unless={<span style="color:#0ff;font-weight:bold">&#34;login&#34;</span>, <span style="color:#0ff;font-weight:bold">&#34;authenticate&#34;</span>, <span style="color:#0ff;font-weight:bold">&#34;logout&#34;</span>})

    <span style="color:#fff;font-weight:bold">static</span> <span style="color:#fff;font-weight:bold">void</span> checkAccess() <span style="color:#fff;font-weight:bold">throws</span> Throwable {
</code></pre></div><p>At that point, I just decided to just debug it later. It works by just forcing it anyway. So I put <code>Check.java</code> back in and proceeded to just refresh until it compiled and ran.</p>
<p>The next problem is a sort of common thing most webapp developers have to solve one way or another. How do you set up the initial admin account? Phrased a different way, how do I login to this thing? The first thing I tried was just add a user into the <code>account</code> table directly. One problem though was how to set the password correctly. Plaintext obviously wouldn&rsquo;t work.</p>
<p>Another note regarding Play 1.x, it provides the <a href="http://www.playframework.com/documentation/1.2.5/secure">secure module</a> which handles logins and keeping state, you simply need to implement the method <code>boolean authenticate(String username, String password)</code>. It leaves the actual process of verifying the login to the programmer. This can be exploited by just making the method return <code>true</code> and then any login would work. No need to actually set the password. Excellent.</p>
<p>And we&rsquo;re logged in, just in time to encounter a runtime exception. This also works much like compilation errors in Play. It shows a page with the error and the relevant source lines. Now we get, <code>IndexOutOfBoundsException occured : Index: 0, Size: 0</code></p>
<div class="highlight"><pre tabindex="0" style="color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#fff;font-weight:bold">if</span>(session.<span style="color:#007f7f">get</span>(<span style="color:#0ff;font-weight:bold">&#34;agencyId&#34;</span>) == <span style="color:#fff;font-weight:bold">null</span>) {

    Agency agency = agencies.<span style="color:#007f7f">get</span>(0); <span style="color:#007f7f">// error here
</span><span style="color:#007f7f"></span>
    session.<span style="color:#007f7f">put</span>(<span style="color:#0ff;font-weight:bold">&#34;agencyId&#34;</span>, agency.<span style="color:#007f7f">id</span>);
    session.<span style="color:#007f7f">put</span>(<span style="color:#0ff;font-weight:bold">&#34;agencyName&#34;</span>, agency.<span style="color:#007f7f">name</span>);
</code></pre></div><p>Apparently, we need to have an agency. That&rsquo;s generally simple enough. You just manually insert an agency into the <code>agency</code> table. After that&rsquo;s done, we finally have a view of the actual application. It&rsquo;s very Bootstrap-y, but that&rsquo;s just fine. The workflow though, is not perfectly intuitive, but I&rsquo;ll talk about that some other day.</p>
<p>That&rsquo;s not the end of it though, we still have to fix these bugs. The developer obviously didn&rsquo;t have to put up with this when they were working, so what happened? Also, the log is showing some weird things,</p>
<div class="highlight"><pre tabindex="0" style="color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">~        _            _
~  _ __ | | __ _ _  _| |
~ | &#39;_ \| |/ _&#39; | || |_|
~ |  __/|_|\____|\__ (_)
~ |_|            |__/
~
~ play! 1.2.5, http://www.playframework.org
~
~ Ctrl+C to stop
~
CompilerOracle: exclude jregex/Pretokenizer.next
Listening for transport dt_socket at address: 8000
23:32:14,943 INFO  ~ Starting /Users/thomas/Workspace/maps/gtfs-editor
23:32:14,948 WARN  ~ Declaring modules in application.conf is deprecated. Use dependencies.yml instead (module.secure)
23:32:14,948 INFO  ~ Module secure is available (/Users/thomas/.root/opt/play-1.2.5/modules/secure)
23:32:15,830 WARN  ~ You&#39;re running Play! in DEV mode
23:32:15,952 INFO  ~ Listening for HTTP on port 9000 (Waiting a first request to start) ...
23:32:28,792 ERROR ~

@6f02fa9dd
Internal Server Error (500) for request GET /

Compilation error (In /app/controllers/Check.java around line 10)
The file /app/controllers/Check.java could not be compiled. Error raised is : The type Check is already defined

play.exceptions.CompilationException: The type Check is already defined
	at play.classloading.ApplicationCompiler$2.acceptResult(ApplicationCompiler.java:246)
	at org.eclipse.jdt.internal.compiler.Compiler.handleInternalException(Compiler.java:672)
	at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:516)
	at play.classloading.ApplicationCompiler.compile(ApplicationCompiler.java:282)
	at play.classloading.ApplicationClassloader.getAllClasses(ApplicationClassloader.java:426)
	at play.Play.start(Play.java:516)
	at play.Play.detectChanges(Play.java:630)
	at play.Invoker$Invocation.init(Invoker.java:198)
	at Invocation.HTTP Request(Play!)
23:32:31,551 INFO  ~ Connected to jdbc:postgresql://127.0.0.1/gtfs_editor
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/thomas/Workspace/maps/gtfs-editor/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/thomas/.root/opt/play-1.2.5/framework/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
23:32:32,490 INFO  ~ Initializing HBSpatialExtension
23:32:32,492 INFO  ~ Attempting to load Hibernate Spatial Provider org.hibernatespatial.postgis.DialectProvider
23:32:32,494 INFO  ~ Checking for default configuration file.
23:32:32,496 INFO  ~ No configuration file hibernate-spatial.cfg.xml on the classpath.
23:32:34,077 INFO  ~ Application &#39;gtfs-editor&#39; is now started !
23:32:34,151 INFO  ~ Bootstrapping Database...
23:32:34,297 DEBUG ~ select count(*) as col_0_0_ from Agency agency0_ limit ?
play.exceptions.UnexpectedException: Unexpected Error
	at play.vfs.VirtualFile.contentAsString(VirtualFile.java:180)
	at play.templates.TemplateLoader.load(TemplateLoader.java:78)
	at play.test.Fixtures.loadModels(Fixtures.java:174)
	at jobs.BootstrapDatabase.doJob(BootstrapDatabase.java:57)
	at play.jobs.Job.doJobWithResult(Job.java:50)
	at play.jobs.Job.call(Job.java:146)
	at play.jobs.Job.run(Job.java:132)
	at play.jobs.JobsPlugin.afterApplicationStart(JobsPlugin.java:116)
	at play.plugins.PluginCollection.afterApplicationStart(PluginCollection.java:531)
	at play.Play.start(Play.java:547)
	at play.Play.detectChanges(Play.java:630)
	at play.Invoker$Invocation.init(Invoker.java:198)
	at play.server.PlayHandler$NettyInvocation.init(PlayHandler.java:189)
	at play.Invoker$Invocation.run(Invoker.java:276)
	at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:229)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:680)
Caused by: play.exceptions.UnexpectedException: Unexpected Error
	at play.vfs.VirtualFile.inputstream(VirtualFile.java:111)
	at play.vfs.VirtualFile.contentAsString(VirtualFile.java:178)
	... 22 more
Caused by: java.io.FileNotFoundException: /Users/thomas/.root/opt/play-1.2.5/modules/docviewer/app/initial-agencies-data.yml (No such file or directory)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:120)
	at play.vfs.VirtualFile.inputstream(VirtualFile.java:109)
	... 23 more
23:32:34,316 ERROR ~ java.lang.RuntimeException: Cannot load fixture initial-agencies-data.yml: Unexpected Error
23:32:40,989 DEBUG ~ select account0_.id as id15_, account0_.active as active15_, account0_.admin as admin15_, account0_.agency_id as agency9_15_, account0_.email as email15_, account0_.lastLogin as lastLogin15_, account0_.password as password15_, account0_.passwordChangeToken as password7_15_, account0_.username as username15_ from Account account0_ where account0_.username=? limit ?
23:32:40,994 DEBUG ~ select count(*) as col_0_0_ from Account account0_ limit ?
23:32:40,999 DEBUG ~ select nextval (&#39;hibernate_sequence&#39;)
23:32:41,051 DEBUG ~ insert into Account (active, admin, agency_id, email, lastLogin, password, passwordChangeToken, username, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
23:32:41,061 DEBUG ~ select agency0_.id as id24_, agency0_.color as color24_, agency0_.defaultLat as defaultLat24_, agency0_.defaultLon as defaultLon24_, agency0_.defaultRouteType_id as default12_24_, agency0_.gtfsAgencyId as gtfsAgen5_24_, agency0_.lang as lang24_, agency0_.name as name24_, agency0_.phone as phone24_, agency0_.systemMap as systemMap24_, agency0_.timezone as timezone24_, agency0_.url as url24_ from Agency agency0_ order by agency0_.name
23:32:41,175 ERROR ~

@6f02fa9dg
Internal Server Error (500) for request GET /

Execution exception (In /app/controllers/Application.java around line 57)
IndexOutOfBoundsException occured : Index: 0, Size: 0

play.exceptions.JavaExecutionException: Index: 0, Size: 0
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:237)
	at Invocation.HTTP Request(Play!)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.RangeCheck(ArrayList.java:547)
	at java.util.ArrayList.get(ArrayList.java:322)
	at controllers.Application.initSession(Application.java:57)
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:510)
	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:484)
	at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:479)
	at play.mvc.ActionInvoker.handleBefores(ActionInvoker.java:328)
	at play.mvc.ActionInvoker.invoke(ActionInvoker.java:142)
	... 1 more
</code></pre></div><p>After <code>23:32:34</code> is when I get the login page. <code>23:32:40</code> is after I&rsquo;ve logged in.</p>
<h3 id="second-pass">Second Pass</h3>
<p>So how did you do? First, the error that <code>type Check already defined</code> usually does mean that <code>Check</code> was already defined elsewhere. Looking in the app folder though, there was nothing of the sort. It&rsquo;s the only one there that was <code>Check.java</code>. But remember the secure module? Modules work by providing source files and Play just compiles them all together. Bingo, <code>Check.java</code>. Doing a diff shows nothing was changed. So the solution really was just simply delete <code>Check.java</code> and also <code>Secure.java</code>. No more compilation errors!</p>
<p>The next question is, how do you get the initial user? There actually is some code that looks like it creates the default admin user,</p>
<div class="highlight"><pre tabindex="0" style="color:#e5e5e5;background-color:#000;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#fff;font-weight:bold">if</span>(Security.<span style="color:#007f7f">isConnected</span>()) {
    ...
    Account account = Account.<span style="color:#007f7f">find</span>(<span style="color:#0ff;font-weight:bold">&#34;username = ?&#34;</span>, Security.<span style="color:#007f7f">connected</span>()).<span style="color:#007f7f">first</span>();
    ...
    <span style="color:#fff;font-weight:bold">if</span>(account == <span style="color:#fff;font-weight:bold">null</span> &amp;&amp; Account.<span style="color:#007f7f">count</span>() == 0) {
        account = <span style="color:#fff;font-weight:bold">new</span> Account(<span style="color:#0ff;font-weight:bold">&#34;admin&#34;</span>, <span style="color:#0ff;font-weight:bold">&#34;admin&#34;</span>, <span style="color:#0ff;font-weight:bold">&#34;admin@test.com&#34;</span>, <span style="color:#fff;font-weight:bold">true</span>, <span style="color:#fff;font-weight:bold">null</span>);
        account.<span style="color:#007f7f">save</span>();
    }
    ...
}
</code></pre></div><p>You can actually see this in action at <code>23:32:41,051</code> in the log. So what&rsquo;s wrong with all of this? The account creation happened after I&rsquo;ve already logged in. In fact, <code>Security.isConnected()</code> checks whether the user is already logged in or not. How does this even make sense?</p>
<p>Lastly, we have the problem of the agencies. Just by looking at the log, you can safely say we&rsquo;re missing a file called <code>initial-agencies-data.yml</code>. Ok, apparently it&rsquo;s a <a href="http://www.playframework.com/documentation/1.2.5/test#fixtures">fixture</a> like you would use for testing. It&rsquo;s easy enough to infer what the file&rsquo;s contents should be. We just copy it over from the GTFS data.</p>
<p>But then where do you put the file? If you look at the log, it says <code>/Users/thomas/.root/opt/play-1.2.5/modules/docviewer/app/initial-agencies-data.yml</code> but that doesn&rsquo;t look right. That&rsquo;s in the Play distribution directory, probably not somewhere something app-specific should go into. Well, a fixture is used for testing, so maybe the <code>test/</code> directory? No, that doesn&rsquo;t work either since we&rsquo;re not running a test.</p>
<p>What I ended up doing was just looking at the sources for <code>Fixtures.load</code>. If you follow the stack trace, you end up finding <code>Play.javaPath</code> which sort of works like PATH for Fixtures and some other things. So where can we put the file? <code>app/</code> and <code>conf/</code>. And with that, we&rsquo;re done.</p>
<!-- raw HTML omitted -->
<p>GTFS Editor is very much in development. Just getting it to run was problematic. There also seem to be a lot of missing issues judging from the Github Issues page. If you want to try it out for yourself, I suggest you clone <a href="https://github.com/thatsmydoing/gtfs-editor">my branch</a> as I&rsquo;ve fixed the issues discussed earlier. The default login is <code>admin:admin</code>.</p>
<p>Even after getting it to run, it&rsquo;s still not quite usable. Not in the UX sense, but you really can&rsquo;t do much with it. There is no way to import the GTFS data into the webapp. There is something like import from TransitWand but even that is unclear to me. And even if we do get that running as well, we still don&rsquo;t have any data we can play around with. We would need database dumps from the already running tools for these to be of any use right now.</p>

	</div>
	<aside class="postpromonav">
		<nav>
			<ul class="pager clearfix">
				
				<li class="previous">
					<a href="/posts/one-bus-or-maybe-jeep-away.html" rel="prev" title="One Bus (or maybe Jeep) Away">&larr; Previous post</a>
				</li>
				
				
				<li class="next">
					<a href="/posts/fare-data.html" rel="next" title="Fare Data">Next post &rarr;</a>
				</li>
				
			</ul>
		</nav>
	</aside>
	<section class="comments">
		<script
	data-isso="https://isso.pleasantprogrammer.com/"
	data-isso-require-author="true"
	data-isso-vote="false"
	src="https://isso.pleasantprogrammer.com/js/embed.min.js">
</script>
<section id="isso-thread"></section>

	</section>
</article>
</main>


	<footer id="footer" role="contentinfo">
		<p>
		<a rel="license" href="http://creativecommons.org/licenses/by-sa/3.0/deed.en_US">
			<img alt="CC-BY-SA" style="border-width:0" src="https://licensebuttons.net/l/by-sa/3.0/80x15.png">
		</a> &copy; 2022 Thomas Dy - Powered by <a href="http://gohugo.io">Hugo</a></p>
	</footer>
</div>

<script src="/assets/js/konami.js"></script>
<script>
var easter_egg = new Konami();
easter_egg.code = function() {
	var el = document.getElementById('thomas');
	if(el.className == "whoa") {
		el.className = "";
	}
	else {
		el.className = "whoa";
	}
	document.body.scrollTop = document.documentElement.scrollTop = 0;
}
easter_egg.load();


</script>


</body>
</html>