Tynamo is model-driven, full-stack web framework based on Apache Tapestry 5

Ever since Tapestry 5.5 was released late March, I had been thinking that I really should update the Tynamo libraries and fix the issues that had slowly been piling up. However, it was really thanks to performance issues with Spring ACL in an older JSP webapp that caused me to pick up the open source hammer and start hammering. If you know Spring ACL, you know that its ACL rules are detached from the domain data. So, what happens is that a few tens of thousand of data rows can quickly balloon to several million of ACL join table rows with all combinations of individual (read, update, etc.) instance permissions. Worse yet, the ACL rules are often (especially for row level access) matched post data query. Figured it was a great case to demonstrate how it could be rewritten better, so I busted open my trustworthy Tynamo toolbox and pulled out tapestry-security-jpa from there. Obviously, the tools needed a bit of sharpening and some fixes here and there, but I was generally happy to find out how solid and innovative some of the ideas in them were and still are (perhaps an old school JSP/Spring app is not much of a competition, but still).

Another library I was happy to update was tapestry-resteasy. Alejandro had once upon a time made a swagger integration for it, but it was left rotting there in src/test/java. Swagger had moved ahead by a few generations (to v3) but it wasn’t too hard to lift the old code, update it to work with the latest and package it up as a separate module, tapestry-resteasy-swagger, that can be used by simply adding it as a dependency to your project. You’ll get the swagger UI in addition to (v2) openapi.json.

Updating tapestry-security was also a no-brainer, but functionally I really didn’t need to do much, mainly it was matter of updating dependencies and cleaning up some obsolete code paths. It’s updated to Shiro 1.5.2 as a bonus.

All in all, I made no less than seven Tynamo releases, all updated to work with Tapestry 5.5:

• tapestry-conversations 0.3.0
• tapestry-jpa-seedentity 0.2.0
• tapestry-security 0.8.0
• tapestry-security-jpa 0.2.0
• tapestry-resteasy 0.6.0
• tapestry-resteasy-swagger 0.0.2
• tynamo-test 0.2.1

Not bad for a weekend’s worth of work, enjoy :)

Recently, I had a chance to work with websockets in a Single Page Application (SPA) that visualizes incoming real-time data. To build the SPA, I used re-frame, my new favorite SPA client framework in ClojureScript. For websockets, I had several choices but zeroed in on Sente (written in amazingly compact crossover code to make it work both in ClojureScript as well as in Clojure), a thin abstraction layer that takes care of detecting and re-establishing the connection and downgrading to XHR polling if needed. If you don’t know anything about re-frame, I’ll give you the basics - it’s like an event bus or a loop where dispatched UI events cause a computational change in the atomic state of the data (literally, the whole state is kept in one giant hashmap). The UI components then subscribe to react (it’s all React/Reagent based) to specific branches/sections in the state of the data. As the re-frame folks almost euphorically state, it’s quite beautiful really.

Anyhow. What’s even more euphoric, is that after some amount of coding, I realized that the signature of the Sente messages and the re-frame events are almost exactly the same. There’s really no reason for me to have an additional layer to translate a dispatched Sente message from the server before re-dispatching it as a re-frame event. I might just as well dispatch any incoming websocket message as a re-frame event, therefore making an “outer event loop” (see graph below). .

Since the project had a lot of real-time, in-memory data, I figured I can just as well use re-frame on the server as well. So now I have an atomic re-frame state going on both on the client and on the server which immensely pleases my desire for symmetry. Although, I obviously need to be a bit careful of what I stick in the server state because it holds the state of all of the clients at the same time. Not that different from any multiplayer online game though.

SPAs always have some rarely changing data bits that in classic request-response based implementations are typically fetched only once and even if those bits are altered afterwards, the changes are not refectled on the UI after the app is initialized. However, with a two-way communication through the “outer loop” (to the server and back), the cost of propagating the new state to the UI is effectively zero even with any rarely changing data, since the initialization logic is minimal and there’s no reason to aggregate the responses together. As an experiment, log on to your favorite webapp from two different devices, change your name on one and see if the change gets reflected on the other devices.

What I’m advocating is that SPAs should take another look at websockets which I feel is still an underutilized part of the HTML5 spec. Scalability issues with websockets are well understood but you should not see it as binary choice between XMLHttpRequests (XHR) and websockets but the latter as complementary to standard HTTP request/response cycles. If the communication through the “active state” of the webapp was handled through websockets instead of XHR, the programming logic would be vastly simpler. Success/error handlers for most XHR requests are like checked exceptions - you are forced to handle them but often don’t know how to make the error handling actually meaningful - or it may too expensive to handle them locally. Instead, if you establish a communication channel just when you need it, you can tear it down right after and you only need one central location to handle communication channel breakdowns. For the rest of the part-of-the-task messages, you can just optimistically assume they go through. Adding another technology adds complexity to but depending on the application, the gained simplicity in programming logic might well be worth it. Don’t you think?

Two years ago I read a blog post about some guy writing a Tetris webapp with Clojure in two hours and I remember thinking there’s no way. There’s simply no way I can develop anything that complete in just two hours. But it was right then and there I decided to pick up Clojure. In the last 12 months, I’ve not only developed, but published three Clojure/Script projects and I can confidently say I’m able to develop at the same pace I’m designing an application.

One of the projects I developed is Coachaide. In all honesty, it did take a bit longer than two hours to “complete”, but I’ll use it as an example here because it’s still a simple hobby project. I had written the first, client-side/in-browser only version as an experiment in two weeks last Summer. At the beginning of the year, I had some time to kill, so I added touch gestures, simplified and reworked the UI, generalized the concept to support multiple game types, added multiple balls and wrote user management and a whole server side for it, complete with Postgres backed database in three manic weeks (thanks family for putting up with me!).

The world is of course full of scrappy little tacticboards, but far fewer of more intelligent user interfaces like the one in Coachaide. It understands passes, tracks players positions in time, works online and offline (because it aggressively caches all resources on the client thanks to the programmable cache of a progressive webapp), scales to any screen size (go ahead and try resizing your browser) and works on all modern devices with touch or mouse interfaces. Certainly you can implement the same with older and more conventional web technologies, but the effort required for a free-to-use tool is too big and too costly, which is why so many of these hobby projects stay in the drawer, never quite reaching the “good-enough-to-publish” state.

For another example, let’s compare two competing implementations I had developed for the same unnamed, UI-heavy project. Earlier, I had already implemented the project as a complete Tapestry app with Ember frontend. It took me months to write it (although that includes design). For practice, I rewrote it with Clojure in less than two weeks (obviously it’s far easier when you know what the goal is). These two web apps are functionally almost fully equivalent except I had added a few more bells and whistles in the Clojure version - just because I could.

Lines of code - Tapestry version with Ember

(output by github.com/AlDanial/cloc v 1.72 from “src” using “–exclude-dir=assets” to remove Ember .js files)

Lines of code - Clojure/Script version with Reagent

(output by github.com/AlDanial/cloc v 1.72 from “src” and “resources” combined)

As a summary, reaching the equivalent functionality (and more) required less than 1/3 of the lines of code in the Tapestry implementation, and if you’ve ever written Tapestry, you know it’s fairly succinct for Java. The lines of code is still hardly the right measurement stick, but the development time is. It’s hard to explain why it’s so much faster to develop in pure Clojure if you haven’t experienced it yourself. While the stateless nature of functional programming is clearly part of it, the advantages go far beyond it. One big part is there is no constant switching between the template and the code, but just one view, as provided by Reagent (the better React). Like this guy described it: “Clojure fits into the React model like toothpaste in tubes and the community has switched almost universally … to one of the React wrappers”.

Another one of those things is that I feel don’t have bugs anymore, I just have logical errors :) It may be a bit difficult to grasp the higher order functions at first but like they say, it’s very easy to reason with Clojure code. Since you are mainly dealing with pure values, there’s barely any null checking or other major sources for classic bugs in object-oriented world, like non-initialized values, missing references or stale object state.

Also, did you see that ClojureC in the cloc summary? That’s Clojure crossover code, same code that’s compiled both to Java and Javascript. Except for node.js and Scala.js, there’s really nothing out there that allows you to write the same code and then let you decide whether you want to run it on the client or on the server or both.

But I digress. Expectations for good user interfaces have risen at the same time as it’s become easier to build them. It’s gotten more common to require a single, responsive UI that works on all devices instead of building a different interface for each each platform and screen format. Usage patterns have also changed. https://www.coachaide.com sees northward of 70% of the userbase from mobile and I expect that trend to only accelerate. Increased usage of mobile requires more design effort because screen real estate on mobile is so limited. Mobile usage also requires more tapping and less typing.

Personally, I’ve never really learned to love developing user interfaces. Back-end algorithms (I’ve developed a few fairly complex ones over the years) often seem so much simpler to me. There are fewer possible inputs and once you get your algorithms right, they might just work for years without requiring any fixing or updating. In the user interface, there are always too many ways to achieve the same goal or present the same data and I’m never really quite sure what the right answer is - whether it’s positioning elements, choosing colors etc. That, together with typical issues building user interaces with conventional object-oriented/component-based patterns and the usual problems associated with local state, UI dispatches, notifier patterns etc. have always been a turn off for me.

Now though, armed with Reagent and re-frame, I feel confident building arbitrarily complex user interfaces. My pace of development doesn’t seem to suffer like before when I develop the application further and the complexity increases. One of the simple measures is lines of code per file. From 10,000 feet high, the more lines in a file, the more complex it is. One of the files in coachaide.com has over 2000 lines. If it was a class file, I would have refactored that class (file) into smaller classes a long time ago, following a classic composite pattern. But Clojure, and functional languages in general, are remarkably shallow, and this seems to apply both to my own user interface code as well as Clojure ecosystem and supporting libraries in general (for example, consider Ring+Compojure vs servlets+any Java web framework).

As great as this brave, new world of user interface development is, the one thing I’m still wondering is: where are all these great apps? I see surprisingly few successful examples of scalable, responsive, progressive webapps out there. Am I too far ahead of the curve? Are there just too many existing apps out there with expensive resources thrown at them that nobody cares about making stuff better, faster & cheaper?

PS. Don’t worry, haven’t forgotten about Tapestry. I’ll be happy to continue maintaining Tynamo libs even if I dabble with other stuff every once and awhile :)

PPS. Originally, there were no “neutral” players in Coachaide. It took me 30 mins to implement the end-to-end functionality, from the button in the UI to persistence changes for storing the new info in the DB.

PPPS. I hadn’t planned on localizing Coachaide but after finding out about tempura, I decided to give it a go, and ended up internationalizing and localizing it to Finnish over one weekend. Thanks to my neighbor Laura, there is also Spanish translation!

PPPPS. Before publishing this, I decided to implement full-text search for Coachaide. That took a bit longer, but just for the heck of it, I implemented both remote and local (as in client-side) full-text searches for it!

It’s that time of the year again. Actually, the community has been asking these to be released for a while. Thanks to @homburgs, you can now contribute chains with RegExPatternMatcher to tapestry-security. Also, 0.7.1 is now using the latest version of Tapestry and I cleaned up a few annoying default configuration errors that used to pop up in the logs. See docs for tapestry-security for more.

Tapestry-resteasy 0.5.0 now comes with the latest version of Tapestry and resteasy (3.1.4.Final) as well. Thanks to @jochenberger for tirelessly keeping the versions in sync. See docs for tapestry-resteasy for more.

As a side-note, while our Maven & OSS based release process has worked well for the community for years, it does feel clunky and almost pointless for those of us who’ve ever used Bintray’s Github-based release “process”. Using Bintray also nicely solves the problem of building snapshots since you can use ant git(hub) hash as a version in case you need access to some unreleased features. I don’t know if I have appetite to make changes to move over though, given that most of our libraries are pretty mature and slow moving at this point. Anyhow, something to keep in mind in case we come up with a brand new module.

Enjoy!

After an extended hiatus, I figured it’s time to do some releases. First up is the old workhorse, tapestry-security. The 0.7.0 release represents a true community effort, my job was mainly to merge pull requests from multiple authors. Thank you @jochenberger, @fkretzer, @pedrocborges2, @ascandroli, @bobharner! We are all updated to use the latest Shiro and Tapestry 5.4, with a host of important fixes and even a few new new features. To highlight one of them, see #40 add facility to remove/add/update security chains at runtime, from @jochenberger. Note that there was one breaking change made as part of this issue - The contribution to SecurityConfiguration was changed to OrderedConfiguration, so you need to name all your contributions, see examples in the tapestry-security documentation. Thanks to all and enjoy!

See release notes at Github

While Dokku, containerization and the like steal all of the deployment hype these days, the good old skinny wars are still alive and kicking. I was recently giving CPR to maintaining an older Tapestry 5 app that was moved from one of those ultra-cool but expensive PaaS platforms to an el cheapo VPS server (because nowadays, they offer just ridiculously good price/performance ratio and for whatever reason, the app didn’t achieve world domination at its time).

The PaaS had its own proprietary deployment API but the app had everything and the kitchen sink as dependencies so I quickly got tired of deploying the whole WAR when less than 1% of it had really changed. Since I’ve done plenty of skinny war deployments in the past and this app was in maintenance mode anyway, I figured a skinny WAR was the way to go, only to find out that the deployment script I had copied from another old project just didn’t work anymore in Tomcat 8.

Seems like there’s been lots of internal configuration changes in Tomcat 8. The problem is that there’s so much old information available (when you start googling for answers, nobody ever seems to care about specifying the version of Tomcat their advice is for) and that Tomcat’s own User Guide is hopelessly succinct, lacking any examples for the common cases or references to what and why something has changed.

Instead of utilizing the classloader hierarchy (common/shared classloaders) or using VirtualWebappLoader (which is removed), in Tomcat 8 you can configure external mount points for webapp Context resources. This is clearly documented in the relevant class in Tomcat 8’s documentation, the only problem is you have to find the class first. In any case, this is how you’d do it:

<Context path="/" docBase="ROOT.war" debug="0">
	<Resources className="org.apache.catalina.webresources.StandardRoot">
		<JarResources className="org.apache.catalina.webresources.DirResourceSet"
			base="${catalina.base}/shared/lib"
			webAppMount="/WEB-INF/lib">
		</JarResources>
	</Resources>
</Context>

The base can obviously point to anything. These virtual resources are a bit like symbolic links and clearly offer way more flexibility than the previous methods should you ever need it. Now for Tapestry5 apps, loading the app and its classes from these virtual directories did not work, probably because Tapestry does its own classpath scanning for annotation support etc. I had to package my own application classes into jar and deploy it to /WEB-INF/lib to make it work (for Maven, configure the war plugin with true ). That opens up interesting possibilities, since I really don’t need to even deploy the skinny war but only the application jar to update the app. The problem with that though is that I lose the built-in benefits of parallel and hot deploying WAR files (see more about versioned parallel deployment to Tomcat ) but I guess I’ll just live with it for now.

While the Apache Shiro team (me included) are still mulling over how to best fix a serious security vulnerability with Shiro’s default rememberMe cookies and object deserialization SHIRO-550, I went ahead and implemented a fix for the vulnerability in tapestry-security, following a similar approach I had proposed for Shiro as well. In essence, an attacker can potentially gain access to the server using a deserialization vulnerability (see more at https://www.contrastsecurity.com/security-influencers/java-serialization-vulnerability-threatens-millions-of-applications ). It’s a bit more convoluted for Shiro in general, but once again, service contributions make this a simple matter to handle in Tapestry. In 0.6.4 of tapestry-security, only known principal types can be deserialized. Long, Integer and String (which likely cover 99% of the use cases) are sanctioned by default and users can easily contribute additional principal types to the org.apache.shiro.io.Serializer service, for example:

@Contribute(Serializer.class)
public static void addSafePrincipalTypes(Configuration<Class> configuration) {
	configuration.add(UID.class);
}

You are advised to update immediately. Currently the fix is only available for Tapestry 5.4 but I’ll make other releases available as requested. You can also easily copy the relevant classes from the source tree and override services as needed in your application.

Release notes

• Upgrade Shiro to 1.2.5 #26
• Safely deserialize principals from rememberMe cookie #27
• Update to Tapestry 5.4.1 #28

We aim to serve. A user asked for a T5.4 compatible version of conversations module and we delivered. Upgrading the module was pure joy and it is a testament to the outstanding backwards compatibility record T5 has had throughout its existence. Note that I had written the original version for Tapestry 5.0 while we were still called the Trails framework! Checkout the tapestry-conversations guide for more info.

Release notes

• Update to tapestry 5.4 #1

While working on securing an Ember frontend with Tapestry backend, I noticed that tynamo-resteasy didn’t want to play well with security. I’m fairly surprised this issue surfaced only now because tapestry-security is fairly well battle tested. Anyhow, if you’ve ever had a problem with T5.4 and tapestry-security not being able obtain assigned roles or user identity from the request, it’s time to upgrade.

Release notes

• Update to T5.4-rc-1 #25
• Store ShiroHttpServletRequest to RequestGlobals before processing #24

Finally, we have an officially compatible release tapestry-model for T5.4 available. We had one long running issue with the changes in tapestry core and it wasn’t until T5.4-rc1 before it was all sorted out. The official release note is short but there’s been lots of great functional changes to the core.

Release notes

• Fix the test suite to use changed form element names #4
• 0.5.2 has a dependency on eclipselink #2

Here’s a few screenshots of the brand spanking new tapestry-model in action (just to put down any potential qualms abouts us doing nothing but smoking weed all day long here). Also notice the search options.