When to use PythonCaller (or RCaller or TclCaller) with FME

According to Wikipedia, "a domain-specific language (DSL) is a computer language specialized to a particular application domain." As a DSL (among other things), FME is incredibly productive when it's used to solve the key problems in its domain: computation and conversion of geospatial and other data features. In my experience, it's roughly 3 to 5 times faster to implement such a solution with FME than with a traditional general-purpose programming language...even a relatively productive one like Python. Its pipe and filter architecture is ideal for the flow of features, modifying, splitting and merging them from their sources to their destinations.

But often in the course of my work, I find myself facing a challenge that FME was just not designed to address. That's fine, as every tool has its uses and limitations. Any tool that tries to be all things to all people usually winds up doing nothing at all well. That's why FME's PythonCaller, RCaller, and TclCaller transformers exist: to allow a workspace author to dip into a full-blown programming language environment whenever the genuine need arises. With every release, FME becomes more powerful, handling more use cases that once required programming. Yet I still encounter plenty of situations where some programming is necessary, or at least is the better option. Here are three broad categories of such situations.

Attribute processing with dynamic schemas

I often use dynamic readers and writers in workspaces and transformers that need a lot of flexibility to handle frequently changing data schemas. However, to process each attribute individually using standard FME transfomers, one has to split a feature into multiple copies (one per target attribute), apply the necessary transformers to that attribute, then merge the copies back together to produce the required output features Exploding and reassembling lists only multiplies the madness. The result is usually an unintelligible Rube Goldberg contraption with terrible runtime performance.

This is where programming logic shines. The code in a PythonCaller can use all the power of the language's control structures to process the individual attributes in any fashion that is required, in place, cleanly and efficiently. No need to bust a feature into umpteen copies, work with the attributes, then put it all together again. Additionally, the logic can handle dependencies between different attributes much more cleanly than under the split-manipulate-and-merge scenario.

Complex intra-feature manipulation logic needs

Beyond the challenges of dynamic schemas, the "control flow" structure of Python and similar languages lends itself very well to manipulating attributes, lists, and geometries with complex interdependencies. Trying to do the same with FME's pipe-and-filter "data flow" architecture would be an insane effort of mental acrobatics...not only to build it, but also to maintain it. Who wants to twist themselves into a pretzel merely to avoid a little programming?

This is a problem of mismatched control paradigms that signals the need to switch tools. In computer science, there is a similar concept called impedance mismatch, which normally refers to the challenges of taking the object-oriented data structures many programming languages use to hold data in memory, and "mapping" them onto the different paradigm relational databases use to store data on a persistent medium such as a disk. However, this term is often applied to other situations where there's a fault line between two different paradigms, requiring a "mapping" between them. Luckily, FME makes shifting between the "data flow" and the "control flow" paradigms relatively easy, thanks to the existence of the PythonCaller, RCaller, and TclCaller transformers.

Anything FME flat-out can't do

Finally, if there's something FME just can't do, either because the capability hasn't yet been implemented by the good folks at Safe Software, or because it's something FME doesn't (and perhaps shouldn't) do by design, a PythonCaller or one of its brethren might just fit the bill. For example, I've recently had to interact with web services that don't follow standard REST principles. Python to the rescue! In fact, I've used third party Software Development Kits in Python that make quick work of connecting to and using advanced APIs. I'm sure there are plenty of other possible examples, but this should be enough to get the point across.

The Upshot

This is why engaging an FME author with software engineering skills is so critical. And if you are an FME author and would like to acquire or expand your programming skills, you should strongly consider proceeding. That being said, it's still best to dip into traditional programming only when and where necessary, if only because FME is so very powerful and productive for the major portions of data solutions in its domain. But please don't turn an attempt to avoid all programming into a circus act!