FME Flow is commonly used to integrate business systems through data transformation processes run on a schedule. Often synchronising the data repositories of different applications such as GIS, Asset Management, Customer Relationship Management (CRM) etc on an hourly, daily, or weekly basis is acceptable, however, FME Flow supports alternate usage patterns that may be a better fit for some scenarios.
This blog post describes at a high level a pattern where FME Flow provides the application service/middleware layer for a user-facing web app built using Esri’s ArcGIS Web AppBuilder (WAB). The result is low-code, Just-In-Time (JIT) integration between GIS and any other format that FME can connect to, either natively or via an API.
Architecture
The design discussed in this post is a standard four-tier web solution architecture that includes a Web Client, a Web Service backend, an Application Server, and a Data Layer that consists of one or more databases as well as internal/external web services that provide additional data, or perform supplementary work, as and when required.
Each tier delivers important functionality to the complete solution:
- Web client allows users to view and interact with data.
- Presentation tier/web service backend hosts the user-facing application and provides additional services such as authentication. This tier also acts as a façade for the business logic tier to decouple it from the client, to enhance security as well as improve maintainability of the business logic tier.
- Business logic tier interacts with the data tier on behalf of the client. In a lot of cases, it only interacts with a single domain-specific database, but in more complex scenarios it can interact with multiple databases and/or third-party APIs.
- Data tier manages the information viewed and changed through the client.
Utilising the applications available in many of Locus’ clients’ GIS stacks, the above diagram can be modified as shown below.
Client tier (Web AppBuilder)
Esri’s no-code solution for building secure, map-centric web applications. While it is considered a legacy app now and will eventually be replaced by the new Experience Builder application, it is currently the only option for apps that require the Geoprocessing widget.
The out-of-the-box (OOTB) geoprocessing widget can connect to an ArcGIS Enterprise Geoprocessing service and automatically knows how to present its input parameters as a web form, without the need for custom code.
Web service tier (ArcGIS Enterprise)
ArcGIS Enterprise delivers different types of web services to support desktop, web, and mobile workloads. Map services serve images that provide context in a map while Feature services allow users to interact with individual map objects (features) and support both reading and writing of information.
Geoprocessing services support implementing custom functionality not available through OOTB tools and can be built in different ways, e.g. via a Python toolbox. As Geoprocessing services are a regular service type in the Esri platform they can be easily secured and shared with an intended audience using standard ArcGIS Enterprise functionality. This allows us to use a geoprocessing service as a secure façade for FME Flow without having to expose FME Flow tokens to the client or manage a pool of users and their permissions that mirrors the identities already available in ArcGIS Enterprise.
The main goal for the geoprocessing service is to keep the amount of custom code to a minimum. Rather than using ArcGIS geoprocessing tools or third-party Python modules to process data, the only tasks the geoprocessing service should carry out are:
- Parse input parameters provided to the web client by a user
- Call FME Flow with some or all of the parsed input parameters
- Inspect the FME Flow response and send results back to the client
Limiting the geoprocessing service to only these tasks means that each of its tools (operations) can be implemented with only a few lines of code.
Business logic tier (FME Flow)
FME Flow provides two types of securable web services that can be called by clients to execute workspaces built in FME Form. Clients can control workspace behaviour at runtime by passing information to FME Flow as part of the call.
The JobSubmitter service accepts GET requests and can map URL parameters to workspace parameters, the DataStreaming service can also accept client data submitted via a POST request body.
Which of these two services to use depends on functional requirements as well as overall FME Flow configuration. Where a simple fire-and-forget pattern is sufficient, e.g., users of the client app don’t need to know about the processing result immediately, or where the associated workspace takes a long time to execute (minutes or more), using the JobSubmitter service with published parameters is an easy solution.
In cases where more data needs to be sent than can be managed via URL parameters, or where the client needs to use workspace results, e.g., to make them available to the user or perform further work, the DataStreaming service is a better option. This service type acts just like a web service written in .NET (or a similar framework), but with no custom code. While very powerful, this approach should only be used for workspaces that execute quickly (seconds) and with FME Flow installations that reliably have engines available for ad-hoc requests.
Data tier
Detail is solution-specific, but a common pattern we’ve implemented is one or more geodatabases and one or more web services/APIs, either internal to the organisaton or third party. As an example Microsoft’s Power Automate provides a very quick and easy way of building a secure web service end point that can take a JSON payload and perform work against any of the many applications it can natively connect to, e.g. SharePoint, Dynamics CRM etc.
In Conclusion
This blog post explored at a high level a generic architecture for building low-code custom solutions based on Esri and FME technology. While conceptually simple, each solution that implements this pattern will have specific requirements that require careful consideration.
Additionally, the technical limitations of the GIS stack (and wider IT environment) need to be taken into account as they may dictate the approach.
Contact Locus for more information or to discuss a design and implementation approach
