Do you experience unpredictable workloads in your FME environment? Imagine a solution that allows you to effortlessly scale your processing power up or down based on demand.
Let me introduce you to FME CPU-Usage Engines. A concept that Safe Software rolled out 4 years ago under the name Dynamic Engines, CPU-Usage Engines offer a flexible and cost effective way to handle fluctuating workloads.
Assuming like us, you may not have appreciated the opportunities presented by CPU-Usage Engines, let’s jump in and take a closer look.
How do CPU-Usage Engines work?
FME Flow supports two engine types; the Standard Engines which are the traditionally licensed engines, the type we all know and love and the CPU-Usage Engines. These are licensed by pre-bought packs of credits. Each credit equals one hour of processing time (not idle time or session duration).
Esri users are familiar with credits, but it’s often unclear what they buy. With Safe’s FME, it’s simple: 1 credit equals 1 hour of actual processing time. For example, an FME session that lasts 58.6 seconds would only cost 20.9 credits because you’re only charged for the time the system is actively working on your data. In other words, CPU time plus user time.
In all other respects, CPU-Usage Engines are the same as standard engines with the same functionality and capabilities. However, the key benefit of CPU-Usage Engines is their flexibility. You can dynamically adjust the number of engines based on your workload, paying only for the CPU time used. This eliminates the cost of idle engines. FME Flow administrators can easily manage engine numbers through the FME Flow Web UI.
When to use CPU-Usage Engines
Handling fluctuating demand (expected or unexpected!): Demand can often exceed available capacity, and CPU-Usage Engines can help you scale your resources to meet the demand without overpaying. Perhaps you have a client or public facing web app where usage is unpredictable, CPU-Usage Engines would be perfect for that scenario. Spin up some engines that can deal with the peak demand and remember that you are only paying for CPU time, not idle time
Large data transformations: Need to process massive datasets quickly? Augment your existing environment with additional CPU-Usage Engines to remove job queues and accelerate the process
Monitoring and streaming data: A champion use for CPU-Usage Engines (and FME in general) is for monitoring IoT or streaming data. Streaming requires a dedicated engine which can get expensive during periods of inactivity. CPU-Usage Engines offer a cost-effective alternative.
Building a highly available environment: CPU-Usage Engines can also be used as part of a highly available, fault tolerant FME environment with additional engines ready to go as needed.
You might be wondering how to manage job assignments to different Engines. This is where the Queue Control functionality comes in. It allows you to create job routing and Engine assignment rules to direct jobs efficiently. For example, you might like to say that jobs will only be run on CPU-Usage Engines if Standard Engine queues are full, or you might base a rule on the amount of CPU time required to run a job.
It is also possible to use the FME Flow REST API to monitor the number of jobs waiting in each queue and then re assign engines to queues with the greatest job load. With a bit of careful planning mixing Standard Engines with CPU-Usage Engines is a fantastic way to optimise the FME Platform.
If CPU-Usage Engines sounds like a good fit for your organisation, please reach out to Chris Morris of the Locus team to discuss.
