Skip to main content

Amazon Simple Workflow Service

Why SWF?

Amazon Simple Workflow Service (Amazon SWF) helps developers build, run, and scale background jobs that have parallel or sequential steps. You can think of Amazon SWF as a fully-managed state tracker and task coordinator in the Cloud.

If your app's steps take more than 500 milliseconds to complete, you need to track the state of processing, and you need to recover or retry if a task fails, Amazon SWF can help you.

If you are looking for a low-code visual process flow service to orchestrate AWS services, automate business processes, or build serverless applications, learn more about AWS Step Functions.

Benefits

Amazon SWF promotes a separation between the control flow of your background job's stepwise logic and the actual units of work that contain your unique business logic. This allows you to separately manage, maintain, and scale "state machinery" of your application from the core business logic that differentiates it. As your business requirements change, you can easily change application logic without having to worry about the underlying state machinery, task dispatch, and flow control.

Amazon SWF runs within Amazon’s high-availability data centers, so the state tracking and task processing engine is available whenever applications need them. Amazon SWF redundantly stores the tasks, reliably dispatches them to application components, tracks their progress, and keeps their latest state.

Amazon SWF replaces the complexity of custom-coded process flow solutions and process automation software with a fully managed cloud process flow web service. This eliminates the need for developers to manage the infrastructure plumbing of process automation so they can focus their energy on the unique functionality of their application.

Amazon SWF seamlessly scales with your application’s usage. No manual administration of the process flow service is required as you add more cloud process flows to your application or increase the complexity of your process flows.

Amazon SWF lets you write your application components and coordination logic in any programming language and run them in the cloud or on-premises.

Amazon SWF product details

Using Amazon SWF to manage workflows within your application is easy.

Amazon SWF functionality

Amazon SWF acts as the coordination hub for all of the different components of your application:

  • Maintaining application state
  • Tracking workflow executions and logging their progress
  • Holding and dispatching tasks
  • Controlling which tasks each of your application hosts will be assigned to execute

To use Amazon SWF you simply:

  • Use the AWS Management Console or the Amazon SWF APIs to specify the names of workflows.
  • Use the Amazon SWF APIs to “start” a new workflow, which results in a particular sequence of workflow tasks, called a “workflow execution,” being kicked into action.
  • Use the Amazon SWF APIs from your worker machines (a "worker" is a component of your application which handles specific tasks) to establish the task order, manage conditional flows, and execute loops for a workflow execution.
  • Use the Amazon SWF APIs from your worker machines to request and execute workflow tasks in the cloud or on premises.
  • Monitor the status and progress of workflow executions and their associated tasks in the AWS Management Console.

The sample walkthrough in the AWS Management Console takes you through the steps for registering and running a sample application. Please refer to the Documentation for more information.

Detailed description

Amazon SWF is a fully managed workflow service for building scalable, resilient applications. Amazon SWF provides simple API calls that can be executed from code written in any language and run on your EC2 instances, or any of your machines located anywhere in the world that can access the Internet.

Amazon SWF acts as a coordination hub with which your application hosts interact. You create desired workflows with their associated tasks and any conditional logic you wish to apply and store them with Amazon SWF. Each time you execute a workflow, it is considered a distinct workflow execution. You program your application components to request various tasks to be performed in your workflow execution and Amazon SWF coordinates the tasks getting done, in the order you specified, on your hosts (whether the hosts are located in the cloud or on premises). Applications communicate with Amazon SWF using APIs to record success or failure of tasks. Amazon SWF then either continues the sequence of tasks by assigning the next workflow task to an application host, or by re-running a failed task depending on your business logic.

Log in to the AWS Management Console to start an Amazon SWF workflow execution. Alternatively, you can start a workflow execution via our Amazon SWF APIs. Amazon SWF employs a simple web service interface that is easy to use and highly flexible:

  • StartWorkflowExecution: starts a sequence of workflow tasks and makes the first task available to one of your application hosts.
  • DescribeWorkflowExecution: provides status of your workflow executions and tasks.
  • PollForActivityTask: your application hosts (in the cloud or on premises) request and execute workflow tasks in a continuous loop.
  • RespondActivityTaskCompleted: an application host tells Amazon SWF that it successfully completed a task. Amazon SWF will then continue the workflow execution by making the next task available to an application host.
  • TerminateWorkflowExecution: stop driving a particular workflow execution forward. Amazon SWF will not assign any more tasks of this particular workflow execution to application host.

Intended usage and restrictions

Your use of this service is subject to the Amazon Web Services Customer Agreement.

Use cases

Video encoding using Amazon S3 and Amazon EC2. In this use case, large videos are uploaded to Amazon S3 in chunks. The upload of chunks has to be monitored. After a chunk is uploaded, it is encoded by downloading it to an Amazon EC2 instance. The encoded chunk is stored to another Amazon S3 location. After all of the chunks have been encoded in this manner, they are combined into a complete encoded file which is stored back in its entirety to Amazon S3. Failures could occur during this process due to one or more chunks encountering encoding errors. Such failures need to be detected and handled through Amazon SWF's cloud process flow management.

Migrating components from the datacenter to the cloud. Business critical operations are hosted in a private datacenter but need to be moved entirely to the cloud without causing disruptions. Amazon SWF-based applications can combine workers that wrap components running in the datacenter with workers that run in the cloud. To transition a datacenter worker seamlessly, new workers of the same type are first deployed in the cloud. The workers in the datacenter continue to run as usual, along with the new cloud-based workers. The cloud-based workers are tested and validated by routing a portion of the load through them. During this testing, the application is not disrupted because the workers in the datacenter continue to run. After successful testing, the workers in the datacenter are gradually stopped and those in the cloud are scaled up, so that they move entirely to a cloud process flow management application. This cloud process flow process can be repeated for all other workers in the datacenter so that the application moves entirely to the cloud. If for some business reason, certain processing steps must continue to be performed in the private data center, those workers can continue to run in the private data center and still participate in the application.

Processing large product catalogs using Amazon Mechanical Turk. While validating data in large catalogs, the products in the catalog are processed in batches. Different batches can be processed concurrently. For each batch, the product data is extracted from servers in the datacenter and transformed into CSV (Comma Separated Values) files required by Amazon Mechanical Turk’s Requester User Interface (RUI). The CSV is uploaded to populate and run the HITs (Human Intelligence Tasks). When HITs complete, the resulting CSV file is reverse transformed to get the data back into the original format. The results are then assessed and Amazon Mechanical Turk workers are paid for acceptable results. Failures are weeded out and reprocessed, while the acceptable HIT results are used to update the catalog. As batches are processed, the system needs to track the quality of the Amazon Mechanical Turk workers and adjust the payments accordingly. Failed HITs are re-batched and sent through the pipeline again.

AWS Flow Framework

The AWS Flow Framework is a collection of convenience libraries that make it faster and easier to build applications with Amazon Simple Workflow.

How it works

The AWS Flow Framework has a straightforward programming model that makes it easy to build applications that perform work across many machines. With the framework, you quickly create tasks, coordinate them, and express how these tasks depend on each other -- as you would do in a typical program. For example, you can run a method in an application on a “remote” computer simply by calling a method in your application logic that is hosted on a separate “local” computer. The AWS Flow Framework takes care of the complex back-and-forth needed to execute the remote method and returns its result to the local application by using information that is stored by the Amazon Simple Workflow service. The output of any executed method can be used to connect separate parts of your logic that depend on each other. The framework allows you use straightforward syntax to express dependencies between methods with a simple “block and wait for a callback” approach. The framework also lets you handle a failure on a remote machine as if it were a local error and gives you easy ways to define how you’d like to retry important methods in your application if they happen to fail.

Note

The AWS Flow Framework for Ruby is no longer under active development. While existing code will continue to work indefinitely, there will be no new features or versions. The source code of the Flow Framework for Ruby will continue to be available on the AWS SWF GitHub archive. If your Ruby-based Simple Workflow application is evolving and changing, there are options you may want to consider, including migrating to AWS Step Functions or using the native Simple Workflow API. For details, see the AWS Simple Workflow Developer Guide.

Using the AWS Flow Framework, you write simple code and let the framework’s pre-built objects and classes handle the details of Amazon Simple Workflow APIs. The AWS Flow Framework takes care of creating and executing your application’s steps, keeps track of their progress, lets you define retry rules for when steps fail, and much more. Meanwhile, the Amazon Simple Workflow Service maintains your application’s execution state, hands out tasks to available workers, and keeps an audit history of all of your tasks.

The AWS Flow Framework is Java open source, and developed and managed by AWS. You can use the links below to get libraries, source code, documentation, samples, and more.