Release Highlights

Schema changes

Before upgrading to your Temporal Cluster to this release 1.16, you must upgrade your storage schema version to the following:

MySQL schema version 1.8
PostgreSQL schema version 1.8
Cassandra schema version 1.7

Use the Schema Upgrade Tool to upgrade your schema version. How to use the Schema Upgrade Tool: Upgrade Server version.

Note that schema version is NOT the same as the storage type (database) version, see “Latest SQL database support” below.

Docker Version

This version (1.16) will be the last version that we build our images based on Alpine 3.13. In the next release (1.17), we are going to upgrade our base image to use Alpine 3.15, which may not be able to run on docker versions lower than 20.10.0. Please check more details here.

Golang version

Upgraded golang to 1.18

Tctl

tctl implementation has been moved to a new repository

API

New Operator service for managing namespace and search attribute
New DeleteNamespace API in OperatorService. Please note that the delete namespace feature is still under active development and not production ready
New TERMINATE_IF_RUNNING workflowIdReusePolicy. If a workflow is running using the same workflow ID, this policy will terminate it and start a new one; otherwise, the behavior is the same as ALLOW_DUPLICATE
RequestCancelWorkflowExecution and RequestCancelExternalWorkflowExecutionCommand now supports reason field. The reason will also be recorded in WorkflowExecutionCancelRequestedEvent and RequestCancelExternalWorkflowExecutionInitiatedEvent
NonRetryableErrorTypes now supports TemporalTimeout:StartToClose and TemporalTimeout:Heartbeat for customizing retry behavior upon timeout failure
New last_heartbeat_details field in RespondActivityTaskFailedRequest and RespondActivityTaskFailedByIdRequest
New activity_failure_include_heartbeat system capability indicating if RespondActivityTaskFailed API supports including heartbeat details
New ListHistoryTasks API in admin service for listing transfer, timer, replication and visibility tasks
New RebuildMutableState API in admin service

Configuration

New frontend.namespaceRPS.visibility and frontend.namespaceBurst.visibility dynamic config for per instance visibility API rate limit
Removed per service Metrics config
New RemoteClusters field for controlling TLS settings for takling to remote cluster
New SKIP_DB_CREATE environment variable for skipping DB creation in auto-setup
If you use SQL persistence layer, please add the following configuration to your dynamicconfig file so that the ID length limit matches the actual limit used in sql schema. A sample configuration file can be found here
```
limit.maxIDLength:
- value: 255
    constraints: {}
```

Metric

New matching service task_lag_per_tl metric for estimated task lag
New service_name metrics tag
New matching task_write_throttle_count and task_write_latency metrics
New worrkflow_task_attempt metric
New task_skipped metrics for number of skipped history tasks

Tool

New disable-initial-host-lookup flag in temporal-cassandra-tool for disabling initial host lookup

Namespace deletion

System workflows for marking namespace as deleted and reclaiming resources
Exclude deleted namespaces from ListNamespace response
Allow delete namespace by NamesapceId
Delete namespace feature is still under active development and not production ready

Activity local dispatch

An optimization for skipping matching service and task polling for running activity tasks
This optimization requires temporal SDK support and it's still under development

Improvements and fixes

Buffer event as local namespace after promotion
Use type from type map before type from Metadata
Add sqlite persistence tests
Fix history client NPE for invalid shardID
Fix timer queue cluster ack level update
Return ErrExecutionsStillExist if executions are still exist
Fix record child workflow complete mutable state stale check
Fix namespace not active error metric
Unify error handling of CRUD operation of workflow within shard context
Ensure minimal context timeout for shard context APIs
Handle workflow not found in replication
Handle activity retry timer in passive
Allow delete open workflows from passive sides
De-duplicate reapply event
Fix replication error handling for reseting workflow with pending child
Change operator service error codes if Search attributes are not found or already exist
Limit timer look ahead
Use ListWorkflow instead of ScanWorkflow for batch operation
Remove replication start event sanity check
Copy version history before returning
Fix shard context error state check
Add SetWorkflowExecution persistence API
Pass useTLS in GetFrontendClientConfig's callback
Handle delete history branch when mutable state does not exist
Rework health check
Support pagination for admin get task queue tasks
Fix unloading shard after task id sanity check
Fix get history scheduled tasks query
Do not generate sync activity task for deleted activity
Support pagination for SQL GetTransfer/VisibilityTasks
Fix potential shard deadlock issue
Update error code and message for missing activity task
Return NOT_FOUND for invalid workflow ID
Fixing syntax in first migration to the Cassandra schema
Backoff failed workflow task
Add task queue to SQL visibility
Check if global ns enabled in handler
Preserve the info when updating is_global_namepsace_enabled
Add dynamic config change diff logging
Force metrics.Reporter as metrics extension point
Improve extensibility of optentelemetry metrics
Fix DeleteWorkflowExecution API when delete non current execution
Generate namespace replication only with 2+ clusters
Add tag support to sdkLogger
Expose metrics reporters constructors to end-user
Treat enablement of TLS separately for server and client config
Add support for GetWorkflowHistory in reverse order
Fix timerMaxReadLevel map init and update
Do not blindly retry timer task read
Exit retry for long poll if context is near deadline
Convert panic when it is not error

Refactoring

Delete bootstrap params
Add context parameter to persistence interface and implementations
Update iterator to use go generics
Remove custom build info script
Replacing SdkClient injection with SdkClientFactory injection
Make matching persistence range queries [inclusive, exclusive)
Remove deprecated admin list tasks APIs
Refactor task filter function and interface
Unify history task persistence range query behavior
Refactor persistence task serializer and factory
Unify history queue processor and task generation implementation
Provide serverOptions as fx.Out struct
Make queue processor depend on history engine interface
Refactor handleWorkflowTaskCompleted retry loop

Others

Upgrade web images to the latest versions
Remove docker files
Removing vim from docker images due to CVE
Added mockgen to base-ci-builder Docker image
[docs] improve documentation for building images
Add Temoporal Web 2.0 to development docker-compose
Split goreleaser archive in two:cgo/nocgo

All changes

2022-04-11 - bcf9e57a0 - Client error log level (#2701) 2022-04-06 - 659c05253 - Change Invalid search attribute value message (#2705) 2022-04-01 - c1d63b713 - buffer event as local namespace after promotion (#2699) 2022-04-01 - 232ea210c - Use type from type map before type from Metadata (#2697) 2022-03-30 - 306eda804 - Add sqlite persistence tests (#2681) 2022-03-30 - ad56ef249 - Upgrade web images to the latest versions (#2684) 2022-03-30 - 2ebc73eed - Delete bootstrap params (#2652) 2022-03-29 - b440f924c - Fix history client NPE for invalid shardID (#2679) 2022-03-29 - ccba09b1f - Fix timer queue cluster ack level update (#2678) 2022-03-29 - 6b15af388 - Return ErrExecutionsStillExist if executions are still exist (#2676) 2022-03-29 - c6243eb79 - Set WorkflowID for child workflows (#2675) 2022-03-29 - a274580ed - Fix record child workflow complete mutable state stale check (#2673) 2022-03-29 - 3af96c64e - Persistence context part 6: visibility store interfaces (#2674) 2022-03-29 - 163234048 - Update iterator to use go generics (#2672) 2022-03-29 - 22f58a91d - Add DeleteNamespace API unit tests (#2670) 2022-03-29 - 7bff5ee6b - Improve deletenamespace workflow logging (#2667) 2022-03-28 - 946961847 - Store namespaces in DELETED state in registry (#2665) 2022-03-28 - b87a3ead8 - Fix parent close policy processor for deleted namespace (#2664) 2022-03-28 - c6a0cfc7a - Move namespace cache delay to reclaim resources workflow (#2658) 2022-03-28 - 30c286787 - Fix namespace not active error metric (#2663) 2022-03-28 - bfaee75cd - Unify error handling of CRUD operation of workflow within shard context (#2662) 2022-03-28 - f066d97df - Ensure minimal context timeout for shard context APIs (#2660) 2022-03-28 - d6f85df4f - Persistence context part 5: persistence store interfaces (#2659) 2022-03-28 - 642d64a11 - Merge SQL schema changes for 1.16 release (#2661) 2022-03-25 - e66040395 - Handle workflow not found in replication (#2633) 2022-03-25 - 6b8986c05 - Add reason field for workflow cancellation (#2653) 2022-03-25 - cc5171143 - Add DeleteNamespace API (#2645) 2022-03-25 - 752ff502b - Exclude deleted namespaces from ListNamespace response (#2646) 2022-03-25 - 4152b17dd - Select the correct proto commit (#2651) 2022-03-25 - 7c8ac02ac - Suppress nonuseful namespaceReplicationMessageProcessor log (#2648) 2022-03-25 - 12f47312b - Handle activity retry timer in passive (#2640) 2022-03-25 - 82ab54f6e - Allow delete namespace by Id (#2643) 2022-03-24 - f0d20df45 - Allow delete open workflows from passive sides (#2636) 2022-03-24 - 8a8f0aac1 - Fix merge conflit (#2644) 2022-03-24 - d663e2cb8 - Persistence Context Part 4: Visibility Manager (#2639) 2022-03-23 - 90b749427 - Fix merge confict (#2642) 2022-03-23 - 559679073 - Rename activity local dispatch request field (#2641) 2022-03-23 - b6de7d55f - Delete namespace workflow (#2569) 2022-03-23 - e60d6bde8 - Persistence Context Part 3: Namespace Replication Queue (#2638) 2022-03-23 - a0c900120 - Persistence Context Part 2: Shard, Task, Metadata, ClusterMetadata Manager (#2632) 2022-03-22 - 7fb1bbea6 - Activity local dispatch (#2618) 2022-03-22 - af9130dcd - Remove custom build info script (#2635) 2022-03-22 - 715d9ce55 - Upgrade go version to 1.18 (#2626) 2022-03-22 - 32977ed8a - De-duplicate reapply event (#2629) 2022-03-22 - 5f776150d - Remove tctl code (#2461) 2022-03-21 - 856089a0b - Handle reset workflow with pending child wf (#2624) 2022-03-21 - f7ed39491 - Change operator service error codes if Search attributes are not found or already exist (#2627) 2022-03-21 - f4c08526e - Persistence Context Part 1: Execution Manager (#2622) 2022-03-21 - b7100aeab - Remove docker files (#2456) 2022-03-21 - cd67ba3d9 - TerminateIfRunning workflowID reuse policy (#2608) 2022-03-20 - b577e89fb - Limit timer look ahead (#2623) 2022-03-18 - 32410dcce - Use ListWorkflow instead of ScanWorkflow for batch operation (#2620) 2022-03-18 - e2d6b31c0 - Bump SDK. (#2601) 2022-03-17 - 60bb20771 - Update server capabilities for sdk usage (#2619) 2022-03-17 - 2bcfb231c - Replacing SdkClient injection with SdkClientFactory injection. (#2611) 2022-03-17 - 650d33704 - Remove replication start event sanity check (#2616) 2022-03-16 - fa3610a82 - Copy version history before returning (#2613) 2022-03-15 - 28aebc21e - Fix shard context error state check (#2612) 2022-03-15 - c8960b527 - Revert "Added mockgen to base-ci-builder Docker image (#2574)" (#2588) 2022-03-15 - bb7b984ef - Make matching persistence range queries [inclusive, exclusive) (#2599) 2022-03-14 - 9732f77eb - Emit estimated matching task lag metric (#2605) 2022-03-14 - 3a3057e03 - Misc. improvements for error checking (#2602) 2022-03-12 - cafdda9ec - removing vim from docker images due to CVE (#2604) 2022-03-11 - da95e7abb - Add a test for #2563 (#2607) 2022-03-11 - 750c61a4b - Add SetWorkflowExecution to workflow context (#2606) 2022-03-11 - ee8dda326 - Updating sql schema to expand VARCHAR (#2568) 2022-03-10 - 4c4652dfe - Ignore parent close policy if child namespace is deleted (#2596) 2022-03-10 - c04f879f9 - Continue to delete workflow execution if namespace is deleted (#2576) 2022-03-10 - f994ea5e6 - Rename namespace persistence API (#2595) 2022-03-09 - 7d1d1c114 - Pass useTLS in GetFrontendClientConfig's callback (#2563) 2022-03-09 - 169d03bcf - Adds flag to temporal-cassandra-tool to disable Initial Host Lookup #2594 2022-03-08 - 1e3bc68f4 - Remove deprecated admin list tasks APIs (#2592) 2022-03-08 - 4375f9428 - Handle delete history branch (#2591) 2022-03-08 - 6fa5b4a00 - Add dynamic config for per instance visibility API rate limit and burst (#2585) 2022-03-08 - 5fb4f1c23 - Rework health check (#2579) 2022-03-08 - 9d69465c3 - Support pagination for admin get task queue tasks (#2583) 2022-03-08 - 4054170bb - Admin list history tasks API (#2580) 2022-03-08 - 72fb979e2 - Match history and proto task category definition (#2586) 2022-03-08 - 11c67aafa - Adding heartbeat details in RespondActivityTaskFailed apis (#2560) 2022-03-07 - 0a2d2e5c4 - Add test for an expected format of Datetime search attribute (#2589) 2022-03-07 - ac1ba9839 - Export cassandra test setup definitions for reuse (#2581) 2022-03-07 - df18e3d8e - Fix unloading shard after task id sanity check (#2571) 2022-03-07 - 810302264 - Fix get history scheduled tasks query (#2578) 2022-03-04 - 9cb3f5a61 - Added mockgen to base-ci-builder Docker image (#2574) 2022-03-04 - 2c6b76f32 - Refactor task filter function and interface (#2556) 2022-03-04 - db40f0217 - Do not generate sync activity task for deleted activity (#2572) 2022-03-04 - 441f7e0ed - Support pagination for SQL GetTransfer/VisibilityTasks (#2564) 2022-03-04 - 1a894659a - Remove deleted namespace from registry (#2567) 2022-03-04 - 40e9fe098 - Fix potential shard deadlock issue (#2570) 2022-03-02 - 19cee07b6 - Update error code and message for missing activity task. (#2561) 2022-03-01 - 8433f9f43 - Update execution tests (#2562) 2022-02-28 - 716b3cd23 - Return NOT_FOUND for invalid workflow ID (#2559) 2022-02-28 - c3e7dbec0 - Unify history task persistence range query behavior (#2547) 2022-03-01 - b431869d6 - [docs] improve documentation for building images (#2557) 2022-02-28 - 4c3810885 - Refactor persistence task serializer (#2555) 2022-02-28 - 3b070308a - Refactor persistence factory implementation (#2544) 2022-02-28 - 9778fa361 - Bump Web to v1.14.0 (#2550) 2022-02-26 - aa817b204 - #2551 Fixing syntax in first migration to the Cassandra schema (#2552) 2022-02-25 - 87a48bb9c - Backoff failed workflow task (#2548) 2022-02-25 - 6107df5ab - reduce unnecessary logging (#2549) 2022-02-25 - f7952b427 - Deprecate per service metrics config (#2546) 2022-02-25 - 9964786cd - Add service type tag to metrics (#2545) 2022-02-24 - f92cc9382 - Sql task queue visibility (#2537) 2022-02-24 - 9e47a655f - Queue refactor part 7: injection (#2534) 2022-02-24 - f79eeef65 - Check is global ns enabled in handler (#2541) 2022-02-24 - 60e5a8b29 - Preserve the info when updating is_global_namepsace_enabled (#2540) 2022-02-24 - 4d129d1e6 - Operator API (#2503) 2022-02-23 - 4f0135572 - provide serverOptions as fx.Out struct (#2536) 2022-02-23 - 4c212e5bb - Add dynamic config change diff logging (#2494) 2022-02-23 - 0c402433d - Queue refactor part 6: shard context (#2511) 2022-02-23 - 3dfccf02c - Queue refactor part 5: persistence layer (#2507) 2022-02-23 - c479ea309 - Update interfaces and structs for admin use (#2533) 2022-02-22 - a5a91ffac - Add RebuildMutableState API (#2532) 2022-02-22 - 253c820ea - Force metrics.Reporter as metrics extension point (#2529) 2022-02-22 - ef1ca2a57 - Fix flaky matching AddTaskStandby test (#2527) 2022-02-22 - acb96fd97 - Adds ExecutionInfo reference to InternalWorkflowMutation #2530 2022-02-22 - 4d06e1aca - Make queue processor depend on history engine interface (#2514) 2022-02-22 - f6acc48e8 - Allow customizing retry behavior for timeout failure (#2524) 2022-02-22 - 0a76b2ae2 - Log build CGO_ENABLED state (#2509) 2022-02-20 - 61b9b6574 - Improve extensibility of optentelemetry metrics (#2521) 2022-02-18 - 73f40bf3e - Add SetWorkflowExecution persistence API (#2523) 2022-02-18 - 2e69c936c - Define type and name for task categories (#2522) 2022-02-18 - faa1dc135 - Clean up auto-setup.sh (#2516) 2022-02-18 - ede203533 - Add Temoporal Web 2.0 to development docker-compose (#2520) 2022-02-17 - 305a6fc97 - Add config and handling for remote cluster cert (#2475) 2022-02-17 - cbd1b5e2d - Fix DeleteWorkflowExecution API when delete non current execution (#2484) 2022-02-16 - 1d33c3ed9 - Refactor handleWorkflowTaskCompleted retry loop (#2483) 2022-02-16 - d2f41f050 - Add SKIP_DB_CREATE env to auto-setup.sh (#2495) 2022-02-16 - ed29a48b7 - Fix metrics scope misuse (#2515) 2022-02-16 - 1f2b29162 - Generate namespace replication only with 2+ clusters (#2513) 2022-02-16 - 24f6dc36c - Emit matching task writer latency and throttle count (#2504) 2022-02-15 - b0f9d3a68 - Add tag support to sdkLogger (#2512) 2022-02-15 - 615fb91d2 - Trigger Docker builds on release branches (#2510) 2022-02-15 - eed1fde32 - Expose metrics reporters constructors to end-user (#2481) 2022-02-14 - 632c4f7a8 - Treat enablement of TLS separately for server and client config (#2501) 2022-02-14 - f6e84df10 - Move goreleaser scripts to Makefile (#2508) 2022-02-14 - dee4abc26 - Update go version of goreleaser to to 1.17.6 (#2506) 2022-02-14 - f0d98a89d - Add support for GetWorkflowHistory in reverse order (#2442) 2022-02-14 - f5a39b7fd - Split goreleaser archive in two:cgo/nocgo (#2505) 2022-02-14 - 27cc0b164 - Fix timerMaxReadLevel map init and update (#2498) 2022-02-14 - 2a57ba95c - Upgrade dependencies (#2499) 2022-02-13 - be1f9084e - Do not blindly retry timer task read (#2500) 2022-02-11 - b99986f05 - Suppress two info logs that are not useful (#2493) 2022-02-11 - 3afd57dff - Queue refactor part 4: task serializer (#2490) 2022-02-11 - 131563b03 - Queue refactor part 3: task notification (#2489) 2022-02-11 - 855ef8e15 - Replace Cyrillic c with c in caCert (#2492) 2022-02-11 - 1552628ce - Emit workflow task attempt stats (#2487) 2022-02-10 - 61dac76ed - Exit retry for long poll if context is near deadline (#2488) 2022-02-10 - d54dd19bb - Queue refactor part 2: task generation (#2479) 2022-02-10 - 0f0c853db - convert panic when it is not error (#2486) 2022-02-09 - ff18623ec - Emit history task skipped metric (#2485) 2022-02-09 - 6a25df34b - Queue refactor part 1: define processor interface and category (#2478)

Helpful links to get you started with Temporal

Temporal Docs Server Docker Compose Helm Chart

Docker images for this release (use tag `1.16.0`)

Server Server With Auto Setup (what is Auto-Setup?) Admin-Tools

v1.16.0