Using TeamCity Detached Builds to Invoke AWS Services

TLDR; Use TeamCity detached builds and callbacks to provide a more seamless integration between TeamCity and AWS Services.

Asynchronous External Service

One of the challenges with invoking a process in AWS from TeamCity is that the call to AWS is usually asynchronous. An example of when you might want to do this is to make CloudFormation updates or initiate a deployment using CodeDeploy. Unlike most build tasks executed in TeamCity, the AWS call returns as soon as the message is received. It does not block until the deployment is complete. This is good because it doesn’t consume an agent while waiting, but bad because there is no indication in TeamCity that the deployment is still executing, no indication of the current status, and no reporting of the final result of the deployment. This makes it difficult to trigger other builds or use the deployment build as a snapshot dependency because it’s not clear when the deployment is actually complete.

TeamCity Detached Builds to the Rescue!

Starting with the 2020.2 build of TeamCity, there is a feature called detached builds. This was designed for calling external asynchronous services from a TeamCity agent. It makes it possible to simulate the behavior of a synchronous operation on a TeamCity agent even when the call to the external service is asynchronous. This is accomplished by using a callback approach. When the build is detached the TeamCity console shows the build as continuing to execute.

A build step can detach the agent by emitting the service message ##teamcity[buildDetachedFromAgent]. This can be done inside PowerShell or a command line build step by echoing the service message to the console:

echo ##teamcity[buildDetachedFromAgent]

Detached Builds Release the Agent

As soon as the build receives the buildDetachedFromAgent service message it releases the build agent back to the agent pool so that it can service a new request. However, from the UI and the TeamCity Server perspective the build continues to be in progress. So the UI shows the build as still running even though there is no longer an agent actively executing it.

To show status messages, log errors, or finish the build requires an API call back to TeamCity. The external service that was called before the build was detached needs to call back to TeamCity through the REST API. The calls look something like below and full documentation can be found in TeamCity BuildApi REST documentation.

https://{team_city_host}/app/rest/builds/id:{build_id}/finish

This call tells TeamCity to mark the build as completed. The {build_id} parameter here is the global team city build id (teamcity.build.id) that uniquely identifies an instance of a running build.

Detached Build Licensing (TeamCity)

Even though detached builds do not consume an agent, the TeamCity server has a limit to the number of detached build agents that can be running at one time. The limit is the same as the number of agent licenses on the server. The count of detached builds is global, so regardless of how many agents are in a pool available to run a particular build, the detached agent count can consume all detached agent licenses on a server. Even if the build is only available to run on the 5 agents in the pool, it could consume all detached build licenses available on the server.

Important safety tip – When the number of detached agents licenses are consumed, TeamCity will no longer allow builds to detach and will continue to execute on actual build agents(consuming the agent).

Conclusion

Asynchronous external services like those found in AWS can be awkward to integrate with TeamCity. Using detached builds allows the invocation of these services with a more seamless synchronous integration user experience in the TeamCity UI. This approach allows these builds to report status as they execute, be used as triggering builds, be used as snapshot dependencies, and provide a status indicating the result of the external service invocation.

One thought on “Using TeamCity Detached Builds to Invoke AWS Services

  1. Great article about a little known feature. I had never heard of it.

    I’ve been doing some work with GitHub Actions. It’s nice, but doesn’t feel totally mature yet. The one thing I absolutely love about it, though, is how you define your pipeline using simple YAML files, and it’s all source controlled.

    Looking forward to your next article.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.