Although I last wrote about technical topics here a while ago, I would like to share some ideas that are helping me reduce friction in creating tools for my colleagues at ClarityAI.
As the Platform Lead, among other things, I am involved in the development experience, which means that my team creates tooling (as part of our platform) for the rest of the engineering and data science teams in the company. This tooling includes command-line tools that help users use our platform with minimal friction and autonomously (without requiring support).
In our case, this means we want command-line tools with the following characteristics:
- Usable on Linux and Mac
- Low friction for usage (easy installation and usage)
- Easy to evolve
- Easy to distribute to all our users
- Few external dependencies so that our users don't have to install additional tools
In this context, and considering that we are already heavy users of Docker, it is a good idea to implement these types of tools as Docker images that our users can use.
This would allow us to include any external tools our users may need in the Docker images and eliminate friction by not requiring them to install those tools.
At the same time, using Docker presented its own challenges:
- Difficulty in distributing a Docker image from a private repository
- Difficulty in usage for execution requiring many parameters (volumes, environment variables, execution parameters and arguments, etc.)
Below, I will explain how we have solved (or are in the process of solving) each of the mentioned challenges.
Controlled Environment
Any modern development platform is implemented using different technologies and solutions.
This means the user must use different clients for each of these technologies (kubectl, AWS CLI, Git, Teleport, etc.).
To eliminate friction, avoid installation problems, and reduce the number of potential issues due to misuse or using incorrect versions, what has worked very well for us is to include all these tools (with controlled versions and environment) in a Docker image along with code to control their usage and configuration.
This allows the platform team to forget about problems caused by using the wrong version of kubectl or incorrect AWS client configuration.
This means the user must use different clients for each of these technologies (kubectl, AWS CLI, Git, Teleport, etc.).
To eliminate friction, avoid installation problems, and reduce the number of potential issues due to misuse or using incorrect versions, what has worked very well for us is to include all these tools (with controlled versions and environment) in a Docker image along with code to control their usage and configuration.
This allows the platform team to forget about problems caused by using the wrong version of kubectl or incorrect AWS client configuration.
Installation
If the installation process of a tool is complicated, you can be sure that a significant percentage of users will encounter problems and require support.
It is often assumed that good documentation is sufficient, but no matter how good the documentation is, if there are many steps to follow and many tools to install, there will be problems.
A better solution is to create a specific installer without configuration and with the fewest possible dependencies.
In our case, the tool to be distributed was built with Docker and stored in a private repository (AWS ECR).
As a lean team, we have iteratively improved the solution, reducing friction at each step:
It is often assumed that good documentation is sufficient, but no matter how good the documentation is, if there are many steps to follow and many tools to install, there will be problems.
A better solution is to create a specific installer without configuration and with the fewest possible dependencies.
In our case, the tool to be distributed was built with Docker and stored in a private repository (AWS ECR).
As a lean team, we have iteratively improved the solution, reducing friction at each step:
- Installation instructions are documented in the engineering handbook. It depends on Docker, properly configured AWS CLI, and knowledge of ECR.
- Installer script in the code repository. It improved the experience and generated a helper script for future usage. It depends on Docker and properly configured AWS CLI but does not require knowledge of ECR.
- Cross-platform installer implemented with Golang. This is the version currently in development. It only requires Docker, and the user has an AWS Key. (Example: ECR Docker image puller / installer)
Execution
When we create an open-source solution or a product that will be used in different environments, the ability to configure and adapt it to different environments is crucial.
But when we develop a tool for our colleagues and our goal is to minimize friction and reduce cognitive load, we need to provide the minimum necessary flexibility.
We can even avoid the need for any parameters.
This becomes a challenge when you have a tool built with Docker that requires multiple credentials and the ability to modify files on the user's machine, as it requires running Docker with many parameters and environment variables.
Once again, iteration has allowed us to improve the solution. The steps we have taken are:
But when we develop a tool for our colleagues and our goal is to minimize friction and reduce cognitive load, we need to provide the minimum necessary flexibility.
We can even avoid the need for any parameters.
This becomes a challenge when you have a tool built with Docker that requires multiple credentials and the ability to modify files on the user's machine, as it requires running Docker with many parameters and environment variables.
Once again, iteration has allowed us to improve the solution. The steps we have taken are:
- Document all the parameters and environment variables for tool execution in the README. This serves us well for development, but we never intended for our users to use the tool following this documentation.
- Using an execution script that hides all the parameters and variables. In our case, this execution script was generated by the installer.
- Cross-platform wrapper implemented in Golang. This allows us to distribute a binary that only depends on having Docker and hides the tool's complexity. This is the version we will distribute soon. (Example: Simple wrapper to execute (interactive/non interactive) docker image)
Continuous Updating
In addition to facilitating installation, it is essential that the platform team can quickly push new versions. In this case, having Docker already facilitates this because simply uploading new versions to the Docker repository makes them available to our users.
On the other hand, to streamline the update process, what has worked well for us is to include an option in the execution script/wrapper to check and download new versions.
Initially, we made it so that each execution would check for new versions, but we found that it caused too much delay during startup, so now we prefer to let the user decide when to update.
On the other hand, to streamline the update process, what has worked well for us is to include an option in the execution script/wrapper to check and download new versions.
Initially, we made it so that each execution would check for new versions, but we found that it caused too much delay during startup, so now we prefer to let the user decide when to update.
Conclusions:
- Docker allows packaging applications with specific versions in a controlled environment.
- Creating a cross-platform wrapper in Go is an excellent solution to eliminate friction in the distribution and execution of tools implemented using Docker.
- Investing in making installation and updating simple allows tools to evolve rapidly without generating much friction.
