Application Configuration is either managed as static or dynamic.
At MoEngage, most of our systems are multi-tenant, which means a lot of our infrastructure is dynamically mapped. The mappings are changed to meet our business requirements. For example, client X using a shared infrastructure has now got a dedicated infrastructure.
The key issues we faced when our application configuration was static, were:
As our user base grew (or added more services), changing static configuration became a more frequent activity. So we started exploring other options to ensure enough dynamicity for our applications.
MoEngage Platform Team’s mission was to make application configuration dynamic, reliable and abstracted away from applications. Our main requirements were:
Our requirements were met with Consul. A distributed, highly available, and a scalable tool built by HashiCorp. Consul is widely used to help with service discovery and configuration management featuring a key-value stores, health checks, and DNS forwarding.
We compared the HashiCorp envconsul and consul-template for the dynamic configuration of the application.
The Twelve-Factor App rule suggests storing the configurations in the environment.
Using envconsul had the overhead of restarting the application process to make the application use updated environment variables. Further, environment variables do not interact well with nested objects/complex configurations.
On the other hand, using a consul-template, we can use a template file for rendering configuration files giving great flexibility in terms of utilizing templating power to format the application configuration without needing to restart the application process.
Hence, consul-template is a clear winner here.
Achieving true dynamic application configuration requires ensuring that the consul-template is available for application configuration and a mechanism is in place for updating application state. The application is made aware of the fact that its configuration is modified and the application should continue to run with the updated configuration.
To achieve this, we use the python watchdog module, to initiate a watcher to keep on listening to the directory where our consul-template rendered files are present. Now, any update in values in the Consul KV store is captured by consul-template daemon, and all the files having that key-value mapping are updated on the file system. This results in issuing file system modification events that are captured by the event handler and watcher to eventually update the application state asynchronously. We use an in-memory data structure to hold the latest configurations that act as a local source of truth when configurations are accessed by the application components.
Components In Action
With all the components working together, we get a distributed, highly available, and scalable dynamic configuration management system as described in the following diagram:
1. Bundling Application
User bundles application code with all the static configurations (local configs) and the corresponding template files (config templates). The config values (key-value mappings) are in a separate folder such as configs in the same git repository where the application code is stored.
2.Populating config values to Consul KV store
Config values are fetched from their respective git repositories and populated to the Consul KV store using git2consul.
Git2consul is one of the key components we use to transfer all our configurations from different git repositories to the Consul KV store. Git2consul allows git as a backing store that keeps all our configurations and utilizes Consul as a delivery mechanism to provide dynamic configurations. Some key features of Git2consul are :
We are using the Go based version of Git2consul : git2consul-go , which is an improved version of the classic one.
3. Publish bundled application to artifact repository
Bundled application created in the first step is moved to the artifact repository called jfrog.
4. Deploy application
At deployment, the artifact is deployed to the server, where the consul-template daemon is already running. Artifact is parsed to extract config templates and local configs to their respective locations.
5. Fetch Configs
After the config templates are extracted to the relevant location, consul template daemon starts rendering the template file to its equivalent configuration file by fetching specific keys from the Consul KV store.
We provide config.hcl file to consul-template daemon to make it render and monitor multiple files used in production:
Note: source path is the config template location and destination is the path where the dynamic configuration files are rendered.
You can pass the .hcl file to the consul template agent as described and start rendering the dynamic configurations.
Rendering dynamic configs
consul-template renders all the sources(templates) to the destinations(configs) and makes the files available on the disk of the application machine. The application startup waits for this to happen (first config to be present on the disk).
6. Application Startup
Starting of the application creates a config reader, which maintains an in-memory data structure, the dictionary that stores fileName as key, and its latest configuration as value. The reader also has a watcher running as a separate thread, which listens for any modification events in the config filesystem. On receiving a file system modification event, the watcher updates the in-memory data structure provides the latest configuration to the application and maintains a dynamic state.
7. Dynamic synchronization of configuration files
Any change in the Consul KV store is captured by the consul-template daemon and the linked dynamic configuration files are updated.
Note: Changes in the Consul KV store happen when the user modifies the config values in the git repository or modification the values from the Consul server directly.
Now, every time a need comes to dynamically update our infrastructure mappings, we need not build the application again with new mapping and redeploy it, rather update the mapping in the Consul KV store and gracefully handle the application state update instantly.
We at MoEngage have started adopting the Dynamic Configurations to give our applications
We have seen significant savings in development time by approximately one hour every week and redeployment overheads using dynamic configuration management. We are excited to move all our applications to this new configuration management system.
We would be happy to hear from you regarding any other interesting approaches that you have followed in your organization to make application configurations dynamic. Send an email to email@example.com.
We are also looking to expand our tech team and in case this excites you, do check our open opportunities and let us know what you think!
Here are actionable resources we've curated for you!