#TechAtMoEngage:How Does MoEngage Perform Application Configuration Management

Application Configuration is either managed as static or dynamic.

At MoEngage, most of our systems are multi-tenant, which means a lot of our infrastructure is dynamically mapped. The mappings are changed to meet our business requirements. For example, client X using a shared infrastructure has now got a dedicated infrastructure.

The key issues we faced when our application configuration was static, were:

• Static configuration is built and versioned along with the application code
• Changes in the configuration required us to rebuild and redeploy our applications.

As our user base grew (or added more services), changing static configuration became a more frequent activity. So we started exploring other options to ensure enough dynamicity for our applications.

MoEngage Platform Team’s mission was to make application configuration dynamic, reliable and abstracted away from applications. Our main requirements were:

1. Application configurations should be dynamically changed without rebuilding our application.
2. When we change a configuration, it should reflect in all application instances instantly.
3. We need to abstract away configuration management from applications. In other words, applications should not know where and how configurations are kept and provided for them.
4. Our configuration management system must be highly available and scalable.

Our requirements were met with Consul. A distributed, highly available, and a scalable tool built by HashiCorp. Consul is widely used to help with service discovery and configuration management featuring a key-value stores, health checks, and DNS forwarding.

We compared the HashiCorp envconsul and consul-template for the dynamic configuration of the application.

• envconsul: A tool that can be used to fetch dynamic config changes from the Consul KV store and restart your application to take the updated values.
• consul-template: A tool that provides a convenient way to populate values from the Consul KV store directly into the file system using the consul-template daemon. Features of consul-template that makes dynamic configuration management available are:
  • Consul Template renders the templates by fetching the values of the keys from the consul server.
  • After the config template is rendered, consul-template agent continues to monitor the rendered file and any change in the config value present in the Consul KV store is reflected to the rendered file immediately.

The Twelve-Factor App rule suggests storing the configurations in the environment.

Using envconsul had the overhead of restarting the application process to make the application use updated environment variables. Further, environment variables do not interact well with nested objects/complex configurations.

On the other hand, using a consul-template, we can use a template file for rendering configuration files giving great flexibility in terms of utilizing templating power to format the application configuration without needing to restart the application process.

Hence, consul-template is a clear winner here.

How to achieve Dynamic Application Configuration?

Achieving true dynamic application configuration requires ensuring that the consul-template is available for application configuration and a mechanism is in place for updating application state. The application is made aware of the fact that its configuration is modified and the application should continue to run with the updated configuration.

To achieve this, we use the python watchdog module, to initiate a watcher to keep on listening to the directory where our consul-template rendered files are present. Now, any update in values in the Consul KV store is captured by consul-template daemon, and all the files having that key-value mapping are updated on the file system. This results in issuing file system modification events that are captured by the event handler and watcher to eventually update the application state asynchronously. We use an in-memory data structure to hold the latest configurations that act as a local source of truth when configurations are accessed by the application components.

Note: Java has a similar filesystem watcher available using the WatchService interface available in java.nio package.

Sample consul-template

Components In Action

With all the components working together, we get a distributed, highly available, and scalable dynamic configuration management system as described in the following diagram:

1. Bundling Application
User bundles application code with all the static configurations (local configs) and the corresponding template files (config templates). The config values (key-value mappings) are in a separate folder such as configs in the same git repository where the application code is stored.

2.Populating config values to Consul KV store
Config values are fetched from their respective git repositories and populated to the Consul KV store using git2consul.
Git2consul is one of the key components we use to transfer all our configurations from different git repositories to the Consul KV store. Git2consul allows git as a backing store that keeps all our configurations and utilizes Consul as a delivery mechanism to provide dynamic configurations. Some key features of Git2consul are :

• • Can start as a daemon and any change in the configuration in a particular git branch (master/main in most cases) available in the Consul KV store based on the polling interval set in the configuration of Git2consul.
  • Can synchronize a subpath under your git repository to consul with a proper naming convention. This allows us to keep configurations at a datacenter level, but still co-located with our application.

We are using the Go based version of Git2consul : git2consul-go , which is an improved version of the classic one.

3. Publish bundled application to artifact repository
Bundled application created in the first step is moved to the artifact repository called jfrog.

4. Deploy application
At deployment, the artifact is deployed to the server, where the consul-template daemon is already running. Artifact is parsed to extract config templates and local configs to their respective locations.

5. Fetch Configs
After the config templates are extracted to the relevant location, consul template daemon starts rendering the template file to its equivalent configuration file by fetching specific keys from the Consul KV store.

We provide config.hcl file to consul-template daemon to make it render and monitor multiple files used in production:

Note: source path is the config template location and destination is the path where the dynamic configuration files are rendered.

You can pass the .hcl file to the consul template agent as described and start rendering the dynamic configurations.

Rendering dynamic configs
consul-template renders all the sources(templates) to the destinations(configs) and makes the files available on the disk of the application machine. The application startup waits for this to happen (first config to be present on the disk).

6. Application Startup
Starting of the application creates a config reader, which maintains an in-memory data structure, the dictionary that stores fileName as key, and its latest configuration as value. The reader also has a watcher running as a separate thread, which listens for any modification events in the config filesystem. On receiving a file system modification event, the watcher updates the in-memory data structure provides the latest configuration to the application and maintains a dynamic state.

7. Dynamic synchronization of configuration files
Any change in the Consul KV store is captured by the consul-template daemon and the linked dynamic configuration files are updated.
Note: Changes in the Consul KV store happen when the user modifies the config values in the git repository or modification the values from the Consul server directly.

Now, every time a need comes to dynamically update our infrastructure mappings, we need not build the application again with new mapping and redeploy it, rather update the mapping in the Consul KV store and gracefully handle the application state update instantly.

Conclusion

We at MoEngage have started adopting the Dynamic Configurations to give our applications

• The power of hot reloadable configurations
• Make the development hassle-free without needing to worry about configuration management.

We have seen significant savings in development time by approximately one hour every week and redeployment overheads using dynamic configuration management. We are excited to move all our applications to this new configuration management system.

We would be happy to hear from you regarding any other interesting approaches that you have followed in your organization to make application configurations dynamic. Send an email to [email protected].

We are also looking to expand our tech team and in case this excites you, do check our open opportunities and let us know what you think!