LinkedIn Replaces Argo CD with Custom GitOps Engine for Kubernetes

LinkedIn has replaced Argo CD with its own GitOps engine, which deploys applications across over 250 Kubernetes clusters, managing more than 100,000 deployments weekly (1). The new system improves the median time to detect and resolve failed deployments from 16.1 to 2.45 minutes. LinkedIn identified seven challenges with Argo CD at scale, including the absence of observedGeneration, which made it impossible to link sync and health status to specific deployments (1). [1]

The new system uses PUT for full replacements instead of PATCH, retries on resourceVersion conflicts with exponential backoff, and uses a 3-way diff to detect drift (1). LinkedIn's deployment infrastructure failures fell from 925 to 63 despite a 3.7x increase in deployment volume (1). The team lists seven problems that emerged at scale, including the lack of observedGeneration and issues with server-side-apply PATCH (1). [1]

LinkedIn created a Lua health check that uses a tracking annotation to address the issue of linking sync and health status to specific deployments (1). The team also found that maintaining 200+ Argo CD instances made each upgrade a multi-week project (1). LinkedIn's new engine runs one ephemeral pod per deployment with no shared controller state (1). [1]

LinkedIn has replaced Argo CD with its own GitOps engine, which deploys applications across over 250 Kubernetes clusters, managing more than 100,000 deployments weekly (1). The new system improves the median time to detect and resolve failed deployments from 16.1 to 2.45 minutes. This change was driven by seven challenges identified with Argo CD at scale, including the absence of observedGeneration, which made it impossible to link sync and health status to specific deployments (1). [1]

To address these challenges, LinkedIn's new engine uses PUT for full replacements instead of PATCH, retries on resourceVersion conflicts with exponential backoff, and uses a 3-way diff to detect drift (1). The company also created a Lua health check that uses a tracking annotation to address the issue of linking sync and health status to specific deployments (1). These changes were implemented to improve the reliability and performance of deployments in large-scale Kubernetes environments. [1]

LinkedIn's migration from Argo CD to its custom engine was accompanied by improvements in the deployment orchestrator's error classification and reliability (1). While the median duration for successful deployments remained similar between the two systems, the mean and tail durations improved, which the team attributes to the use of a namespace-scoped informer cache (1). The new system is internal, and no open-source release is mentioned (1). [1]

LinkedIn's replacement of Argo CD with a custom GitOps engine has been accompanied by significant improvements in deployment reliability and performance, according to InfoQ (1). The new system manages over 100,000 deployments weekly across more than 250 Kubernetes clusters, with deployment infrastructure failures decreasing from 925 to 63 despite a 3.7x increase in deployment volume (1). LinkedIn's migration was driven by seven challenges identified with Argo CD at scale, including the absence of observedGeneration, which made it impossible to link sync and health status to specific deployments (1). [1]

To address these challenges, LinkedIn's new GitOps engine employs several technical strategies, including the use of PUT for full replacements instead of PATCH, retries on resourceVersion conflicts with exponential backoff, and a 3-way diff to detect drift (1). Additionally, LinkedIn developed a Lua health check that uses a tracking annotation to help link sync and health status to specific deployments (1). These changes are part of a broader effort to improve the deployment orchestrator's error classification and reliability, which has also contributed to the overall improvement in deployment performance (1). [1]

Despite these improvements, several open questions remain regarding the long-term impact of LinkedIn's custom GitOps engine and how it may influence broader industry practices (1). For instance, the decision to build an internal engine rather than extend or adopt an existing solution raises questions about the trade-offs involved in such a strategy (1). Additionally, LinkedIn's approach contrasts with other large operators like Uber and Airbnb, which have also developed their own deployment tools tailored to their specific needs (1). These differences highlight the importance of operational models in determining the scalability and reliability of deployment systems (1). [1]

Sources

InfoQ

Prepared with AI assistance from the cited sources.