Handling Overload With Concurrency Control And Load Shedding — Part 3

Concurrency Limits Library In Action

  • CL: Current concurrency limit
  • W : Maximum number of worker threads (Let’s use the default value of 200)

CL ≥ W

CL < W

Issues With This Approach

It Does Not Measure True Latency

It Almost Makes Queuing An All-or-Nothing Proposition

An Alternative Proxy Based Approach

  • In-Flight Requests (IFR) or Current Concurrency: In the earlier approach, the IFR was the number of requests that are currently being processed by worker threads. So IFR is always less than or equal to the maximum number of worker threads (W). In the proxy-based approach, IFR is the number of requests sent to the service by proxy and are currently pending. It includes requests being processed by worker threads as well as requests waiting in the queue. For example, if W = 200, then IFR can be 500, which means that 200 requests are being processed and the rest of 300 are waiting in queue.
  • Measured Response Time (MRT) or Round Trip Time (RTT): In the library-based approach, the MRT or RTT was the time taken by worker threads to process requests i.e. T1. Using the proxy, MRT also includes the time requests spend in queue i.e. T1 + T2. Hence it measures the actual latency experienced by callers.

How Proxy Based Approach Would Work

Advantages

Options Available For Proxy-Based Approach

Envoy

Kanaloa

Further Reading

Principal Engineer @ OLX Group. https://twitter.com/vikaskx

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

FIVE MINUTE EBITDA BRIDGE

My experience on “Pointers Workshop”…

Innovation at the Intersection

How to install simultaneously Docker and Podman on RHEL 8/CentOS 8

AWS SNS Topic, SNS Subscriptions & Lambda DLQ Config, Serverless Framework

My journey with Swift in 4 days

Crosswise Weekly Recap: Week 47 & 48/2021

Fixed a python App deployed on a Kubernetes Cluster

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vikas Kumar

Vikas Kumar

Principal Engineer @ OLX Group. https://twitter.com/vikaskx

More from Medium

Error Budget Is All You Need - Part 1

Cloud native meani

The Benefits of Environment as a Service (EaaS) for Any Software Business

Cloud Foundry Advisory Board Call, Mar 2022: Open Service Broker API