Scalability

How to we arrive at size of Thread pools 

Understand the load 

Peak request per second : 30 

Dependency Latencies 

Median : 40 ms

99th : 200 ms 

99.5th : 300 ms

30 rps X 0.2 seconds = 6 + breathing room = 10 Threads 

Anything load that is like aa tide or burst 

prefer to migrate to batching system . 

these are in place to add constraints , as long as we are bigger than normal traffic flow bigger pools are not going to increase thru-put 

Linked Blocking queues size around 5-10 , prefer NON blocking if you are in design phase 

Queues are not free :  

Thread pool Size + Queue Size 

connect timeout : 100ms , read timeout : 250ms retries : 1 

Set network timeouts as high and rely on Hystrix concurrency restraints  for actions and circuit breaking 

Cost of threads for circuit breaker  : 

1 thread pool size 10 @ 60RPS i box 

median : 0ms 

90th 3ms 

99th 9ms 

the cost of 9ms is the we pay for protection we get using circuit breaker 

75 rps 

median : 0ms

90 : 2ms 

semaphores , effectively no cost @5000 rps all in memory 

Circuit breaker netflix  : 

10 second rolling window , 

min threashold of rps 20  and if 50% fail 

trips circuit trip 

every 5 second try again if any passes circuit reset 

Fail fast is default behaviour 

Fail Silent usually UI ,. like that functionality be hidden return null , empty list 

Stubbed fall back - get fresh data , if fails , get cached data , else get some global static data