[Tech] The effect of slow nodes

[Michael Rogers]

toad wrote:
> Well there are two issues here:
> 1. Simulations don't seem to show a consistent lead for backoff plus
> throttling over just throttling. At least, recent ones - do yours? If
> this is true, then why?

Recent simulations show that backoff plus throttling and throttling 
alone are pretty evenly matched. I would guess this is because 
throttling keeps load at a low enough level that backoff rarely occurs. 
That would also explain why backoff alone slightly outperforms the other 
two mechanisms when there are slow nodes - backoff is a local reaction 
to overload whereas throttling is a global reaction, so backoff 
misroutes around the slow nodes, whereas throttling slows down the whole 
network.

> 2. Why is the real network so slow? It uses far less bandwidth than the
> specified limit; this suggests that the throttling has gone mad, or
> something (pre-emptive rejection) has broken it. Can we reproduce this
> in the simulation, and show why it happens / that it doesn't happen if
> we slightly change things?

I hope so, but there are a lot of places a bug could be hiding. Are 
there any clues in the logs about what's triggering RejectedOverloads? 
At one point inserts were much slower than requests - is that still the 
case? Is the insert throttle slower than the request throttle, and if 
so, why are inserts more likely to trigger RejectedOverloads than 
requests are?

Cheers,
Michael