The connection table and locking over it has been a long source of issues and bottlenecks in our code. As a result, it's become a hugely complex part of the application, which makes it even harder to continue to work upon and improve.

We should replace the current thread worker and connection table design with nunc-stans and a connection tree.

Key to this, is that with nunc-stans we no longer require iterating and polling over the connection table to determine work to be completed. This means we remove a lock (serialisation point) and potential source of issue.

We should start a set of nunc-stans worker threads, rather than start_thread()/connection_threadmain().

When a socket is accepted, it's connection is inserted into a tree ( so we maintain a list of current connections), and the fd is put into nunc-stans as an io read job. When nunc-stans detects IO, it would call some new work function followed by connection_read_operation(). From there the work proceeds as normal.

At this point, we have already eliminated a large burden on the server, as we are not iterating over the connection table for new work.

Additionally, because we choose to use a tree, not a table, adding new connections is fast. When we close a connection, we can quickly remove it from the tree too. Iterating over the tree content is a trivial operation (BFS, DFS, B+Tree walk).

Connection's can be timed out by using PRStatus ns_add_io_timeout_job, and the timeout is reset when they are re-added to the work queues. So this is pretty easy to achieve.