FWD » Core Development » Runtime Infrastructure

Every client spawn calls TemporaryAccount.open(), which acquires TemporaryAccountPool.taskLock, hands one task to the single TemporaryAccountWorker thread, and blocks on callerLatch.await() until that task completes. So N connection serialize behind N sequential worker task executions — no overlap.

What the worker does per task includes:

- HashPassword.hashPassword() at 210,000 PBKDF2-HMAC-SHA256 iterations.

- each create triggers two full rebuilds of the temp-account list (SecurityCache.refreshTempAccounts), once for the admin cache and once for the live cache.

A solution for HashPassword.hashPassword() slowness problem could look in the following way:

- Generate and hash the credentials on the posting thread, when the CreateAccountTask task is created. CreateAccountTask's creation-mode constructor does the random subject/password generation, the PBKDF2 hash, and the ordinal allocation, filling a ready-to-use TempUserDef. TemporaryAccountPool.createUser() constructs the task before entering synchronized (taskLock), so client threads hash in parallel instead of queueing to hash one at a time on the worker.
- Use a reduced iteration count for these ephemeral secrets. New HashPassword.EPHEMERAL_ITERATIONS (1,000) plus a hashPassword(String, int) overload. And will be used only for temporary users.

Every web login calls TemporaryAccount.open(), which reaches TemporaryAccountPool.createUser(). That method hands a CreateAccountTask to a single worker thread and blocks until it completes:

CreateAccountTask account = new CreateAccountTask();
synchronized (taskLock)
{
   poolTask.execute(account);
}

In trunk, CreateAccountTask.createUser() computes the account password hash (HashPassword.hashPassword(), PBKDF2, 210,000 iterations) inside that serialized section. Under load the generation of hash takes ~2 s, and every concurrent login queues behind it. I got this result by 70 concurrent logins on hotel.

I made some changes to generate the credentials and compute the hash in the CreateAccountTask constructor, on the calling thread, so the expensive work runs in parallel across all cores. Only the directory write and the security cache refresh remain inside the lock.

100 concurrent logins on hotel_gui. Timings from instrumentation around createUser().

As a separate thing I tried lowering the PBKDF2 iteration count (210000 -> 1000), because the password is generate at temoprary user creation and is deleted at the end of the seesion. But it does not contribute significantly to the the time drop in congested user login.

Measurement of creation of temporary user account shows a significat imporovment, but I still hit the same issue described #11521-194.

Eduard Soltan wrote:

Measurement of creation of temporary user account shows a significat imporovment, but I still hit the same issue described #11521-194.

For #11521-194 are we talking about a process-storm and thus the system is under heavy load and can't execute all in time, or again lock contention?

Please add any logging from Serban to this branch.

Constantin Asofiei wrote:

I'm OK with the changes.

I could test ETF and clustered application, anything else should be tested?

Eduard Soltan wrote:

Constantin Asofiei wrote:

I'm OK with the changes.

I could test ETF and clustered application, anything else should be tested?

This affects agent spawning (or other load-testing). It does not affect runtime or anything else. Test something with classic agents and is enough.

Constantin Asofiei wrote:

This affects agent spawning (or other load-testing). It does not affect runtime or anything else. Test something with classic agents and is enough.

Tested web_api project with a clasic appserver, plus large gui application was tested on load with 300 concurrent users.