Revision control
Copy as Markdown
Other Tools
# profiler-cli walkthrough: NSS deadlock from DoH heuristics freezing all networking
Companion to `macos-network-nss-doh-deadlock.md`. Reproduces the same findings using `profiler-cli` with annotated output.
---
## Load the profile and get an overview
```
profiler-cli profile info --session nss-deadlock
```
```
Name: Firefox 141 – macOS 15.6.0
Platform: macOS 15.6.0
This profile contains 239 threads across 11 processes.
Top processes and threads by CPU usage:
p-0: Parent Process [pid 87060]
t-18: Renderer [tid 2195712] - 2131.337ms
t-0: GeckoMain [tid 2195644] - 1391.608ms
t-54: BackgroundThreadPool #2 [tid 2195920] - 550.293ms
t-42: WRRenderBackend#1 [tid 2195852] - 488.968ms
t-31: Compositor [tid 2195725] - 204.447ms
...
t-3: Socket Thread [tid 2195672] - 0.000ms
t-35: osclientcerts [tid 2195803] - 0.000ms
t-15: BgIOThreadPool #1 [tid 2195697] - 0.000ms
```
239 threads across 11 processes. The profile spans 42 seconds. The most important observation: the Socket Thread (t-3), the osclientcerts thread (t-35), and BgIOThreadPool #1 (t-15) all show 0.000ms CPU. Three threads that should be doing work are completely inactive over 42 seconds. That is not normal.
---
## Inspect the Socket Thread
```
profiler-cli thread select t-3 --session nss-deadlock
profiler-cli thread samples-top-down --session nss-deadlock
```
```
Thread: Socket Thread
Top-Down Call Tree:
f-0. (root) [total: 100.0%, self: 0.0%]
└─ f-2789. libsystem_pthread.dylib!_pthread_start [total: 100.0%, self: 0.0%]
f-5562. libnss3.dylib!_pt_root [total: 100.0%, self: 0.0%]
f-7184. XUL!nsThread::ThreadFunc(void*) [total: 100.0%, self: 0.0%]
...
f-7419. XUL!mozilla::net::nsSocketTransport::OnSocketEvent(...) [total: 100.0%, self: 0.0%]
f-7430. XUL!mozilla::net::nsSocketTransport::InitiateSocket() [total: 100.0%, self: 0.0%]
f-7431. XUL!mozilla::net::nsSocketTransport::BuildSocket(...) [total: 100.0%, self: 0.0%]
f-7432. XUL!nsSSLSocketProvider::NewSocket(...) [total: 100.0%, self: 0.0%]
f-7433. XUL!nsSSLIOLayerNewSocket(...) [total: 100.0%, self: 0.0%]
f-7434. XUL!nsSSLIOLayerAddToSocket(...) [total: 100.0%, self: 0.0%]
f-7435. XUL!NSSSocketControl::SetResumptionTokenFromExternalCache(PRFileDesc*) [total: 100.0%, self: 0.0%]
f-7420. libnss3.dylib!SSLExp_SetResumptionToken [total: 100.0%, self: 0.0%]
f-7421. libnss3.dylib!ssl_DecodeResumptionToken [total: 100.0%, self: 0.0%]
f-7422. libnss3.dylib!CERT_NewTempCertificate [total: 100.0%, self: 0.0%]
f-7423. libnss3.dylib!STAN_GetCERTCertificateOrRelease [total: 100.0%, self: 0.0%]
f-7424. libnss3.dylib!stan_GetCERTCertificate [total: 100.0%, self: 0.0%]
f-7425. libnss3.dylib!nssTrustDomain_FindTrustForCertificate [total: 100.0%, self: 0.0%]
f-7426. libnss3.dylib!nssToken_FindTrustForCertificate [total: 100.0%, self: 0.0%]
f-7427. libnss3.dylib!nssToken_FindObjectsByTemplate [total: 100.0%, self: 0.0%]
f-7428. libnss3.dylib!find_objects [total: 100.0%, self: 0.0%]
f-7429. libnss3.dylib!nssSession_EnterMonitor [total: 100.0%, self: 0.0%] ← waiting for NSS session lock
f-2903. libnss3.dylib!PR_Lock [total: 100.0%, self: 0.0%]
f-531. libsystem_pthread.dylib!_pthread_mutex_firstfit_lock_slow [total: 100.0%, self: 0.0%]
f-532. libsystem_kernel.dylib!__psynch_mutexwait [total: 100.0%, self: 100.0%] ← 100% of all samples
```
Every sample in the 42-second profile shows the Socket Thread blocked on `__psynch_mutexwait` inside `nssSession_EnterMonitor`. It called into NSS to look up certificate trust for a TLS session resumption token while initiating a connection, and has been waiting for an NSS session lock ever since.
The 0ms CPU figure from `profile info` is confirmed here: this thread has not executed a single instruction in 42 seconds. Something else holds that lock.
---
## Find the lock holder: BgIOThreadPool #1
```
profiler-cli thread select t-15 --session nss-deadlock
profiler-cli thread samples-top-down --session nss-deadlock
```
```
Thread: BgIOThreadPool #1
Top-Down Call Tree:
f-0. (root) [total: 100.0%, self: 0.0%]
└─ f-2789. libsystem_pthread.dylib!_pthread_start [total: 100.0%, self: 0.0%]
...
f-7612. XUL!nsThreadPool::Run() [total: 100.0%, self: 0.0%]
f-7613. XUL!mozilla::detail::RunnableFunction<nsNSSCertificateDB::AsyncHasThirdPartyRoots(...)::$_0>::Run() [total: 100.0%, self: 0.0%]
f-7614. XUL!nsNSSCertificateDB::AsyncHasThirdPartyRoots(...)::$_0::operator()() const [total: 100.0%, self: 0.0%]
f-7617. XUL!nsNSSCertificateDB::AsyncHasThirdPartyRoots(...)::$_0::operator()() const::operator()() const [total: 100.0%, self: 0.0%]
f-7618. XUL!nsNSSCertificateDB::GetCerts(nsTArray<RefPtr<nsIX509Cert> >&) [total: 100.0%, self: 0.0%]
f-7602. libnss3.dylib!PK11_ListCerts [total: 100.0%, self: 0.0%]
f-7603. libnss3.dylib!NSSTrustDomain_TraverseCertificates [total: 100.0%, self: 0.0%]
f-7604. libnss3.dylib!nssToken_TraverseCertificates [total: 100.0%, self: 0.0%]
f-7605. libnss3.dylib!nssPKIObjectCollection_AddInstanceAsObject [total: 100.0%, self: 0.0%]
f-7606. libnss3.dylib!cert_createObject [total: 100.0%, self: 0.0%]
f-7607. libnss3.dylib!nssTrustDomain_AddCertsToCache [total: 100.0%, self: 0.0%]
f-7608. libnss3.dylib!add_cert_to_cache [total: 100.0%, self: 0.0%]
f-7609. libnss3.dylib!STAN_GetCERTCertificate [total: 100.0%, self: 0.0%]
f-7424. libnss3.dylib!stan_GetCERTCertificate [total: 100.0%, self: 0.0%]
f-7610. libnss3.dylib!fill_CERTCertificateFields [total: 100.0%, self: 0.0%]
f-7611. libnss3.dylib!nssTrust_GetCERTCertTrustForCert [total: 100.0%, self: 0.0%]
f-7425. libnss3.dylib!nssTrustDomain_FindTrustForCertificate [total: 100.0%, self: 0.0%]
f-7426. libnss3.dylib!nssToken_FindTrustForCertificate [total: 100.0%, self: 0.0%]
f-7427. libnss3.dylib!nssToken_FindObjectsByTemplate [total: 100.0%, self: 0.0%]
f-7428. libnss3.dylib!find_objects [total: 100.0%, self: 0.0%] ← inside NSS, holds session lock
f-7619. XUL!osclientcerts::C_FindObjectsInit [total: 100.0%, self: 0.0%]
f-7620. XUL!rsclientcerts::manager::Manager<B>::start_search [total: 100.0%, self: 0.0%]
f-7621. XUL!rsclientcerts::manager::Manager<B>::maybe_find_new_objects [total: 100.0%, self: 0.0%]
f-7622. XUL!<osclientcerts::backend_macos::Backend as ...>::find_objects [total: 100.0%, self: 0.0%]
f-7623. XUL!futures_executor::local_pool::block_on [total: 100.0%, self: 0.0%]
f-7624. XUL!futures_executor::local_pool::run_executor [total: 100.0%, self: 0.0%]
...
f-7628. XUL!std::thread::park [total: 100.0%, self: 0.0%]
f-7629. XUL!std::thread::Thread::park [total: 100.0%, self: 0.0%]
f-7630. XUL!std::sys::sync::thread_parking::darwin::Parker::park [total: 100.0%, self: 0.0%]
f-7615. libdispatch.dylib!_dispatch_semaphore_wait_slow [total: 100.0%, self: 0.0%]
f-7616. libsystem_kernel.dylib!semaphore_wait_trap [total: 100.0%, self: 100.0%] ← blocked
```
BgIOThreadPool #1 is running `nsNSSCertificateDB::AsyncHasThirdPartyRoots`, a function called by the DoH heuristics system at startup. It acquired the NSS session lock (inside `PK11_ListCerts` -> `find_objects`), and then the `find_objects` callback dispatched async work to the osclientcerts thread. The thread is now parked via `block_on`, waiting for osclientcerts to complete the work.
The NSS session lock that the Socket Thread is waiting for is held here, by this blocked thread.
---
## Inspect the osclientcerts thread
```
profiler-cli thread select t-35 --session nss-deadlock
profiler-cli thread samples-top-down --session nss-deadlock
```
```
Thread: osclientcerts
Top-Down Call Tree:
f-0. (root) [total: 100.0%, self: 0.0%]
└─ f-2789. libsystem_pthread.dylib!_pthread_start [total: 100.0%, self: 0.0%]
...
f-10190. XUL!moz_task::dispatcher::RunnableFunction<F>::allocate::Run [total: 100.0%, self: 0.0%]
f-10191. XUL!moz_task::dispatcher::RunnableFunction<F>::Run [total: 100.0%, self: 0.0%]
f-10192. XUL!moz_task::executor::schedule::{{closure}} [total: 100.0%, self: 0.0%]
...
f-10194. XUL!async_task::runnable::Runnable::run [total: 100.0%, self: 0.0%]
f-10195. XUL!async_task::raw::RawTask<F,T,S>::run [total: 100.0%, self: 0.0%]
f-10196. XUL!<osclientcerts::backend_macos::Backend as ...>::find_objects::{{closure}} [total: 100.0%, self: 0.0%]
f-10197. XUL!osclientcerts::backend_macos::find_objects [total: 100.0%, self: 0.0%]
f-10198. XUL!core::ptr::drop_in_place<(core::result::Result<...>,...)> [total: 100.0%, self: 0.0%]
f-10199. XUL!core::ptr::drop_in_place<osclientcerts::backend_macos::Key> [total: 100.0%, self: 0.0%]
f-10200. XUL!core::ptr::drop_in_place<osclientcerts::backend_macos::ThreadSpecificHandles> [total: 100.0%, self: 0.0%]
f-10201. XUL!<osclientcerts::backend_macos::ThreadSpecificHandles as core::ops::drop::Drop>::drop [total: 100.0%, self: 0.0%]
f-7623. XUL!futures_executor::local_pool::block_on [total: 100.0%, self: 0.0%] ← Drop calls block_on on its own thread
f-7624. XUL!futures_executor::local_pool::run_executor [total: 100.0%, self: 0.0%]
...
f-7628. XUL!std::thread::park [total: 100.0%, self: 0.0%]
f-7629. XUL!std::thread::Thread::park [total: 100.0%, self: 0.0%]
f-7630. XUL!std::sys::sync::thread_parking::darwin::Parker::park [total: 100.0%, self: 0.0%]
f-7615. libdispatch.dylib!_dispatch_semaphore_wait_slow [total: 100.0%, self: 0.0%]
f-7616. libsystem_kernel.dylib!semaphore_wait_trap [total: 100.0%, self: 100.0%] ← blocked
```
The osclientcerts thread was already running a `find_objects` task. During that work, a `ThreadSpecificHandles` value went out of scope. Its `Drop` implementation calls `futures_executor::local_pool::block_on` to run async cleanup. That `block_on` parks the osclientcerts thread, waiting for the future to complete on the thread-local executor.
The problem: the future needs the osclientcerts thread to run it. But the osclientcerts thread is parked inside `block_on`. The thread is waiting for itself. It will never make progress.
---
## Confirm the self-deadlock with the bottom-up view
```
profiler-cli thread samples-bottom-up --session nss-deadlock
```
```
Bottom-Up Call Tree:
f-7616. libsystem_kernel.dylib!semaphore_wait_trap [total: 100.0%, self: 100.0%]
└─ f-7615. libdispatch.dylib!_dispatch_semaphore_wait_slow [total: 100.0%, self: 0.0%]
f-7630. XUL!std::sys::sync::thread_parking::darwin::Parker::park [total: 100.0%, self: 0.0%]
f-7629. XUL!std::thread::Thread::park [total: 100.0%, self: 0.0%]
f-7628. XUL!std::thread::park [total: 100.0%, self: 0.0%]
f-7627. XUL!futures_executor::local_pool::run_executor::{{closure}} [total: 100.0%, self: 0.0%]
...
f-7623. XUL!futures_executor::local_pool::block_on [total: 100.0%, self: 0.0%]
f-10201. XUL!<osclientcerts::backend_macos::ThreadSpecificHandles as core::ops::drop::Drop>::drop [total: 100.0%, self: 0.0%]
f-10200. XUL!core::ptr::drop_in_place<osclientcerts::backend_macos::ThreadSpecificHandles> [total: 100.0%, self: 0.0%]
f-10199. XUL!core::ptr::drop_in_place<osclientcerts::backend_macos::Key> [total: 100.0%, self: 0.0%]
f-10198. XUL!core::ptr::drop_in_place<(core::result::Result<...>,...)> [total: 100.0%, self: 0.0%]
f-10197. XUL!osclientcerts::backend_macos::find_objects [total: 100.0%, self: 0.0%]
f-10196. XUL!<osclientcerts::backend_macos::Backend as ...>::find_objects::{{closure}} [total: 100.0%, self: 0.0%]
f-10195. XUL!async_task::raw::RawTask<F,T,S>::run [total: 100.0%, self: 0.0%]
f-10194. XUL!async_task::runnable::Runnable::run [total: 100.0%, self: 0.0%]
...
f-10190. XUL!moz_task::dispatcher::RunnableFunction<F>::allocate::Run [total: 100.0%, self: 0.0%]
f-26. XUL!nsThread::ProcessNextEvent(bool, bool*) [total: 100.0%, self: 0.0%]
...
f-0. (root) [total: 100.0%, self: 0.0%]
```
The bottom-up view confirms the full call chain reading upward from the leaf: `semaphore_wait_trap` is reached from `block_on`, which was called by the `Drop` implementation of `ThreadSpecificHandles`, inside an active `find_objects` async closure running on the thread's own executor. The thread is blocked waiting for the executor to run a future that the executor cannot run because the thread is blocked.
---
## The deadlock cycle
Three threads form a cycle:
```
Socket Thread
blocked waiting for: NSS session lock
held by: BgIOThreadPool #1
blocked waiting for: osclientcerts thread
blocked waiting for: itself (Drop handler calling block_on)
```
- The osclientcerts thread is self-deadlocked and will never complete.
- BgIOThreadPool #1 is blocked waiting for osclientcerts and will never release the NSS lock.
- The Socket Thread is blocked waiting for the NSS lock and will never proceed.
All network connections that require TLS certificate work are frozen. No new HTTPS connection can be set up. The browser becomes unable to load any page.
The root trigger is `nsNSSCertificateDB::AsyncHasThirdPartyRoots`, invoked at Firefox startup by the DoH heuristics module (`DoHHeuristics.sys.mjs`). It should not have been acquiring the NSS session lock in a way that could overlap with the osclientcerts thread's internal async work. The fix removed the `thirdPartyRoots` check from DoHHeuristics entirely.
---
## Clean up
```
profiler-cli stop --session nss-deadlock
```