Content Comparison

...

From the examples mentioned above, it is evident that the TLSDESC access mode provides significant optimization for accessing TLS variables on the static TLS block compared to GD. The TLSDESC mode directly returns the offset of the static TLS block in the GOT table entry, while GD requires accessing the dtv array to calculate the address.

动态加载库初始化TLS

bionic通过dlopen动态加载库，初始TLS的步骤如下：

...

Dynamic Loading Library Initialization of TLS

In bionic, when dynamically loading a library through dlopen, the initialization steps for TLS are as follows:

In the do_dlopen ->find> find_library ->soinfo> soinfo::register_soinfo_tls→registertls -> register_tls_module流程中，获得module id，并加入至TlsModules类型变量中在soinfomodule flow, the module ID is obtained and added to the TlsModules variable.
In the soinfo::relocate ->plain> plain_relocate ->plain> plain_relocate_impl ->process_relocation→process> process_relocation -> process_relocation_impl流程中重定位初始化TLS变量的GOT表项
GD：重定位类型包括R
impl flow, the relocation initializes the TLS variables' GOT table entries.

For GD (Global Dynamic): The relocation types include R_AARCH64_TLS_

...

DTPMOD64 and R_AARCH64_TLS_

...

LD：aarch64对LD的实现与GD相同

...

IE/LE：不支持

...

TLSDESC：重定位类型为R_AARCH64_TLSDESC，将其相邻的GOT表项初始化为tlsdesc_resolver_dynamic函数地址和TlsDynamicResolverArg类型变量地址

a. 初始化TlsDynamicResolverArg中TlsIndex的module id以及offset，offset的值为TLS变量在其TLS程序段的偏移量。另外初始化更新标志为库的更新标志，该标志表示动态库是否有更新;
b. 为了存储TlsDynamicResolverArg类型变量，将变量保存在soinfo::tlsdescargs数组中，为处理数组重新分配内存，Relocator::deferred_tlsdesc_relocs缓冲重定位信息，当该库的所有重定位操作完成后，再更新TLS变量的GOT表项.

线程创建过程中初始化TLS

调用pthread_create创建线程，需要对主程序上的所有TLS数据结构进行拷贝。（pthread_create->__allocate_thread）

调用__allocate_thread_mapping分配线程栈空间，包含了静态TLS块空间。（Allocate in order: stack guard, stack, static TLS, guard page）
调用__init_static_tls，将TlsModules类型变量中动态库的TLS段内容拷贝至静态TLS块空间中。
调用__init_tcb更新bionic_tcb
调用__init_tcb_dtv初始bionic_tcb中的TLS_SLOT_DTV，其更新标志值为0。
调用__init_bionic_tls_ptrs更新bionic_tls地址
调用clone，将静态TLS块地址传递给clone，由内核设置段寄存器tpidr_el0值

__tls_get_addr函数实现

GD/LD访问方式使用tls_get_addr函数获取TLS变量绝对地址。tls_get_addr函数涉及对dtv数据更新，其更新的条件由3个更新标志（generation）控制

全局generation，保存在__libc_tls_generation_copy，为TlsModules::generation一个副本，每次新增拥有TLS程序段的动态库时，递增该值，表示有动态库新增。不需要处理动态库删除问题
dtv数组中的generation，保存在数组中的第一个元素，初始化为0，每次更新dtv数组时，更新generation只为当时的全局generation值。与全局generation不相等，说明有新的动态库加载，需要更新dtv数组内容
动态库的generation，保存在TlsModule::first_generation，该值初始化为加载该库时全局generation的值。该值用于判断dtv指向的动态库是否有变化，即是否为旧的动态库

代码块

struct TlsIndex {
  size_t module_id;
  size_t offset;
};

// ti的值保存在动态库的GOT表项中，在重定位时初始化，占两个表项内容

extern "C" void* __tls_get_addr(const TlsIndex* ti){

//  获取dtv数组

TlsDtv* dtv = __get_tcb_dtv(__get_bionic_tcb());

// 获取全局动态库更新标志

size_t generation = atomic_load(&__libc_tls_generation_copy);
  if (__predict_true(generation == dtv->generation)) {

void* mod_ptr = dtv->modules[__tls_module_id_to_idx(ti->module_id)];
    if (__predict_true(mod_ptr != nullptr)) {

      // 无动态库更新，且内存已分配，则进入快速路径，返回TLS变量偏移地址
      return static_cast<char*>(mod_ptr) + ti->offset + TLS_DTV_OFFSET;
    }

    // 延时分配动态库的动态TLS块内存，只有访问该动态库的TLS变量时才分配内存，进入慢速路径
  }

  // 有动态库更新或者第一次访问，进入dtv和动态TLS块的分配和初始化

  return tls_get_addr_slow_path(ti);
}

tls_get_addr_slow_path函数包含dtv和动态TLS块的分配和初始化.

...

DTPREL64. The adjacent GOT table entries are initialized with the module ID and the variable's offset within its TLS segment.
For LD (Local Dynamic): The implementation for AArch64 is the same as GD.
For IE (Initial Executable) / LE (Local Executable):They are not supported.
For TLSDESC: The relocation type is R_AARCH64_TLSDESC. The adjacent GOT table entries are initialized with the address of the tlsdesc_resolver_dynamic function and the address of the TlsDynamicResolverArg variable.
- a. Initialize the TlsIndex in TlsDynamicResolverArg with the module ID and the offset of the TLS variable within its TLS program segment. Additionally, initialize the update flag with the library's update flag, which indicates whether the dynamic library has been updated.
- b. To store the TlsDynamicResolverArg variable, it is saved in the soinfo::tlsdescargs array. To handle reallocation of the array, the Relocator::deferred_tlsdesc_relocs buffer defers relocation information. The TLS variable's GOT table entries are updated once all relocation operations for the library are completed.

Initialization of TLS during Thread Creation

When creating a thread using pthread_create, the following steps are involved in initializing TLS:

The TLS data structures on the main program are copied. This is done within the pthread_create function, specifically in the __allocate_thread function.
The __allocate_thread_mapping function is called to allocate the thread's stack space, which includes the static TLS block. The allocation order includes the stack guard, stack, static TLS block, and guard page.
The __init_static_tls function is called to copy the contents of the TLS segment from the TlsModules variable of dynamic libraries to the static TLS block.
The __init_tcb function is called to update the bionic_tcb (Thread Control Block).
The __init_tcb_dtv function is called to initialize the TLS_SLOT_DTV in the bionic_tcb, with the update flag set to 0.
The __init_bionic_tls_ptrs function is called to update the bionic_tls addresses.
The clone system call is invoked, passing the address of the static TLS block to clone. The kernel then sets the value of the tpidr_el0 register, which represents the thread pointer, according to the provided static TLS block address.

Implementation of the __tls_get_addr Function

The __tls_get_addr function is used by GD (Global Dynamic) and LD (Local Dynamic) access methods to retrieve the absolute address of a TLS variable. This function involves updating the dtv (Dynamic Thread Vector) data, and the update conditions are controlled by three generation flags.

Global Generation: The global generation is stored in __libc_tls_generation_copy, which is a copy of TlsModules::generation. Each time a dynamic library with a TLS program segment is added, this value is incremented to indicate the addition of a new dynamic library. There is no need to handle dynamic library removal.
Generation in dtv Array: The generation value in the dtv array is stored in the first element of the array, initialized to 0. When updating the dtv array, the generation is updated to match the current global generation value. If it is not equal to the global generation, it indicates that a new dynamic library has been loaded, and the contents of the dtv array need to be updated.
Generation in Dynamic Library: The generation value in the dynamic library is stored in TlsModule::first_generation. This value is initialized with the global generation value when the library is loaded. It is used to determine if the dynamic library pointed to by dtv has changed, i.e., whether it is an old dynamic library.

代码块

struct TlsIndex {
  size_t module_id;
  size_t offset;
};

// The value of "ti" (Thread Index) is stored in the GOT (Global Offset Table) entries of the dynamic library. It is initialized during relocation and occupies two entries in the table.

extern "C" void* __tls_get_addr(const TlsIndex* ti) {
  TlsModules& modules{

//  get the dtv

TlsDtv* dtv = __libcget_sharedtcb_globalsdtv()->tls_modules;
  bionic_tcb* tcb = __get_bionic_tcb());

 ScopedSignalBlocker ssb;

  // 互斥写，防止多线程同时修改// retrieve the global dynamic library update flag

size_t generation = atomic_load(&__libc_shared_globals()->tls_modules全局变量
  ScopedWriteLock locker(&modules.rwlocktls_generation_copy);
   // 更新dtv数组或者重新分配数组内存

  update_tls_dtv(tcb);

  TlsDtv* dtv = __get_tcb_dtv(tcb);
  const size_t module_idx = if (__predict_true(generation == dtv->generation)) {

void* mod_ptr = dtv->modules[__tls_module_id_to_idx(ti->module_id)];
  void*   if (__predict_true(mod_ptr != dtv->modules[module_idx];
nullptr)) {
 if (mod_ptr == nullptr)
{      // 不存在，则分配内存，将动态库TLS程序段内容拷贝至新内存，并初始化该模块指针
    const TlsSegment& segment = modules.module_table[module_idx].segment;return static_cast<char*>(mod_ptr) + ti->offset + TLS_DTV_OFFSET;
    }

  }
mod_ptr
=  return tls_get_libcaddr_sharedslow_globalspath()->tls_allocator.memalign(segment.alignment, segment.size);
    if (segment.init_size > 0ti);
}

The tls_get_addr_slow_path function includes the allocation and initialization of dtv (Dynamic Thread Vector) and the dynamic TLS block.

代码块

__attribute__((noinline)) static void* tls_get_addr_slow_path(const TlsIndex* ti) {
  TlsModules& modules =  memcpy(mod_ptr, segment.init_ptr, segment.init_size)__libc_shared_globals()->tls_modules;
  bionic_tcb* tcb }
    dtv->modules[module_idx] = mod_ptr= __get_bionic_tcb();
  ScopedSignalBlocker ssb;

  // ReportsTo theprevent allocationmultiple tothreads thefrom listener,simultaneously ifmodifying any.
    if (modules.on_creation_cb != nullptr) {
      modules.on_creation_cb(mod_ptr, static_cast<void*>(static_cast<char*>(mod_ptr) + segment.size))the __libc_shared_globals()->tls_modules global variable, you can use a mutex to enforce mutual exclusion
  ScopedWriteLock locker(&modules.rwlock);

   }
  }
// update the dtv array or reallocate its memory

 return static_cast<char*>(mod_ptr) + ti->offset + TLS_DTV_OFFSET;
}

update_tls_dtv动态分配dtv数组空间，代码太多不再列出，其实现步骤如下：

更新dtv数组的条件：dtv数组的更新标志与全局动态库更新标志不相等，说明动态库有更新，当前指向的为旧的动态库
重新分配dtv数组，条件条件为拥有TLS程序段的动态库总数量大于dtv数组大小
- a. 根据动态库数量重新分配dtv数组空间
- b. 将dtv数组中的内容备份至新的dtv数组空间
- c. 调用__set_tcb_dtv更新为新的dtv数组
- d. 为实现无锁操作，不释放旧的dtv数组空间，而是将其插入一个垃圾回收队列中，待程序结束时回收
重新更新静态TLS块对应动态库的dtv数组元素
重新更新动态TLS块对应动态库的dtv数组元素，条件：动态库的更新标志大于dtv数组更新标志（表明dtv数组指向的为旧的动态库）
- a. 释放旧的动态库动态TLS块内存
- b. 将dtv数组元素清零
更新dtv数组的更新标志为全局动态库更新标志

TLSDESC访问方式实现

TLSDESC访问方式有两种方式获取TLS变量相对静态TLS块的偏移地址：一种对于静态TLS块上的TLS变量由tlsdesc_resolver_static获取，另一种对于动态TLS块的TLS变量由tlsdesc_resolver_dynamic获取。这两种方式都采用汇编实现，不遵循C/C++函数调用的寄存器传参规范。其使用规范中的返回值寄存器传参，如aarch64的x0寄存器，x86_64的rax寄存器。

代码块

/* Type used to represent a TLS descriptor in the GOT.  */

struct TlsDescriptor {
  TlsDescResolverFunc* func;
  size_t arg;
};

// tlsdesc_resolver_static函数，其TlsDescriptor::arg值为TLS变量的相对偏移

// tlsdesc_resolver_dynamic函数，其TlsDescriptor::arg值为下列类型变量地址

struct TlsDynamicResolverArg {
  size_t generation;
  TlsIndex index;
};

struct TlsIndex {
  size_t module_id;
  size_t offset;
};

tlsdesc_resolver_static的实现相当简单，返回TlsDescriptor::arg值即可。

tlsdesc_resolver_dynamic对更新标志的判断有所优化，共有4个generation更新标志：

全局generation，保存在__libc_tls_generation_copy，为TlsModules::generation一个副本，每次新增拥有TLS程序段的动态库时，递增该值，表示有动态库新增。不需要处理动态库删除问题
dtv数组中的generation，保存在数组中的第一个元素dtv[0]，初始化为0，每次更新dtv数组时，更新generation只为当时的全局generation值。与全局generation不相等，说明有新的动态库加载，需要更新dtv数组内容
动态库的generation，保存在TlsModule::first_generation，该值初始化为加载该库时全局generation的值。该值用于判断dtv指向的动态库是否有变化，即是否为旧的动态库
TLS变量GOT表项中指向的TlsDynamicResolverArg::generation，该值初始化为TlsModule::first_generation值，该值只要不大于dtv数组中的generation不需要重新分配dtv数组，否则表示该动态库的TLS程序段在dtv数组中未初始化。

tlsdesc_resolver_dynamic实现步骤：

快速路径，条件：TlsDynamicResolverArg::generation <= dtv[0] && dtv[mod_id] != NULL a. 返回 dtv[mod_id] + TlsDynamicResolverArg::TlsIndex::offset相对于静态TLS块的偏移
慢速路径，调用__tls_get_addr获取TLS变量的绝对地址，返回与静态TLS块的相对偏移

gnu TLS数据结构初始化流程

原理与bionic TLS数据结构初始化流程类似，不再详述。两者的差异包括：

静态加载库初始化TLS时，gnu会为静态TLS块预留144字节空间
动态加载库初始化TLS时，先从预留的静态TLS块获取空间，不足时采用动态TLS块，这种方式可满足调用dlopen动态加载的TLS变量IE访问方式的动态库
TLSDESC实现函数名称不同：_dl_tlsdesc_return/_dl_tlsdesc_dynamic/_dl_tlsdesc_undefweak -> tlsdesc_resolver_static/tlsdesc_resolver_dynamic/tlsdesc_resolver_unresolved_weak

预留静态TLS空间的作用有两个，一个是支持动态加载IE访问模式的库，另一个是优化TLSDESC访问模式性能。

在linux环境下，图形加速库（OpenGL/EGL）的使用预留静态TLS空间的典型应用。一般linux应用程序会使用图形API转发库，如glvnd，图形API转发库通过dlopen动态加载OpenGL/EGL库，而OpenGL/EGL库一般都使用了IE访问模式的TLS变量，通常是一个指针变量，指向一个数据结构，从而减少静态TLS块预留空间的占用。

预留静态TLS空间注意事项：

...

分配时机：glibc在重定位时尝试分配静态TLS空间，支持的两个重定位类型，分别为R_AARCH64_TLSDESC和R_AARCH64_TLS_TPREL

...

初始化数据：对所有线程的静态TLS空间进行初始化，TLS数据结构在线程栈上，有的线程使用用户栈，有的使用系统栈，在_dl_init_static_tls函数中实现

...

update_tls_dtv(tcb);

  TlsDtv* dtv = __get_tcb_dtv(tcb);
  const size_t module_idx = __tls_module_id_to_idx(ti->module_id);
  void* mod_ptr = dtv->modules[module_idx];
  if (mod_ptr == nullptr) {

    // If the dtv array does not exist, you would need to allocate memory, copy the contents of the dynamic library's TLS program segment to the new memory, and initialize the module pointer.
    const TlsSegment& segment = modules.module_table[module_idx].segment;
    mod_ptr = __libc_shared_globals()->tls_allocator.memalign(segment.alignment, segment.size);
    if (segment.init_size > 0) {
      memcpy(mod_ptr, segment.init_ptr, segment.init_size);
    }
    dtv->modules[module_idx] = mod_ptr;

    // Reports the allocation to the listener, if any.
    if (modules.on_creation_cb != nullptr) {
      modules.on_creation_cb(mod_ptr, static_cast<void*>(static_cast<char*>(mod_ptr) + segment.size));
    }
  }

  return static_cast<char*>(mod_ptr) + ti->offset + TLS_DTV_OFFSET;
}

The steps to implement the update_tls_dtv function for dynamically allocating dtv array space are as follows:

Condition for updating the dtv array: If the update flag of the dtv array is not equal to the global dynamic library update flag, it indicates that the dynamic library has been updated and the current dtv array points to the old dynamic library.
Reallocate the dtv array if the total number of dynamic libraries with TLS program segments is greater than the size of the dtv array.

Allocate new space for the dtv array based on the number of dynamic libraries.
Backup the contents of the dtv array to the new space.
Call __set_tcb_dtv to update to the new dtv array.
To achieve lock-free operation, don't free the old dtv array space. Instead, insert it into a garbage collection queue for later cleanup at the end of the program.

Update the dtv array elements corresponding to the static TLS block and dynamic TLS blocks of the dynamic libraries.

Free the memory of the old dynamic TLS block
Clear the dtv array element.

Update the update flag of the dtv array to the global dynamic library update flag.

The access method of TLSDECS

The TLSDESC access method provides two ways to obtain the offset address of a TLS variable relative to the static TLS block: tlsdesc_resolver_static for TLS variables on the static TLS block and tlsdesc_resolver_dynamic for TLS variables on the dynamic TLS block. Both of these methods are implemented using assembly language and do not adhere to the register parameter passing conventions of C/C++ function calls. Instead, they use the return value register for passing parameters, such as the x0 register for AArch64 or the rax register for x86_64.

代码块

/* Type used to represent a TLS descriptor in the GOT.  */

struct TlsDescriptor {
  TlsDescResolverFunc* func;
  size_t arg;
};

// tlsdesc_resolver_static, Its TlsDescriptor::arg value is the relative offset of the TLS variable
// tlsdesc_resolver_dynamic, tlsdesc_resolver_dynamic function, its TlsDescriptor::arg value is the address of the following type variable

struct TlsDynamicResolverArg {
  size_t generation;
  TlsIndex index;
};

struct TlsIndex {
  size_t module_id;
  size_t offset;
};

The implementation of tlsdesc_resolver_static is straightforward as it simply returns the value of TlsDescriptor::arg.

For tlsdesc_resolver_dynamic, there are several optimizations based on the update flags:

Global generation: Stored in __libc_tls_generation_copy, which is a copy of TlsModules::generation. It is incremented each time a dynamic library with TLS program segments is added, indicating the addition of a new dynamic library. There is no need to handle dynamic library removal.
Generation in the dtv array: Stored in the first element of the dtv array, dtv[0]. It is initialized to 0 and updated to the value of the global generation whenever the dtv array is updated. If it is different from the global generation, it indicates that new dynamic libraries have been loaded, and the dtv array needs to be updated.
Generation in the dynamic library: Stored in TlsModule::first_generation. It is initialized with the value of the global generation when the library is loaded. This value is used to determine if the dynamic library pointed to by dtv has changed, indicating an old dynamic library.
Generation in the TLS variable's GOT entry: Stored in TlsDynamicResolverArg::generation. It is initialized with TlsModule::first_generation. As long as it is not greater than the generation in the dtv array, there is no need to reallocate the dtv array. If it is greater, it means that the TLS program segment of that dynamic library is not initialized in the dtv array.

The implementation steps for tlsdesc_resolver_dynamic are as follows:

Fast path (conditions: TlsDynamicResolverArg::generation <= dtv[0] && dtv[mod_id] != NULL): Return dtv[mod_id] + TlsDynamicResolverArg::TlsIndex::offset as the relative offset to the static TLS block.
Slow path:

Call __tls_get_addr to obtain the absolute address of the TLS variable. and Calculate the relative offset to the static TLS block.

The initialization process of the GNU TLS data structure

It is similar to the Bionic TLS data structure, and the principles are comparable. The differences between them include:

When initializing TLS for statically loaded libraries, GNU reserves 144 bytes of space for the static TLS block.
When initializing TLS for dynamically loaded libraries, GNU first tries to obtain space from the reserved static TLS block. If it is insufficient, it falls back to the dynamic TLS block. This approach allows dynamic libraries using the IE access mode for TLS variables to be accessed properly when loaded with dlopen.
The function names for TLSDESC implementation differ: _dl_tlsdesc_return, _dl_tlsdesc_dynamic, and _dl_tlsdesc_undefweak in GNU correspond to tlsdesc_resolver_static, tlsdesc_resolver_dynamic, and tlsdesc_resolver_unresolved_weak respectively.

The purpose of reserving the static TLS space is twofold. Firstly, it supports dynamic libraries loaded with IE access mode for TLS variables. Secondly, it optimizes the performance of the TLSDESC access mode.

In the Linux environment, a typical use case for reserving static TLS space is in graphics acceleration libraries (such as OpenGL/EGL). Linux applications generally use graphics API dispatch libraries like glvnd, which dynamically load OpenGL/EGL libraries using dlopen. These OpenGL/EGL libraries often use TLS variables in IE access mode, typically a pointer variable pointing to a data structure. This reduces the space occupied by the reserved static TLS block.

Some considerations regarding reserving static TLS space are as follows:

Allocation Timing: glibc attempts to allocate the static TLS space during relocation and supports two relocation types: R_AARCH64_TLSDESC and R_AARCH64_TLS_TPREL.
Initialization Data: The static TLS space for all threads is initialized. The TLS data structures are located on the thread stack, with some threads using the user stack and others using the system stack. Initialization is performed in the _dl_init_static_tls function.
Concurrent Access: The allocation of reserved static TLS space is implemented in dl_open_worker_begin, and a large lock (dl_load_tls_lock) is acquired before calling the function. When initializing static TLS data, a lock (dl_stack_cache_lock) is obtained within the _dl_init_static_tls function to ensure thread safety.

版本	旧版本 4	新版本当前的
做出修改的是	kyy	kyy
保存于	2月 02, 2024	2月 04, 2024

版本比较

密钥

动态加载库初始化TLS

Dynamic Loading Library Initialization of TLS

线程创建过程中初始化TLS

__tls_get_addr函数实现

Initialization of TLS during Thread Creation

Implementation of the __tls_get_addr Function

TLSDESC访问方式实现

gnu TLS数据结构初始化流程

The access method of TLSDECS

The initialization process of the GNU TLS data structure

Content Comparison

版本比较

密钥

动态加载库初始化TLS​

Dynamic Loading Library Initialization of TLS

线程创建过程中初始化TLS​

__tls_get_addr函数实现​

Initialization of TLS during Thread Creation​

Implementation of the __tls_get_addr Function​

TLSDESC访问方式实现​

gnu TLS数据结构初始化流程​

The access method of TLSDECS​

The initialization process of the GNU TLS data structure

动态加载库初始化TLS

线程创建过程中初始化TLS

__tls_get_addr函数实现

Initialization of TLS during Thread Creation

Implementation of the __tls_get_addr Function

TLSDESC访问方式实现

gnu TLS数据结构初始化流程

The access method of TLSDECS