Zsim

简介

Zsim模拟器是

• 一个快速的x86-64多核模拟器，它利用英特尔Pin工具包收集进程的内存访问跟踪，并在zsim模拟器中重放跟踪。
• 一种基于 PIN 的二进制插桩工具，可以在运行时对指令进行插桩，并插入自定义的函数调用，用于模拟内存子系统的行为。
• 它最初是为了评估ZCache而编写的，但后来它扩展了其功能
• zsim的主要目标是快速，简单和准确，重点是模拟内存层次结构和大型异构系统

模拟器发展的部分比较

• Interval Simulation: Raising the Level of Abstraction in Architectural Simulation(HPCA 2010) 提出Interval Simulation
  • Sniper: exploring the level of abstraction for scalable and accurate parallel multi-core simulation(SC’2011)
    • 速度：2.0 MIPS when simulating a 16-core system on an 8-core SMP machine.
• ZSim: Fast and Accurate Microarchitectural Simulation of Thousand-Core Systems(ISCA 2013)
  • 与sniper的区别
    • 模型不同Zsim 是instruction-driven timing models
    • OOO scoreboard + uops based ROB (所以需要知道每条uops的execution time)
  • 速度
    • 单核速度 The combination of DBT and instruction-driven models allows us to achieve 20 MIPS per simulated core, and slowdowns of about 200×. This is faster than Sniper [8], which uses approximate OOO models (though slower memory system simulation likely plays a role in the difference),
    • Sniper多核模拟不理想是因为频繁的多核同步，Zsim发现可以记录访存行为集中时间统一同步, 可以做到近线性的加速比
• An Evaluation of High-Level Mechanistic Core Models 2014, (TACO) 提出instruction-window centric (IW-centric) core model完善了Interval Simulation的细节错误
  • 比如将old-new windows替换成ROB，并且关注其内指令依赖

工作原理

利用了X86上的pin来控制进程，但是pin没有开源代码，底层实现不知

Recommended Installations

1. It is highly recommended to install the respective open-source Docker versions for your system. [^1]
2. Note that the official version, last updated in 2015, is outdated and corresponds to Ubuntu 12.04 and Pin 2.14.
3. For an unofficial but improved version that supports Ubuntu 18.04 and Pin 3.7, you can explore the following link: Ubuntu 18.04 + Pin 3.7.

各版本Zsim开源魔改版本比较

Zsim 魔改版本根据需求有所不同，我接触过的是访存接入HMC 3D堆叠内存，和对地址翻译有支持两类。

1. PIM支持：
   1. ramulator-pim
   2. DAMOV
2. TLB + 地址翻译：HSCC

安装：Zsim+tlb

基于Zsim+tlb的版本

STEP1: boost

The Boost C++ libraries are a collection of open-source C++ libraries that provide support for tasks and functionalities commonly used in C++ development. Install-ref

# newest 1.83 not support for missing api, use github specific 1.59 version passed
wget http://sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.tar.gz
# tar & cd
./bootstrap.sh --prefix=$HOME/.local/icarus1
./b2 install # will new include and lib in the prefix_path

STEP2: HDF5

The Hierarchical Data Format version 5 (HDF5) is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes and is designed for flexible and efficient I/O and high volume and complex data.
Install-Ref

wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.14/hdf5-1.14.2/src/hdf5-1.14.2.tar.gz
# tar & cd
./configure --prefix=$HOME/.local/icarus1
make -j16
make install

STEP3: zsim+tlb

cd zsim-nvmain
# install
source env.sh # configed
# can SConstruct and active O3, 10X speed up
# env["LINKFLAGS"] = " -O3 -finline "
python2 `which scons` -j16

# remove
source env.sh # configed
python2 `which scons` -c # clean
# rm -rf build
# rm -rf bin

安装：multipim (failure)

According to github, the multipim seems a combination of Zsim-ramulator +
Booksim2 and HSCC.

# Install
cd MultiPIM
sudo sh setup.sh
# `SConstruct` is written in `python2`, you should change all python2 `print` error.
sh compile.sh opt

1. check file env.sh
2. replace the include zsim_hooks.h path in testcase code to misc/hooks/zsim_hooks.h
3. REPLACE: all /u/yc9uf/Workspace/MultiPIM and /home/yuchao/Workspace/PIM/MultiPIM in project to your path

# Compile Test-case
cd tests/benchmarks/Polybench/
./compile.sh

# test simulation
cd tests
mkdir output
source ./run_pim.sh

# Fucking failure: [snode6 ]Pin app terminated abnormally due to signal 4(means Illegal Instruction). [icarus0] signal 6(system call 'abort()') 
# And nothing more debug info or file.

Install-problem

1. aclocal-1.15: command not found
   1. FIX: add autoreconf -f -i before ./configure command
2. makeinfo: command not found
   1. FIX: sudo apt-get install texinfo

[SKIP] Bug: Pin 2.14 Fails on Ubuntu 20.04.6 LTS (snode6)

1. The initial approach was to replace it with zsim-tlb pim (but it turned out to be a bad idea, even for the older Pin version 2.13.)
   1. after replace the pin folder content by ../HSCC/zsim-nvmain/pin_kit, you should reinstall all the multipim from sh compile.sh opt clean. (of course chmod +x bin) which lead to the same error Pin app terminated abnormally due to signal 6.
2. The next idea is try the newest pin version pin3.28
   6. First TODO: After repeatedly adding and modifying include files to solve various compilation issues, we encountered even more problems. Maybe we should analyse how pintools include nwest pin head files
      1. the point is how pintool utilize the newest CRT (C Runtime) part code of pin
3. TODO: We should delve deeper into understanding Pin’s usage of grammar.

Running in docker

Utilizing docker like DAMOV[^1], is an effective approach to circumvent version conflicts between the Linux kernel and tools like Pin.

Here’s a Dockerfile snippet for reference:

FROM gfojunior/damov:latest

WORKDIR /root
RUN apt update \
   && apt install -y libcurl4 \
   vim \
   libxerces-c-dev \
   libboost-all-dev \
   libcurl4-openssl-dev \
   texinfo \
   ack-grep \
   && apt clean \
   && rm -rf /var/lib/apt/lists/* \

Docker is built on top of the Linux kernel

And pin is conflict with linux kernel. But by using Docker and modifying the underlying Ubuntu version to 18.04, you effectively address the compatibility issue. This suggests that the conflict may be rooted in differences between the Ubuntu libraries or other aspects of the system environment.

基本使用

app target code instrumentation

#include "zsim_hooks.h"
foo(){
    /*
    * zsim_roi_begin() marks the beginning of the region of interest (ROI).
    * It must be included in a serial part of the code.
    */
	zsim_roi_begin();
	zsim_PIM_function_begin(); // Indicates the beginning of the code to simulate (hotspot).
	...
	zsim_PIM_function_end(); // Indicates the end of the code to simulate.
    /*
    * zsim_roi_end() marks the end of the ROI.
    * It must be included in a serial part of the code.
    */
	zsim_roi_end();
}

It seems hard to simulate command like mpirun -n 32 exe, even instrumentate the executable file. Because mpirun start 32 individual processes which will be zombie after interupting the zsim simulation.

there is a slolution

zsim config

配置参考官方

1. 问题1: 可执行文件需要在特定目录执行，如何解决cd PWD; zsim config
   1. set env="PWD=xxx" seems useless
2. 问题1导致无法并行，会写同一个目录的文件？
   1. Can config the output directory? default zsim not support, but our version of zsim support this.

zsim模拟多进程的资源冲突

使用zsim的config，可以添加process0和process1，对应两个进程

代码部分阅读

zsim tlb part

基于代码 zsim-nvmain

1. futex usage is very helpful
2. using 4MB page pgt_walker : mode = "Legacy_Huge";

zsim 异构计算核模拟

https://github.com/s5z/zsim/blob/master/tests/het.cfg

Result file

zsim.out

contention: # Contention simulation stats
 domain-0: # Domain stats
  time: 311,889,628 # Weave simulation time

It seems something about event contention according to the blog. And it explains how zsim use a bound-weave model to simulate the event contention(eg. multiple threads access to the cache).

This can be easily understood if you read the zsim paper. The idea is gathering the contention traces to weave phase to be dealed with.

time: # Simulator time breakdown
 init:    7,058,723,186
 bound: 323,052,948,592 # real time 5:35 = 335 seconds
 weave:   4,540,545,954
 ff: 0                  # finishing time

the real time simulation about the bound-weave model.

Bugs

zsim-tlb simulation encounter bugs, such as

pinbin: build/opt/zsim.cpp:816: LEVEL_BASE::VOID VdsoCallPoint(LEVEL_VM::THREADID): Assertion `vdsoPatchData[tid].level' failed.
Pin app terminated abnormally due to signal 6.

需要进一步的研究学习

暂无

遇到的问题

1. ZSim+Ramulator, 中计算单元是怎么模拟的。
   1. 如果Zsim是基于Pin，那么怎么模拟计算单元的行为呢？

开题缘由、总结、反思、吐槽~~

参考文献

zsim code implementation details in this blog

[^1]: damov github