Posted Updated Programming21 minutes read (About 3103 words)

The CUDA Execution Model

导言

The CUDA Execution Model: how the CUDA architecture handles the execution of code in parallel.

这篇将聚焦于CUDA的基础知识，基本概念，

Read more
Posted Updated Tutorials11 minutes read (About 1603 words)

Github Access

导言

作为程序员，最经常遇到的问题就是无法访问github，这无异于和世界断开连接。

  1. 由于http代理可以代理DNS请求，所以不太可能是DNS污染的问题。
  2. github加速访问两种思路：
    1. VPN加速
    2. warp或者wg转发到墙外(linux 服务器)
  3. 之前能访问，但是现在不能访问，可能是wg配置重启掉了。

Proxy in terminal

ssh config

linux下通过按照如下修改.ssh/config设置账号密码，并 ssh -vT [email protected]，成功后输出Hi Kirrito-k423! You've successfully authenticated, but GitHub does not provide shell access.。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# .ssh/config
Host github.com
  User [email protected]
  Hostname ssh.github.com
  PreferredAuthentications publickey
  ProxyCommand nc -X 5 -x 127.0.0.1:7890 %h %p #如果通过代理需要这句话
  IdentityFile ~/.ssh/id_rsa
  Port 443

Host *
  # Win报错取消下面三行 getpeername failed: Not a socket getsockname failed: Not a socket
  ControlMaster auto
  ControlPath /tmp/sshcontrol-%C
  ControlPersist 1d
  ServerAliveInterval 30

Windows PowerShell 平台

假如是windows下，如果安装了git bash，会有connect.exe的程序

配置如下[^1]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Host github.com
  User git
  Port 22
  Hostname github.com
  # 注意修改路径为你的路径
  IdentityFile "C:\Users\Administrator\.ssh\id_rsa"
  TCPKeepAlive yes
  # 这里的 -a none 是 NO-AUTH 模式，参见 https://bitbucket.org/gotoh/connect/wiki/Home 中的 More detail 一节
  ProxyCommand E:\\commonSoftware\\Git\\mingw64\\bin\\connect.exe -S 127.0.0.1:7890 -a none %h %p

Host ssh.github.com
  User git
  Port 443
  Hostname ssh.github.com
  # 注意修改路径为你的路径
  IdentityFile "C:\Users\Administrator\.ssh\id_rsa"
  TCPKeepAlive yes

debug ssh clone/push

ssh-git 与 https-git的不同

1
2
3
4
git config --global http.proxy localhost:7890 # PowerShell proxy
git config --global http.proxy "http://127.0.0.1:7890"
git config --global https.proxy "http://127.0.0.1:7890"
GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone [email protected]:Kirrito-k423/autoUpdateIpconfigPushGithub.git

1
GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://github.com/llvm/llvm-project.git

Windows PowerShell 平台 git push –verbose

不同于linux平台的GIT_TRACE=1 git push,Windows PowerShell 平台应该如下设置：

1
2
3
$env:GIT_CURL_VERBOSE = 1
$env:GIT_TRACE = 1
git push

ssh成功，但是git操作还是失败

没使用上指定config文件，git操作需要明确指定。

1
2
$env:GIT_SSH_COMMAND = 'ssh -F /path/to/your/ssh_config'
git push

http代理

There are tons of identical solutions over the internet for defining proxy tunnel for git’s downloads like this one, which all is by setting git’s https.proxy & http.proxy config. but those answers are not working when you try to clone/push/pull etc. over the ssh protocol!

For example, by setting git config --global https.proxy socks5://127.0.0.1:9999 when you try to clone git clone [email protected]:user/repo.git it does not go through the defined sock5 tunnel!

环境实在是只有https代理, 可以利用github_token的https协议

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Method 1. git http + proxy http
git config --global http.proxy "http://127.0.0.1:1080"
git config --global https.proxy "http://127.0.0.1:1080"

# Method 2. git http + proxy shocks
git config --global http.proxy "socks5://127.0.0.1:1080"
git config --global https.proxy "socks5://127.0.0.1:1080"

# to unset
git config --global --unset http.proxy
git config --global --unset https.proxy

# Method 3. git ssh + proxy http
vim ~/.ssh/config
Host github.com
HostName github.com
User git
ProxyCommand socat - PROXY:127.0.0.1:%h:%p,proxyport=1087

# Method 4. git ssh + proxy socks
vim ~/.ssh/config
Host github.com
HostName github.com
User git
ProxyCommand nc -v -x 127.0.0.1:1080 %h %p

%h %p 是host和post的意思

或者

After some visiting so many pages, I finally find the solution to my question:

1
2
3
4
5
6
7
8
9
10
11
12
# [step 1] create a ssh-proxy
  ssh -D 9999 -qCN [email protected]

# [step 2] make git connect through the ssh-proxy
  # [current script only]
  export GIT_SSH_COMMAND='ssh -o ProxyCommand="connect -S 127.0.0.1:9999 %h %p"'
  # OR [git global setting] 
  git config --global core.sshCommand 'ssh -o ProxyCommand="connect -S 127.0.0.1:9999 %h %p"'
  # OR [one-time only use]
  git clone -c=core.sshCommand 'ssh -o ProxyCommand="connect -S 127.0.0.1:9999 %h %p"' [email protected]:user/repo.git
  # OR [current repository use only]
  git config core.sshCommand 'ssh -o ProxyCommand="connect -S 127.0.0.1:9999 %h %p"'

To install connect on Ubuntu:

1
sudo apt install connect-proxy

ssh代理

1
2
3
4
5
ssh -vT -o "ProxyCommand connect -S 127.0.0.1:7890 %h %p" [email protected]
ssh -vT -o "ProxyCommand nc -X 5 -x 127.0.0.1:7890 %h %p" [email protected]
# 使用HTTP 代理
ssh -o ProxyCommand='corkscrew proxy.net 8888 %h %p' [email protected]
ssh -o ProxyCommand='proxytunnel -p proxy.net:8888 -P username -d %h:%p' [email protected]

Wireguard 代理

post request forward is an all-in-one solution.

1
2
3
4
5
6
7
8
9
10
11
interface: warp
  public key: fcDZCrGbcpz3sKFqhBw7PtdInygUOtEJfPAs08Wwplc=
  private key: (hidden)
  listening port: 51825

peer: bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo=
  endpoint: [2606:4700:d0::a29f:c001]:1701
  allowed ips: 172.16.0.0/24, 0.0.0.0/0, ::/0
  latest handshake: 89 days, 23 hours, 15 minutes, 28 seconds ago
  transfer: 3.51 GiB received, 1.71 GiB sent
  persistent keepalive: every 25 seconds

latest handshake: 89 days ago demonstrate wg is done for a long time. At the same time mtr github.com shows no output prove the bad situation.

STEP1: first try is to bring the wg-proxy up again

1
2
3
4
python register.py #自动生成warp-op.conf,warp.conf和warp-helper
mv warp-helper /etc/default
vim /etc/config/network #填写warp-op.conf内容，修改只用替换option private_key 和 ipv6 的 list addresses 即可
ifup warp #启动warp, 代替wg-quick up warp.conf

and test brainiac machine is back online

常见情形

大文件提交

Sometimes，it‘s the big log fault.

1
2
3
4
5
6
7
8
9
10
11
# find file
find . -type f -name "zsim.log.0" -size +10M
# find the most repeated lines
head -n 10000 your_file.txt | sort | uniq -c | sort -nr | head
# delete partten line in files
sed -i '/\[S 0\] WARN: \[6\] ContextChange, reason SIGRETURN, inSyscall 1/d' /staff/shaojiemike/github/PIA_huawei/log/zsim/chai-n/hsti/1000/cpu_tlb/zsim.log.0

# conbine two command
find . -type f -name "zsim.log.0" -size +10M -print0 | xargs -0 sed -i '/字符串模式/d'
# or just save the tail (sth wrong needed test)
find . -type f -name "zsim.log.0" -size +1M -exec bash -c 'tail -n 2000 "$1" > "$1"_back ' _ {} \;

请求被拦截

1
2
3
4
t00906153@A2305023964 MINGW64 ~/github
$ git clone https://github.com/jeremy-rifkin/cpptrace.git
Cloning into 'cpptrace'...
fatal: unable to access 'https://github.com/jeremy-rifkin/cpptrace.git/': SSL certificate problem: self-signed certificate in certificate chain

常见问题

error

出现于使用https协议，下载大仓库时，出现该错误。

  1. 首先使用retry脚本，多重试。
  2. 脚本中有http下载的限制修改。
  3. 使用depth=1参数，只下载最新提交。之后下好后能适应unshallow复原
  4. 使用更快的https代理。 e.g., 华为xgate使用HK或者Auto

expecting SSH2_MSG_KEX_ECDH_REPLY

设置mtu解决：

STEP1：

1
2
3
4
5
6
7
8
9
10
eno0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 202.38.73.217  netmask 255.255.255.0  broadcast 202.38.73.255
        inet6 fe80::ae1f:6bff:fe8a:e4ba  prefixlen 64  scopeid 0x20<link>
        inet6 2001:da8:d800:811:ae1f:6bff:fe8a:e4ba  prefixlen 64  scopeid 0x0<global>
        inet6 2001:da8:d800:730:ae1f:6bff:fe8a:e4ba  prefixlen 64  scopeid 0x0<global>
        ether ac:1f:6b:8a:e4:ba  txqueuelen 1000  (以太网)
        RX packets 12345942  bytes 2946978044 (2.9 GB)
        RX errors 0  dropped 1438318  overruns 0  frame 0
        TX packets 4582067  bytes 675384424 (675.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

STEP2：

1
ifconfig eno0 mtu 1200

STEP3：

1
2
3
4
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno0
MTU=1200            #MTU设置

[root@localhost ~]# systemctl restart network

参考文献

Posted Updated network8 minutes read (About 1233 words)

Cloudflare warp proxy

简介

  • Cloudflare 作为全球最大的网络服务提供商，全球大约有30%的网络服务由它提供。
  • WARP是Cloudflare提供的免费的，没有带宽上限或限制的VPN服务，可以让你在任何地方访问互联网，而不会受到地域限制。
    • WARP软件是基于wireguard魔改的。
    • WARP有安卓和Windows的客户端，但是使用人数过多，体验并不好
    • Linux下通过WARP代理能实现20MB/s的下载速度。

WARP on Linux

安装配置

  • 缘由：WARP下的PT做种快得多。(不是，是因为网络硬盘，所以下载多少要占用多少上传)
  • 参考教程。
  • 脚本和所需文件在Z:\shaojiemike\Documents\文献\计算机网络目录下。这里先使用fjw的脚本。
  1. 通过注册脚本register.py，获得私钥和分配的ip
  2. 配置wg。其中Endpoint端口可从官方文档中找到，默认的2408很可能被封。WARP with firewall · Cloudflare Zero Trust docs 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    [Interface]
    PrivateKey = xxxx         #register.py的私钥
    Address = xxx/32,xxx/128  #register.py的ipv4和ipv6
    Table = off              # off禁止wg修改路由表

    [Peer]
    PublicKey = bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo=
    AllowedIPs = 0.0.0.0/0,::/0
    Endpoint = [2606:4700:d0::a29f:c001]:500    #2408, 1701, 500, 4500
    PersistentKeepalive = 25
  3. warp使用的不是标准的wg协议，root下运行，需要通过一个nft脚本main.sh修改包的3个字节。
    1. 安装nft apt-get install nftables
    2. 需要/etc/default/warp-helper文件填写对应的
    3. ROUTING_ID对应register.py的ROUTING_ID。注意三个数之间没空格
    4. UPSTREAM对应wg-conf里Endpoint。比如：
    5.   ROUTING_ID=11,45,14
  UPSTREAM=[2606:4700:d0::a29f:c001]:500
  
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      2. 最后开启路由表,Root权限运行` ip route add default dev warp proto static scope link table default`

      ## WARP on OpenWRT 

      * 目的：为了防止大量流量通过WARP，导致被官方封禁，所以只在OpenWRT上配置WARP分流github的流量。
      * 实现思路：
        * 运行python脚本，通过github的API获得所有的github域名ip，
        * 使用iptables的warp_out表，将目的地址为github域名ip路由到WARP的虚拟网卡上。

      ### WARP Wireguard Establishment

      ```bash
      python register.py #自动生成warp-op.conf,warp.conf和warp-helper
      mv warp-helper /etc/default
      # cat main.sh
      # cat warp-op.conf
      vim /etc/config/network #填写warp-op.conf内容，默认只转发172.16.0.0/24来测试连接
      ifup warp #启动warp, 代替wg-quick up warp.conf
      bash main.sh #启动防火墙实现报文头关键三字节修改
      nft list ruleset #查看防火墙，是否配置成功
      wg #查看warp状态，测试是否连接成果

这时还没创建warp_out路由表，所以还不能通过WARP出数据。

1
2
3
4
5
6
7
8
#/etc/config/network
config interface 'warp'
        option proto 'wireguard'
        option private_key 'wKpvFCOk4sf8d/RD001TF7sNQ39bWnllpqaFf8QnHG4='
        option listen_port '51825'
        list addresses '172.16.0.2/32'
        list addresses '2606:4700:110:8466:d4ea:ffb8:10da:470f/128'
        #option disabled '1'

然后WebUI点击apply 或者命令行运行ifconfig warp down && ifup

Network planning and design

添加了WARP的网络出口后，路由器不在只是通过WAN出数据。防火墙需要更新：

  • 原路返回规则。
    • 针对有公网ip的接口，需要原路返回。
      • 配置来自wan和WARP的信报，使用wan和WARP的路由表，优先级3
      • 来自wan的比如来自外部的ssh，为了防止失联。
      • 来自WARP的比如wget --bind-address=WARP_ip来模拟
    • 内网地址没有必要配置，因为通过内网地址访问host，则dst必然也是内网地址。因此会匹配main中的内网地址规则。
  • wan和WARP的路由表内各自走wan和WARP的网卡
  • 为了使得原本wg正常运行，10: from all lookup main suppress_prefixlength 1
    • 假如warp_out是defualt规则，该项也是为了防止失联。
  • 创建warp_out的空路由表1000: from all lookup warp_out,优先级1000
1
2
3
4
5
6
7
8
root@tsjOp:~/warp# ip rule
0:      from all lookup local
3:      from 114.214.233.141/22 iif eth1 lookup wan
3:      from 172.16.0.2 iif warp lookup warp
10:     from all lookup main suppress_prefixlength 1
1000:   from all lookup warp_out
32766:  from all lookup main
32767:  from all lookup default

填充warp_out路由表

1
2
3
cd ip_route
mv ../github_ipv4.txt .
python fill_ip_table.py --table warp_out --iface warp --p2p -f github_ipv4.txt

对所有github域名的ip执行类似ip ro add 192.30.252.0/22 dev warp proto static table warp_out操作。

测试

1
2
3
mtr www.github.com
ssh -vT [email protected]
git clone https://github.com/llvm/llvm-project

添加到启动项

修改/etc/rc.local

1
2
3
4
5
6
7
8
9
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.


sleep 30 && cd /root/warp/ip_route && python fill_ip_table.py --table warp_out --iface warp --p2p -f github_ipv4.txt

/root/warp/main.sh #重新添加防火墙

exit 0

WARP on Windows

基于1.1.1.1 的安装windows版本直接白嫖

需要进一步的研究学习

暂无

遇到的问题

暂无

开题缘由、总结、反思、吐槽~~

参考文献

https://gist.github.com/iBug/3107fd4d5af6a4ea7bcea4a8090dcc7e

glados

Follow

Categories

Subscribe for updates

follow.it

Links

Recents

Archives

Tags

5G1
64bits vs 32bits1
AI3
AMAT1
AMD3
ASPLOS1
ATI1
AVX2
Algorithm1
Alpha1
Analysis1
Apt1
Assembly4
BFS1
BHive1
BT3
BTL1
Baka Mitai1
Bash1
Big-Endian1
×