[RLHF] OpenRLHF食用指南

使用OpenRLHF
https://github.com/OpenRLHF/OpenRLHF?tab=readme-ov-file
的过程.

---

基本上是按照README的知道一步步来.
https://github.com/OpenRLHF/OpenRLHF?tab=readme-ov-file

零. 大战 nvidia-docker

To use OpenRLHF, first launch the docker container (Recommended) and pip install openrlhf inside the docker container:

由于之前已经安装好了docker, 所以主要是在搞nvidia-docker的东西.

过程不算轻松. 具体可见另一篇blog.

一. 大战 pip install openrlhf

根据README的指导:

# Launch the docker container
docker run --runtime=nvidia -it --shm-size="8g" --name docker-OpenRLHF --cap-add=SYS_ADMIN -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:24.07-py3 bash
pip uninstall xgboost transformer_engine flash_attn -y

# pip install
pip install openrlhf

# If you want to use vLLM acceleration (Install vLLM 0.6.4.post1)
pip install openrlhf[vllm]
# latest vLLM is also supported
pip install openrlhf[vllm_latest]

# pip install the latest version
pip install git+https://github.com/OpenRLHF/OpenRLHF.git

# Or git clone
git clone https://github.com/OpenRLHF/OpenRLHF.git
cd OpenRLHF
pip install -e .

安装镜像源, 运行容器, pip install openrlhf 即可. 在搞定nvidia-docker的前提下不会遇到什么问题.

和README里的命令有一些不同. 1. 添加了--name的选项, 给容器命名. 2. 去除了-rm选项, 对容器进行复用 3. docker 内部的命令不需要 sudo

---

发现始终无法完成 命令, 一直卡在 Building wheel for flash-attn (setup.py) ... -这一步.

尝试按照README上的指引重新安装了nvidia-docker (需要把之前的docker给删掉, 算是白折腾了). 这一次倒是没有遇到网络问题, 一路把镜像源下载了下来. 但是进去之后还是卡在了 flash-attn这一步.

中间还遇到了

sudo apt-get update 

N: Ignoring file 'skypeforlinux.lis' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension
E: Conflicting values set for option Signed-By regarding source https://download.docker.com/linux/ubuntu/ focal: /usr/share/keyrings/docker-ce-archive-keyring.gpg != /usr/share/keyrings/docker-archive-keyring.gpg
E: Conflicting values set for option Signed-By regarding source https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64/ /: /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg != 
E: The list of sources could not be read.

之类的问题. 找了找发现是两次安装导致密钥重复了. 这和gpt的解决方案是一样的. 只需要进入
/etc/apt/sources.list.d/, 然后把 nvidia-container-toolkit.list中的重复内容剔除, 只剩下一个就行.
https://github.com/NVIDIA/nvidia-docker/issues/1626

后面试着解决进度条不动的问题, 找了两个方法都不奏效. 试着单独安装 pip install flash-attn 也是同样的卡住了.
https://github.com/Dao-AILab/flash-attention/issues/224
https://github.com/adithya-s-k/omniparse/issues/36

后面找到了一个网页 https://til.simonwillison.net/python/installing-flash-attention, 上面貌似表示安装这个东西卡顿是个正常现象. (For some reason attempting to install this runs a compilation process which can take multiple hours.). 所以我打算给它一些时间. 只要它没有报错就让它在那跑吧.

二. Prepare Datasets