最近大半年一直在参与公司针对OpenStack的一些定制开发,主要是与Neutron组件相关。大大小小的各个项目中,与华为、中兴、EasyStack等都有过合作,每个项目基于的OpenStack版本都不一样,所以经常来回的安装部署OpenStack环境,比较繁琐,工作之余一直在寻求一种部署环境的便捷方式。目前比较主流的部署方式是devstack,也非常方便,但观察OpenStack的发展趋势,已经有越来越多的公司在做OpenStack的容器化部署,OpenStack社区也有对应的项目:kolla,虽然现在用的不是特别多,但利用容器产生的一些优势,相信在将来会有更多的公司采用容器化部署,本人在空闲时间尝试用kolla部署了一套mitaka版本的OpenStack环境(ALL IN ONE),以下是详细步骤,如果有任何问题也欢迎底下留言。

1. 环境信息

操作系统: Ubuntu 14.04
Docker: Docker version 1.12.5, build 7392c3b
注意: Docker版本不能使用最新版本,推荐1.12.x。使用最新版本部署的时候会有问题,主要是python的docker api库版本不匹配,网络上其他教程基本都有这个问题。

2. 环境准备

2.1 升级内核

apt update
apt upgrade
apt-get install linux-image-generic-lts-wily
reboot

2.2 升级python版本

sudo add-apt-repository ppa:fkrull/deadsnakes-python2.7
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5BB92C09DB82666C
apt update
apt upgrade
apt-get install python-dev libffi-dev libssl-dev gcc

2.3 安装pip

这里推荐手动安装pip工具,而不要使用apt来安装,原因是用apt安装会同时安装几个python库,这几个python库是无法被pip管理的,后面安装过程中可能会有冲突。

wget https://files.pythonhosted.org/packages/c8/89/ad7f27938e59db1f0f55ce214087460f65048626e2226531ba6cb6da15f0/pip-19.0.1.tar.gz
tar xvf pip-19.0.1.tar.gz
cd pip-19.0.1/
python setup.py install

pip install -U pip

安装完了pip以后,使用dpgk命令查看下系统当前安装了哪些python库,用apt remove将它们都删除,用pip再安装一遍。

dpkg -l | grep python-

2.4 安装Docker

使用最新版本的docker部署mitaka会报错,本文使用1.12.5版本的docker。

wget https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.12.5-0~ubuntu-trusty_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd-journal0_204-5ubuntu20_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/main/libt/libtool/libltdl7_2.4.2-1.7ubuntu1_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/libg/libgcrypt20/libgcrypt20_1.6.1-2ubuntu1.14.04.1_amd64.deb
 
dpkg -i libltdl7_2.4.2-1.7ubuntu1_amd64.deb
dpkg -i libgcrypt20_1.6.1-2ubuntu1.14.04.1_amd64.deb
dpkg -i libsystemd-journal0_204-5ubuntu20_amd64.deb
dpkg -i docker-engine_1.12.5-0~ubuntu-trusty_amd64.deb

reboot

安装完了以后,需要对docker配置shared mount flag,否则部分容器部署会失败。

mount --make-shared /run
service docker restart

推荐将上述两条命令加入到rc.local脚本中,否则每次重启都需要执行。

2.4 安装Python lib库

2.4.1 安装docker-py

网上也有教程说安装docker库,本人没有试验过。

pip install -U docker-py

2.4.2 安装ansible

mitaka版本需要使用1.9.4,不能使用最新版本。

pip install -U ansible==1.9.4

2.4.3 安装kolla

OpenStack官方的仓库中已经废弃了mitaka版本的分支,所以现在无法从官方下载,我从github中查找到了一个包含mitaka版本的仓库,但是有些小问题,复制到自己的github仓库:https://github.com/zpzhoudev/kolla-deprecate.git 进行了修复。

apt install git
git clone -b stable/mitaka https://github.com/zpzhoudev/kolla-deprecate.git
mv kolla-deprecate kolla
pip install -r kolla/requirements.txt -r kolla/test-requirements.txt
pip install kolla/

利用tox生成配置文件

pip install -U tox
cd kolla/
tox -e genconfig
cp -rv etc/kolla /etc/

2.4.4 安装OpenStack Client

pip install -U python-openstackclient python-neutronclient

2.5 配置本地docker仓库

因为本次部署的all in one,所以这步跳过了,多节点的话是必须的。

2.5.1创建registry容器

docker pull registry:latest
docker run -d -p 4000:5000 -e REGISTRY_STORAGE_DELETE_ENABLED="true" --restart=always --name registry registry:latest

2.5.2 修改docker启动参数

vi /etc/default/docker

DOCKER_OPTS="--insecure-registry 10.180.52.80:4000"
service docker restart

3. 编译部署

3.1 编译镜像

本次只搭建基础环境,所以需要的镜像也不多,没必要全部编译。

 kolla-build -b ubuntu -t binary horizon cinder heat nova neutron glance keystone rabbitmq keepalived haproxy heka kolla-toolbox mariadb memcached cron openvswitch
 
 # 如果使用docker本地仓库,需要添加参数: --registry {registry-ip}:4000 --push

以上命令是基于ubuntu系统来编译,现在貌似只有ubuntu才有mitaka版本的软件包源。
编译完成后镜像如下所示:

root@ubuntu:~# docker images
REPOSITORY                                          TAG                 IMAGE ID            CREATED             SIZE
kollaglue/ubuntu-binary-kolla-toolbox               2.0.1               04ad51ce7150        26 hours ago        824.1 MB
kollaglue/ubuntu-binary-nova-compute                2.0.1               260bd29ab0ac        2 days ago          800.8 MB
kollaglue/ubuntu-binary-nova-libvirt                2.0.1               9cb64d949a52        2 days ago          822.9 MB
kollaglue/ubuntu-binary-nova-compute-ironic         2.0.1               47a68f4d4c88        2 days ago          724.2 MB
kollaglue/ubuntu-binary-nova-novncproxy             2.0.1               38f6cd8037b1        2 days ago          471.1 MB
kollaglue/ubuntu-binary-nova-spicehtml5proxy        2.0.1               c77f67bcae77        2 days ago          448.9 MB
kollaglue/ubuntu-binary-nova-api                    2.0.1               645b11126977        2 days ago          450.3 MB
kollaglue/ubuntu-binary-nova-conductor              2.0.1               f005defaba9a        2 days ago          448.6 MB
kollaglue/ubuntu-binary-nova-consoleauth            2.0.1               5ca76067f86a        2 days ago          448.6 MB
kollaglue/ubuntu-binary-nova-scheduler              2.0.1               49577fddeac3        2 days ago          448.6 MB
kollaglue/ubuntu-binary-horizon                     2.0.1               638e97eb8575        2 days ago          490.3 MB
kollaglue/ubuntu-binary-nova-ssh                    2.0.1               4f89ffb93a8b        2 days ago          452.5 MB
kollaglue/ubuntu-binary-nova-network                2.0.1               8cf2779ac36c        2 days ago          451.4 MB
kollaglue/ubuntu-binary-cinder-volume               2.0.1               4a3486e3e3b0        2 days ago          513.7 MB
kollaglue/ubuntu-binary-cinder-rpcbind              2.0.1               ec6088366536        2 days ago          506.8 MB
kollaglue/ubuntu-binary-cinder-backup               2.0.1               392b20fbe1e4        2 days ago          506.3 MB
kollaglue/ubuntu-binary-cinder-scheduler            2.0.1               d9faab863d82        2 days ago          506.3 MB
kollaglue/ubuntu-binary-cinder-api                  2.0.1               6c50baa045b6        2 days ago          506.3 MB
kollaglue/ubuntu-binary-nova-base                   2.0.1               fda510c8876b        2 days ago          446.6 MB
kollaglue/ubuntu-binary-cinder-base                 2.0.1               dce5291bdbca        2 days ago          504.2 MB
kollaglue/ubuntu-binary-neutron-metadata-agent      2.0.1               1988f1ca228e        2 days ago          420.3 MB
kollaglue/ubuntu-binary-neutron-l3-agent            2.0.1               d5cd6f437820        2 days ago          425.9 MB
kollaglue/ubuntu-binary-neutron-openvswitch-agent   2.0.1               2babf6b4b944        2 days ago          420.5 MB
kollaglue/ubuntu-binary-neutron-linuxbridge-agent   2.0.1               5043fcb29f0b        2 days ago          420.9 MB
kollaglue/ubuntu-binary-neutron-dhcp-agent          2.0.1               dea8dee1cf79        2 days ago          421 MB
kollaglue/ubuntu-binary-keystone                    2.0.1               2672e9a5612d        2 days ago          412.4 MB
kollaglue/ubuntu-binary-heat-api-cfn                2.0.1               ce00524c2649        2 days ago          403.6 MB
kollaglue/ubuntu-binary-neutron-server              2.0.1               aed7e7adc997        2 days ago          418.2 MB
kollaglue/ubuntu-binary-neutron-base                2.0.1               4d0957a67168        2 days ago          418.2 MB
kollaglue/ubuntu-binary-glance-api                  2.0.1               9014d2e097fa        2 days ago          428.7 MB
kollaglue/ubuntu-binary-glance-registry             2.0.1               5a0fdf9afefb        2 days ago          428.7 MB
kollaglue/ubuntu-binary-heat-engine                 2.0.1               0ecbcfa285fb        2 days ago          403.6 MB
kollaglue/ubuntu-binary-heat-api                    2.0.1               ff613b152586        2 days ago          403.6 MB
kollaglue/ubuntu-binary-glance-base                 2.0.1               d605ff7f7a54        2 days ago          428.7 MB
kollaglue/ubuntu-binary-heat-base                   2.0.1               7a404658b38f        2 days ago          401.7 MB
kollaglue/ubuntu-binary-openstack-base              2.0.1               e73bbe926a69        2 days ago          348.7 MB
kollaglue/ubuntu-binary-mariadb                     2.0.1               bfc07c71e3a8        2 days ago          445.6 MB
kollaglue/ubuntu-binary-rabbitmq                    2.0.1               be7794769c8f        2 days ago          276.7 MB
kollaglue/ubuntu-binary-heka                        2.0.1               91757ed86dbc        2 days ago          283.1 MB
kollaglue/ubuntu-binary-openvswitch-vswitchd        2.0.1               99ccfb2278a8        2 days ago          246.6 MB
kollaglue/ubuntu-binary-openvswitch-db-server       2.0.1               0b50b1151baf        2 days ago          246.6 MB
kollaglue/ubuntu-binary-openvswitch-base            2.0.1               8e05dc556547        2 days ago          246.6 MB
kollaglue/ubuntu-binary-haproxy                     2.0.1               a2ab0c4c6e72        2 days ago          240.6 MB
kollaglue/ubuntu-binary-keepalived                  2.0.1               1c9c4cc82e86        2 days ago          246.3 MB
kollaglue/ubuntu-binary-memcached                   2.0.1               a4f0cb710477        2 days ago          239.2 MB
kollaglue/ubuntu-binary-cron                        2.0.1               530021fed789        2 days ago          237.6 MB
kollaglue/ubuntu-binary-base                        2.0.1               23a9cb3b1b0e        2 days ago          237.6 MB
ubuntu                                              14.04               5dbc3f318ea5        12 days ago         188.1 MB

3.2 编写配置文件

3.2.1 密码相关

使用kolla-genpwd命令生成密码文件。

kolla-genpwd

生成的文件保存在/etc/kolla/passwords.yml,为方便后面的使用,可以将password结尾的配置项的值都改为固定的易识别的字符串。

3.2.2 全局配置

编辑文件/etc/kolla/globals.yml,根据实际情况修改,以下是我的配置:

kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "2.0.1"

# 当不是高可用的情况下,使用network_interface接口上的ip,prechecks可能不通过,可忽略
kolla_internal_address: "10.180.52.80"
kolla_internal_fqdn: "controller"
kolla_external_fqdn: "controller"
network_interface: "eth0"
neutron_external_interface: "eth1"
neutron_plugin_agent: "openvswitch"
enable_ceilometer: "no"
enable_haproxy: "no"
enable_cinder: "yes"
enable_horizon: "yes"

3.3 修改host名和hosts文件

hostnamectl set-hostname ubuntu

将本机的host名对应的ip修改成外部可访问的ip,不能用127.0.0.1,再添加kolla_external_fqdnkolla_internal_fqdn对应的hosts记录

10.180.52.80  ubuntu
10.180.52.80  controller

3.4 预检查

kolla-ansible prechecks

如果检查没有问题,就可以开始部署了。

3.5 部署容器

kolla-ansible deploy

# 部署指定容器
kolla-ansible deploy -t mariadb

如果部署失败,需要重新部署,kolla提供了几个脚本方便我们清理环境:

#清理容器
tools/cleanup-containers 

 #清理配置
tools/cleanup-host

#清理docker镜像
tools/cleanup-images

3.4 收尾工作

1.使用kolla自动生成环境变量文件,保存在/etc/kolla/目录下

kolla-ansible post-deploy

2.修改horizon的监听host地址:
修改/etc/kolla/horizon/horizon.conf文件中的Listen配置项。

4. 遇到的问题一览

4.1 kolla build错误

build marathon error: KeyError: 'logs'

INFO:kolla.cmd.build:marathon:Removing intermediate container a0952c7ad4f7
ERROR:kolla.cmd.build:marathon:Error'd with the following message
ERROR:kolla.cmd.build:marathon:The command '/bin/sh -c useradd --user-group marathon     && chmod 755 /usr/bin/marathon' returned a non-zero code: 9
Traceback (most recent call last):
  File "/usr/local/bin/kolla-build", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 739, in main
    kolla.summary()
  File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 559, in summary
    for line in image['logs'].split('\n'):
KeyError: 'logs'

找到对应代码,将log输出注释即可。

4.2 deploy错误

TASK: [ceph | Fetching Ceph keyrings] ***************************************** 
fatal: [localhost -> ubuntu] => SSH Error: ssh: Could not resolve hostname ubuntu: Name or service not known
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

FATAL: all hosts have already failed -- aborting

增加host名到/etc/hosts


failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.8.132.72' login_port='3306' login_user='root' login_password='hillstone' name='haproxy' password='' host='%' priv=*.*:USAGE"], "delta": "0:00:00.622472", "end": "2019-02-01 15:18:25.299043", "failed": true, "rc": 0, "start": "2019-02-01 15:18:24.676571", "stdout_lines": ["/usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.", "  utils.DeprecatedIn23,", "localhost | SUCCESS => {", "    \"changed\": false, ", "    \"user\": \"haproxy\"", "}"], "warnings": []}
stdout: /usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
  utils.DeprecatedIn23,
localhost | SUCCESS => {
    "changed": false, 
    "user": "haproxy"
}
msg: Task failed as maximum retries was encountered

镜像中的python版本太低,需要修改base的dockerfile,升级python,在我的github仓库中已经修改。


TASK: [rabbitmq | fail msg="Hostname has to resolve to IP address of api_interface"] *** 
failed: [localhost] => (item={'cmd': ['getent', 'ahostsv4', 'ubuntu'], 'end': '2019-02-03 13:23:55.317541', 'stderr': '', 'stdout': '10.180.169.193  STREAM ubuntu\n10.180.169.193  DGRAM  \n10.180.169.193  RAW    ', 'changed': False, 'rc': 0, 'item': 'localhost', 'warnings': [], 'delta': '0:00:00.004081', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'getent ahostsv4 ubuntu'}, 'stdout_lines': ['10.180.169.193  STREAM ubuntu', '10.180.169.193  DGRAM  ', '10.180.169.193  RAW    '], 'start': '2019-02-03 13:23:55.313460'}) => {"failed": true, "item": {"changed": false, "cmd": ["getent", "ahostsv4", "ubuntu"], "delta": "0:00:00.004081", "end": "2019-02-03 13:23:55.317541", "invocation": {"module_args": "getent ahostsv4 ubuntu", "module_complex_args": {}, "module_name": "command"}, "item": "localhost", "rc": 0, "start": "2019-02-03 13:23:55.313460", "stderr": "", "stdout": "10.180.169.193  STREAM ubuntu\n10.180.169.193  DGRAM  \n10.180.169.193  RAW    ", "stdout_lines": ["10.180.169.193  STREAM ubuntu", "10.180.169.193  DGRAM  ", "10.180.169.193  RAW    "], "warnings": []}}
msg: Hostname has to resolve to IP address of api_interface

FATAL: all hosts have already failed -- aborting

host名必须解析到apt_interface所配置的接口的IP

4.3 运行错误

4.3.1 没有_member_角色

提示没有_member_角色
执行命令手动创建:openstack role create _member_

4.3.2 无法访问admin的systemInfo

修改horizon容器中的cinderclient:/usr/lib/python2.7/site-packages/cinderclient/v2/services.py

class Service(base.Resource):

    def __repr__(self):
        # return "<Service: %s>" % self.service
        return "<Service: %s>" % self._info

4.3.3 neutron openvswitch agent不在agent list中

原因是neutron openvswitch agent启动卡住了,neutron执行sudo命令被要求输入密码,是rootwrap配置的问题,还未发现原因,比较粗暴的解决办法是:
进入容器修改sudo配置文件,让neutorn可以免密码执行任意命令

visudo

# 添加以下配置
neutron ALL=(ALL) NOPASSWD: ALL

所有的neutron相关的容器都有这个问题,都需要修改。

4.3.4 cinder-volume服务启动失败

原因有两个:
1、cinder用户没办法免密码执行相关命令,进入容器修改sudo配置文件

visudo

# 添加以下配置
cinder ALL=(ALL) NOPASSWD: ALL

2、host上没有cinder-volume的volume group

dd if=/dev/zero of=./cinder-volumes bs=1 count=0 seek=200G
losetup /dev/loop0 cinder-volumes
pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
vgdisplay

4.3.5 创建volume失败

这个问题非常难搞,耗费了整整大半天才找到原因,主要是因为udev的同步问题导致容器中执行lvcreate或lvdelete命令时,一直卡在那里了,只要关掉lvm的udev同步即可。

 vi /etc/lvm/lvm.conf
 
 udev_sync = 0
 udev_rules = 0