在ubuntu上安裝openstack的swift組件-installing openstack object storage


這幾天正在自己搭建一個swift的環境。看了一些資料,結合自己的實踐,寫下了它:

1.物理環境說明:
[html] view plain copy
Linux系統版本:Ubuntu Server 12.04  
Proxy Server IP: 10.214.0.181  
Storage Server One: 10.214.0.179  
Storage Server Two: 10.214.0.180  
Storage Server three: 10.214.0.182  

2.重要組件說明:
node:運行一個或多個object storage service
proxy node:運行proxy services
auth node:運行Auth service
storage:運行account,container,and object services
Ring:是Openstack object storage 到物理設備的映射集合

在所有節點上的準備工作: 安裝服務 [html] view plain copy apt-get install swift openssh-server rsync memcached python-netifaces python-xattr python-memcache 配置路徑 [html] view plain copy mkdir -p /etc/swift chown -R swift:swift /etc/swift/ 新建文件vim /etc/swift/swift.conf [html] view plain copy [swift-hash] # random unique string that can never change (DO NOT LOSE) swift_hash_path_suffix = fLIbertYgibbitZ 3.安裝和配置storage nodes 安裝:Storage node packages,所有storage都要進行操作。 [html] view plain copy aptitude install swift-account swift-container swift-object xfsprogs 新建xfs系統分區,當然,(這需要一個空分區),如果硬盤的分區已經被使用而數據你又不想要了,那就刪除分區後進行下列操作。否則添加一塊新硬盤是最方便的選擇。所有storage都要進行操作。 [html] view plain copy fdisk /dev/sdb Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-488281249, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-488281249, default 488281249): Using default value 488281249 Command (m for help): p Disk /dev/sdb: 250.0 GB, 250000000000 bytes 255 heads, 63 sectors/track, 30394 cylinders, total 488281250 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000081 Device Boot Start End Blocks Id System /dev/sdb1 2048 488281249 244139601 83 Linux Command (m for help): w The partition table has been altered! 識別並掛載新的分區: [html] view plain copy partprobe#如果不行,就重啟一下吧。 [html] view plain copy mkdir -p /srv/node/sdb1 [html] view plain copy mkfs.xfs -i size = 1024 /dev/sdb1 -f echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier, logbufs = 8 0 0" > > /etc/fstab mount /srv/node/sdb1 [html] view plain copy chown -R swift:swift /srv/node 創建並配置vim /etc/rsyncd.conf [python] view plain copy uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 10.214 . 0.179 [account] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/account.lock [container] max connections = 2 path =/srv/node/ read only = false lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/object.lock 注意:path 寫你實際的(剛才新建的)分區的位置。address修改為當前storage的地址。 編輯vim /etc/default/rsync(即設置為了開機啟動) [python] view plain copy RSYNC_ENABLE = true 重新/啟動服務 [html] view plain copy service rsync restart 創建或檢查vim /etc/swift/account-server.conf [html] view plain copy [DEFAULT] devices = /sdb1 mount_check = false bind_port = 6002 user = swift bind_ip = 0 .0.0.0 workers = 2 [pipeline:main] pipeline = account -server [app:account-server] use = egg :swift#account [account-replicator] [account-auditor] [account-reaper] 創建或檢查vim /etc/swift/container-server.conf [html] view plain copy [DEFAULT] devices = /sdb1 mount_check = false bind_ip = 0 .0.0.0 bind_port = 6001 workers = 2 [pipeline:main] pipeline = container -server [app:container-server] use = egg :swift#container [container-replicator] [container-updater] [container-auditor] [html] view plain copy [container-sync] [html] view plain copy [html] view plain copy 注意:[container-sync]這個是添加上去的,不要忘記了 創建或檢查vim /etc/swift/object-server.conf [html] view plain copy [DEFAULT] devices = /sdb1 mount_check = false bind_ip = 0 .0.0.0 bind_port = 6000 workers = 2 [pipeline:main] pipeline = object -server [app:object-server] use = egg :swift#object [object-replicator] [object-updater] [object-auditor] 注意去掉:object-expirer 啟動storage service: [html] view plain copy swift-init object-server restart swift-init object-replicator restart swift-init object-updater restart swift-init object-auditor restart swift-init container-server restart swift-init container-replicator restart swift-init container-updater restart swift-init container-auditor restart swift-init account-server restart swift-init account-replicator restart swift-init account-auditor restart [html] view plain copy 也可以用命令 [html] view plain copy swift-init all restart 4.安裝和配置Proxy Node [html] view plain copy apt-get install swift-proxy swift-doc memcached Memcached 是一個高性能的分佈式內存對象緩存系統,用於動態Web應用以減輕數據庫負載。它通過在內存中緩存數據和對象來減少讀取數據庫的次數,從而提供動態、數據庫驅動網站的速度。Memcached基於一個存儲鍵/值對的hashmap。其守護進程(daemon )是用C寫的,但是客戶端可以用任何語言來編寫,並通過memcached協議與守護進程通信。 為ssl建立簽名證書 [html] view plain copy cd /etc/swift openssl req -new -x509 -nodes -out cert.crt -keyout cert.key 修改/etc/memcched.conf [html] view plain copy -l 127.0.0.1 #改為proxy節點的ip -l 10.214.0.181 重新啟動memcached服務 [html] view plain copy service memcached restart 新建vim /etc/swift/proxy-server.conf [html] view plain copy [DEFAULT] bind_port = 8080 [html] view plain copy bind_ip = 10 .214.0.181 user = swift [html] view plain copy [pipeline:main] pipeline = catch_errors healthcheck cache authtoken keystone proxy-server [app:proxy-server] use = egg :swift#proxy account_autocreate = true [filter:keystone] paste.filter_factory = keystone .middleware.swift_auth:filter_factory operator_roles = admin , swiftoperator [filter:authtoken] paste.filter_factory = keystone .middleware.auth_token:filter_factory # Delaying the auth decision is required to support token-less # usage for anonymous referrers ('.r:*'). delay_auth_decision = true service_port = 5000 service_host = 10 .214.0.179 auth_port = 35357 auth_host = 10 .214.0.179 auth_token = 123456 admin_token = 123456 auth_protocol = http auth_uri = http ://10.214.0.179:5000/ admin_tenant_name = admin admin_user = admin admin_password = 123456 [filter:cache] use = egg :swift#memcache [html] view plain copy memcache_servers = 10 .214.0.181:11211 set log_name = cache [filter:catch_errors] use = egg :swift#catch_errors [filter:healthcheck] use = egg :swift#healthcheck 注意:如果你運行多個memcache ,把多個ip:端口添加到在[filter:cache]中,例如: [html] view plain copy memcache_servers = 10 .1.2.3:11211,10.1.2.4:11211 如果不想用keystone可以用下面的配置內容代替: [html] view plain copy [DEFAULT] bind_port = 8080 bind_ip = 10 .214.0.181 user = swift [pipeline:main] pipeline = healthcheck cache tempauth proxy-server [app:proxy-server] use = egg :swift#proxy allow_account_management = true account_autocreate = true [filter:tempauth] use = egg :swift#tempauth user_admin_admin = admin .admin .reseller_admin user_test_tester = testing .admin user_test2_tester2 = testing2 .admin user_test_tester3 = testing3 [filter:healthcheck] use = egg :swift#healthcheck [filter:cache] use = egg :swift#memcache memcache_servers = 10 .214.0.181:11211 創建環,建立account,container,object的ring,參數18代表“環”是2^18(http://blog.csdn.net/zoushidexing/article/details/7852014),參數3 代表每一個存儲對像有3份拷貝,當然這是有至少3個storage的情況下,所有的所有服務都在一個節點上,那就寫1吧。參數1代表1小時可以移動分區一次(即延遲時間)。 [html] view plain copy cd /etc/swift swift-ring-builder account.builder create 18 3 1 swift-ring-builder container.builder create 18 3 1 swift-ring-builder object.builder create 18 3 1 添加ring節點,例如,我們在Zone 1中建立的一個存儲節點,ip地址為10.214.0.182,分區為/sdb1, 路徑在rsyncd.conf(存儲節點)文件path中。100 代表設備的權重。可以根據磁盤的容量設定對應值,比如2T的100,則1T的可以是50。端口要與配置文件中的對應。 [html] view plain copy swift-ring-builder account.builder add z1-10.214.0.179:6002/sdb1 93 swift-ring-builder container.builder add z1-10.214.0.179:6001/sdb1 93 swift-ring-builder object.builder add z1-10.214.0.179:6000/sdb1 93 注意:假定有若干個zones,每一個zones有一個node。zone應該起始於1,以1遞增。例如:swift-ring-builder account.builder add z2-10.214.0.180:6002/sdb1 28 (為了以後重新建立環境方便,可以把它們寫在vim set_devices.sh中: (一條條複製也挺煩的。) [html] view plain copy swift-ring-builder account.builder add z1-10.214.0.179:6002/sdb1 93 swift-ring-builder container.builder add z1-10.214.0.179:6001/sdb1 93 swift-ring-builder object.builder add z1-10.214.0.179:6010/sdb1 93 swift-ring-builder account.builder add z2-10.214.0.180:6002/sdb1 28 swift-ring-builder container.builder add z2-10.214.0.180:6001/sdb1 28 swift-ring-builder object.builder add z2-10.214.0.180:6010/sdb1 28 swift-ring-builder account.builder add z3-10.214.0.182:6002/sdb1 23 swift-ring-builder container.builder add z3-10.214.0.182:6001/sdb1 23 swift-ring-builder object.builder add z3-10.214.0.182:6000/sdb1 23 swift-ring-builder account.builder swift-ring-builder container.builder swift-ring-builder object.builder [html] view plain copy 執行腳本。 sh set_devices.sh) 如果添加錯了而你有不止到怎麼修改,那可以刪除/etc/swift下的account.builder、container.builder、 object.builder後重新創建環即重新執行swift-ring-builder account.builder create 18 3 1等。 核實ring的內容 [html] view plain copy swift-ring-builder account.builder [html] view plain copy swift-ring-builder container.builder swift-ring-builder object.builder 平衡rings [html] view plain copy swift-ring-builder account.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance 成功之後會在當前目錄生成account.ring.gz 文件,把/etc/swift/下的account.ring.gz, container.ring.gz, and object.ring.gz拷貝到每一個proxy節點和storage節點。 確定所有的配置文件的權限: [html] view plain copy chown -R swift:swift /etc/swift 啟動proxy服務 [html] view plain copy swift-init proxy restart 5.重啟服務 [html] view plain copy swift-init main restart swift-init rest restart [html] view plain copy swift-init all restart [html] view plain copy [html] view plain copy Proxy節點:swift-init proxy start 各個Storage節點:swift-init all start 6.(可選)添加一個額外的proxy server 為了提高可靠性,可以添加額外的proxy server。安裝過程和上述安裝proxy node 的過程類似。但是要進行一些配置。 一旦擁有了多個proxy,就可能需要用到負載均衡。有多種負載均衡的方法可以選擇,比如:輪詢、在proxy前加一個負載均衡器,指定特定的storage。 添加proxy需要進行一些配置,當然這些配置需要需要在其proxy同時進行。 更新/etc/swift/proxy-server.conf文件,如果使用的多個memcache servers,那麼,要如下添加IP. [html] view plain copy [filter:cache] use = egg :swift#memcache memcache_servers = 10 .214.0.179:11211,10.214.0.182:11211 更改默認的default_cluster_url 指向負載均衡的url以代替第一次創建在/etc/swift/proxy-server.conf配置的。 [html] view plain copy [app:auth-server] use = egg :swift#auth default_cluster_url = https ://10.214.0.180/v1 # Highly recommended to change this key to something else! super_admin_key = devauth 當你改變了default_cluster_url設置,就需要刪除auth database 並且重新創建openstack object storage的users,或者也可以在auth database中手動為每一個account更新正確的url。 下一步,需要拷貝ring的信息到所有幾點,包括你新建的proxy節點。並且確定ring可以到達所有的存儲節點。 當同步了所有的節點之後,確保admin在/etc/swift有key,並且對ring文件有充分的權限。 7.驗證設置。 使用正確的服務Identity service URL,通過export ADMINPASS=secretword導入對ADMINPASS設置。(可以通過proxy-server.conf查看用戶和密碼) [html] view plain copy swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 stat 得到an X-Storage-Url 和X-Auth-Token [html] view plain copy curl -k -v -H 'X-Storage-User: adminUser:admin' -H 'X-Storage-Pass: $ADMINPASS' http:// < AUTH_HOSTNAME > :5000/auth/v1.0 例如: [html] view plain copy curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://10.214.0.181:8080/auth/v1.0 檢查你de賬號 [html] view plain copy curl -k -v -H 'X-Auth-Token: < token-from-x-auth-token-above > ' < url-from-xstorage-url-above > 例如: 使用swift刪除一些小文件,命名為'bigfile[1-2].tgz' 給一個container 名字叫'myfiles': [html] view plain copy $ swift -A http:// < AUTH_HOSTNAME > :5000/v2.0 -U adminUser:admin -K $ADMINPASS upload myfiles bigfile1.tgz $ swift -A http:// < AUTH_HOSTNAME > :5000/v2.0 -U adminUser:admin -K $ADMINPASS upload myfiles bigfile2.tgz 使用swift從'myfiles' container下載所有文件 [html] view plain copy $ swift -A http:// < AUTH_HOSTNAME > :5000/v2.0 -U adminUser:admin -K $ADMINPASS download myfiles 8.錯誤解決提示 如果遇到問題,可以查看日誌文件var/log/syslog 同時,在/var/log/kern.log中也會有錯誤日誌。 注意端口的佔用情況,我在安裝的時候由於8080端口被佔用,出了好多莫名奇妙的情況。 查看端口的命令為: [html] view plain copy netstat -anp | grep 8080 如果你的服務不能夠啟動,而這個端口還被佔用這,那請kill 到你用查看端口命令看到的進程的ID號。 關聯關係: keystone user-role-add --user 83cd1180b3794811848265cd012dffb7 --tenant_id 3804546967574be38aaa08faca23c32d --role d8c278323c9e4663bbf4ff48ebcb24d6 endpoint 配置再次提醒, 上面的service id需要換成你自己創建的service的id, 在publicurl和internalurl裡出現的AUTHxxxx,這裡的xxxx需要換成你的adminTenant的id 這裡的格式需要參考你的swift\_auth裡定義的格式, 有個reseller_prefix的設置, 這裡我們設置成AUTH。 keystone endpoint-create --region RegionOne \ --service_id c7a8827453ef4364af3b6809a174c6bb \ --publicurl http://10.214.0.181:8080/v1/AUTH_3804546967574be38aaa08faca23c32d \ --adminurl http://10.214.0.181:8080 \ --internalurl http:/ /10.214.0.181:8080/v1/AUTH_3804546967574be38aaa08faca23c32d 9.常用命令 查看文件統計結果 創建一個文件夾 [html] view plain copy swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 post myfiles myfiles 是文件夾的名字 上傳文件 [html] view plain copy swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 upload myfiles set_devices.sh set_devices.sh 是文件的名稱 下載文件 [html] view plain copy swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 download myfiles set_devices.sh 參考文獻: 官方文檔:http://docs.openstack.org/essex/openstack-compute/install/apt/openstack-install-guide-essex.pdf atkisc的[Openstack] swift 安裝配置過程(all in one) :http://bbs.linuxtone.org/thread-16276-1-1.html free_coder的Ubuntu 12.04 LTS 上安裝swift1.4.8:http://www.cnblogs.com/free--coder/archive/2012/05/28/2521135.html cywosp的使用Swauth認證多節點安裝Swift:http://blog.csdn.net/cywosp/article/details/7428769 和http://blog.csdn.net/cywosp/article/details/7439440 趣雲的Swift部署和動態擴展:http://blog.lightcloud.cn/?p=68#sec-4.1 Openstack Hands on lab 2: Swift安裝並使用Keystone做身份驗證:http://liangbo.me/index.php/2012/03/29/openstack-hands-on-lab-2-swift-installation-with-keystone /

留言

這個網誌中的熱門文章

Json概述以及python對json的相關操作

遠程控制管理工具ipmitool

從Kubernetes到Cloud Native——雲原生應用之路