-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathrtfm.txt
212 lines (165 loc) · 9.45 KB
/
rtfm.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
This document describes how to build the infrastructure from scratch, as well manage it in general.
'SCRIPTS' refers to the scripts repo, found at https://github.com/vtluug/scripts
'ANSIBLE' refers to the ansible repo, found at https://github.com/vtluug/ansible
For ansible:
- Install it
- Install any additional packages in the ansible readme
Table of Contents
0. Summary
1. Installation
2. Networking
3. Configuration
Summary
======
NFS Configuration:
- All hosts:
- 755 root:root /home 10.98.0.7:/cistern/nfs/home
- 755 root:root /nfs
- 755 root:root /nfs/tank
- 775 nobody:luugusers /nfs/tank/share 10.98.0.6:/tank/nfs/share
- 777 root:root /nfs/tank/scratch 10.98.0.6:/tank/nfs/scratch
- 755 root:root /nfs/cistern
- 775 nobody:luugusers /nfs/cistern/share 10.98.0.7:/cistern/nfs/share
- 775,g+s nobody:officers /nfs/cistern/files 10.98.0.7:/cistern/nfs/files
- 755 root:root /nfs/keg
- 775 nobody:luugusers /nfs/keg/share 10.98.0.8:/keg/nfs/share
- Libvirt hosts only (meltdown, spectre):
- 755 root:root /nfs/cistern/libvirt 10.98.0.7:/cistern/nfs/libvirt
- 755 root:root /nfs/cistern/libvirt/images
- 755 root:root /nfs/cistern/libvirt/images/meltdown (For meltdown vms)
- 755 root:root /nfs/cistern/libvirt/images/spectre (For spectre vms)
- 755 root:root /nfs/cistern/libvirt/isos
- 775 root:root /var/lib/docker /dev/$hostname-vg/docker-lv # For restricting docker container sizes
- Docker Hosts only:
- 770 root:root /nfs/cistern/docker
- 755 root:root /nfs/cistern/docker/data 10.98.0.7:/cistern/nfs/docker/data
- Sczi Hosts only:
- 755 root:root /nfs/cistern/srv 10.98.0.7:/cistern/nfs/srv
- ZFS:
- All of the paths listed above should be ZFS filesystems on their respective hosts uexcept the images and isos directory within libvirt, which should be manually created
Installation (Baremetal)
========================
All hosts
- Create the papatux user with the password in the vtluug-admin repo
- This is used as a general admin account
- Aside from meltdown and spectre, use 2 boot drives in raid if possible
- Open ssh port 2222 (and 22) since this is externally blocked unless the IPs have exemptions.
- Ensure ssh is enabled and python is installed
- Ensure /etc/hostname contains the host's fqdn
Meltdown & Spectre
- Install using LVM:
- Name the Volume Group 'meltdown-vg'.
- Create a 50G root Logical Volume called 'root-lv'.
- Create a Logical Volume taking up 90% of free space called 'docker-lv'. This is for setting size limits on containers.
- Format 'docker-lv' as xfs. Ansible will take care of mounting it.
Router
- Disable root password login over ssh.
More configuration will be done in the 'Configuration' section after Networking
Networking
==========
- IPs and MACs of hosts are listed in scripts/router/lan/local_hosts
- Hosts get IPv4 address assigned using DHCP based on MAC, and IPv6 are assigned using SLAAC
- Set up physical boxes based on architecture_diagram.png TODO
Router (Debian)
---------------
Configure IP addresses
- Copy SCRIPTS/router/ip-config/interfaces into /etc/network/interfaces
Configure ARP Proxying
- Copy SCRIPTS/router/proxy/arp_proxy.sh into /usr/local/bin/
- Copy SCRIPTS/router/proxy/arp_proxy.service into /etc/systemd/system
- Start and enable arp_proxy.service
Configure dnsmasq (DHCP/SLAAC/DNS) & resolv.conf
- Install dnsmasq
- Copy SCRIPTS/router/lan/dnsmasq.conf into /etc/dnsmasq.conf
- Copy SCRIPTS/router/lan/local_hosts into /usr/local/bin
- Copy SCRIPTS/router/lan/resolv.conf into /etc/resolv.conf
- Start and enable dnsmasq.service
Configure iptables (NAT/Firewall)
- Install iptables
- Copy SCRIPTS/router/lan/vtluug_iptables.sh into /usr/local/bin
- Copy SCRIPTS/router/lan/vtluug_iptables.service into /etc/systemd/system
- Start and enable vtluug_iptables.service
Libvirt hosts (Ubuntu via netplan)
----------------------------------
- Configure the bridge interface
- Copy SCRIPTS/libvirt-hosts/ip-config/01-netcfg.yaml to /etc/netplan/01-netcfg.yaml
- Change $INTERFACE to the interface being bridged
- Change $MAC to the mac address specified in $SCRIPTS/router/lan/local_hosts
- Run 'netplan apply'
- Update dnsmasq on joey with the correct MAC address & apply changes (Only if new host is added w/o any records in dnsmasq)
- Restart networking on the libvirt hosts
Configuration
=============
These steps MUST be done in order. YMMV otherwise.
At this point, all bare metal hosts should have an admin user created, so root ssh will be disabled in the following step
NFS
---
To build each zfs pool, first look up the device ids in /dev/disk/by-id since these are guaranteed to not change (unlike /dev/sdX)
- Install 'zfsutils-linux'. If the following commands produce errors reboot first.
- Cyberdelia - 7 1TB drives in a raidz2 zfs pool named 'tank'
- Similarly, run 'zpool create -f -o ashift=12 keg raidz2 <disk_1_id> <disk_2_id> ... <disk_7_id>
- Dirtycow - 8 2TB drives in a raidz2 pool named 'cistern'
- Similarly, run 'zpool create -f -o ashift=12 cistern raidz2 <disk_1_id> <disk_2_id> ... <disk_8_id>
- Shellshock - 3 500GB drives in a raidz2 zfs pool named 'cistern'
- Run 'zpool create -f -o ashift=12 keg raidz1 <disk_1_id> <disk_2_id> <disk_3_id>
- Run 'zfs create <directory>' accoring to the 'NFS Configuration' section in the beginning of the document.
Baremetal Hosts
---------------
- Make sure you've copied your ssh key to all the baremetal servers so ansible works
- Run 'ansible-playbook main.yml -i hosts.cfg -K -u papatux --limit=<host>.vtluug.org --skip-tags needs_freeipa' for each host in nfs
- NFS needs to work first before we can do anything and freeipa needs to work before proper permissions can be set
- You could do the nfs instead of each host individually, but doing 1 host a time makes things easier to debug
- Create a homedir for papatux on nfs with the correct umask settings since it won't be a user in freeipa
- Run 'ansible-playbook main.yml -i hosts.cfg -K -u papatux --limit=<host>.vtluug.org --skip-tags needs_freeipa' for each host in baremetal and not in nfs group
- You could do the nfs instead of each host individually, but doing 1 host a time makes things easier to debug
Create the VMs
--------------
- Run 'ansible-playbook deploy.yml -i hosts.cfg -u papatux -k -K -@/path/to/accounts/file''
- Configuring FreeIPA Server
- Copy 'SCRIPTS/freeipa/ipa-config/ifcfg-eth0:0' to '/etc/sysconfig/network-scripts/eth0:0'.
- The configuration is not automated because it is only done once, and you should be aware of how it's configured
- Install packages freeipa-server, ipa-server-dns, and rng-tools
- In /etc/hosts, add chimera's v4 & v6 address, fqdn, and hostname to the bottom (on 2 separate lines)
- In /etc/NetworkManager/NetworkManager.conf, add 'dns=none' under [main]
- Reboot
- In /etc/resolv.conf, replace the content with a v4 & v6 address (preferably VTs)
- Reboot again
- Run 'ipa-server-install' (as root)
- Use these options:
- Do you want to configure integrated DNS (BIND)? [no]: yes
- Server host name: chimera.vtluug.org
- Domain name: krb.vtluug.org
- Realm name: KRB.VTLUUG.ORG
- Directory Manager PW: See accounts file in gitolite-admin
- Admin PW: See accounts file in gitolite-admin
- Configure DNS forwarders: yes
- Search for missing reverse zones: yes
- (Press Enter until you get to the next step)
- Continue to configure system with these values: yes
- Run 'kinit admin' to authenticate as admin
- Create groups, set shell/home/email, and disable private groups
- 'ipa group-add luugusers --desc "VTLUUG Members"'
- 'ipa group-add officers --desc "VTLUUG Officers"'
- 'ipa config-mod --defaultgroup luugusers'
- 'ipa config-mod --defaultshell=/bin/bash'
- 'ipa config-mod --homedirectory=/nfs/cistern/home'
- 'ipa config-mod --emaildomain=vtluug.org'
- 'ipa-managed-entries -e "UPG Definition" -p $DIRECTORY_MANAGER_PW disable'
- Reboot one last time
- Do NOT try to use letsencrypt. FreeIPA can be very fragile sometimes so just don't do it.
- Run 'ansible-playbook main.yml -i hosts.cfg -u papatux -k -K --limit chimera.vtluug.org' to configure the rest of the system
Configure baremetal hosts for freeipa login & correct folder permissions
------------------------------------------------------------------------
- Run 'ipa-client-install --domain=krb.vtluug.org --server=chimera.vtluug.org --principal=admin'
- For NFS hosts, set the proper permissions for nfs folders in the 'NFS Configuration' now that freeipa is working
- Run the ansible scripts from 2 sections ago, but with '--tags' instead of '--skip-tags'
- Run 'mount -a' on nfs hosts to mount dirs with correct permissions
Configure the new VMs (mostly)
------------------------------
- Run 'ansible-playbook main.yml -i hosts.cfg -u papatux -k -K --limit=vms'
- This mostly configures all the new hosts
Docker Services Config
--------------------------------------
- Ansible should've pulled https://github.com/vtluug/apps.git into /nfs/cistern/docker/apps on meltdown, spectre, & sczi (+ any other docker hosts)
- Follow directions in the README for each of the services
Acidburn specific config ???