ULab NIUInstallNotes
Revision as of 14:14, 7 July 2014 by imported>TimothyPearson (Fix attachment)
Installation notes from uLab system at Northern Illinois University
Required files
Extract the following file(s) to a flash drive for use where indicated in the installation instructions below
- ulab-config.tar.gz: Referenced uLab configuration files
Installation instructions
==================================
Bare Metal Hardware
Main Server (x1)
8x Opteron 3.3GHz
8GB RAM
2TB x6 hardware RAID10
Mellanox Infiniband 4x CX4
GbE x2
Client Nodes (x1)
8x Opteron
10GB RAM
Mellanox Infiniband 4x CX4
GbE x2
Router
Xeon E3
GbE x2
500GB x2 RAID1
==================================
Initial setup
Connect 1 router GbE port to 1 main server GbE port; designate master uplink
Connect other router GbE port to public LAN/WAN
Connect the first Infiniband port on each server to the Infiniband switch
Connect first network port on all servers excet the main server to the GbE router
Connect UPS to main server
INSTALL NETWORK ROUTER
Install pfSense on router Flash drive via zcat
Create internal DMZ network on 172.16.0.0/24
Navigate to https://172.16.0.1 and complete setup
Enable SSH server on port xxxx
Add WAN rule to allow traffic to WAN port xxxx
Add NAT rule to forward traffic from WAN TCP port 3390 to TCP port 3389 on master001 (172.16.0.10)
Add NAT rule to forward traffic from WAN TCP port 80 to TCP port 80 on master001 (172.16.0.10)
Add NAT rule to forward traffic from WAN TCP port 4004 to TCP port 4004 on master001 (172.16.0.10)
*****NOTE: NIU has assigned WAN address 131.156.yyy.zzz to the system
*****NOTE: Port 3390 is used on the WAN due to NIU blocking port 3389 as a Windows antihack measure
INSTALL MAIN SERVER
Create RAID10 arrays
Dell utility
Boot main server from LiveUSB disk
Make root filesystem
mkfs.ext4 /dev/sda1
Mount system disk array
mkdir /disk
mount /dev/sda /disk
Install base Debian Wheezy system
debootstrap --arch amd64 wheezy /disk/ http://<debian archive>
Install Linux kernel and grub
mount -o bind /dev /disk/dev
mount -o bind /proc /disk/proc
mount -o bind /sys /disk/sys
chroot /disk/
apt-get install linux-image grub-pc mdadm
grub-install /dev/sda
grub-install /dev/sdb
dpkg-reconfigure grub-pc
passwd
exit
reboot
Set hostname
nano /etc/hostname
master001
nano /etc/hosts
192.168.1.10 master001.cluster90.local master001
192.168.1.11 ldap001.cluster90.local ldap001
Setup networking
nano /etc/network/interfaces
auto eth0
iface eth0 inet static
address 192.168.1.10
netmask 255.255.255.0
auto eth1
iface eth1 inet dhcp
ifup eth1
Install secure shell server and client
apt-get install ssh
Configure Infiniband networking
apt-get install infiniband-diags
nano /etc/modules
Append these lines:
ib_umad
ib_ipoib
nano /etc/network/interfaces
Append these lines:
auto ib0
iface ib0 inet static
address 192.168.2.10
netmask 255.255.255.0
Install apt-mirror
apt-get install apt-mirror
nano /etc/apt/mirror.list
Edit to taste; need main package archives, TDE repository, LibreOffice TDE repository, and uLab repository
Also need i386 and armel binaries (deb-i386 and deb-armel)
Should also grab source (deb-src)
Initialize local package mirror
apt-mirror
Enable cron apt-mirror runs
nano /apt-mirror
#!/bin/bash
apt-mirror
/var/spool/apt-mirror/var/clean.sh
chmod +x /apt-mirror
crontab -e
0 4 * * * /apt-mirror &> /dev/null
Install Apache web server
apt-get install apache2
Configure Apache to serve package mirrors
ln -s /var/spool/apt-mirror/mirror/ftp.us.debian.org/debian /var/www/debian
ln -s /var/spool/apt-mirror/mirror/ppa.quickbuild.pearsoncomputing.net/trinity/trinity-nightly-builds/ubuntu /var/www/tde-nightly
ln -s /var/spool/apt-mirror/mirror/ppa.quickbuild.pearsoncomputing.net/trinity/trinity-nightly-build-dependencies/ubuntu /var/www/tde-nightly-deps
ln -s /var/spool/apt-mirror/mirror/ppa.quickbuild.pearsoncomputing.net/trinity/libreoffice-tde /var/www/libreoffice-tde
ln -s /var/spool/apt-mirror/mirror/ppa.quickbuild.pearsoncomputing.net/trinity/ulab/ubuntu ulab
Configure apt to use the local package mirror
nano /etc/apt/sources.list
deb http://localhost/debian wheezy main
deb http://localhost/tde-nightly wheezy main
deb http://localhost/tde-nightly-deps wheezy main
deb http://localhost/ulab wheezy main
apt-get update
Install bare remote management software
apt-get install x11vnc chromium xorg
Install DHCP server
apt-get install dhcp3-server
Configure DHCP server
Create /etc/dhcp/dhcpd.conf from /config/dhcp/dhcpd.conf on flash drive
chown root /etc/dhcp/dhcpd.conf
chgrp root /etc/dhcp/dhcpd.conf
chmod 644 /etc/dhcp/dhcpd.conf
/etc/init.d/isc-dhcp-server restart
Install DNS server
apt-get install bind9
Configure DNS server
Create /etc/bind/named.conf.local from /config/bind/named.conf.local on flash drive
Create /var/lib/bind/cluster90.edu.db from /config/bind/cluster90.edu.db on flash drive
Create /var/lib/bind/rev.1.168.192.in-addr.arpa from /config/bind/rev.1.168.192.in-addr.arpa on flash drive
mkdir -p /var/log/bind9
chown -R bind /var/log/bind9
chgrp -R bind /var/log/bind9
chown root /etc/bind/named.conf.local
chown root /var/lib/bind/cluster90.edu.db
chown root /var/lib/bind/rev.1.168.192.in-addr.arpa
chgrp root /etc/bind/named.conf.local
chgrp root /var/lib/bind/cluster90.edu.db
chgrp root /var/lib/bind/rev.1.168.192.in-addr.arpa
chmod 644 /etc/bind/named.conf.local
chmod 644 /var/lib/bind/cluster90.edu.db
chmod 644 /var/lib/bind/rev.1.168.192.in-addr.arpa
nano /etc/bind/named.conf
Comment out /etc/bind/named.default-zones ***FIXME***
nano /etc/bind/named.conf.options
Add these lines below the commented out forwarders block:
forwarders {
172.16.0.1;
};
Comment out dnssec-validation auto;
Add this below that newly commented line:
dnssec-validation no;
/etc/init.d/bind9 restart
Configure local DNS resolver
nano /etc/resolv.conf
nameserver 192.168.1.10
search cluster90.edu
*****NOTE: If this step is omitted, Kerberos will not be able to communicate with the KDC!
Install TFTP server
apt-get install tftpd-hpa
Configure TFTP server
Create /etc/default/tftpd-hpa from /config/default/tftpd-hpa on flash drive
chown root /etc/default/tftpd-hpa
chgrp root /etc/default/tftpd-hpa
chmod 644 /etc/default/tftpd-hpa
mkdir -p /var/lib/tftpboot
Copy PXE base files to /var/lib/tftpboot from /config/tftpboot on flash drive
chmod -R 655 /var/lib/tftpboot
/etc/init.d/tftpd-hpa restart
cd /var/lib/tftpboot
cp -Rp /boot/vmlinuz-3.2.0-4-amd64 vmlinuz-node
cp -Rp /etc/initramfs-tools/initramfs.conf /etc/initramfs-tools/initramfs.conf.good
nano /etc/initramfs-tools/initramfs.conf
MODULES=netboot
update-initramfs -u -k all
cp -Rp /boot/initrd.img-3.2.0-4-amd64 initrd-node
mv /etc/initramfs-tools/initramfs.conf.good /etc/initramfs-tools/initramfs.conf
update-initramfs -u -k all
Install NTP server
apt-get install ntp
Configure NTP server
Append the following line to /etc/ntp.conf:
broadcast 192.168.1.255
/etc/init.d/ntp restart
Set timezone
dpkg-reconfigure tzdata
Install NFS server
apt-get install nfs-kernel-server
Create NFS root directories
mkdir -p /nfsroots/amd64
mkdir -p /nfsroots/armel
Create user home directory
mkdir -p /userdata/homes
Create user home template directory
mkdir -p /userdata/usertemplate
Create common read only data files directory
mkdir -p /readonlydata
Create program data directory
mkdir -p /applications
Configure NFS server
nano /etc/exports
/nfsroots/amd64 192.168.1.1/24(rw,async,no_subtree_check,no_root_squash)
/nfsroots/armel 192.168.1.1/24(rw,async,no_subtree_check,no_root_squash)
/userdata/homes 192.168.2.1/24(rw,async,no_subtree_check,no_root_squash)
/applications 192.168.2.1/24(rw,async,no_subtree_check,no_root_squash)
/readonlydata 192.168.2.1/24(ro,async,no_subtree_check,no_root_squash)
/userdata/usertemplate 192.168.2.1/24(ro,async,no_subtree_check,no_root_squash)
/etc/init.d/nfs-kernel-server restart
Install Network UPS Tools server and client
apt-get install nut
Configure Network UPS Tools server and client
cd /etc/nut
nano ups.conf
Append the following lines:
[dell]
driver = usbhid-ups
port = auto
nano nut.conf
MODE=netserver
nano upsd.users
Append the following lines:
[upsmaster]
password = internalmaster
allowfrom = localhost internal
upsmon master
[upsslave]
password = internalslave
allowfrom = localhost internal
upsmon slave
nano upsmon.conf
Append the following line:
MONITOR dell@localhost 1 upsmaster internalmaster master
nano upsd.conf
Append the following lines:
LISTEN 127.0.0.1
LISTEN 192.168.1.10
ACL all 0.0.0.0/0
ACL localhost 127.0.0.1/32
ACL internal 192.168.1.0/24
ACCEPT localhost
ACCEPT internal
REJECT all
upsc -l
upsc dell
/etc/init.d/nut-server restart
/etc/init.d/nut-client restart
Install debootstrap
apt-get install debootstrap
Install MySQL server
apt-get install mysql-server
Configure MySQL server
nano /etc/mysql/my.cnf
Comment out "bind-address"
/etc/init.d/mysql restart
Generate master SSH keys
ssh-keygen -t rsa
Install git
apt-get install git
Obtain uLab hardware access source code
cd /
git clone http://scm.trinitydesktop.org/scm/git/remotelaboratory
Obtain uLab terminal services source code
cd /
git clone http://scm.trinitydesktop.org/scm/git/xrdp-proprietary
Install uLab databases
mysql -p < /remotelaboratory/database/mysql/remotelab.sql
mysql -p < /xrdp-proprietary/database/mysql/remotelab_sm_database.sql
mysql -p
GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY '<masterpassword>';
FLUSH PRIVILEGES;
quit
Install base TDE system
apt-get install tdebase-trinity
/etc/init.d/tdm-trinity start
Install LDAP bonding utility
apt-get install kcontrol-ldap-bonding-trinity
***** Set up LDAP server here
Enable root logins
nano /etc/trinity/tdm/tdmrc
AllowRootLogin=true
reboot
Join master server to LDAP realm
Start TDE and log in
Start TDE Menu->Settings->Control Center
Under "TDE Components", select "LDAP Realm Bonding"
Check "Enable LDAP Realm Support"
Click "Bond to New Realm" and follow the prompts
***** Set up terminal services node here
Set up initial user template
rm -rf /userdata/usertemplate
cp -Rp /nfsroots/amd64/node001/etc/skel /userdata/usertemplate
mkdir -p /userdata/usertemplate/.trinity/share/config
nano /userdata/usertemplate/.trinity/share/config/kpersonalizerrc
[General]
FirstLogin=false
nano /userdata/usertemplate/.trinity/share/config/twinrc
[Windows]
AnimateMinimize=false
AnimateShade=false
mkdir -p /userdata/usertemplate/Documents
mkdir -p /userdata/usertemplate/Desktop
mkdir -p /userdata/usertemplate/.config
nano /userdata/usertemplate/Desktop/remote_laboratory_client.desktop
[Desktop Entry]
Categories=Qt;TDE;Utility;
Comment=Remote laboratory client
Comment[en_US]=Remote laboratory client
Encoding=UTF-8
Exec=remote_laboratory_client master001.cluster90.edu
GenericName=Remote Laboratory Client
GenericName[en_US]=Remote Laboratory Client
Icon=remote_laboratory_client.png
MimeType=
Name=Remote Laboratory Client
Path=
StartupNotify=true
Terminal=false
TerminalOptions=
Type=Application
X-DCOP-ServiceType=
X-TDE-SubstituteUID=false
X-TDE-Username=
nano /userdata/usertemplate/Desktop/xilinx_ise_14_4.desktop
[Desktop Entry]
Comment=
Comment[en_US]=
Exec=/applications/xilinx/start_ise_14.4
GenericName=
GenericName[en_US]=
Icon=/applications/xilinx/14.4/ISE_DS/ISE/data/images/pn-ise.png
MimeType=
Name=Xilinx ISE 14.4
Name[en_US]=Xilinx ISE 14.4
Path=
StartupNotify=true
Terminal=false
TerminalOptions=
Type=Application
X-DCOP-ServiceType=
X-TDE-SubstituteUID=false
X-TDE-Username=
nano /userdata/usertemplate/Desktop/common_data_files.desktop
[Desktop Entry]
Name=Common Data Files
Icon=kfm
Type=Link
URL=/readonlydata/
nano /userdata/usertemplate/.config/Trolltech.conf
[Qt]
Font%20Substitutions/arial=helvetica
Font%20Substitutions/courier%20new=courier
Font%20Substitutions/sans%20serif=helvetica
Font%20Substitutions/times%20new%20roman=times
GUIEffects=none
Palette/active=#000000, #dddfe4, #ffffff, #eeeff1, #aaacb0, #b8babe, #000000, #ffffff, #000000, #ffffff, #efefef, #000000, #000080, #ffffff, #0000ff, #ff00ff, #e8e8e8
Palette/disabled=#6e6f72, #dddfe4, #ffffff, #eeeff1, #aaacb0, #939598, #6e6f72, #ffffff, #6e6f72, #ffffff, #efefef, #000000, #000080, #ffffff, #0000ff, #ff00ff, #e8e8e8
Palette/inactive=#000000, #dddfe4, #ffffff, #eeeff1, #aaacb0, #b8babe, #000000, #ffffff, #000000, #ffffff, #efefef, #000000, #000080, #ffffff, #0000ff, #ff00ff, #e8e8e8
XIMInputStyle=On The Spot
cursorFlashTime=1000
doubleClickInterval=400
embedFonts=true
font="Sans Serif,9,-1,5,50,0,0,0,0,0"
fontPath=@Invalid()
globalStrut/height=0
globalStrut/width=0
resolveSymlinks=false
style=Windows
useRtlExtensions=false
wheelScrollLines=3
chown -R root /userdata/usertemplate
chgrp -R root /userdata/usertemplate
chmod -R o-rwx /userdata/usertemplate
Configure laboratory workspaces database
Fire up your favorite MySQL client and connect to the master server as root
Add workspace system user
USE `remotelab`;
GRANT USAGE on *.* to workspacesystem@"%" identified by '<workspacepassword>';
GRANT ALL PRIVILEGES ON remotelab.* to workspacesystem@"%";
FLUSH PRIVILEGES;
Register custom FPGA service types
USE `remotelab`;
INSERT INTO `servicetypes`(`serviceid`,`name`,`description`,`client_library`,`version`,`single_instance`) values (256,'Remote FPGA','Remote FPGA access [X3CS200-FT256]','libremotelab_fpgaviewer',1,1);
INSERT INTO `servicetypes`(`serviceid`,`name`,`description`,`client_library`,`version`,`single_instance`) values (257,'Remote FPGA','Remote FPGA access [XC6LX16-CS324]','libremotelab_fpgaviewer',1,1);
Register workspace stations (2 administrative and 5 FPGA stations)
USE `remotelab`;
INSERT INTO `stations`(`pk`,`name`,`description`) values (1,'Session Management','Session Management');
INSERT INTO `stations`(`pk`,`name`,`description`) values (2,'User Access Management','User Access Management');
INSERT INTO `stations`(`pk`,`name`,`description`) values (20,'Remote FPGA Access [X3CS200-FT256]','Remote FPGA Access');
INSERT INTO `stations`(`pk`,`name`,`description`) values (21,'Remote FPGA Access [XC6LX16-CS324]','Remote FPGA Access');
INSERT INTO `stations`(`pk`,`name`,`description`) values (22,'Remote FPGA Access [XC6LX16-CS324]','Remote FPGA Access');
INSERT INTO `stations`(`pk`,`name`,`description`) values (23,'Remote FPGA Access [XC6LX16-CS324]','Remote FPGA Access');
INSERT INTO `stations`(`pk`,`name`,`description`) values (24,'Remote FPGA Access [XC6LX16-CS324]','Remote FPGA Access');
Add services to stations
USE `remotelab`;
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (1,6,'master001.cluster90.edu',4015);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (2,7,'master001.cluster90.edu',4016);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (20,256,'pi001.cluster90.edu',4010);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (21,257,'pi002.cluster90.edu',4010);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (22,257,'pi003.cluster90.edu',4010);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (23,257,'pi004.cluster90.edu',4010);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (24,257,'pi005.cluster90.edu',4010);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (20,3,'pi001.cluster90.edu',4012);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (21,3,'pi002.cluster90.edu',4012);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (22,3,'pi003.cluster90.edu',4012);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (23,3,'pi004.cluster90.edu',4012);
INSERT INTO `services`(`station`,`servicetype`,`hostname`,`port`) values (24,3,'pi005.cluster90.edu',4012);
Set initial permissions
USE `remotelab`;
INSERT INTO `permissions`(`station`,`groupname`) values ('1','realmadmins');
INSERT INTO `permissions`(`station`,`groupname`) values ('2','realmadmins');
INSERT INTO `permissions`(`station`,`groupname`) values ('20','realmadmins');
INSERT INTO `permissions`(`station`,`groupname`) values ('21','realmadmins');
INSERT INTO `permissions`(`station`,`groupname`) values ('22','realmadmins');
INSERT INTO `permissions`(`station`,`groupname`) values ('23','realmadmins');
INSERT INTO `permissions`(`station`,`groupname`) values ('24','realmadmins');
Mark workspace servers as online
USE `remotelab`;
INSERT INTO `status`(`hostname`,`online`) values ('master001.cluster90.edu','1');
INSERT INTO `status`(`hostname`,`online`) values ('pi001.cluster90.edu','1');
INSERT INTO `status`(`hostname`,`online`) values ('pi002.cluster90.edu','1');
INSERT INTO `status`(`hostname`,`online`) values ('pi003.cluster90.edu','1');
INSERT INTO `status`(`hostname`,`online`) values ('pi004.cluster90.edu','1');
INSERT INTO `status`(`hostname`,`online`) values ('pi005.cluster90.edu','1');
Install development packages for compilation of xrdp forwarder
apt-get install bzip2 build-essential autoconf libtool gettext pkg-config libssl-dev libpam0g-dev libx11-dev libxfixes-dev libmysqlclient-dev
Compile and install xrdp forwarder
cd /xrdp-proprietary
./bootstrap
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make -j12
make install
Install development packages for compilation of X11rdp display server
apt-get install xsltproc flex bison libxml-parser-perl
Compile X11rdp
cd /xrdp-proprietary/xorg/X11R7.6
mkdir /opt/X11rdp
/bin/sh buildx.sh /opt/X11rdp
Create binary archive for deployment
cd /
tar -cjvf x11rdp_binaries.tar.bz2 opt/X11rdp
Install PulseAudio
apt-get install pulseaudio
Enable passwordless SSH access to terminal services nodes
ssh-copy-id -i ~/.ssh/id_rsa.pub root@node001.cluster90.edu
Configure terminal services database
Fire up your favorite MySQL client and connect to the master server as root
Add terminal services system user
USE `remotelab_sm`;
GRANT USAGE on *.* to terminalsystem@"%" identified by '<terminalpassword>';
GRANT ALL PRIVILEGES ON remotelab_sm.* to terminalsystem@"%";
FLUSH PRIVILEGES;
Register terminal servers (1 terminal server)
USE `remotelab_sm`;
INSERT INTO `servers`(`name`,`maxusers`,`online`) values ('node001.cluster90.edu',10,1);
Set initial permissions and limits
USE `remotelab_sm`;
INSERT INTO `allowed_servers`(`groupname`,`server`) values ('realmadmins','node001.cluster90.edu');
INSERT INTO `groups`(`groupname`,`sesslimit`) values ('realmadmins',65535);
Configure xrdp forwarder
nano /etc/xrdp/xrdp.ini
Append the following lines
[Database]
Server=master001.cluster90.edu
Database=remotelab_sm
User=terminalsystem
Password=<terminalpassword>
chmod 600 /etc/xrdp/xrdp.ini
Restart xrdp forwarder
/etc/xrdp/xrdp.sh restart
Obtain Kerberos keys for uLab arbiter from KDC
kadmin -p ldapadmin@CLUSTER90.EDU
ext ulab/master001.cluster90.edu
Set up a cron job to refresh the uLab arbiter keytab file
nano /root/ulab_arbiter_password
Same password as given for the ulab account above
chown root /root/ulab_arbiter_password
chgrp root /root/ulab_arbiter_password
chmod 400 /root/ulab_arbiter_password
nano /refresh_ulab_arbiter_keytab
#!/bin/bash
kinit --password-file=/root/ulab_arbiter_password ulab
chown root /refresh_ulab_arbiter_keytab
chgrp root /refresh_ulab_arbiter_keytab
chmod 700 /refresh_ulab_arbiter_keytab
crontab -e
Append this line:
0 3 * * * /refresh_ulab_arbiter_keytab &> /dev/null
Install the uLab arbiter daemon
apt-get install ulab-authserver
Configure the uLab arbiter daemon
mkdir -p /root/.trinity/share/config
nano /root/.trinity/share/config/ulab_authserver.conf
[Database]
server=master001.cluster90.edu
username=workspacesystem
password=<workspacepassword>
database=remotelab
driver=TQMYSQL3
chmod 600 /root/.trinity/share/config/ulab_authserver.conf
/etc/init.d/ulab-authserver restart
Install the uLab user management daemon
apt-get install ulab-adminusermgmtserver
Configure the uLab user management daemon
nano /root/.trinity/share/config/ulab_adminusermgmtserver.conf
[Terminals Database]
server=master001.cluster90.edu
username=terminalsystem
password=<terminalpassword>
database=remotelab_sm
driver=TQMYSQL3
[Workspaces Database]
server=master001.cluster90.edu
username=workspacesystem
password=<workspacepassword>
database=remotelab
driver=TQMYSQL3
[Security]
masteruser=ulab
masterrealm=
chmod 600 /root/.trinity/share/config/ulab_adminusermgmtserver.conf
/etc/init.d/ulab-adminusermgmtserver restart
Install the uLab system management daemon
apt-get install ulab-adminsysctlserver
Configure the uLab user management daemon
nano /root/.trinity/share/config/ulab_adminsysctlserver.conf
[Terminals Database]
server=master001.cluster90.edu
username=terminalsystem
password=<terminalpassword>
database=remotelab_sm
driver=TQMYSQL3
[Workspaces Database]
server=master001.cluster90.edu
username=workspacesystem
password=<workspacepassword>
database=remotelab
driver=TQMYSQL3
[Security]
masteruser=ulab
masterrealm=
chmod 600 /root/.trinity/share/config/ulab_adminsysctlserver.conf
/etc/init.d/ulab-adminsysctlserver restart
Configure user permissions
Log in to the RDP services as a realm administrator
Start the uLab Remote Laboratory Client
remote_laboratory_client master001.cluster90.edu
Click Connect
Select "Manage user permissions" and click OK
Launch the User Management Console
In the top pane, select "standardusers" and click "Modify Group"
Chamge "Maximum Simultaneous Users" to 10
Add "node001.cluster90.edu" to the Selected list
Click OK
In the bottom pane, select "standardusers" and click "Modify Group"
Add "fpgaaccess" to the Selected list
Click OK
Install VirtualBox
apt-get install virtualbox
Add virtual machine for license server
Start VirtualBox client
Click New, then Next
Enter "License Server" as the name, select Linux and RedHat 64, then click Next
Enter 2048MB for the memory size, then click Next
Select Create New Hard Disk, then click Next
Select VDI, then click Next
Select Fixed Size, then click Next
Select /root/VirtualBox\ VMs as the location and set the size to 128GB, then click Next
Click Create
Click Create
Install license server
Download RHEL 6 install DVD and copy to /root/CD Images
Start VirtualBox client
Select the License Server entry and click Settings
Select the Storage list entry
Click the Add CD/DVD Device button
Click Choose Disk
Select the RHEL 6 install DVD from /root/CD Images and click OK
Select the Network list entry
In the "Attached to" drop-down list, select "Bridged Adapter"
Click OK
Click Start
Follow the on-screen prompts to install RHEL 6 with the following settings:
Name: license001.cluster90.edu
IP address: 192.168.1.9
Network mask: 255.255.255.0
Gateway: 192.168.1.10
DNS: 192.168.1.10
After installation, shut down the virtual machine
Select the License Server entry and click Settings
Select the System list entry
Select the Hard Disk entry and move it to the top of the list
Click OK
***** FIXME
***** TO BE CONTINUED
Publish read-only FPGA template files and sample project(s)
mkdir -p /readonlydata/uLab/FPGA
cd /readonlydata/uLab/FPGA
mkdir common
cd common
wget http://git.trinitydesktop.org/cgit/ulab/plain/fpga/common/remote_access.v
cd ..
<copy other desired data, such as sample projects, into this directory>
INSTALL LDAP/KERBEROS REALM CONTROLLER SERVER
Boot LDAP server from LiveUSB disk
Create RAID partitions on disks
fdisk /dev/sda
1 partition, type fd
fdisk /dev/sdb
1 partition, type fd
Create RAID1 array
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb
mkfs.ext4 /dev/md0
Mount system disk array
mkdir /disk
mount /dev/sda /disk
Install base Debian Wheezy system
debootstrap --arch amd64 wheezy /disk/ http://192.168.10.1
Install Linux kernel and grub
mount -o bind /dev /disk/dev
mount -o bind /proc /disk/proc
mount -o bind /sys /disk/sys
chroot /disk/
apt-get install linux-image grub mdadm
grub-install /dev/sda
grub-install /dev/sdb
dpkg-reconfigure grub-pc
passwd
exit
reboot
Set hostname
nano /etc/hostname
ldap001
nano /etc/hosts
192.168.1.11 ldap001.cluster90.local master001
Setup networking
nano /etc/network/interfaces
auto eth0
iface eth0 inet static
address 192.168.1.11
netmask 255.255.255.0
auto eth1
iface eth1 inet dhcp
ifup eth1
nano /etc/resolv.conf
nameserver 192.168.1.10
search cluster90.edu
Install secure shell server and client
apt-get install ssh
Install NTP client daemon
apt-get install ntp
Configure NTP client daemon
nano /etc/ntp.conf
Remove all server lines except one, and change the remaining line to use master001.cluster90.edu
/etc/init.d/ntp restart
Set timezone
dpkg-reconfigure tzdata
Configure apt to use the local package mirror
nano /etc/apt/sources.list
deb http://master001.cluster90.edu/debian wheezy main
deb http://master001.cluster90.edu/tde-nightly wheezy main
deb http://master001.cluster90.edu/tde-nightly-deps wheezy main
apt-get update
Install base TDE system
apt-get install tdebase-trinity
/etc/init.d/tdm-trinity start
Install LDAP controller and management utilities
apt-get install kcontrol-ldap-controller-trinity kcontrol-ldap-manager-trinity
Enable root logins
nano /etc/trinity/tdm/tdmrc
AllowRootLogin=true
reboot
Configure LDAP
Start TDE and log in
Start TDE Menu->Settings->Control Center
Under "TDE Components", select "LDAP Realm Controller"
Select "Primary Realm Controller" from the drop-down list and follow the prompts
Add uLab LDAP service
Start TDE Menu->Settings->Control Center
Under "TDE Components", select "LDAP Realm Manager" and log in as the LDAP admin user
Select the Services tab and click New
Enter "ulab" as the service name and select "master001.cluster90.edu" as the host name, then click OK
Add uLab arbiter user
Start TDE Menu->Settings->Control Center
Under "TDE Components", select "LDAP Realm Manager" and log in as the LDAP admin user
Select the Users tab and click New
Select the POSIX tab
Enter "ulab" as the Login Name
Enter "uLab Arbiter" as the display name
Select "standardusers" as the Primary Group
Enter "/home/ulab" as the Home Directory
Enter "/dev/null" as the Shell
Select the User Information tab
Enter "uLab" as the Last Name
Select the Password and Security tab
Enter a strong password and remember it
Click OK
Install Network UPS Tools client
apt-get install nut-client
Configure Network UPS Tools server and client
cd /etc/nut
nano nut.conf
MODE=netclient
nano upsmon.conf
Append the following line:
MONITOR dell@master001.cluster90.edu 1 upsmaster internalmaster master
upsc dell@master001.cluster90.edu
/etc/init.d/nut-client restart
SET UP ARMEL XILINX PROGRAMMER TEMPLATE
Download Debian Wheezy armel image from the Raspberry Pi website
Write image to SD card
dd if=<armel wheezy image name> of=<sd card>
Update bootloader
Download firmware zip file from https://github.com/raspberrypi/firmware/ and extract
Mount first partition on SD card (boot)
Remove all files on that partition
Copy contents of the boot directory in the firmware download to the boot partition of the SD card
Install SD card into Raspberry Pi board
Boot Raspberry Pi with network cable and TV attached
In first run wizard, enable SSH then select "Finish"
Reboot the Raspberry Pi
Login with pi/raspberry
Set root password
sudo passwd
exit
Login as root
Delete pi account
deluser pi
rm -rf /home/pi
Prepare Raspberry Pi system files for nfsroot and create deployment file
Shutdown Raspberry Pi and insert its SD card into a reader on a normal computer
Mount second partition on the SD card
cd <path to Raspberry Pi system partition>
nano etc/fstab
Remove the root (/) mount line
nano etc/inittab
Comment out the "T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100" line with a "#"
nano etc/ntp.conf
Remove all server lines except one, and change the remaining line to use master001.cluster90.edu
nano etc/resolv.conf
Change nameserver to 192.168.1.10
nano etc/apt/sources.list
deb http://master001.cluster90.edu/debian wheezy main contrib non-free
deb http://master001.cluster90.edu/tde-nightly wheezy main
deb http://master001.cluster90.edu/tde-nightly-deps wheezy main
deb http://master001.cluster90.edu/ulab wheezy main
tar -cjvf rpi_system_image.tar.bz2 .
INSTALL ARMEL XILINX PROGRAMMER
Deploy Raspberry Pi system files to NFS server
cd /nfsroots/armel
mkdir pi001
cd pi001
tar -xjvf <path to storage>/rpi_system_image.tar.bz2
nano etc/hostname
pi001
nano etc/hosts
Edit the last local network line to include "pi001.cluster90.edu pi001" instead of "raspberrypi"
Create nfsroot boot SD card
Mount first partition of SD card and open terminal in it
nano cmdline.txt
dwc_otg.lpm_enable=0 root=/dev/nfs nfsroot=192.168.1.10:/nfsroots/armel/pi001 ip=dhcp rootfstype=nfs
Close terminal and unmount SD card
fdisk <device node of SD card>
Delete all partitions except partition 1
Install nfsroot boot card into Raspberry Pi
Add DNS entries
nano /var/lib/bind/cluster90.edu.db
Append:
pi001 A 192.168.1.50
nano /var/lib/bind/rev.1.168.192.in-addr.arpa
Append:
50 PTR pi001.cluster90.edu.
/etc/init.d/bind9 restart
Enable Raspberry Pi boot from master server
Plug in Raspberry Pi. including network, and obtain MAC address
nano /etc/dhcp/dhcpd.conf
Append:
host pi001 {
hardware ethernet <xx:xx:xx:xx:xx:xx>;
fixed-address 192.168.1.50;
}
/etc/init.d/isc-dhcp-server restart
Reboot the Raspberry Pi
Install VNC server
apt-get install tightvncserver
Install base TDE system
apt-get remove lightdm
apt-get install tdebase-trinity
/etc/init.d/tdm-trinity start
Install LDAP bonding utility
apt-get install kcontrol-ldap-bonding-trinity
Enable root logins
nano /etc/trinity/tdm/tdmrc
AllowRootLogin=true
reboot
Join Raspberry Pi to LDAP realm
vncserver
EXPORT DISPLAY=:1
/opt/trinity/bin/starttde
From remote VNC viewer
Start TDE Menu->Settings->Control Center
Under "TDE Components", select "LDAP Realm Bonding"
Check "Enable LDAP Realm Support"
Click "Bond to New Realm" and follow the prompts
Log out
***** Add ulab services for this host on the LDAP controller here
Obtain Kerberos keys for hardware access server from KDC
kadmin -p ldapadmin@CLUSTER90.EDU
ext ulab/pi001.cluster90.edu
Install the uLab FPGA viewer daemon
apt-get install ulab-fpgaserver
Configure the uLab FPGA viewer daemon
nano /ulab_fpgaserver.conf
[Security]
masteruser=ulab
masterrealm=
[FPGA]
serialdevice=/dev/ttyUSB0
baudrate=115200
chmod 600 /ulab_fpgaserver.conf
/etc/init.d/ulab-fpgaserver restart
Install the uLab FPGA programmer daemon
apt-get install ulab-fpgaprogserver
Install the precompiled uLab FPGA programming utilities
cd /usr/bin
wget http://ulab.trinitydesktop.org/binaries/armel/bit2svf
wget http://ulab.trinitydesktop.org/binaries/armel/xsvf-rpi
wget http://ulab.trinitydesktop.org/binaries/armel/bitdevice
wget http://git.trinitydesktop.org/cgit/ulab/plain/fpga/xilinx/programmer/program_device.sh
chmod 755 bit2svf
chmod 755 xsvf-rpi
chmod 755 bitdevice
chmod 755 program_device.sh
mkdir -p /usr/share/bit2svf
cd /usr/share/bit2svf
wget http://git.trinitydesktop.org/cgit/ulab/plain/fpga/xilinx/programmer/bit2svf/templates/alg_Spartan_6.svft
wget http://git.trinitydesktop.org/cgit/ulab/plain/fpga/xilinx/programmer/bit2svf/templates/DEVICES
Configure the uLab FPGA programmer daemon
nano /ulab_fpgaprogserver.conf
[Security]
masteruser=ulab
masterrealm=
[Programming]
script=/usr/bin/program_device.sh %f
chmod 600 /ulab_fpgaprogserver.conf
/etc/init.d/ulab-fpgaprogserver restart
INSTALL DISKLESS TERMINAL SERVICES NODE
Install base Debian Wheezy system
cd /nfsroots/amd64
mkdir node001
cd node001
debootstrap --arch amd64 wheezy /nfsroots/amd64/node001/ http://master001.cluster90.edu/debian/
Set hostname
nano etc/hostname
node001
Install secure shell server and client
chroot .
apt-get install ssh
exit
Set root password
chroot .
passwd
exit
Assign IP address
Obtain MAC address of node server and setup PXE boot as the only boot option in the node server's BIOS
node001: 08:9e:01:17:a2:04
nano /etc/dhcp/dhcpd.conf
Append:
host node001 {
hardware ethernet <xx:xx:xx:xx:xx:xx>;
fixed-address 192.168.1.20;
}
/etc/init.d/isc-dhcp-server restart
Add DNS entries
nano /var/lib/bind/cluster90.edu.db
Append:
node001 A 192.168.1.20
nano /var/lib/bind/rev.1.168.192.in-addr.arpa
Append:
20 PTR node001.cluster90.edu.
/etc/init.d/bind9 restart
*****FIXME
*****There is a bug in the Wheezy DHCP server that prevents a client from requesting its address after being booted with PXE
*****Workaround follows...
nano /var/lib/tftpboot/pxelinux.cfg/C0A80114
Copy APPEND line and comment out (#) the first APPEND line
In the uncommented APPEND line, change ip=dhcp to ip=192.168.1.20:192.168.1.10:192.168.1.10:255.255.255.0
*****NOTE
*****This workaround requires the IP address of each node to be set in its corresponding pxelinux.cfg file!
*****Failure to update the IP address on pxelinux.cfg file WILL result in the new node stomping on the IP address of another node!
Configure Infiniband networking
apt-get install infiniband-diags
nano /etc/modules
Append these lines:
ib_umad
ib_ipoib
nano /etc/network/interfaces
Append these lines:
auto ib0
iface ib0 inet static
address 192.168.2.20
netmask 255.255.255.0
Set up apt
nano /etc/apt/sources.list
deb http://master001.cluster90.edu/debian wheezy main
deb http://master001.cluster90.edu/tde-nightly wheezy main
deb http://master001.cluster90.edu/tde-nightly-deps wheezy main
deb http://master001.cluster90.edu/libreoffice-tde wheezy main
deb http://master001.cluster90.edu/ulab wheezy main
apt-get update
Install NTP client daemon
apt-get install ntp
Configure NTP client daemon
nano /etc/ntp.conf
Remove all server lines except one, and change the remaining line to use master001.cluster90.edu
/etc/init.d/ntp restart
Set timezone
dpkg-reconfigure tzdata
Install kernel modules
apt-get install linux-image-3.2.0-4-amd64
Install base TDE system
apt-get install tdebase-trinity
/etc/init.d/tdm-trinity start
Install LDAP bonding utility
apt-get install kcontrol-ldap-bonding-trinity
Enable root logins
nano /etc/trinity/tdm/tdmrc
AllowRootLogin=true
reboot
Join terminal services node to LDAP realm
Start TDE and log in
Start TDE Menu->Settings->Control Center
Under "TDE Components", select "LDAP Realm Bonding"
Check "Enable LDAP Realm Support"
Click "Bond to New Realm" and follow the prompts
Disable graphical login manager startup
update-rc.d -f tdm-trinity remove
Install Network UPS Tools client
apt-get install nut-client
Configure Network UPS Tools server and client
cd /etc/nut
nano nut.conf
MODE=netclient
nano upsmon.conf
Append the following line:
MONITOR dell@master001.cluster90.edu 1 upsslave internalslave slave
upsc dell@master001.cluster90.edu
/etc/init.d/nut-client restart
Install full TDE system
apt-get install redmond-default-settings-ii-trinity
Install utility applications
apt-get install kerberostray-trinity
Transfer X11rdp binaries to node and install
cd /
scp root@master001.cluster90.edu:/x11rdp_binaries.tar.bz2 .
tar -xjvf x11rdp_binaries.tar.bz2
rm x11rdp_binaries.tar.bz2
ln -s /opt/X11rdp/bin/X11rdp /usr/bin/
Install PulseAudio
apt-get install pulseaudio
Configure PAM
nano /etc/pam.d/common-session
Append this line at the bottom:
session required pam_mkhomedir.so skel=/etc/skel umask=0077
session optional pam_umask.so umask=0077
nano /etc/login.defs
Change UMASK from 022 to 077
Install NFS client
apt-get install nfs-client
Configure NFS mounts
mkdir -p /applications
mkdir -p /readonlydata
nano /etc/fstab
Append these lines:
192.168.2.10:/userdata/homes /home nfs defaults,rw,hard,nointr,tcp,vers=3 0 0
192.168.2.10:/applications /applications nfs defaults,rw,hard,nointr,tcp,vers=3 0 0
192.168.2.10:/readonlydata /readonlydata nfs defaults,rw,hard,nointr,tcp,vers=3 0 0
192.168.2.10:/userdata/usertemplate /etc/skel nfs defaults,ro,hard,nointr,tcp,vers=3 0 0
Install FOSS utility applications
apt-get install dos2unix
Install FOSS uLab client
apt-get install remote-laboratory-client-trinity
Install FOSS development applications
apt-get install geda gerbv pcb-gtk qtoctave wxmaxima kpicosim-trinity tdevelop-trinity
Install FOSS office and graphics applications
apt-get install libreoffice-trinity libreoffice-style-crystal gimp kolourpaint-trinity
Install FOSS Web browser
apt-get install iceweasel
Install FOSS realm management utility for administrator use
apt-get install kcontrol-ldap-manager-trinity
ln -s /usr/sbin/kadmin /usr/bin/kadmin
Remove unwanted screensaver applications
apt-get remove xscreensaver xscreensaver-gl xscreensaver-data
Install Xilinx ISE
Download "Full Installer for Linux" from http://www.xilinx.com and extract to temporary directory
Install in /applications/xilinx (./xsetup)
Can use 'ssh -C -Y root@master001' from a graphical node for ease of setup
Remove temporary files
Set permissions
chown -R root /applications/xilinx
chgrp -R root /applications/xilinx
chmod -R o+g /applications/xilinx
chmod -R o-w /applications/xilinx
Create startup script
nano /applications/xilinx/start_ise_14.4
#!/bin/bash
# License server
# export XILINXD_LICENSE_FILE=2100@<ip_of_flexlm_license_server>
export XILINXD_LICENSE_FILE=/applications/<path_to_webpack_license_file>
cd /applications/xilinx/14.4/ISE_DS
source settings64.sh
ise
chmod 755 /applications/xilinx/start_ise_14.4
Enable and install basic 32-bit application support
dpkg --add-architecture i386
apt-get install libc6:i386 libx11-6:i386 libncurses5:i386 libxext6:i386 libxt6:i386 libxpm4:i386 libxmu6:i386 libxp6:i386
Install Windows application support (wine):
apt-get install wine:i386
==================================