Reinstalling a Juniper SRX100 with corrupt flash via CLI

 - 

srx100At work we have several old SRX100H’s lying around which are not able to boot anymore. I took one home last week to see if I could resurrect the unit and make it operational again. Based on the specs of the device it would make a very nice router/firewall. The problem is that the device will not boot and simply will not start JunOS (the OS on the device). Hooking up the console shows that the device gets a kernel panic as the internal flash disk has filesystem issues. This is a common issue on the SRX100 and mainly occurs when the power is plugged from the device while it’s in operational state. This post will explain what steps needs to be taken to revive the unit.

Specifications:
CPU: Octeon CN5020 dualcore MIPS64 @ 500MHz
RAM: 1GB
LAN: 8x 100Mbit
MGT: 1x serial (RJ45) @ 9600 baud
OTHER: Has USB port for installation and recovery. Also has full UTM functions with appropriate licensing (not default)

The CPU has offloading support for various common used ciphers to speed up encryption tasks.

Requirements:
For the recovery I will use TFTP as I already have this configured on my network. You also need access to the firmware files within the Juniper download center which means you need to be a valid Juniper partner in order to access the firmware. I found a collegue of mine which has access to this area and downloaded the firmware for me. You will also need a serial cable for managing the device, something like a RJ-45 -> DB9 (null modem cable).

Recovery via USB:
Recovery via USB is also possible. For this a USB-stick of at least 256MB is needed and formatted with the FAT32 filesystem. Just copy the firmware file to the USB-stick. You can skip the networking setup and go straight to the Starting installation part. Make sure that you set up the console connection though!

Getting the firmware:
This is the hardest part as you may only download firmware updates for Juniper devices if you have a support plan with Juniper. As I personally don’t have a contract with them I’ve found a colleague of mine who normally manages this for the company and downloaded the latest firmware updates for me. If you don’t have a contract with Juniper it’s not possible to download the firmware. You should contact a network engineer in your company or personal area to see if they can help you further. At the time of writing version “JunOS 12.1X46-D55” was the latest release with a size of approx ~150MB.

A newer release is available as well which starts with JunOS 12.3X… but that is not available for the SRX100H and SRX100B models. So here JunOS 12.1X46-D55 is the latest firmware update that is available for these models.

Connecting the switch via serial:
Now connect the switch to your serial console. 9600 baud 8N1 should do the trick to communicate with the switch. Turn the switch on after you’ve connected the console as you have to halt the 3 second autoboot process!

You should see something like below:
U-Boot 1.1.6-JNPR-2.0 (Build time: Nov 17 2010 - 07:04:52)

SRX_100_HIGHMEM board revision major:0, minor:0, serial #: AT0812AF0793
OCTEON CN5020-SCP pass 1.1, Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM: 1024 MB
Starting Memory POST...
Checking datalines... OK
Checking address lines... OK
Checking 512K memory for U-Boot... OK.
Running U-Boot CRC Test... OK.
Flash: 4 MB
USB: scanning bus for devices... 3 USB Device(s) found
scanning bus for storage devices... 1 Storage Device(s) found
Clearing DRAM........ done
BIST check passed.
Boot Media: nand-flash usb
Net: pic init done (err = 0)octeth0
POST Passed
Press SPACE to abort autoboot in 1 seconds
=>

Preparing the downloaded firmware:
If you have downloaded the firmware you should end up with a .tgz file. You don’t have to extract it and can copy it directly to servers your TFTP folder! The installer will extract it for you during installation.

Setting network parameters for installation:
Before we can start loading the firmware from the network we need to set the IP parameters in uBoot so the installer can reach your TFTP server. My TFTP server is at 192.168.0.75 and I’ve given the Juniper for this install the IP 192.168.0.210. My gateway is traditionally at 192.168.0.1:
setenv ipaddr 192.168.0.210
setenv serverip 192.168.0.75
setenv gatewayip 192.168.0.1

Save your settings:
saveenv
Saving Environment to Flash...
Un-Protected 1 sectors
Erasing Flash...
. done
Erased 1 sectors
Writing to Flash... writing to flash...
done
Protected 1 sectors

Starting the installation:
Now reset the device via uBoot:
reset

And wait until you see the following output with especially the last line:
reset

U-Boot 1.1.6-JNPR-2.0 (Build time: Nov 17 2010 - 07:04:52)


SRX_100_HIGHMEM board revision major:0, minor:0, serial #: AT0812AF0793
OCTEON CN5020-SCP pass 1.1, Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM: 1024 MB
Starting Memory POST...
Checking datalines... OK
Checking address lines... OK
Checking 512K memory for U-Boot... OK.
Running U-Boot CRC Test... OK.
Flash: 4 MB
USB: scanning bus for devices... 3 USB Device(s) found
scanning bus for storage devices... 1 Storage Device(s) found
Clearing DRAM........ done
BIST check passed.
Boot Media: nand-flash usb
Net: pic init done (err = 0)octeth0
POST Passed
Press SPACE to abort autoboot in 1 seconds
ELF file is 32 bit
Loading .text @ 0x8f000078 (244960 bytes)
Loading .rodata @ 0x8f03bd58 (13940 bytes)
Loading .rodata.str1.4 @ 0x8f03f3cc (16648 bytes)
Loading set_Xcommand_set @ 0x8f0434d4 (100 bytes)
Loading .rodata.cst4 @ 0x8f043538 (20 bytes)
Loading .data @ 0x8f044000 (5608 bytes)
Loading .data.rel.ro @ 0x8f0455e8 (120 bytes)
Loading .data.rel @ 0x8f045660 (136 bytes)
Clearing .bss @ 0x8f0456e8 (11656 bytes)
## Starting application at 0x8f000078 ...
Consoles: U-Boot console
Found compatible API, ver. 2.0


FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.0
(builder@warth.juniper.net, Wed Nov 17 07:07:32 UTC 2010)
Memory: 1024MB
[9]Booting from nand-flash slice 3
Un-Protected 1 sectors
writing to flash...
Protected 1 sectors
\
can't load '/kernel'
can't load '/kernel.old'
Press Enter to stop auto bootsequencing and to enter loader prompt.

Here press the Enter key so you will be dropped at the loader prompt. You should be now left at the following prompt:
Type '?' for a list of commands, 'help' for more detailed help.
loader>

To start the installer via TFTP run the following command:
install tftp://192.168.0.75/junos-srxsme-12.1X46-D55.3-domestic.tgz

Should you want to start the installation from the USB-stick:
install file:///junos-srxsme-12.1X46-D55.3-domestic.tgz

Starting the installer may take several minutes as the archive is being copied to the device and extracted for further installation. Note that the output of the installer will be extremely long and has 2 stages where the 1st stage is the actual installation and the 2nd stage is the first startup of the device in which it will generate it’s host keys etcetera. Overall the installation time may vary from 15 ~ 30 minutes.

The first part looks like (prepare for flood!):
octeth0: Up 1000 Mbps Full duplex (port 0)
/kernel data=0xb16d5c+0x134b2c syms=[0x4+0x8bbd0+0x4+0xcadc3]
Kernel entry at 0x801000e0 ...
init regular console
Primary ICache: Sets 64 Size 128 Asso 4
Primary DCache: Sets 1 Size 128 Asso 64
Secondary DCache: Sets 128 Size 128 Asso 8
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Copyright (c) 1996-2016, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
JUNOS 12.1X46-D55.3 #0: 2016-07-08 18:46:54 UTC
builder@quoarth.juniper.net:/volume/build/junos/12.1/service/12.1X46-D55.3/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
JUNOS 12.1X46-D55.3 #0: 2016-07-08 18:46:54 UTC
builder@quoarth.juniper.net:/volume/build/junos/12.1/service/12.1X46-D55.3/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory = 1073741824 (1024MB)
avail memory = 559992832 (534MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
netisr_init: !debug_mpsafenet, forcing maxthreads from 2 to 1
cpu0 on motherboard
: CAVIUM's OCTEON 5020 CPU Rev. 0.1 with no FPU implemented
L1 Cache: I size 32kb(128 line), D size 8kb(128 line), sixty four way.
L2 Cache: Size 128kb, 8 way
obio0 on motherboard
uart0: on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
dwc0: on obio0
usb0: on dwc0
usb0: USB revision 2.0
uhub0: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1: vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 2
uhub1: single transaction translator
uhub1: 2 ports with 1 removable, self powered
umass0: STMicroelectronics ST72682 High Speed Mode, rev 2.00/2.10, addr 3
cpld0 on obio0
pcib0: on obio0
Disabling Octeon big bar support
PCI Status: PCI 32-bit: 0xc041b
pcib0: Initialized controller
pci0: on pcib0
pci0: at device 2.0 (no driver attached)
pci0: at device 2.1 (no driver attached)
pci0: at device 2.2 (no driver attached)
gblmem0 on obio0
octpkt0: on obio0
cfi0: on obio0
octpkt_attach: Initializing octpkt0 interface
Timecounter "mips" frequency 500000000 Hz quality 0
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
md0: Preloaded image 10022912 bytes at 0x80ea2224
da0 at umass-sim0 bus 0 target 0 lun 0
da0: Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C)
Trying to mount root from cd9660:/dev/md0
WARNING: preposterous time in file system
WARNING: clock 10662 days greater than file system time
tty: not found
Starting JUNOS installation:
Source Package: net0:/junos-srxsme-12.1X46-D55.3-domestic.tgz
Target Media : internal
Product : srx100h
add default: gateway 192.168.0.1
PING 192.168.0.75 (192.168.0.75): 56 data bytes
64 bytes from 192.168.0.75: icmp_seq=0 ttl=64 time=1.493 ms


--- 192.168.0.75 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.493/1.493/1.493/0.000 ms
Computing slice and partition sizes for /dev/da0 ...
Media check on da0
Attempting to save existing configuration...
** /dev/da0s3e
Cannot find file system superblock


LOOK FOR ALTERNATE SUPERBLOCKS? yes


32 is not a file system superblock
USING ALTERNATE SUPERBLOCK AT 12768
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
UNKNOWN FILE TYPE I=2
CLEAR? yes

Here you can see that the internal filesystem is corrupt. In my case this CLEAR message happens around 500 times which has to be displayed over a 9600 baud connection :). Once this is done it will resume with:

** Phase 2 - Check Pathnames
ROOT INODE UNALLOCATED
ALLOCATE? yes


CG 0: BAD MAGIC NUMBER
CG 0: BAD MAGIC NUMBER
** Phase 3 - Check Connectivity
UNREF DIR I=1792 OWNER=0 MODE=40755
SIZE=512 MTIME=Nov 2 13:43 2012
RECONNECT? yes


NO lost+found DIRECTORY
CREATE? yes


CG 0: BAD MAGIC NUMBER
CG 0: BAD MAGIC NUMBER
DIR I=1792 CONNECTED. PARENT WAS I=2


** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
CG 0: BAD MAGIC NUMBER
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes


SUMMARY INFORMATION BAD
SALVAGE? yes


BLK(S) MISSING IN BIT MAPS
SALVAGE? yes


8 files, 8 used, 12430 free (14 frags, 1552 blocks, 0.1% fragmentation)


UPDATE CORRUPTED DUPLICATE SUPERBLOCKS? yes


UPDATE STANDARD SUPERBLOCK? yes


***** FILE SYSTEM WAS MODIFIED *****
Could not find any existing configuration.
Formatting target media /dev/da0 ...
Preparing to create slices on /dev/da0
/dev/da0: 2048000 sectors [C:1000 H:64 S:32 SS:512]
Shrinking slice 1 by 256 blocks for alignment
1+0 records in
1+0 records out
512 bytes transferred in 0.000671 secs (763143 bytes/sec)
Creating slices:
g c1000 h64 s32
p 1 0xA5 256 610048
p 2 0xA5 610304 610304
p 3 0xA5 1220608 763904
p 4 0xA5 1984512 63488
a 1
******* Working on device /dev/da0 *******
Computing layout of partitions in /dev/da0s1...
Shrinking partition a by 1792 blocks for alignment
Labeling /dev/da0s1:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 608000 256 unused 0 0
c: 610048 0 unused 0 0 # "raw" part, don't edit
/dev/da0s1a: 296.9MB (607996 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 74.22MB, 4750 blks, 9600 inodes.
super-block backups (for fsck -b #) at:
32, 152032, 304032, 456032
Computing layout of partitions in /dev/da0s2...
Labeling /dev/da0s2:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s2:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 610048 256 unused 0 0
c: 610304 0 unused 0 0 # "raw" part, don't edit
/dev/da0s2a: 297.9MB (610044 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 74.47MB, 4766 blks, 9600 inodes.
super-block backups (for fsck -b #) at:
32, 152544, 305056, 457568
Computing layout of partitions in /dev/da0s3...
Shrinking partition e by 256 blocks for alignment
Labeling /dev/da0s3:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s3:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 763904 0 unused 0 0 # "raw" part, don't edit
e: 50944 256 unused 0 0
f: 712704 51200 unused 0 0
/dev/da0s3e: 24.9MB (50940 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 6.22MB, 398 blks, 896 inodes.
super-block backups (for fsck -b #) at:
32, 12768, 25504, 38240
/dev/da0s3f: 348.0MB (712700 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 87.00MB, 5568 blks, 11136 inodes.
super-block backups (for fsck -b #) at:
32, 178208, 356384, 534560
Computing layout of partitions in /dev/da0s4...
Shrinking partition e by 256 blocks for alignment
Labeling /dev/da0s4:
bsdlabel: write to disk label supressed - label was as follows:
# /dev/da0s4:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 57344 6144 unused 0 0
c: 63488 0 unused 0 0 # "raw" part, don't edit
e: 5888 256 unused 0 0
/dev/da0s4a: 28.0MB (57340 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 7.00MB, 448 blks, 896 inodes.
super-block backups (for fsck -b #) at:
32, 14368, 28704, 43040
/dev/da0s4e: 2.9MB (5884 sectors) block size 16384, fragment size 2048
using 3 cylinder groups of 1.00MB, 64 blks, 128 inodes.
super-block backups (for fsck -b #) at:
32, 2080, 4128
Creating root filesystem layout ...
Downloading /junos-srxsme-12.1X46-D55.3-domestic.tgz from 192.168.0.75 ...
Time and ticks drifted too much, resetting synchronization...
Verified SHA1 checksum of /a/cf/install/junos-boot-srxsme-12.1X46-D55.3.tgz
Verified SHA1 checksum of /a/cf/install/junos-srxsme-12.1X46-D55.3-domestic
Creating var filesystem layout ...
Creating bsdlabel recovery information ...
Initializing alternate root ...
machdep.bootsuccess: 0 -> 1
machdep.nextbootdev: usb -> nand-flash
JUNOS requires BIOS version upgrade from 2.0 to 2.8
Upgrading to BIOS 2.8 ...
boot.upgrade.uboot="0xbfc00000"
boot.upgrade.loader="0xbfe00000"
Upgrading Loader...
#####################################
Verifying the loader image... OK
Upgrading U-Boot...
###############################################################################
Verifying the new U-Boot image... OK
WARNING: The new boot firmware will take effect when the system is rebooted.
Installation completed successfully, rebooting ...
Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 done


syncing disks... All buffers synced.
Uptime: 9m9s
Rebooting...
cpu_reset: Stopping other CPUs

This was the first phase. The device will now reboot and start booting the fresh installed image and prepare itself for use:
U-Boot 1.1.6-JNPR-2.8 (Build time: Feb 10 2015 - 01:03:41)

Initializing memory this may take some time...
Measured DDR clock 266.62 MHz
SRX_100_HIGHMEM board revision major:0, minor:0, serial #: AT0812AF0793
OCTEON CN5020-SCP pass 1.1, Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM: 1024 MB
Starting Memory POST...
Checking datalines... OK
Checking address lines... OK
Checking 512K memory for U-Boot... OK.
Running U-Boot CRC Test... OK.
Flash: 4 MB
USB: scanning bus for devices... 3 USB Device(s) found
scanning bus for storage devices... 1 Storage Device(s) found
Clearing DRAM........ done
BIST check passed.
Boot Media: nand-flash usb
Net: pic init done (err = 0)octeth0
POST Passed
Press SPACE to abort autoboot in 1 seconds
ELF file is 32 bit
Loading .text @ 0x8f0000a0 (246560 bytes)
Loading .rodata @ 0x8f03c3c0 (14144 bytes)
Loading .reginfo @ 0x8f03fb00 (24 bytes)
Loading .rodata.str1.4 @ 0x8f03fb18 (16516 bytes)
Loading set_Xcommand_set @ 0x8f043b9c (96 bytes)
Loading .rodata.cst4 @ 0x8f043bfc (20 bytes)
Loading .data @ 0x8f044000 (5760 bytes)
Loading .data.rel.ro @ 0x8f045680 (120 bytes)
Loading .data.rel @ 0x8f0456f8 (136 bytes)
Clearing .bss @ 0x8f045780 (11600 bytes)
## Starting application at 0x8f0000a0 ...
Consoles: U-Boot console
Found compatible API, ver. 2.8


FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.8
(slt-builder@svl-ssd-build-vm06.juniper.net, Tue Feb 10 00:32:30 PST 2015)
Memory: 1024MB
[0]Booting from nand-flash slice 1
Un-Protected 1 sectors
writing to flash...
Protected 1 sectors
Loading /boot/defaults/loader.conf
/kernel data=0xb16d5c+0x134b2c syms=[0x4+0x8bbd0+0x4+0xcadc3]


Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]...
Kernel entry at 0x801000e0 ...
init regular console
Primary ICache: Sets 64 Size 128 Asso 4
Primary DCache: Sets 1 Size 128 Asso 64
Secondary DCache: Sets 128 Size 128 Asso 8
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Copyright (c) 1996-2016, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
JUNOS 12.1X46-D55.3 #0: 2016-07-08 18:46:54 UTC
builder@quoarth.juniper.net:/volume/build/junos/12.1/service/12.1X46-D55.3/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
JUNOS 12.1X46-D55.3 #0: 2016-07-08 18:46:54 UTC
builder@quoarth.juniper.net:/volume/build/junos/12.1/service/12.1X46-D55.3/obj-octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory = 1073741824 (1024MB)
avail memory = 509661184 (486MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
netisr_init: !debug_mpsafenet, forcing maxthreads from 2 to 1
cpu0 on motherboard
: CAVIUM's OCTEON 5020 CPU Rev. 0.1 with no FPU implemented
L1 Cache: I size 32kb(128 line), D size 8kb(128 line), sixty four way.
L2 Cache: Size 128kb, 8 way
obio0 on motherboard
uart0: on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
dwc0: on obio0
usb0: on dwc0
usb0: USB revision 2.0
uhub0: vendor 0x0000 DWC OTG root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1: vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 2
uhub1: single transaction translator
uhub1: 2 ports with 1 removable, self powered
umass0: STMicroelectronics ST72682 High Speed Mode, rev 2.00/2.10, addr 3
cpld0 on obio0
pcib0: on obio0
Disabling Octeon big bar support
PCI Status: PCI 32-bit: 0xc041b
pcib0: Initialized controller
pci0: on pcib0
pci0: at device 2.0 (no driver attached)
pci0: at device 2.1 (no driver attached)
pci0: at device 2.2 (no driver attached)
gblmem0 on obio0
octpkt0: on obio0
cfi0: on obio0
Timecounter "mips" frequency 500000000 Hz quality 0
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
da0 at umass-sim0 bus 0 target 0 lun 0
da0: Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C)
Trying to mount root from ufs:/dev/da0s1a
MFSINIT: Initialising MFSROOT
Process-1 beginning MFSROOT initialization...
Creating MFSROOT...
/dev/md0: 20.0MB (40956 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 5.00MB, 320 blks, 640 inodes.
super-block backups (for fsck -b #) at:
32, 10272, 20512, 30752
Populating MFSROOT...
Creating symlinks...
Setting up mounts...
Continuing boot from MFSROOT...
Attaching /cf/packages/junos via /dev/mdctl...
Mounted junos package on /dev/md1...
S
chflags: /var/packages/*: No such file or directory
Media check on da0
Automatic reboot in progress...
** /dev/da0s1a (NO WRITE)
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no


SUMMARY INFORMATION BAD
SALVAGE? no


BLK(S) MISSING IN BIT MAPS
SALVAGE? no


161 files, 75562 used, 73964 free (44 frags, 9240 blocks, 0.0% fragmentation)
mount reload of '/' failed: Operation not supported


Verified junos signed by PackageProductionEc_2016 method ECDSA
Verified jboot signed by PackageProductionEc_2016 method ECDSA
Verified junos-12.1X46-D55.3-domestic signed by PackageProductionEc_2016 method ECDSA
Checking integrity of BSD labels:
s1: Passed
s2: Passed
s3: Passed
s4: Passed
** /dev/bo0s3e
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 12436 free (28 frags, 1551 blocks, 0.2% fragmentation)
** /dev/bo0s3f
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 175282 free (34 frags, 21906 blocks, 0.0% fragmentation)
Checking integrity of licenses:
Checking integrity of configuration:
rescue.conf.gz: No recovery data
Loading configuration ...
Time and ticks drifted too much, resetting synchronization...
mgd: error: Cannot open configuration file: /config/juniper.conf
mgd: warning: activating factory configuration
Interface control process: [edit interfaces fe-0/0/1 unit 0]
Interface control process: 'family'
Interface control process: Ethernet-switching family not supported in HA mode for srx100h platform
mgd: error: configuration check-out failed
Warning: Commit failed, activating partial configuration.
Warning: Edit the router configuration to fix these errors.
Setting initial options: .
Starting optional daemons: usbd.
Doing initial network setup:
.
Initial interface configuration:
additional daemons: eventd.
Generating RSA key /etc/ssh/ssh_host_key
Generating public/private rsa1 key pair.
Your identification has been saved in /etc/ssh/ssh_host_key.
Your public key has been saved in /etc/ssh/ssh_host_key.pub.
The key fingerprint is:
e3:ba:b3:06:65:26:4f:5b:a9:92:02:c5:5c:99:46:e9 root@
The key's randomart image is:
+--[RSA1 2048]----+
| o oo+ |
| + = |
| . o . |
|. E = o |
| . O +S |
| . + +. . |
| . o . |
| o. |
| .++ |
+-----------------+
Generating DSA key /etc/ssh/ssh_host_dsa_key
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
47:85:2c:5b:fa:a1:b2:bd:ba:fe:ef:51:2d:7b:5d:22 root@
The key's randomart image is:
+--[ DSA 1024]----+
| . .. |
| . +. |
| =. |
| o.. . |
| So..E o .|
| . .... + o.|
| + . . . .|
| . . . . |
| .++o+o |
+-----------------+
Generating RSA2 key /etc/ssh/ssh_host_rsa_key
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
40:7a:16:30:b7:a0:ed:c0:2c:6c:2d:be:5e:00:f0:28 root@
The key's randomart image is:
+--[ RSA 2048]----+
|. +.+ |
|o=.o * o |
|E+*.o = |
|=..o o . |
| o . S |
| o |
| . . |
|. . |
| . |
+-----------------+
Generating ECDSA key /etc/ssh/ssh_host_ecdsa_key
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
The key fingerprint is:
59:8f:da:93:f7:8b:0c:0f:2c:1c:b7:fb:7d:ba:91:d4 root@
The key's randomart image is:
+--[ECDSA 256]---+
| |
| |
| . |
| o o . |
| S o . . E|
| . * o . . |
| + O . o |
| . O + ..|
| ..= *= |
+-----------------+
Generating ED25519 key /etc/ssh/ssh_host_ed25519_key
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
The key fingerprint is:
75:52:bb:29:4d:06:6d:b9:e4:ce:6f:18:e2:d8:ce:7c root@
The key's randomart image is:
+--[ED25519 256--+
| .... |
| o=. |
| o+=. |
| . *oo |
| S .o+ |
| ..+ |
| + . + |
| .oo E o |
| .+. . |
+-----------------+
Ignoring watchdog timeout during boot/reboot
Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boot/modules;/modules/ifpfe_drv;/modules;
kld netpfe drv: ifpfed_dialer ipsec kld.
Doing additional network setup:.
Starting final network daemons:.
setting ldconfig path: /usr/lib /opt/lib
starting standard daemons: cron.
Initial rc.mips initialization:.
Local package initialization:.
starting local daemons:set cores for group access
.
kern.securelevel: -1 -> 1
Creating JAIL MFS partition...
JAIL MFS partition created
boot.upgrade.uboot="0xBFC00000"
boot.upgrade.loader="0xBFE00000"
JUNOS requires backup BIOS version upgrade from 2.0 to 2.8
Upgrading to BIOS 2.8 ...
Upgrading Secondary U-Boot...
###############################################################################
Verifying the new U-Boot image... OK
Boot media /dev/da0 has dual root support
** /dev/da0s2a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 74476 free (28 frags, 9306 blocks, 0.0% fragmentation)
Fri Sep 9 18:12:31 UTC 2016


Amnesiac (ttyu0)


login:

Continue configuration:
The default account is “root” with no password. If you have no experience with the setup of such device purely via CLI I advise you to hook up a LAN cable between your machine and the first ethernet port of the device (that is port 0/0) and set up a static IP in the 192.168.1.0/24 range. You should then be able to browse to the URL:
http://192.168.1.1/

The welcome screen should be like below:
j-web

You can login with “root” and will be welcomed by a setup wizard which will guide you through the rest of the setup based on several simple questions.


Updating the firmware on a Juniper SSG5 via CLI

 - 

SSG5_Recently I got an old Juniper SSG5 router which was taken out of service. The unit itself dates back from 2010 and never received updates while it was operational. Before taking it into production again I want to make sure that the firmware is up-to-date.

Prerequisites:
Before we can start updating the unit we need to download the firmware. Since the device is EOL we can download the firmware free of charge (but requires you to register once). You can find the download page here.

Although not required, a TFTP server is needed to save the initial configuration and load the new firmware from. In this guide I assume that a TFTP server is available in the network. In this guide my TFTP server listens on IP-address 192.168.0.75.

If you are going to upgrade from a firmware earlier than the 6.3 release you need to download a new key file as well which holds the MD5 checksums for the new images.

Factory resetting the switch:
As this unit came out of production it holds a certain configuration and password I don’t know. So in order to use the device we need to reset it to the factory defaults first. With a paperclip you can reset the device (reset button is on the back) as follows:

1: While the device is running press and hold the reset button for ~6 seconds and release
2: Wait ~2 seconds
3: Press the reset button again for around ~6 seconds and release

The unit should now reboot and restore the factory image into flash. This may take up to 5 minutes to finish.

If you already have this unit in production do not perform these steps as it will wipe your entire configuration!

Logging in on the switch:
The switch has a serial console port which is the most convenient way to administer it. It is also possible to use telnet and/or SSH to log in but it has to be enabled manually per port on the device and is by default disabled.

The console settings are 9600baud 8N1 with no flow control.

The default username and password are both “netscreen“. When logged in successfully you will see the following prompt:
login: netscreen
password:
ssg5-serial->

Checking the current software version:
When logged in to the switch perform the following command to see the current version:

ssg5-serial-> get system
Product Name: SSG5-Serial
Serial Number: 0162122010003603, Control Number: 00000000
Hardware Version: 0710(0)-(00), FPGA checksum: 00000000, VLAN1 IP (0.0.0.0)
Flash Type: Samsung
Software Version: 6.2.0r5.0, Type: Firewall+VPN
Feature: AV-K
Compiled by build_master at: Thu Jan 28 11:42:26 PST 2010
Base Mac: 28c0.dae8.e140
File Name: ssg5ssg20.6.2.0r5.0, Checksum: 23364b4e
, Total Memory: 256MB

There is more output to be shown, you can press CONTROL + C to cancel the listing.

Saving the configuration:
This step is only needed when you already have the device in production. If you have just factory reset the device this step can be skipped.

When logged in to the console enter the following to save the configuration:
save config to tftp 192.168.0.75 ssg5_31-8-2016.cfg

This will save the current configuration with filename “ssg5_31-8-2016.cfg” to the TFTP server running on 192.168.0.75. If this goes well you should see:
Read the current config.
Save configurations (4935 bytes) to ssg5_31-8-2016.cfg on TFTP server 192.168.0.75.
!!!!!!!!!!!!!!!!!!!!!!
tftp transferred records = 10
tftp success!


TFTP Succeeded
ssg5-serial->

Preparing the firware:
The latest firmware for the SSG5 as of writing is version 6.3.0r22. You should have a file called “ssg5ssg20.6.3.0r22.0.zip”. Unzip the contents of the zip into your TFTP directory (usually /srv/tftp/).

Optionally you may have a bootloader update as well as I had to update as well. You will also have a file called “Loadssg5ssg20v132.d.zip” which needs to be unzipped in the TFTP directory as well.

Updating the firmware:
If you want to update the firmware and are coming from a release before 6.3 you need to update the image MD5 keys first. Make sure you have extracted the zip file containing the keys in your TFTP directory. After unzipping you will have a “imagekey.cer” file that we need to flash first.

You may want to check the keys installed first. you can do this with:
ssg5-serial-> exec pki test skey
exec pki test .
Flash base = 0x51000000, Flash end = 0x0, sector size= 0x4000


KEY1 N/A len =432
308201ac02010002818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f800


KEY2 N/A len =432
308201ac02010002818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f800


KEY3 N/A len =432
308201ac02010002818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f800

You see that the string starts with 308201ac. It’s eigth character is a c which indicates that you have a firmware prior to 6.3 running and you have to update the keys first. Firmware releases in the 6.3 branch have the c replaced with a d.

Issue the following command to load the new keys:
ssg5-serial-> save image-key tftp 192.168.0.75 imagekey.cer
Load file from TFTP 192.168.0.75 (file: imagekey.cer).
!!!!!
tftp received octets = 863
tftp success!
Done


TFTP Succeeded
ssg5-serial->

Now we can check the keys again and you should notice that the “c” is changed to a “d”:
ssg5-serial-> exec pki test skey
exec pki test .
Flash base = 0x51000000, Flash end = 0x0, sector size= 0x4000


KEY1 N/A len =433
308201ad02010002818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f800


KEY2 N/A len =433
308201ad02010002818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f800


KEY3 N/A len =433
308201ad02010002818100fd7f53811d75122952df4a9c2eece4e7f611b7523cef4400c31e3f800

Now that the keys are up-to-date we can start the actual firmware update:
ssg5-serial-> save software from tftp 192.168.0.75 ssg5ssg20.6.3.0r22.0 to flash
Load software from TFTP 192.168.0.75 (file: ssg5ssg20.6.3.0r22.0).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
tftp received octets = 13381258
tftp success!


TFTP Succeeded
Save to flash. It may take a few minutes ...platform = 25, cpu = 12, version = 18
update new flash image (02a676a0,13381258)
platform = 25, cpu = 12, version = 18
offset = 20, address = 5800000, size = 13381180
date = 2669, sw_version = 31808000, cksum = 59d01749
Image authenticated!
Program flash (13381258 bytes) ...
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++e
Done

This step takes up to 5 minutes to download and install the firmware.

Now that the new firmware is installed we need to reboot the device to load the new firmware. This can simply be done with the reset command:
ssg5-serial-> reset
System reset, are you sure? y/[n] y
In reset ...

After around 5 minutes the device should be back online again and we want to check if the new version is active. Log in and retrieve the system information:
ssg5-serial-> get system
Product Name: SSG5-Serial
Serial Number: 0162122010003603, Control Number: 00000000
Hardware Version: 0710(0)-(00), FPGA checksum: 00000000, VLAN1 IP (0.0.0.0)
Flash Type: Samsung
Software Version: 6.3.0r22.0, Type: Firewall+VPN
Feature: AV-K
BOOT Loader Version: 1.3.2
Compiled by build_master at: Wed Mar 9 07:57:20 PST 2016
Base Mac: 28c0.dae8.e140
File Name: ssg5ssg20.6.3.0r22.0, Checksum: 11a822d0
, Total Memory: 256MB

In this case the upgrade went with success!

Updating the bootloader firmware:
Updating the bootloader cannot be performed when the OS is running and can only be performed from the bootloader itself. Reset the device and halt the boot process to gain access to the bootloader prompt:

ssg5-serial-> reset
System reset, are you sure? y/[n] y
In reset ...


Juniper Networks SSG5 Boot Loader Version 1.3.2 (Checksum: A1EAB858)
Copyright (c) 1997-2006 Juniper Networks, Inc.


Total physical memory: 256MB
Test - Pass
Initialization - Done


Hit any key to run loader
Hit any key to run loader


Serial Number [0162122010003603]: READ ONLY
HW Version Number [0710]: READ ONLY
Self MAC Address [28c0-dae8-e140]: READ ONLY
Boot File Name [ssg5ssg20.6.3.0r22.0]: Loadssg5ssg20v132.d
Self IP Address [192.168.2.26]: 192.168.0.7
TFTP IP Address [192.168.2.100]: 192.168.0.75


Save loader config (56 bytes)... Done


Loading file "Loadssg5ssg20v132.d"...
rtatatatatatatatatatatatatatatatatatatatatatatatatatat


Loaded Successfully! (size = 407,771 bytes)


Image authenticated!


Save to on-board flash disk? (y/[n]/m)

Enter “y” at the above question to flash the bootloader firmware.

Save to on-board flash disk? (y/[n]/m) Yes!

Saving system image to on-board flash disk...
Done! (size = 407,771 bytes)


Run downloaded system image? ([y]/n)

Enter “y” at the above question to run the bootloader update:

Run downloaded system image? ([y]/n) Yes!
Check on-board Boot Loader... Update needed!

Are you sure you want to update Boot Loader? (y/n)

Enter “y” to finalize the bootloader update:

Read product information of on-board boot flash device:
Manufacturer ID = 1f
Device ID = 13
Additional Device ID = 10


Boot flash device is AT49LV040B


Erase on-board boot flash device.......... Done



Verify Boot Loader... Done


Boot Loader has been updated successfully!


Please hit any key to reboot the system...

Press any key to reboot the system. The bootloader update has finished! After this the bootloader tries to update at every reboot which is very inconvenient. You can solve this by removing the bootloader update file from the TFTP folder.


Testing your internet connection speed from the commandline

 - 

speedometer
Sometimes you may want to check the internet connection of the provider you are connected to. Although there are various versions that are browser related I rather want to have a method like this commandline based.

There is always the method of downloading a large file off the internet to see how fast it performs, but it’s not very convenient. Upon some searching I came across the tool “speedtest-cli” which is able to perform such speedtests directly from the commandline.

Requirements:
Speedtest-cli is a python program and can be installed using pip. If you don’t have pip installed you can install it with the commands below:

Debian/Ubuntu:
apt-get install python-pip

CentOS/RedHat:
yum install python-pip

ArchLinux:
pacman -Sy python-pip

Installing speedtest-cli:
When pip is installed, run the following to install it:
pip install speedtest-cli

The output will be like:
Collecting speedtest-cli
Using cached speedtest_cli-0.3.4-py2.py3-none-any.whl
Installing collected packages: speedtest-cli
Successfully installed speedtest-cli-0.3.4

Using speedtest-cli
Now that it is installed, you can simply start it by running:
speedtest-cli

It will automatically determine the nearest mirror and start the test. Output will be like it is below (IP masked):
[fileserver ~]# speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Ziggo (1.2.3.4)...
Selecting best server based on latency...
b'Hosted by INTERACTIVE 3D B.V. (Rotterdam) [14.50 km]: 16.043 ms'
Testing download speed........................................
Download: 236.53 Mbit/s
Testing upload speed..................................................
Upload: 27.37 Mbit/s

I have a 300/30Mbit internet connection at home and had some downloads ongoing at that time, so the results are what I’ve expected from it.


Install Observium on ArchLinux

 - 

Hamster-transparentRecently I’ve obtained an old 8-port switch from the office (a WWP LightningEdge 310). That switch has a management port for managing the switch via cli but also is capable of sending SNMP data so we can monitor the status of the switch in tools like Cacti or Observium. I’ve tried Cacti before and although it’s very mature and has a lot of functions it’s also a big puzzle to find out how it works with all of it’s templates and graphs. I’ve also tried Observium and this does the job perfectly. Since the installation of Observium is well documented for RedHat and Debian like systems and I only have ArchLinux based systems I’ve tried installing it on Arch, this post will therefore describe the process of installing and running Observium on a ArchLinux install.

Why Observium?
Observium is able to poll SNMP data like Cacti but it’s much more convenient in what it is able to show and graph. By default Cacti has some basic templates for traffic monitoring but it’s not very thorough. Observium is able to graph all SNMP data that is available for processing and does the work almost completely for you. In my case with the LE-310 switch it is able to show all interface statistics, optic status, system status etcetera as where Cacti was able to only monitor the first.

System requirements:
Observium has pretty high system requirements. Since I’m only monitoring 1 switch in this install I can imagine that the actual requirements would be much lower. My testing case was done on a very old HP machine (s7720.nl) with the following specs:

CPU: AMD X2 3800+ 2GHz dualcore
RAM: 4GB DDR2 800 ECC
LAN: Realtek 8101CP 100Mbit
HDD: 1TB Seagate Constellation 7200RPM

As said, the machine is very old (almost 9 years). As it’s only used for testing purposes I found the machine handling Observium pretty well. The web pages are generated pretty fast with PHP7 and no Opcode caching. The only thing that was notably slower was the generation of the graphs which in this case is the CPU intensive part. It could take up to several seconds to load ~15 – ~20 graphs on a page, still okay for me. Should you want more speed you might want to consider a more modern machine with a newer CPU for processing the graph data.

Installing required software:
Before we can install Observium we need to install some packages that consist of a LAMP stack, some PHP modules and software that is needed so Observium can do it’s work properly. From the commandline/terminal elevate to root and install the required software:

pacman -Sy apache mariadb php php-apache php-mcrypt php-gd php-snmp net-snmp fping rrdtool whois mtr ipmitool graphviz imagemagick python-pip nmap

For the cron scripts to work we need to have pymysql installed as well:
pip3 install pymysql

Then go back to your normal user account and install the php-pear package, I’ve done this like (as root):
su jeffrey
yaourt- Sy php-pear
exit

Configuring the installed software:
By default the Apache and MariaDB services are not started and have to be “enabled” so they start at boot and have to be “started” in order for them to work.

Enable the apache and MariaDB services:
systemctl enable httpd.service
systemctl enable mariadb.service

If you have just installed MariaDB and haven’t set it up before it’s best to do now to finish and secure your installation. To finish the installation run:
mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql

This will take a couple of seconds to finish and results in a working MariaDB server setup but with no password for the root account, start MariaDB and set it via:
systemctl start mariadb.service
mysql_secure_installation

The script makes most of the default choices for you, it’s best to keep them as suggested!

We haven’t started Apache yet as we have to change the default config as well and saves us a Apache restart. Open the main configuration file:
vim /etc/httpd/conf/httpd.conf

Locate the following line:
#LoadModule rewrite_module modules/mod_rewrite.so

And replace it with:
LoadModule rewrite_module modules/mod_rewrite.so

Next locate the following line:
LoadModule mpm_event_module modules/mod_mpm_event.so

Replace it with:
#LoadModule mpm_event_module modules/mod_mpm_event.so

And change the line underneath it from:
#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

To:
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

In that same LoadModule list add to the end of the list the following line so that PHP scripts work as well:
LoadModule php7_module modules/libphp7.so

Search in the file for Include and insert the following line underneath the last Include:
Include conf/extra/php7_module.conf

Later in the file there is a DocumentRoot section, within the Directory directive, replace everything with:
DirectoryIndex index.php
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted

Change the DocumentRoot and Directory itself as well from:
DocumentRoot "/srv/http"

To:
DocumentRoot "/srv/http/html"

Save the changes and exit the file.

Now we have to change some PHP settings as well, open the following file:
vim /etc/php/php.ini

Search for the following line:
extension_dir = "/usr/lib/php/modules/"

And add the following lines beneath it:
extension=snmp.so
extension=sockets.so
extension=mysqli.so
extension=pdo_mysql.so
extension=gd.so
extension=mcrypt.so

Save the changes and exit the file. Now let’s start Apache as well:
systemctl start httpd.service

Downloading Observium:
By default the docroot of Apache on ArchLinux is in /srv/http/. One thing that you must know is that Observium must run from within the docroot itself and will not run using a alias or subdirectory. This is because it’s programmed to always run from the docroot (and that is bad).

So first we navigate to the docroot directory:
cd /srv/http/

Download the latest Observium archive:
wget http://www.observium.org/observium-community-latest.tar.gz

And unpack it straight into the current docroot directory:
tar --strip-components=1 -vxzf observium-community-latest.tar.gz

Remove the downloaded installation archive:
rm -f observium-community-latest.tar.gz

Configuring Observium:
In order to use Observium we need to create a configfile with the database credentials and set up a database. First we will setup the database, so log in to MySQL as the root user with the password you’ve set earlier:
mysql -u root -p

Create a database, new database user and password:
mysql> CREATE DATABASE observium DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
mysql> GRANT ALL PRIVILEGES ON observium.* TO 'observium'@'localhost'
-> IDENTIFIED BY '';

Exit MySQL and setup the config file:
cp config.php.default config.php
vim config.php

You have to change the following lines to what you have setup above with the MySQL database setup:
// Database config --- This MUST be configured
$config['db_extension'] = 'mysqli';
$config['db_host'] = 'localhost';
$config['db_user'] = 'USERNAME';
$config['db_pass'] = 'PASSWORD';
$config['db_name'] = 'observium';

You also need to change the Observium root directory from:
#$config['install_dir'] = "/opt/observium";

To:
$config['install_dir'] = "/srv/http";

And save your changes. Now we are going to install the database schema, run the following:
./discovery.php -u

This will import the basic database schema into the Observium database. It may take up to a couple of minutes depending on the speed of the system, mostly disk related. The output is as shown below:
Install initial database schema ... done.
-- Updating database/file schema
252 -> 253 ... (db) done.
253 -> 254 ... (db) done.
254 -> 255 ... (db) done.
255 -> 256 ... (php)
256 -> 257 ... (php)
257 -> 258 ... (php)
258 -> 259 ... (db) done.
259 -> 260 ... (php)
260 -> 261 ... (db) done.
261 -> 262 ... (php)
262 -> 263 ... (db) done.
263 -> 264 ... (db) done.
264 -> 265 ... (db) done.
265 -> 266 ... (db) done.
-- Done.

We also need to create logging and RRD data directories:
mkdir /srv/http/rrd
mkdir /srv/http/logs
chmod 777 /srv/http/{rrd,logs}

The last step is to fix a issue with PEAR. Observium uses PEAR for several additional PHP modules, but looks for the PEAR.php file in it’s own pear directory which isn’t there. If you have the php-pear package installed before from yaourt you can solve this issue by copying the PEAR.php file from the pear directory over to the pear directory from Observium:
cp /usr/share/pear/PEAR.php /srv/http/libs/pear/

If you don’t copy the PEAR.php file you will not be able to add devices to your Observium install!

Creating a new user for Observium:
Before we can log in to the web interface of Observium we need to create a user for this purpose:
cd /srv/http/

Run the following command to create a user called “jeffrey” with a password “password” and admin rights “10”:
./adduser.php jeffrey password 10

Which should result in:
Observium CE 0.16.1.7533
Add User

User jeffrey added successfully.

Logging in to the web interface of Observium:
In your browser you can navigate to the IP-address or hostname, this should result in the login screen of Observium like shown below:
Observium

When logged in you should be able to add hosts and devices. Do note that they cannot be added by IP-address and always have to be entered as hostnames. Should this be an issue you can use the local /etc/hosts file on your Observium machine of fix the hostnames your DHCP server hands out.

Automatic polling for Observium:
It’s possible to poll the data for Obserium automatically. For this we will use the cron mechanism from the system. As root, edit the current cron:
crontab -e

And add the following 3 lines:
33 */6 * * * /srv/http/discovery.php -h all > /dev/null 2>&1
*/5 * * * * /srv/http/discovery.php -h new > /dev/null 2>&1
*/5 * * * * /srv/http/poller-wrapper.py 2 > /dev/null 2>&1

Save the cron and from now on every 5 minutes the new SNMP data will be polled for the devices you have configured!

In Server

Fix ProFTPd [unable to lstat AuthUserFile] error on DirectAdmin

 - 

directadmin
Recently I’ve updated the ProFTPd software on one of the servers at work and ran into the following issue after custombuild finished the update:
SNIP:
Restarting ProFTPd.
Shutting down proftpd: [ OK ]
Starting proftpd: 2016-06-10 10:12:35,133 server.example.net proftpd[3367]: mod_auth_file/1.0: unable to lstat AuthUserFile '/usr/local/directadmin/data/users/morpheus/ftp.passwd': No such file or directory
2016-06-10 10:12:35,134 server.example.net proftpd[3367]: fatal: AuthUserFile: unable to use /usr/local/directadmin/data/users/morpheus/ftp.passwd: No such file or directory on line 4 of '/etc/proftpd.vhosts.conf'
[FAILED]

The issue here is that the ProFTPd server no longer accepts the vhost config file but still parses it (hence the error above). So the real problem is that there is a user “morpheus” on the server (in my case) which has it’s own vhost config in the ProFTPd service but for that user the password file was not present anymore.

Upon looking into the vhost configuration file of ProFTPd learned me that there was indeed a manual created vhost section for the user “morpheus”:

bash-3.00# cat /etc/proftpd.vhosts.conf
[VirtualHost 194.60.207.182]
ServerName "ProFTPd"
ExtendedLog /var/log/proftpd/194.60.207.182.bytes WRITE,READ userlog
AuthUserFile /usr/local/directadmin/data/users/morpheus/ftp.passwd
[/VirtualHost]

This itself isn’t an issue as long as the ftp.passwd file for the user exists. When examining this further I found out that the user “morpheus” didn’t exist on the server anymore! So here it was actually pretty clear why the error showed up as the given ftp.passwd didn’t exist anymore because the user was removed from the system in the past but the vhost config was not properly cleaned.

To solve it in one go we can simply move away the vhost configuration file of ProFTPd to a safe location (as it’s deprecated now anyway):
bash-3.00# mv /etc/proftpd.vhosts.conf /root/

After moving the file out of the way start ProFTPd again and you should see no errors and the service should be reachable again:
bash-3.00# /etc/init.d/proftpd restart
Shutting down proftpd: [FAILED]
Starting proftpd: [ OK ]


Updating the firmware on a Brocade 300 FC switch

 - 

product-300-right-gallery-bannerWith the office cleanup a couple of weeks ago an old Brocade 300 FC-switch (24 port 8Gbit fibrechannel)was found and was taken out of service due to probable hardware issues. Since the unit was already out of warranty no further efforts were made to revive or test the unit. I’ve taken the unit home to use with my 2 FC-cards to see how such a switch works and can be configured.

The first issue I ran into was the firmware. The conclusion at work was that a possible firmware update would be able to solve the issue so I went on a nice journey to find out how to retrieve the firmware and to update the switch (as the unit was never updated). This post will cover the steps and requirements needed to upgrade the firmware on this type of FC-switch. For people interested in the specifications of this switch can have a look at the datasheet here.

Getting firmware updates from Brocade:
This is the hardest part as you may only download firmware updates for Brocade devices if you have a support plan with Brocade. As I personally don’t have a contract with them I’ve found a colleague of mine who normally manages this for the company and downloaded the latest firmware updates for me. If you don’t have a contract with Brocade it’s not possible to download the firmware. You should contact a network or storage engineer in your company or personal area to see if they can help you further. At the time of writing version “7.4.1c” was the latest release with a size of approx ~1.2GB.
Version 8.0.0 and 8.0.1 were available as well, but not recommended by Brocade yet and therefore are not used in this post.

Logging in on the switch:
My unit was already configured so the LAN interface was already set to a fixed IP and had to reset this first. There is a serial console as well and sits next to the LAN interface (the IOIOI port) and with a RJ45 -> DB9 converter you can setup a serial connection with 9600baud 8N1. I had a problem with the serial connection that it would stop working after being idle for a minute and had to reboot the switch in order to revive it. Luckily the engineers never set a password themselves on the switch so when the login prompt arrived I was able to log in using the default password (username is “admin” and the password is “password”:

sw1 login: admin
Password:

-----------------------------------------------------------------
sw1:admin>

I had already updated the password, once logging in with the default password the system will ask you if you want to change this, for security it’s best to do so.

Since I’m logged in I had to change the interface details, this can be done with the following command:

sw1:admin> ipaddrset
Ethernet IP Address [192.168.0.199]:
Ethernet Subnetmask [255.255.255.0]:
Gateway IP Address [192.168.0.1]:
DHCP [Off]:
sw1:admin>

If you want to use DHCP you should set DHCP to On, but it’s best to set a fixed IP here so you always know the IP if you need to log in.

From this point on you don’t need the serial connection and can telnet to the IP address you’ve set above:
telnet 192.168.0.199
Trying 192.168.0.199...
Connected to 192.168.0.199.
Escape character is '^]'.


Fabric OS (sw1)
Fabos Version 6.4.3d

sw1 login:

Retrieving the current firmware version:
You should log in as admin on the console or management interface, with the “firmwareshow” command you can show the current running firmware:
sw1:admin> firmwareshow
Appl Primary/Secondary Versions
------------------------------------------
FOS v6.4.3d
v6.4.3d

As you can see the firmware on my unit is rather old. Using SSH to connect to the management IP was not possible due to weak ciphers and other strange behaviour that started showing up. The unit clearly looked faulty at this point. The unit itself is from 2012 and had it’s last update (according to the eventlog) in 2013 and was the stock firmware the unit shipped with at that point.

Updating the firmware:
The firmware update process is pretty simple but the instructions are not very clear if you are not familiar with it. You can update using a variety of protocols like:
– FTP
– SFTP
– SCP

To download the firmware to the device the “firmwaredownload” function can be used. In this process I will use SCP as this was the most convenient way in my network. To be able to use the SCP method we need to have a server which runs SSH and has a account that has SSH access, the easiest solution is to either use root or a local username (on the server) and extract the firmware software to it’s home directory. So in my case I have a user “jeffrey” on my server with IP 192.168.0.75. I’ve copied the firmware tar.gz file to the homedir of that user and extracted it there. This will result in a “v7.4.1c” directory in the homedir of “jeffrey”.

A note on the firmware archive: There are a lot of files in the archive, this holds the firmware for all supported devices so it’s not only for this switch. You don’t need to know the so-called SWBD number (hardware ID) of the switch. The updater will search in the right SWBD folder automatically and you only need to point the file name location to the directory where the firmware was extracted to.

A note on firmware update versions: You cannot upgrade directly to the latest version if you are running more than 1 release behind. So in my case where I have v6.4.3 I had to update to 7.0.0a -> 7.2.1g -> 7.4.1c instead. Note that you can update in even numbered steps, odd numbered update steps won’t work.See the end of this post for more information. In this case the instructions apply in general for the update process itself but this firmware version only works when running 7.2.x or 7.3.x.

Below are the steps that I took to accomplish this in general and will apply also when the device is running a version behind (with the console output):
sw1:admin> firmwaredownload -s
Server Name or IP Address: 192.168.0.75
User Name: jeffrey
File Name: v7.4.1c
Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 3
Verifying if the public key authentication is available.Please wait ...
The public key authentication is not available.
Password:
Do Auto-Commit after Reboot [Y]:
Reboot system after download [N]: Y
Server IP: 192.168.0.75, Protocol IPv4
Checking system settings for firmwaredownload...
WARNING: Fabric Watch is discontinued in FOS 7.4 and will not run after firmware upgrade. To continue with monitoring capability, it is recommended to migrate to MAPS prior to firmware upgrade. Users can convert existing Fabric Watch thresholds into MAPS policies by using "mapsConfig --fwconvert" CLI command and continue monitoring with the same settings. Fabric Watch thresholds cannot be converted to MAPS policies after firmware upgrade. Please refer to MAPS Administrator's Guide for further information.
System settings check passed.


You are running firmwaredownload with auto-reboot and auto-commit enabled. After the firmware is downloaded the system will reboot and commit firmware automatically.


Do you want to continue (Y/N) [Y]:
Firmware is being downloaded to the switch. This step may take up to 30 minutes.
Preparing for firmwaredownload...
Start to install packages...
dir ##################################################
ldconfig ##################################################
glibc ##################################################
glibc-linuxthreads ##################################################
bash ##################################################
readline ##################################################
terminfo ##################################################
termcap ##################################################
vixie-cron ##################################################
fileutils ##################################################
textutils ##################################################
warning: /etc/group created as /etc/group.rpmnew
warning: /etc/passwd created as /etc/passwd.rpmnew
setup ##################################################
warning: /etc/hosts created as /etc/hosts.rpmnew
swbd12-setup ##################################################
which ##################################################
findutils ##################################################
bzip ##################################################
zlib ##################################################
chkconfig ##################################################
sed ##################################################
procps ##################################################
psmisc ##################################################
modutils ##################################################
sin ##################################################
rcinit ##################################################
misc ##################################################
pam ##################################################
util-linux ##################################################
sh-utils ##################################################
popt ##################################################
grep ##################################################
rpm ##################################################
sysvinit ##################################################
man ##################################################
less ##################################################
gzip ##################################################
tar ##################################################
rsync ##################################################
uuid-libs ##################################################
e2fsprogs ##################################################
cpio ##################################################
dev ##################################################
bootenv ##################################################
wdtd ##################################################
fwdl ##################################################
telnet-server ##################################################
kernel ##################################################
kernel-module-usb ##################################################
swbd21-drivers ##################################################
sysklogd ##################################################
syslog-ng ##################################################
getty ##################################################
net-tools ##################################################
uucp ##################################################
portmap ##################################################
inetd ##################################################
iptables ##################################################
tcpd ##################################################
rsh-server ##################################################
rsh ##################################################
openssl-libs ##################################################
openssh ##################################################
warning: /etc/sshd_config saved as /etc/sshd_config.rpmsave
openssh-server ##################################################
rusers-server ##################################################
rdate ##################################################
logrotate ##################################################
ntp ##################################################
pciutils ##################################################
strace ##################################################
sendmail ##################################################
iproute2 ##################################################
libxml2 ##################################################
fss ##################################################
warning: /etc/fabos/rbac/dynamic created as /etc/fabos/rbac/dynamic.rpmnew
fabos-setup ##################################################
fabos-drivers ##################################################
fabos-libs ##################################################
fabos-diag ##################################################
fabos ##################################################
fabos-daemons ##################################################
fabos-zoning ##################################################
sqlite ##################################################
dhcpcd ##################################################
dhclient ##################################################
fabos-vf ##################################################
fabos-hmon ##################################################
fabos-wwnhs ##################################################
fabos-man ##################################################
fabos-swbd71 ##################################################
apache ##################################################
fastcgi ##################################################
fabos-webtools ##################################################
fabos-webtoolsez ##################################################
tz ##################################################
mtracer-tool ##################################################
sysstat ##################################################
prom-440epx ##################################################
Please avoid powering off the system during prom update.
ipv6 ##################################################
awk ##################################################
ipsec ##################################################
hss-diag ##################################################
Removing unneeded files, please wait ...
Finished removing unneeded files.


INFO: Ciphersuite change on switch
HTTPS ciphers will be modified to be compatible with new firmware version
creating the old storage file
All packages have been downloaded successfully.
Firmware has been downloaded to the secondary partition of the switch.


Broadcast message from root (pts/0) Mon May 23 15:49:45 2016...


The system is going down for reboot NOW !!
Connection closed by foreign host.

After the reboot the switch should be running the newly updated firmware. The reboot procedure varies in each version. I’ve set it to auto commit and reboot as this is a testing unit for me and don’t mind that it just reboots when done. In a production environment you might want to keep this feature disabled (which it is by default!). The download process took a couple of minutes (the management port is 10/100Mbit) and the update process around 15 minutes. Although I just waited for some time you may want to run “firmwaredownloadstatus” which can show the current firmware download eventlog and the current action it’s performing.

An example of the “firmwaredownloadstatus” output can be found below:
sw1:admin> firmwaredownloadstatus
[1]: Mon May 23 15:10:37 2016
Firmware is being downloaded to the switch. This step may take up to 30 minutes.


[2]: Mon May 23 15:15:23 2016
Firmware has been downloaded to the secondary partition of the switch.


[3]: Mon May 23 15:17:04 2016
The firmware commit operation has started. This may take up to 10 minutes.


[4]: Mon May 23 15:20:04 2016
The commit operation has completed successfully.


[5]: Mon May 23 15:20:04 2016
Firmwaredownload command has completed successfully. Use firmwareshow to verify the firmware versions.

Note on availability: During the download and update process the switch will remain available for traffic. During the reboot this is (yes really) interrupted. After the switch is back online the traffic is resumed.

Updating issues:
If you run a firmware older than 7.2.x you cannot directly upgrade to the current 7.4.1 release. If you try to upgrade it will error with:
sw1:admin> firmwaredownload
Server Name or IP Address:
fwdl (pid=3898): signal=2
sw1:admin> firmwaredownload
Server Name or IP Address: 192.168.0.75
User Name: jeffrey
File Name: v7.4.1c/
Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 3
Password:
Server IP: 192.168.0.75, Protocol IPv4
Checking system settings for firmwaredownload...


The following item(s) need to be addressed before downloading the specified firmware:
Cannot upgrade directly to 7.4. Please upgrade to 7.2 first and then upgrade to 7.4.


Firmwaredownload failed.
sw1:admin>

If this occurs you should download the 7.2 firmware update and apply it first and then try updating to 7.4 again. Do note that you cannot update major versions if you are running more than 2 firmware versions behind. So in order to update to 7.4 you either need to run 7.2 or 7.3 on your unit.

So in my case I had to update to 7.0.0a, 7.2.1g and to v7.4.1c. For reference you will find the whole console output below:

Updating from v6.4.3d to v7.4.1c:
Updating from v6.4.3d to v7.0.0a:
sw1:admin> firmwaredownload
Server Name or IP Address: 192.168.0.75
User Name: jeffrey
File Name: v7.0.0a
Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 3
Password:
Server IP: 192.168.0.75, Protocol IPv4
Checking system settings for firmwaredownload...
System settings check passed.


You can run firmwaredownloadstatus to get the status
of this command.


This command will cause a warm/non-disruptive boot but will
require that existing telnet, secure telnet or SSH sessions
be restarted.


Do you want to continue (Y/N) [Y]:
Firmware is being downloaded to the switch. This step may take up to 30 minutes.
Preparing for firmwaredownload...
Start to install packages...
dir ##################################################
ldconfig ##################################################
glibc ##################################################
glibc-linuxthreads ##################################################
bash ##################################################
readline ##################################################
terminfo ##################################################
termcap ##################################################
vixie-cron ##################################################
fileutils ##################################################
textutils ##################################################
setup ##################################################
swbd12-setup ##################################################
which ##################################################
findutils ##################################################
bzip ##################################################
zlib ##################################################
chkconfig ##################################################
sed ##################################################
procps ##################################################
psmisc ##################################################
modutils ##################################################
sin ##################################################
rcinit ##################################################
misc ##################################################
pam ##################################################
util-linux ##################################################
sh-utils ##################################################
popt ##################################################
grep ##################################################
rpm ##################################################
sysvinit ##################################################
man ##################################################
less ##################################################
gzip ##################################################
tar ##################################################
rsync ##################################################
uuid-libs ##################################################
e2fsprogs ##################################################
cpio ##################################################
dev ##################################################
bootenv ##################################################
wdtd ##################################################
fwdl ##################################################
telnet-server ##################################################
kernel ##################################################
kernel-module-usb ##################################################
swbd21-drivers ##################################################
sysklogd ##################################################
getty ##################################################
net-tools ##################################################
uucp ##################################################
portmap ##################################################
inetd ##################################################
iptables ##################################################
tcpd ##################################################
rsh-server ##################################################
rsh ##################################################
openssl-libs ##################################################
openssh ##################################################
openssh-server ##################################################
rusers-server ##################################################
rdate ##################################################
logrotate ##################################################
ntp ##################################################
pciutils ##################################################
strace ##################################################
sendmail ##################################################
iproute2 ##################################################
libxml2 ##################################################
fss ##################################################
fabos-setup ##################################################
fabos-drivers ##################################################
fabos-libs ##################################################
nonet-lib ##################################################
fabos-diag ##################################################
fabos ##################################################
fabos-daemons ##################################################
fabos-zoning ##################################################
sqlite ##################################################
dhcpcd ##################################################
fabos-vf ##################################################
fabos-hmon ##################################################
fabos-wwnhs ##################################################
fabos-man ##################################################
fabos-swbd71 ##################################################
apache ##################################################
fastcgi ##################################################
fabos-webtools ##################################################
fabos-webtoolsez ##################################################
tz ##################################################
mtracer-tool ##################################################
sysstat ##################################################
prom-440epx ##################################################
Please avoid powering off the system during prom update.
ipv6 ##################################################
awk ##################################################
ipsec ##################################################
hss-diag ##################################################
Removing unneeded files, please wait ...
Finished removing unneeded files.


All packages have been downloaded successfully.
Firmware has been downloaded to the secondary partition of the switch.
HA Rebooting ...
Connection closed by foreign host.

Check to see if the update went okay:
[jeffrey@e7440 ~]$ telnet 192.168.0.199
Trying 192.168.0.199...
Connected to 192.168.0.199.
Escape character is '^]'.


Fabric OS (sw1)
Fabos Version 7.0.0a

Updating from v7.0.0.a to 7.2.1g:
sw1:admin> firmwaredownload -s
Server Name or IP Address: 192.168.0.75
User Name: jeffrey
File Name: v7.2.1g
Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 3
Verifying if the public key authentication is available.Please wait ...
The public key authentication is not available.
Password:
Do Auto-Commit after Reboot [Y]:
Reboot system after download [N]:
Server IP: 192.168.0.75, Protocol IPv4
Checking system settings for firmwaredownload...


This action will set default QoS port configuration from AE to OFF because Adaptive Networking License is not installed on the switch.


System settings check passed.


You are running firmwaredownload with auto-reboot disabled. After firmware is downloaded, please reboot the system to activate the new firmware.


Do you want to continue (Y/N) [Y]:
Firmware is being downloaded to the switch. This step may take up to 30 minutes.
Preparing for firmwaredownload...
Removing nonet-lib
Removing hss-diag
Start to install packages...
dir ##################################################
ldconfig ##################################################
glibc ##################################################
glibc-linuxthreads ##################################################
bash ##################################################
readline ##################################################
terminfo ##################################################
termcap ##################################################
vixie-cron ##################################################
fileutils ##################################################
textutils ##################################################
setup ##################################################
swbd12-setup ##################################################
which ##################################################
findutils ##################################################
bzip ##################################################
zlib ##################################################
chkconfig ##################################################
sed ##################################################
procps ##################################################
psmisc ##################################################
modutils ##################################################
sin ##################################################
rcinit ##################################################
misc ##################################################
pam ##################################################
util-linux ##################################################
sh-utils ##################################################
popt ##################################################
grep ##################################################
rpm ##################################################
sysvinit ##################################################
man ##################################################
less ##################################################
gzip ##################################################
tar ##################################################
rsync ##################################################
uuid-libs ##################################################
e2fsprogs ##################################################
cpio ##################################################
dev ##################################################
bootenv ##################################################
wdtd ##################################################
fwdl ##################################################
telnet-server ##################################################
kernel ##################################################
kernel-module-usb ##################################################
swbd21-drivers ##################################################
sysklogd ##################################################
getty ##################################################
net-tools ##################################################
uucp ##################################################
portmap ##################################################
inetd ##################################################
iptables ##################################################
tcpd ##################################################
rsh-server ##################################################
rsh ##################################################
openssl-libs ##################################################
openssh ##################################################
openssh-server ##################################################
rusers-server ##################################################
rdate ##################################################
logrotate ##################################################
ntp ##################################################
pciutils ##################################################
strace ##################################################
sendmail ##################################################
iproute2 ##################################################
libxml2 ##################################################
fss ##################################################
warning: /etc/fabos/rbac/dynamic created as /etc/fabos/rbac/dynamic.rpmnew
fabos-setup ##################################################
fabos-drivers ##################################################
fabos-libs ##################################################
fabos-diag ##################################################
fabos ##################################################
fabos-daemons ##################################################
fabos-zoning ##################################################
sqlite ##################################################
dhcpcd ##################################################
dhclient ##################################################
fabos-vf ##################################################
fabos-hmon ##################################################
fabos-wwnhs ##################################################
fabos-man ##################################################
fabos-swbd71 ##################################################
apache ##################################################
fastcgi ##################################################
fabos-webtools ##################################################
fabos-webtoolsez ##################################################
tz ##################################################
mtracer-tool ##################################################
sysstat ##################################################
prom-440epx ##################################################
Please avoid powering off the system during prom update.
ipv6 ##################################################
awk ##################################################
ipsec ##################################################
Removing unneeded files, please wait ...
Finished removing unneeded files.


All packages have been downloaded successfully.
Firmware has been downloaded to the secondary partition of the switch.

Issue a “firmwarecommit” to apply the update. The switch will reboot for this. After the reboot check if the update went okay:
[jeffrey@e7440 ~]$ telnet 192.168.0.199
Trying 192.168.0.199...
Connected to 192.168.0.199.
Escape character is '^]'.


Fabric OS (sw1)
Fabos Version 7.2.1g

And the last step, from v7.2.1g to v7.4.1c:
sw1:admin> firmwaredownload -s
Server Name or IP Address: 192.168.0.75
User Name: jeffrey
File Name: v7.4.1c
Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 3
Verifying if the public key authentication is available.Please wait ...
The public key authentication is not available.
Password:
Do Auto-Commit after Reboot [Y]:
Reboot system after download [N]: Y
Server IP: 192.168.0.75, Protocol IPv4
Checking system settings for firmwaredownload...
WARNING: Fabric Watch is discontinued in FOS 7.4 and will not run after firmware upgrade. To continue with monitoring capability, it is recommended to migrate to MAPS prior to firmware upgrade. Users can convert existing Fabric Watch thresholds into MAPS policies by using "mapsConfig --fwconvert" CLI command and continue monitoring with the same settings. Fabric Watch thresholds cannot be converted to MAPS policies after firmware upgrade. Please refer to MAPS Administrator's Guide for further information.
System settings check passed.


You are running firmwaredownload with auto-reboot and auto-commit enabled. After the firmware is downloaded the system will reboot and commit firmware automatically.


Do you want to continue (Y/N) [Y]:
Firmware is being downloaded to the switch. This step may take up to 30 minutes.
Preparing for firmwaredownload...
Start to install packages...
dir ##################################################
ldconfig ##################################################
glibc ##################################################
glibc-linuxthreads ##################################################
bash ##################################################
readline ##################################################
terminfo ##################################################
termcap ##################################################
vixie-cron ##################################################
fileutils ##################################################
textutils ##################################################
warning: /etc/group created as /etc/group.rpmnew
warning: /etc/passwd created as /etc/passwd.rpmnew
setup ##################################################
warning: /etc/hosts created as /etc/hosts.rpmnew
swbd12-setup ##################################################
which ##################################################
findutils ##################################################
bzip ##################################################
zlib ##################################################
chkconfig ##################################################
sed ##################################################
procps ##################################################
psmisc ##################################################
modutils ##################################################
sin ##################################################
rcinit ##################################################
misc ##################################################
pam ##################################################
util-linux ##################################################
sh-utils ##################################################
popt ##################################################
grep ##################################################
rpm ##################################################
sysvinit ##################################################
man ##################################################
less ##################################################
gzip ##################################################
tar ##################################################
rsync ##################################################
uuid-libs ##################################################
e2fsprogs ##################################################
cpio ##################################################
dev ##################################################
bootenv ##################################################
wdtd ##################################################
fwdl ##################################################
telnet-server ##################################################
kernel ##################################################
kernel-module-usb ##################################################
swbd21-drivers ##################################################
sysklogd ##################################################
syslog-ng ##################################################
getty ##################################################
net-tools ##################################################
uucp ##################################################
portmap ##################################################
inetd ##################################################
iptables ##################################################
tcpd ##################################################
rsh-server ##################################################
rsh ##################################################
openssl-libs ##################################################
openssh ##################################################
warning: /etc/sshd_config saved as /etc/sshd_config.rpmsave
openssh-server ##################################################
rusers-server ##################################################
rdate ##################################################
logrotate ##################################################
ntp ##################################################
pciutils ##################################################
strace ##################################################
sendmail ##################################################
iproute2 ##################################################
libxml2 ##################################################
fss ##################################################
warning: /etc/fabos/rbac/dynamic created as /etc/fabos/rbac/dynamic.rpmnew
fabos-setup ##################################################
fabos-drivers ##################################################
fabos-libs ##################################################
fabos-diag ##################################################
fabos ##################################################
fabos-daemons ##################################################
fabos-zoning ##################################################
sqlite ##################################################
dhcpcd ##################################################
dhclient ##################################################
fabos-vf ##################################################
fabos-hmon ##################################################
fabos-wwnhs ##################################################
fabos-man ##################################################
fabos-swbd71 ##################################################
apache ##################################################
fastcgi ##################################################
fabos-webtools ##################################################
fabos-webtoolsez ##################################################
tz ##################################################
mtracer-tool ##################################################
sysstat ##################################################
prom-440epx ##################################################
Please avoid powering off the system during prom update.
ipv6 ##################################################
awk ##################################################
ipsec ##################################################
hss-diag ##################################################
Removing unneeded files, please wait ...
Finished removing unneeded files.


INFO: Ciphersuite change on switch
HTTPS ciphers will be modified to be compatible with new firmware version
creating the old storage file
All packages have been downloaded successfully.
Firmware has been downloaded to the secondary partition of the switch.


Broadcast message from root (pts/0) Mon May 23 15:49:45 2016...


The system is going down for reboot NOW !!
Connection closed by foreign host.

After a couple of minutes, log back in to see if the update went okay:
[jeffrey@e7440 ~]$ telnet 192.168.0.199
Trying 192.168.0.199...
Connected to 192.168.0.199.
Escape character is '^]'.


Fabric OS (sw1)
Fabos Version 7.4.1c


Fix the PHP Fatal: [ionCube Loader] issue on DirectAdmin

 - 

directadmin
When updating / recompiling PHP with Custombuild on a DirectAdmin server I sometimes see that Apache seems to start, but immediately crashes due to errors regarding ionCube. You don’t see the error when restarting Apache for exaple, but checking the logs or listing the installed PHP modules results in the following error:

[root@server]# php -m
PHP Fatal error: [ionCube Loader] The Loader must appear as the first entry in the php.ini file in Unknown on line 0

I mostly see this behaviour on servers running CentOS and the first thing that I did was locating all php.ini files and remove all ioncube references from these configs. Unfortunately that didn’t work out and the error persists. Upon some internet searching I found out that the issue is caused by the internal “php.ini” that DirectAdmin itself uses. That file is called “directadmin.ini” and is located in:
/usr/local/lib/php.conf.d/10-directadmin.ini

I’ve removed the ioncube loader reference from the config and Apache could start and continue to work again.

This is a rather small post compared to the ones I normally publish, since this is a easy fix I wanted it to keep it as simple as possible hoping it will help others.


Updating IPMI firmware on a SuperMicro server

 - 

Super_Micro_Computer_Logo.svgLast week we moved to a new office building and a old SuperMicro server popped up. In particular a SuperMicro X8DTL-IF in a 936A-R1200B chassis. It’s a rather old server now, but since I want to use this machine for SAN-like testing and purposes it comes in handy for me! In the past this machine was used as a big storage box containing backupdata and never received system maintance like BIOS updates for example. So upon examining the unit I found out that it was running the stock firmwares that were present at the time of delivery so the IPMI firmware running is version 02.02 whereas the current version is 03.13. In this post I will explain how to update the IPMI firmware on this mainboard using the webinterface.

Management interface:
The machine has a management network interface which works by using IPMI. The management interface can be reached via the webbrowser if you browse to it’s assigned IP-address. Even when the machine itself is powered off (but connected to mains) the management interface is accessible and allows you to perform several maintance related tasks like remote power management, KVM and console redirection. If you access it via the browser a login screen like below appears:
Screenshot_20160606_113508
The default username is ADMIN and the password is ADMIN.

Checking the IPMI firmware version:
Once logged in to the interface go to System -> System Information and the following information should be displayed (version numbers may vary):
Screenshot_20160606_114158

Getting the IPMI firmware update:
I cannot give downloads for this as this is updated regularly and may vary per board used. Please navigate to the SuperMicro website and browse to the board that you use. On the specitications page go to “IPMI Firmware” link. You should end up with a .ZIP file containing a .bin file. Extract that to a convenient place.

Updating the IPMI firmware:
The update process is pretty straight-forward. In the web interface go to Maintenance -> Firmware update. On the page that shows up you will need to enable update mode by clicking the Enter Update Mode and confirm that you want to do so on the popup that shows up.
A file selector button appears, point to the .bin file that is in the downloaded .ZIP file and select the Upload file button. Once the file is uploaded a confirmation screen appears which shows the current running version and the newly installing version:
Screenshot_20160606_120648
If you want your current settings to be preserved, leave the checkbox ticked. Not ticking the box will reset all settings to factory defaults.

Click on the Start Upgrade button to start the upgrade, the progress will be showed after clicking the button:
Screenshot_20160606_121217

When the update is finished the management interface will reboot and you will return back to the login screen.

Checking the new installed firmware:
Log back in to the web interface to check if the firmware was installed successfully, go to System -> System Information to check the new version. In this case the upgrade has succeeded:
Screenshot_20160606_121646


Install Composer on Ubuntu 14.04

 - 

logo-composer-transparentI run a newsgroup indexer privately which has switched to Composer for managing PHP dependencies. As Composer is not in the default Ubuntu 14.04 repositories I had to install it manually. This is a really simple process, but since I had to search a while for this I wanted to spend a small post about it. You will need root (or an account that can sudo) in order to install Composer.

Instaling the requirements:
For Composer to work, some dependencies need to be installed if they are not installed already:
apt-get update
apt-get install php5-cli git curl

Install Composer:
Now we need to download and put the Composer binary into place. Here we will install it directly to /usr/local/bin/:
curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer

Testing Composer:
Now that Composer is installed, run the program to see if it outputs it’s help file:
composer
Running composer as root/super user is highly discouraged as packages, plugins and scripts cannot always be trusted
______
/ ____/___ ____ ___ ____ ____ ________ _____
/ / / __ \/ __ `__ \/ __ \/ __ \/ ___/ _ \/ ___/
/ /___/ /_/ / / / / / / /_/ / /_/ (__ ) __/ /
\____/\____/_/ /_/ /_/ .___/\____/____/\___/_/
/_/
Composer version 1.1.2 2016-05-31 19:48:11

Usage:
command [options] [arguments]

Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
--profile Display timing and memory usage information
--no-plugins Whether to disable plugins.
-d, --working-dir=WORKING-DIR If specified, use the given directory as working directory.
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
about Short information about Composer
archive Create an archive of this composer package
browse Opens the package's repository URL or homepage in your browser.
clear-cache Clears composer's internal package cache.
clearcache Clears composer's internal package cache.
config Set config options
create-project Create new project from a package into given directory.
depends Shows which packages cause the given package to be installed
diagnose Diagnoses the system to identify common errors.
dump-autoload Dumps the autoloader
dumpautoload Dumps the autoloader
exec Execute a vendored binary/script
global Allows running commands in the global composer dir ($COMPOSER_HOME).
help Displays help for a command
home Opens the package's repository URL or homepage in your browser.
info Show information about packages
init Creates a basic composer.json file in current directory.
install Installs the project dependencies from the composer.lock file if present, or falls back on the composer.json.
licenses Show information about licenses of dependencies
list Lists commands
outdated Shows a list of installed packages that have updates available, including their latest version.
prohibits Shows which packages prevent the given package from being installed
remove Removes a package from the require or require-dev
require Adds required packages to your composer.json and installs them
run-script Run the scripts defined in composer.json.
search Search for packages
self-update Updates composer.phar to the latest version.
selfupdate Updates composer.phar to the latest version.
show Show information about packages
status Show a list of locally modified packages
suggests Show package suggestions
update Updates your dependencies to the latest version according to composer.json, and updates the composer.lock file.
validate Validates a composer.json and composer.lock
why Shows which packages cause the given package to be installed
why-not Shows which packages prevent the given package from being installed

That’s it!

In Server

Upgrading MySQL from the commandline on a cPanel server

 - 

cpanel_logo
Recently I had to upgrade MySQL on a cPanel server. Although this can be easily done from within WHM itself I wanted to perform this from the commandline as that is more my way of working. The documentation on upgrading MySQL from the commandline on a cPanel server is not easily found, that’s why I want to share these instructions here.

Important notice:
Be aware that after upgrading MySQL you need to recompile PHP as well if you want the MySQL extension to work with the upgraded MySQL version. The recompile of PHP can be done using the EasyApache option from within WHM or /scripts/easyapache.

Changing the cPanel configuration file:
In order to upgrade MySQL we need to alter the cPanel configuration file. Log in via SSH as the root user and open the following configuration file:
vim /var/cpanel/cpanel.config

Search for the line starting with:
mysql-version=

In my case the server was running MySQL 5.5, so the line looked like this:
mysql-version=5.5

I want to upgrade to MySQL 5.6, so change the line to (for this example):
mysql-version=5.6

Save the changes.

Upgrading MySQL:
Since the configuration is changed, we need to make sure cPanel sees the change and downloads the correct RPM’s for MySQL and install it. Run the following command (the output of the command is underneath it):
/scripts/check_cpanel_rpms
[2016-06-01 10:26:01 +0200]
[2016-06-01 10:26:01 +0200] Problems were detected with cPanel-provided files which are RPM controlled.
[2016-06-01 10:26:01 +0200] If you did not make these changes intentionally, you can correct them by running:
[2016-06-01 10:26:01 +0200]
[2016-06-01 10:26:01 +0200] > /usr/local/cpanel/scripts/check_cpanel_rpms --fix
[2016-06-01 10:26:01 +0200] The following RPMs are missing from your system:
[2016-06-01 10:26:01 +0200] MySQL56-client-5.6.30-1.cp1156
[2016-06-01 10:26:01 +0200] MySQL56-devel-5.6.30-1.cp1156
[2016-06-01 10:26:01 +0200] MySQL56-server-5.6.30-1.cp1156
[2016-06-01 10:26:01 +0200] MySQL56-shared-5.6.30-1.cp1156
[2016-06-01 10:26:01 +0200] MySQL56-test-5.6.30-1.cp1156
[2016-06-01 10:26:03 +0200]
[2016-06-01 10:26:03 +0200] The following RPMs are unneeded on your system and should be uninstalled:
[2016-06-01 10:26:03 +0200] MySQL55-client-5.5.49-1.cp1156
[2016-06-01 10:26:03 +0200] MySQL55-devel-5.5.49-1.cp1156
[2016-06-01 10:26:03 +0200] MySQL55-server-5.5.49-1.cp1156
[2016-06-01 10:26:03 +0200] MySQL55-shared-5.5.49-1.cp1156
[2016-06-01 10:26:03 +0200] MySQL55-test-5.5.49-1.cp1156
Do you want to repair these RPMs?(y/n):

It’s wise to check the versions above before performing the upgrade, make sure that the new version matches the one you set and that the old version is the version you are currently running. If everything is correct we can start the upgrade by saying yes here and the upgrade will start:

y
[2016-06-01 10:51:49 +0200] Removing 0 broken rpms:
[2016-06-01 10:51:49 +0200] rpm: no packages given for erase
[2016-06-01 10:51:50 +0200] Downloading http://httpupdate.cpanel.net/RPM/11.56/centos/6/x86_64/rpm.sha512
[2016-06-01 10:51:50 +0200] Successfully verified signature for cpanel (key types: release).
[2016-06-01 10:51:50 +0200] Downloading http://httpupdate.cpanel.net/RPM/11.56/centos/6/x86_64/MySQL56-shared-5.6.30-1.cp1156.x86_64.rpm
[2016-06-01 10:51:50 +0200] Downloading http://httpupdate.cpanel.net/RPM/11.56/centos/6/x86_64/MySQL56-server-5.6.30-1.cp1156.x86_64.rpm
[2016-06-01 10:51:51 +0200] Downloading http://httpupdate.cpanel.net/RPM/11.56/centos/6/x86_64/MySQL56-client-5.6.30-1.cp1156.x86_64.rpm
[2016-06-01 10:51:51 +0200] Downloading http://httpupdate.cpanel.net/RPM/11.56/centos/6/x86_64/MySQL56-devel-5.6.30-1.cp1156.x86_64.rpm
[2016-06-01 10:51:51 +0200] Downloading http://httpupdate.cpanel.net/RPM/11.56/centos/6/x86_64/MySQL56-test-5.6.30-1.cp1156.x86_64.rpm
[2016-06-01 10:51:52 +0200] Disabling service monitoring.
[2016-06-01 10:51:57 +0200] Hooks system enabled.
[2016-06-01 10:51:57 +0200] Checking for and running RPM::Versions 'pre' hooks for any RPMs about to be installed
[2016-06-01 10:51:57 +0200] All required 'pre' hooks have been run
[2016-06-01 10:51:58 +0200] Uninstalling unneeded rpms: MySQL55-server MySQL55-devel MySQL55-test MySQL55-shared MySQL55-client
[2016-06-01 10:52:22 +0200] Installing new rpms: MySQL56-client-5.6.30-1.cp1156.x86_64.rpm MySQL56-devel-5.6.30-1.cp1156.x86_64.rpm MySQL56-server-5.6.30-1.cp1156.x86_64.rpm MySQL56-shared-5.6.30-1.cp1156.x86_64.rpm MySQL56-test-5.6.30-1.cp1156.x86_64.rpm
[2016-06-01 10:52:22 +0200] Preparing packages for installation...
[2016-06-01 10:52:23 +0200] MySQL56-client-5.6.30-1.cp1156
[2016-06-01 10:52:23 +0200] MySQL56-test-5.6.30-1.cp1156
[2016-06-01 10:52:30 +0200] MySQL56-devel-5.6.30-1.cp1156
[2016-06-01 10:52:30 +0200] Giving mysqld 5 seconds to exit nicely
[2016-06-01 10:52:36 +0200] MySQL56-server-5.6.30-1.cp1156
[2016-06-01 10:52:57 +0200] Waiting for “mysql” to start ……waiting for “mysql” to initialize ………finished.
[2016-06-01 10:52:57 +0200]
[2016-06-01 10:52:57 +0200] Startup Log
[2016-06-01 10:52:57 +0200] Starting MySQL..... SUCCESS!
[2016-06-01 10:52:57 +0200]
[2016-06-01 10:52:57 +0200] Log Messages
[2016-06-01 10:52:57 +0200] 2016-06-01 10:52:56 30359 [Note] /usr/sbin/mysqld: ready for connections.
[2016-06-01 10:52:57 +0200]
[2016-06-01 10:52:57 +0200] mysql started successfully.
[2016-06-01 10:55:44 +0200] Looking for 'mysql' as: /usr/bin/mysql
[2016-06-01 10:55:44 +0200] Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
[2016-06-01 10:55:44 +0200] Running 'mysqlcheck with default connection arguments
[2016-06-01 10:55:44 +0200] Running 'mysqlcheck with default connection arguments
[2016-06-01 10:55:44 +0200] mysql.columns_priv OK
[2016-06-01 10:55:44 +0200] mysql.db OK
[2016-06-01 10:55:44 +0200] mysql.event OK
[2016-06-01 10:55:44 +0200] mysql.func OK
[2016-06-01 10:55:44 +0200] mysql.general_log OK
[2016-06-01 10:55:44 +0200] mysql.help_category OK
[2016-06-01 10:55:44 +0200] mysql.help_keyword OK
[2016-06-01 10:55:44 +0200] mysql.help_relation OK
[2016-06-01 10:55:44 +0200] mysql.help_topic OK
[2016-06-01 10:55:44 +0200] mysql.host OK
[2016-06-01 10:55:44 +0200] mysql.ndb_binlog_index OK
[2016-06-01 10:55:44 +0200] mysql.plugin OK
[2016-06-01 10:55:44 +0200] mysql.proc OK
[2016-06-01 10:55:44 +0200] mysql.procs_priv OK
[2016-06-01 10:55:44 +0200] mysql.proxies_priv OK
[2016-06-01 10:55:44 +0200] mysql.servers OK
[2016-06-01 10:55:44 +0200] mysql.slow_log OK
[2016-06-01 10:55:44 +0200] mysql.tables_priv OK
[2016-06-01 10:55:44 +0200] mysql.time_zone OK
[2016-06-01 10:55:44 +0200] mysql.time_zone_leap_second OK
[2016-06-01 10:55:44 +0200] mysql.time_zone_name OK
[2016-06-01 10:55:44 +0200] mysql.time_zone_transition OK
[2016-06-01 10:55:44 +0200] mysql.time_zone_transition_type OK
[2016-06-01 10:55:44 +0200] mysql.user OK
[2016-06-01 10:55:44 +0200] Running 'mysql_fix_privilege_tables'...
[2016-06-01 10:55:44 +0200] Running 'mysqlcheck with default connection arguments
[2016-06-01 10:55:44 +0200] Running 'mysqlcheck with default connection arguments
SNIP:long output of MySQL repair:SNIP
[2016-06-01 10:55:44 +0200] OK
[2016-06-01 10:55:46 +0200] The 'mysql' service passed the check.
[2016-06-01 10:55:46 +0200] The 'mysql' service passed the check.
[2016-06-01 10:55:52 +0200] Starting MySQL SUCCESS!
[2016-06-01 10:55:52 +0200] Checking MySQL server status after update
[2016-06-01 10:55:52 +0200] The 'mysql' service passed the check.
[2016-06-01 10:55:52 +0200] SUCCESS! MySQL running (30359)
[2016-06-01 10:55:52 +0200] MySQL56-shared-5.6.30-1.cp1156
[2016-06-01 10:55:52 +0200] Hooks system enabled.
[2016-06-01 10:55:52 +0200] Checking for and running RPM::Versions 'post' hooks for any RPMs about to be installed
[2016-06-01 10:55:52 +0200] All required 'post' hooks have been run
[2016-06-01 10:55:52 +0200] Restoring service monitoring.

After this you will be left at the commandline again, let’s check if the server is running the new MySQL version:
mysql --version
mysql Ver 14.14 Distrib 5.6.30, for Linux (x86_64) using EditLine wrapper

That’s it!

As said at the beginning of this post, don’t forget to run EasyApache if you want PHP to work with the upgraded version of MySQL!

In cPanel