Stuck waiting for network configuration after install


#1

This is a copy and paste of a support case opened from a github issue .

./securedrop-admin install fails and mon and app boxes become unreachable

SecureDrop v0.5.2
Tails (admin box) - 3.3

After running ./securedrop-admin install, the app and mon boxes both reboot after some time.

Unfortunately, they both then hang waiting for Network Configuration.

The ansible script on the amdin box waits for the boxes to come back, but when they hang, the script eventually fails.

Also (I think because the ansible script failed), we do not have an app-source-ths or any other ths files and can no longer access the mon and app servers.

We have repeated this twice (after having to wipe the drives completely from app and mon by removing them and putting them in a sled).

Is there any explanation for why the app and mon boxes would not be able to get a Network Configuration? We looked over the firewall configuration and it seems fine…

at that point I asked the user for details on hardware and they responded with

Is there a command I can run that will help to give you various hardware info quickly?

If I boot into the mon server as the root user and less /etc/network/interfaces I get the following:

auto lo
iface lo inet loopback

Is there a command I can run that will help to give you various hardware info quickly?

If I boot into the mon server as the root user and less /etc/network/interfaces I get the following:

auto lo
iface lo inet loopback

The primary network interface

auto em1
iface em1 inet static
addresss 10.20.3.2
netmask 255.255.255.0
network 10.20.3.0
boradcast 10.20.3.255
gateway 10.20.3.1

dns-* options are implementd by the resolveconf package, if installed

dns-nameservers 8.8.8.8

auto em1
iface em1 inet static
addresss 10.20.3.2
netmask 255.255.255.0
network 10.20.3.0
boradcast 10.20.3.255
gateway 10.20.3.1
dns-nameservers 8.8.8.8

I’ve taken a few more cracks at this from the firewall end of the configuration, running the install, reformatting the servers after each attempt, etc. and still running in to the same roadblock.

Scenario:
• I initiate the installer from the admin workstation.
• It proceeds up to the point where it triggers the servers to reboot.
• During the reboot process, both servers hang up at…
Waiting for network configuration…
Waiting up to 60 more seconds for network configuration
• After the timeout, both servers proceed to the log in screen.
• The installer on admin workstation then displays the following error:
RUNNING HANDLER [common : wait_for] **************************************************************************************************************************
fatal: [mon -> localhost]: FAILED! => {“changed”: false, “elapsed”: 301, “failed”: true, “msg”: “Timeout when waiting for search string OpenSSH in 10.20.3.2:22”}
fatal: [app -> localhost]: FAILED! => {“changed”: false, “elapsed”: 301, “failed”: true, “msg”: “Timeout when waiting for search string OpenSSH in 10.20.2.2:22”}

Notes:
• Able to SSH from all three machines pre-install.
• In the troubleshooting process, I’ve enabled all SSH traffic across all three firewall interfaces between the app, mon and admin machines.
• App and mon are reformatted and Ubuntu reloaded after each attempt.
• The admin workstation is a MacBook Air with a Thunderbolt to Ethernet adapter and booting from a Tails SD card.

Any thoughts are greatly appreciated.


#2

So once the playbook lags at the network stage… are you able to login via console and verify the machine still has network connectivity?


#3

Have we confirmed that they are using Ubuntu Trusty on the servers? I have seen at least one client who reported similar errors and that turned out to be the root cause.


#4

Hi Mike. Once the servers reboot and return to the login screen I am able to log in. If I attempt to ssh or ping from the app server I receive a “Network unreachable” error (nor can I ssh to it/ping it from the admin). I am able however to ping from the mon server and I can ssh in to it from the admin workstation. So odd.


#5

HI. I downloaded and installed ubuntu-14.04.5-server-amd64.iso following the SecureDrop “Set up the Servers” guide.


#6

Hi Ian,

So this is sounding like a firewall registration issue. Can you upload a screenshot of your OPT1 and OPT2 rules?


#7

You should look at lshw (debian/ubuntu packaged).


#8

HI. Below are the results of a sh int I have a copy of the full show run if you’d like that as well.

Thanks again!

Interface GigabitEthernet1/3 “OPT1”, is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(10 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 843d.c646.6f06, MTU 1500
IP address 10.20.2.1, subnet mask 255.255.255.0
584493 packets input, 57684800 bytes, 0 no buffer
Received 43 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
250178 packets output, 326639468 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (896/863)
output queue (blocks free curr/low): hardware (1023/865)
Traffic Statistics for “OPT1”:
584489 packets input, 47118028 bytes
250178 packets output, 321863912 bytes
82 packets dropped
1 minute input rate 0 pkts/sec, 0 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 0 bytes/sec
5 minute output rate 0 pkts/sec, 0 bytes/sec
5 minute drop rate, 0 pkts/sec

SD-ASA5506X(config)# show int opt2
Interface GigabitEthernet1/4 “OPT2”, is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 843d.c646.6f07, MTU 1500
IP address 10.20.3.1, subnet mask 255.255.255.0
131105 packets input, 12104463 bytes, 0 no buffer
Received 26 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
346065 packets output, 299454535 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (969/877)
output queue (blocks free curr/low): hardware (1023/887)
Traffic Statistics for “OPT2”:
131089 packets input, 9696501 bytes
346065 packets output, 291079969 bytes
67079 packets dropped
1 minute input rate 1 pkts/sec, 83 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 84 bytes/sec
5 minute output rate 0 pkts/sec, 0 bytes/sec
5 minute drop rate, 0 pkts/sec


#9

@this-is-ian Can you download screenshots from the hardware firewall and show them to us?


#10

Hello. Sorry for the delay, I’ve been out sick. Below is a show run from the ASA. The config has become a bit of a monster with troubleshooting.

!
interface GigabitEthernet1/1
nameif outside
security-level 0
ip address dhcp setroute
!
interface GigabitEthernet1/2
nameif inside
security-level 100
ip address 10.20.1.1 255.255.255.0
!
interface GigabitEthernet1/3
nameif OPT1
security-level 100
ip address 10.20.2.1 255.255.255.0
!
interface GigabitEthernet1/4
nameif OPT2
security-level 100
ip address 10.20.3.1 255.255.255.0
!
interface GigabitEthernet1/5
description ASDM Admin Port
nameif ADMIN
security-level 100
ip address 10.20.4.1 255.255.255.0
!
interface GigabitEthernet1/6
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/7
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/8
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/9
nameif wifi
security-level 100
ip address 1.1.1.1 255.255.255.0
!
interface Management1/1
management-only
nameif MGMT
security-level 100
ip address 192.168.45.45 255.255.255.0
!
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns server-group DefaultDNS
domain-name securedrop.local
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network admin_workstation
host 10.20.1.2
object network app_server
host 10.20.2.2
object network monitor_server
host 10.20.3.2
object network external_dns_1
host 8.8.8.8
object network external_dns_2
host 8.8.4.4
object service OSSEC
service tcp destination eq 1514
description OSSEC per SecureDrop
object service ossec_agent_auth
service tcp destination eq 1515
description OSSEC Per SecureDrop
object network ASDM-Network
subnet 10.20.4.0 255.255.255.0
object network external_dns_3
host 9.9.9.9
object-group network local_servers
network-object object app_server
network-object object monitor_server
object-group network external_dns_servers
network-object object external_dns_1
network-object object external_dns_2
network-object object external_dns_3
object-group network DM_INLINE_NETWORK_1
network-object 10.20.2.0 255.255.255.0
network-object 10.20.3.0 255.255.255.0
object-group network DM_INLINE_NETWORK_2
network-object 10.20.1.0 255.255.255.0
network-object 10.20.3.0 255.255.255.0
object-group network DM_INLINE_NETWORK_3
network-object 10.20.1.0 255.255.255.0
network-object 10.20.2.0 255.255.255.0
object-group network DM_INLINE_NETWORK_4
network-object 10.20.2.0 255.255.255.0
network-object 10.20.3.0 255.255.255.0
object-group network DM_INLINE_NETWORK_5
network-object 10.20.1.0 255.255.255.0
network-object 10.20.2.0 255.255.255.0
network-object 10.20.3.0 255.255.255.0
object-group network DM_INLINE_NETWORK_6
network-object 10.20.1.0 255.255.255.0
network-object 10.20.2.0 255.255.255.0
network-object 10.20.3.0 255.255.255.0
object-group network DM_INLINE_NETWORK_7
network-object 10.20.1.0 255.255.255.0
network-object 10.20.2.0 255.255.255.0
object-group network DM_INLINE_NETWORK_8
network-object 10.20.1.0 255.255.255.0
network-object 10.20.3.0 255.255.255.0
object-group service DM_INLINE_SERVICE_1
service-object object OSSEC
service-object object ossec_agent_auth
object-group service DM_INLINE_SERVICE_2
service-object object OSSEC
service-object object ossec_agent_auth
access-list wifi_access_in extended deny ip any any
access-list inside_access_in extended permit tcp object admin_workstation object-group local_servers eq ssh
access-list inside_access_in extended permit ip any any
access-list inside_access_in extended permit object OSSEC object-group local_servers object admin_workstation
access-list inside_access_in extended permit object ossec_agent_auth object-group local_servers object admin_workstation
access-list inside_access_in extended permit object OSSEC object admin_workstation object-group local_servers
access-list inside_access_in extended permit object ossec_agent_auth object admin_workstation object-group local_servers
access-list inside_access_in extended permit ip object-group DM_INLINE_NETWORK_4 10.20.1.0 255.255.255.0
access-list inside_access_in extended permit icmp object-group DM_INLINE_NETWORK_5 object-group DM_INLINE_NETWORK_6
access-list inside_access_in extended permit object-group DM_INLINE_SERVICE_2 object admin_workstation object monitor_server
access-list inside_access_in extended permit tcp any any eq ssh
access-list OPT2_access_in extended permit ip 10.20.3.0 255.255.255.0 10.20.1.0 255.255.255.0
access-list OPT2_access_in extended permit ip 10.20.3.0 255.255.255.0 10.20.2.0 255.255.255.0
access-list OPT2_access_in extended permit ip object monitor_server any
access-list OPT2_access_in extended permit ip object monitor_server object-group external_dns_servers
access-list OPT2_access_in extended permit udp object monitor_server any eq ntp
access-list OPT2_access_in extended permit icmp 10.20.3.0 255.255.255.0 object-group DM_INLINE_NETWORK_7
access-list OPT2_access_in extended permit object OSSEC object app_server object monitor_server
access-list OPT2_access_in extended permit object ossec_agent_auth object app_server object monitor_server
access-list OPT2_access_in extended permit tcp object-group DM_INLINE_NETWORK_3 10.20.3.0 255.255.255.0 eq ssh
access-list OPT2_access_in extended permit object-group DM_INLINE_SERVICE_1 object admin_workstation object app_server
access-list OPT2_access_in extended permit tcp any any eq ssh
access-list OPT1_access_in extended permit ip 10.20.2.0 255.255.255.0 10.20.1.0 255.255.255.0
access-list OPT1_access_in extended permit ip 10.20.2.0 255.255.255.0 10.20.3.0 255.255.255.0
access-list OPT1_access_in extended permit ip object app_server any
access-list OPT1_access_in extended permit ip object app_server object-group external_dns_servers
access-list OPT1_access_in extended permit udp object app_server any eq ntp
access-list OPT1_access_in extended permit icmp 10.20.2.0 255.255.255.0 object-group DM_INLINE_NETWORK_8
access-list OPT1_access_in extended permit object OSSEC object monitor_server object app_server
access-list OPT1_access_in extended permit object ossec_agent_auth object monitor_server object app_server
access-list OPT1_access_in extended permit tcp object-group DM_INLINE_NETWORK_2 10.20.2.0 255.255.255.0 eq ssh
access-list OPT1_access_in extended permit tcp any any eq ssh
access-list global_access extended permit udp object app_server object monitor_server
access-list ADMIN_access_in extended permit tcp object-group DM_INLINE_NETWORK_1 any eq ssh inactive
pager lines 24
logging asdm informational
mtu outside 1500
mtu inside 1500
mtu OPT1 1500
mtu OPT2 1500
mtu ADMIN 1500
mtu wifi 1500
mtu MGMT 1500
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
!
object network obj_any
nat (any,outside) dynamic interface
access-group inside_access_in in interface inside
access-group OPT1_access_in in interface OPT1
access-group OPT2_access_in in interface OPT2
access-group ADMIN_access_in in interface ADMIN
access-group wifi_access_in in interface wifi
access-group global_access global
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
user-identity default-domain LOCAL
http server enable
http 10.20.2.0 255.255.255.0 OPT1
http 10.20.4.0 255.255.255.0 ADMIN
http 192.168.45.0 255.255.255.0 MGMT
no snmp-server location
no snmp-server contact
service sw-reset-button
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5
ssh version 2
ssh key-exchange group dh-group1-sha1
console timeout 0

dhcpd auto_config outside
!
dynamic-access-policy-record DfltAccessPolicy

!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect ip-options
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:xxxxxxxxxxxxxxxxxxxxxxxxx
: end


#11

Hi @this-is-ian, tagging @mike for additional review on Monday.

This stuck out to me I might be incorrect: As i understand it, SecureDrop explicitly removes sha1 hashes from SSH because it is now considered “insecure”.

ssh stricthostkeycheck
ssh timeout 5
ssh version 2
ssh key-exchange group dh-group1-sha1

From the file install_files/ansible-base/roles/restrict-direct-access/files/ssh_config I believe we only allow the following:

 Cipher blowfish
   Ciphers aes256-gcm@openssh.com,aes256-ctr,chacha20-poly1305@openssh.com
   MACs hmac-sha2-256,hmac-sha2-512
     SendEnv LANG LC_*
     HashKnownHosts yes

Again, tagging Mike because I am very unfamiliar with Cisco network firewalls and their syntax.

Best,
Freddy M


#12

Thank you so much @bmeson and @mike. I’ve been KO’d with the flu but, back amongst the land of the living today. I’ll try giving it a shot from that end. Any other tips are greatly appreciated!


#13

Hey @this-is-ian - can you get a console on one of the boxes and get me the output from iptables? You might have better luck getting into single user mode (via Ubuntu rescue mode at boot --> shell).

$ sudo iptables -L -n -v


#14

Hi @mike. This is from the mon server.

slateadmin@mon:~$ sudo iptables -L -n -v
[sudo] password for slateadmin:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination


#15

The app server has the same results.


#16

Hi @this-is-ian,

I am a bit confused about the DNS being set to 9.9.9.9 in the configuration. Are you able to get a response to the host commands we have documented on both servers?


#17

Hi @bmeson! Yes, I was able to successfully test the connectivity in the Set Up the Servers step. However, after running through the next installation steps, I lost connectivity in the failed SecureDrop installation process.


#18

I’ve removed 9.9.9.9 from the DNS network objects as well.


#19

@bmeson Hi Freddy. Below is the email that I just sent to you. Thanks again.

Hi Freddy,

This is our current status…
• I reformatted the servers and ran the installation again.
• This time, the mon server seemed to complete. Post-install, I am able to ping and ssh to the server.
• The app server on the other hand, is not accessible from the admin workstation nor the mon server.
• I ensured that all configs on the Cisco ASA firewall matched in regards to access rules.
• Reboots of all devices, re-reviewing configs, etc.
• I addressed another workstation the same IP as the app server and plugged it in to the firewall port for the app server. I was able to ping this device and it had network connectivity.
• I used the same cable that I tested the workstation with and plugged it in to the app server’s NIC port. No luck. Unresponsive to pings/ssh attempts from the admin and mon machines.

I agree, I’m kind-of at a loss as well. My next step is to reformat both servers and try again.

Thanks,
Ian


Update:
• After updating some of the rules on the Cisco firewall, we reformatted the servers and gave the install another shot.
• We were able to proceed past the ‘waiting for network configuration’ hangup on both servers.
• The installation completes with one failed task for the app server. Failed task is
TASK [app : Initialize sqlite database.] *********************************************************************************************************************
fatal: [app]: FAILED! => {“changed”: true, “cmd”: “su -s /bin/bash -c “PYTHONPATH=/var/www/securedrop python -c ‘import db; db.init_db()’” www-data”, “delta”: “0:00:00.173039”, “end”: “2018-04-05 14:23:24.047817”, “failed”: true, “rc”: 1, “start”: “2018-04-05 14:23:23.874778”, “stderr”: “Traceback (most recent call last):\n File “”, line 1, in \nAttributeError: ‘module’ object has no attribute ‘init_db’”, “stderr_lines”: [“Traceback (most recent call last):”, " File “”, line 1, in ", “AttributeError: ‘module’ object has no attribute ‘init_db’”], “stdout”: “”, “stdout_lines”: []}

Any thoughts on the cause of this error? Also, is it possible to re-run just this task for the app server?

Thank you again!!!


#20

Hi @this-is-ian!

So, in the 0.6 release of SecureDrop, we changed the way that we setup the SQL database. I suspect that you have the0.5.x branch checked out and not the 0.6 branch. I would make sure you are on the latest code base and then try to install again:

cd ~/Persistent/securedrop
git fetch --tags
git checkout 0.6
gpg --recv-key "2224 5C81 E3BA EB41 38B3 6061 310F 5612 00F4 AD77"
git tag -v 0.6 # Output should include "Good signature"
./securedrop-admin setup
./securedrop-admin install 

Let me know if that helps,
Freddy