Friday 15 June 2018

Oracle Cloud: howto access the public web from private networks via Squid proxy

Building up on my howto guide 'Run a bastion host with ssh-forwarding on Oracle Cloud Infrastructure', this guide is for the opposite direction. Say you have a server in a private subnet, only accessible through the bastion host. If that host needs access to the public internet, a proxy in a public subnet of the VCN (Virtual Compute Network) could be a solution.

The situation is the same as in the former guide. Two hosts, one named bastion in a public subnet and one named webserver in a private subnet. The webserver host has no access to the internet, but needs to access some information from a public URL, so a proxy on the bastionhost needs to be set up.

Summary
  • Install and setup Squid on a host in the public subnet
  • Create an ingress rule for the public subnet to allow traffic to the Squid port
  • Create an egress rule to allow egress traffic for the private subnet
  • Open the Squid port on the Linux firewall of the Squid host
  • Set http_proxy to the Squid host:port
Step-by-step guide

This guide uses Apache Squid, but only uses it as a proxy server. So first install it via yum:

[root@bastion ~] yum install squid

Start Squid and check it's status:

[root@bastion ~] systemctl start squid
[root@bastion ~] systemctl status squid
● squid.service - Squid caching proxy
   Loaded: loaded (/usr/lib/systemd/system/squid.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-06-14 12:34:49 GMT; 10s ago
  Process: 11307 ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 11301 ExecStartPre=/usr/libexec/squid/cache_swap.sh (code=exited, status=0/SUCCESS)
 Main PID: 11309 (squid)
   CGroup: /system.slice/squid.service
           ├─11309 /usr/sbin/squid -f /etc/squid/squid.conf
           ├─11311 (squid-1) -f /etc/squid/squid.conf
           └─11312 (logfile-daemon) /var/log/squid/access.log

Jun 14 12:34:49 bastion systemd[1]: Starting Squid caching proxy...
Jun 14 12:34:49 bastion squid[11309]: Squid Parent: will start 1 kids
Jun 14 12:34:49 bastion squid[11309]: Squid Parent: (squid-1) process 11311...ed
Jun 14 12:34:49 bastion systemd[1]: Started Squid caching proxy.
Hint: Some lines were ellipsized, use -l to show in full.

If everything is fine, enable automatic Squid startup at boot. If you get errors, run a complete yum update and try again. That fixed my problems with startup.

[root@bastion ~] systemctl enable squid
Created symlink from /etc/systemd/system/multi-user.target.wants/squid.service to /usr/lib/systemd/system/squid.service.

To test it from the system Squid is running on, set the HTTP proxy to the Squid default port, which is 3128.

[opc@bastion ~]$ export http_proxy=http://127.0.0.1:3128 
[opc@bastion ~]$ export https_proxy=http://127.0.0.1:3128

Then open a page with curl or lynx, eg.

[opc@bastion ~]$ lynx https://www.oracle.com

That should return the text version of that page.


Check that the proxy has been used in the squid access log.

[root@bastion ~]# tail -f /var/log/squid/access.log
1528980628.887     45 127.0.0.1 TCP_MISS/200 3125 GET http://www.google.com/ - HIER_DIRECT/172.217.5.228 text/html
1528980661.842      5 127.0.0.1 TCP_MISS/301 314 GET http://www.oracle.com/ - HIER_DIRECT/23.15.129.79 -

Now that the squid proxy works and is accessible locally, create a security role on OCI to give the webserver host access to the Squid proxy on the bastionhost.


Find the public subnet in your VCN and click on the security list.


Click on 'Edit all rules' and add the network CIDR block of the webserver subnet (alternatively only the host IP address).


After clicking on 'Save Security List Rules', these should be listed in the Security List.


Also make sure, that in the private networks security list egress traffic is allowed at least for port 3128 to the private network, or like in this case allow all egress traffic. In the default security list, egress traffic is allowed. If you have created your own security list, you might need to add an egress rule.

firewall-cmd --new-zone=webapp --permanent
firewall-cmd --reload
firewall-cmd --zone=webapp --add-source=10.11.111.132/32 --permanent
firewall-cmd --zone=webapp --add-port=3128/tcp --permanent
firewall-cmd --reload

The last step is to open the Squid port on the public host in the Linux firewall.

[opc@webserver ~]$ curl www.google.com
^C
[opc@webserver ~]$ export http_proxy=10.11.111.148:3128
[opc@webserver ~]$ curl www.google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description">....

Now check, if everything works. Try to open any public URL, eg. with curl. That should not work. Wait for the timeout or CTRL-C. Set the proxy to the Squid server and port and try again. That should work now, if not double-check all settings.
So now, the server in the private network with no access to the public internet can access the web through the Squid proxy in a public subnet of your VPN.

Sunday 10 June 2018

How to run Python from Notepad++

Notepad++ can easily be configured to run Python. This is practically the same, as my former 'Running Oracle Cloud Stack Manager CLI from Notepad++' howto. But there are some minor updates.
Most important, all required plugins are now available for 64Bit Notepad++. so install the Notepad++ version of choice.

Install Plugin Manager

The nppPluginManager is now on GitHub. Just follow the steps (copy PluginManager.dll to the plugins and gpup.exe to the updater directory) from the README.md. Restart Notepad++

Install NppExec

Open Plugin Manager (Plugins|Plugin Manager|Show Plugin Manager), pick the NppExec Plugin from the list of available Plugins and click 'Install'. After restart, it should be available.


4. Save a command for Python and run it

With some meaningful Python code opened, hit F6 in Notepad++. Enter the following command for running Python (adjust the path to your environment) and save it.

c:\dev\Python\Python36\python.exe "$(FULL_CURRENT_PATH)" 


Clicking OK opens a console window and runs Python with your last saved code version.

5. Re-run Python while editing

Via CTRL-F6, the last command can be re-run, don't forget to CTRL-S first to save your code.


Friday 8 June 2018

Run a bastion host with ssh-forwarding on Oracle Cloud Infrastructure

This short howto guide will show how to setup ssh forwarding for Oracle Linux on OCI and access from Windows via PuTTY.
The idea of a bastion host is to have a single entry point into your network. The bastion host is the only one that is accessible from outside the network, there is no sensible data on it. All applications and data reside on private hosts, that are not accessible from the outside. To access the applications, ssh into the bastion host and from there ssh to the specific private host. Firewall rules will block all unwanted traffic.
There are several options to do the key handling. One obvious one would be, to install all private keys on the bastion host. But if there would be a successful attack onto the bastion host, the attacker would have access to the private hosts.
Better would be, to use ssh forwarding. Here, the keys will remain on the administrators PC, out of reach for an attacker on the bastion host.


Consider a VCN (Virtual Compute Network) with two subnets, one public and one private.


These contain two compute instances. The bastionhost in the bastion-net with a public IP address and the applicationhost in the private-server-net with no public access.


To access SSH through the firewall, port 22 (ssh) needs to be added to the security lists. By using the Default Security List, this is enabled by default.

login as: opc
Authenticating with public key "putty-LASC-key"
[opc@bastionhost ~]$ sudo su
[root@bastionhost opc]# vi /etc/ssh/sshd_config

To enable ssh forwarding from the bastionhost, edit  /etc/ssh/sshd_config.


Uncomment the line containing 'AllowAgentForwarding yes' and save the file.

systemctl restart sshd.service

Restart the sshd to make the changes effective.


Check both boxes under 'Authentication parameters' to enable SSH forwarding in PuTTY.


To tell PuTTY, which ssh-key to forward, start Pageant which comes with your PuTTY installation. Add the private key for the private host. Close your PuTTY SSH session and open a new one to the bastion host.


Now it is possible to ssh from the bastionhost to the applicationhost without the need for private keys on the bastionhost because these are forwarded from the adminstration PC.


Friday 1 June 2018

Run fnproject on Oracle Linux 7.5

A great thing about fn is that it runs on your (Linux-)Desktop. So for a local installation, Docker needs to be installed first. On Oracle Linux this is an easy task, as everything needed is already available in the yum-repositories. Just make sure, that all needed repositories are enabled.

[root@localhost] yum-config-manager --enable ol7_preview

Docker 17.12.1 is found in the ol7_preview repository, so enable that one.

[root@localhost] yum-config-manager --enable ol7_addons

Docker depends on container-selinux, which is found in the ol7_addons repository.

[root@localhost ~]# yum install docker-engine
...
================================================================================================================
 Package                      Arch              Version                            Repository              Size
================================================================================================================
Installing:
 docker-engine                x86_64            17.12.1.ol-1.0.3.el7               ol7_preview             31 M
Installing for dependencies:
 container-selinux            noarch            2:2.21-1.el7                       ol7_addons              28 k

Transaction Summary
================================================================================================================
...
Installed:
  docker-engine.x86_64 0:17.12.1.ol-1.0.3.el7                                                                   

Dependency Installed:
  container-selinux.noarch 2:2.21-1.el7                                                                         

Complete!


Now that we have the right repositories enabled, just yum install docker-engine to get docker.

[root@localhost ~]# systemctl start docker

Start Docker (and systemctl enable docker, if you want to start it automatically). Then follow the Post-installation steps for Docker.

[oracle@localhost ~]$ sudo usermod -aG docker $USER

The docker group is already available, but the non-root user to run Docker needs to be added (and don't forget to logout/logon again, if you already are that user).

[oracle@localhost ~]$ docker run hello-world

Hello from Docker!

Check, that Docker is running correctly, if you like.
Now that Docker is running, just follow the fn installation guide to get fn.

[oracle@localhost ~]$ curl -LSs https://raw.githubusercontent.com/fnproject/cli/master/install | sh

fn version 0.4.113

        ______
       / ____/___
      / /_  / __ \
     / __/ / / / /
    /_/   /_/ /_/`

This only takes seconds.

[oracle@localhost ~]$ fn start
Unable to find image 'fnproject/fnserver:latest' locally
latest: Pulling from fnproject/fnserver
ff3a5c916c92: Pull complete 
...
Digest: sha256:b1a330d24f07c8297fe8e9ba275eda6e04bf814a20b221be87bb492e9da2ff61
Status: Downloaded newer image for fnproject/fnserver:latest
time="2018-06-01T14:45:37Z" level=info msg="Setting log level to" level=info
time="2018-06-01T14:45:37Z" level=info msg="datastore dialed" datastore=sqlite3 max_idle_connections=256
time="2018-06-01T14:45:37Z" level=info msg="agent starting cfg={MinDockerVersion:17.10.0-ce DockerNetworks: FreezeIdle:50ms EjectIdle:1s HotPoll:200ms HotLauncherTimeout:1h0m0s AsyncChewPoll:1m0s CallEndTimeout:10m0s MaxCallEndStacking:8192 MaxResponseSize:0 MaxRequestSize:0 MaxLogSize:1048576 MaxTotalCPU:0 MaxTotalMemory:0 MaxFsSize:0 PreForkPoolSize:0 PreForkImage:busybox PreForkCmd:tail -f /dev/null PreForkUseOnce:0 PreForkNetworks: EnableNBResourceTracker:false MaxTmpFsInodes:0 DisableReadOnlyRootFs:false}"
time="2018-06-01T14:45:37Z" level=info msg="no docker auths from config files found (this is fine)" error="open /root/.dockercfg: no such file or directory"
time="2018-06-01T14:45:37Z" level=info msg="available memory" availMemory=13503231591 cgroupLimit=9223372036854771712 headRoom=1500359065 totalMemory=15003590656
time="2018-06-01T14:45:37Z" level=info msg="sync and async ram reservations" ramAsync=10802585273 ramAsyncHWMark=8642068218 ramSync=2700646318
time="2018-06-01T14:45:37Z" level=info msg="available cpu" availCPU=1000 totalCPU=1000
time="2018-06-01T14:45:37Z" level=info msg="sync and async cpu reservations" cpuAsync=800 cpuAsyncHWMark=640 cpuSync=200
time="2018-06-01T14:45:37Z" level=warning msg="Severaly Limited CPU: cpuAsync < 1000m (1 CPU)"
time="2018-06-01T14:45:37Z" level=info msg="Fn serving on `:8080`" type=full

        ______
       / ____/___
      / /_  / __ \
     / __/ / / / /
    /_/   /_/ /_/
        v0.3.460

The whole installation is straightforward and only takes minutes (or seconds, if you already have Docker installed), then fn is ready to run. For first steps follow the official tutorial.