Securing OpenClinica

Securing OpenClinca

Installing, configuring and securing the OpenClinica [OC] Community Edition [CE] is definitely an adventure and requires knowledge in a variety of aspects of computer science. With extensive googling and dedication, however, it can be done by anyone who has some affinity working with Linux. I am no computer scientist by trade, but nevertheless took on the path of deploying OC on a webserver and, in the end, managed to get it running successfully. There were points though, where I wished there was an extensive guide on how to manage with the process. With this post I would like to fill this gap, at least with the security aspect of OC, to give a helping hand to anyone attempting the same task.

The list of measures I took are what I could find from numerous sources and I cannot claim responsibility that they are exhaustive of everything that can be/should be done when securing OC. As I mentioned already, I am not an expert in the field, this guide is merely a result of a lot of searching and trial-and-error. Take everything you see here with a grain of salt, and, if you feel you know better, don't hesitate to change aspects of it. Please also feel free to suggest improvements/additions to this process so we can all benefit from this collective knowledge.

I took security as seriously as possible as the software required by the CE of OC is long out-of-date as of 2020, therefore doing everything possible to make it secure is of great importance, especially when working with patient data.

My setup-of-choice was the following:

• OpenClinica CE 3.15
• CentOS 8
• PostgreSQL 8.4.22
• Tomcat 7.0.52
• JVM 1.7.0

I followed this installation guide as I could not find one for v3.15. Getting some of the specific software version listed above was quite a challenge and if I had to restart I would consider going with CentOS 6 instead, as this version is natively supporting some of these older dependencies. CentOS 6 is also out of support as of this year, therefore consider carefully when opting in for it.

As per instructed by the above mentioned guide, the root folder to all the components required to run OC is at: /usr/local

Without further adue, here we go:

SSH

First and foremost accessing the webserver (if it is a remote one) should be secured. There are hundreds of attempts made each day to log-in to webservers using common usernames, therefore this really should be your first line of security.

Modifying the sshd_config file

This file can be found under `/etc/ssh/sshd_config` and controls ssh access to your server. Before making any changes to it, make sure you that you have it backed up. In the file, modify the following parameters:

Only allow the usernames which will need to have access to your server

• AllowUsers <username1> <username2>

Disable root logins

• PermitRootLogin no

• ChallengeResponseAuthentication no

After copying your ssh key to the server (with ssh-copy-id or manually, by copying your public key and pasting it in the `~/.ssh/authorized_keys` file on your server) you should disable password authentication. This way signing in is only possible by ssh key authentication.

• PasswordAuthentication no

For added security, UsePAM can be also set to `no`, however, this messed with my ssh authentication process and had to set it back to `yes`.

After everything is configured restart the sshd service:

sudo systemctl restart sshd

fail2ban

This package provides added SSH security by banning IPs which failed ssh authentication x number of times. Here is a great guide to configuring fail2ban.

Installation

sudo yum install epel-release

sudo yum install fail2ban

Under `/etc/fail2ban` create a file named `jail.local` and paste the following in (use your own e-mail address):

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600
maxretry = 5

[sshd]
enabled = true

destemail = <your@email.com>
sender = <your@email.com>
sendername = Fail2ban
mta = sendmail
action = %(action_mwl)s

Creating this file instead of configuring directly in `jail.conf` is preferred as a fail2ban update would override your configuration otherwise. Now you will receive e-mail notifications when an IP was banned after 5 failed log-in attempts or when fail2ban was stopped/started.

Restart the fail2ban service: sudo systemctl restart fail2ban

If you would like to receive additional information of the banned IP-s, you can install the `whoami.x86_64` package.

You can check the sshd jail status by: sudo fail2ban status sshd

Passwords

While it might sound obvious, it is nevertheless vital to maintain strong and unique passwords on your server. I recommend using KeePass to keep track of them as it also allows you to auto-generate seriously strong passes.

Tomcat

Securing Tomcat is just as vital as securing your webserver, as this is your point of contact with the outside world. I gathered the below listed security measures from the following sites

Access control

This is to make sure that even if an attacker can get hold of the webserver, there is only minimum damage she can do. First and foremost never run your webserver as the root user, create a new user called `tomcat` which has the minimum necessary privileges to run the server.

Make the folder `tomcat` and all its contents owned by the user tomcat and group tomcat:

chown -R tomcat:tomcat /usr/local/tomcat

(without the -R flag you only change the ownership of the folder, but not the contents)

Give the owner of the folder read write and execute permissions:

chmod -R 700 /usr/local/tomcat

(Guide on using chmod: https://www.lifewire.com/uses-of-command-chmod-2201064) Word of caution: to be able to open a folder you need to have execute permission on it. I learned it the hard way as well.

Remove write and execute permissions from the configuration folder. This prevents an attacker to modify the configuration files of Tomcat:

chmod -R u-wx /usr/local/tomcat/conf

However, Tomcat needs to be able to open the folder, therefore add execute permission TO THE FOLDER ONLY (=without the -R flag):

chmod u+x /usr/local/tomcat/conf

Remove read permission from the openclinica.config and openclinica-ws.config (if installed) folders:

chmod -R u-w openclinica*.config

Remove read permission from the logs folder:

chmod -R u-r /usr/local/tomcat/logs

Remove all permissions from the oldwebapps folder, as during deployment Tomcat will not have to do anything with these:

chmod -R u-rwx /usr/local/tomcat/oldwebapps

Set the `datainfo.properties` file to read-only (do the same of OpenClinica-ws if installed):

chmod 400 /usr/local/tomcat/webapps/OpenClinica/WEB-INF/classes/datainfo.properties

With this you are done with the access control aspect of Tomcat.

One additional thing you can do is to add your user to the tomcat group and grant read write execute privileges to group for the usr/local/tomcat folder. This way you can still conveniently edit everything without having to use root/sudo. I am really unsure about the security aspect of it and use it at your own risk. And do not forget to remove yourself from the group and remove the privileges once everything is configured.

HTTPS

As referenced above already, Tomcat's guide on setting up SSL should be the first step you do (https://tomcat.apache.org/tomcat-7.0-doc/ssl-howto.html). You will have to make sure that all the steps laid out in this guide are done using the user `tomcat`, as if it is done otherwise, the .keystore file will be created under that other user's home directory and tomcat won't be able to find and read it. Theoretically you could also probably comment out the HTTP connector as enforcing using HTTPS for OC is good practice. Port forwarding will be enabled on a firewall level, therefore in theory Tomcat should never receive a request to port 8080, but I haven't tested this in action yet.

After this is configured you should be able to use HTTPS to communicate with your webserver (on https://localhost:8443/OpenClinica). If you are using a self-signed certificate your browser will complain, but you can nevertheless access the site. You can enforce using HTTPS within Tomcat, by adding the following at the end of your web.xml file under `tomcat/conf` (before the </web-app> tag):

<security-constraint>
  <web-resource-collection>
    <web-resource-name>Protected Context</web-resource-name>
    <url-pattern>/*</url-pattern>
  </web-resource-collection>
  <user-data-constraint>
    <transport-guarantee>CONFIDENTIAL</transport-guarantee>
  </user-data-constraint>
</security-constraint>

Cookies, Custom Error Pages, Disabling listings ..

These are good to have practices, but not must haves. These include hiding server version from error pages or disabling file system listings to better protect you from DDoS attacks.

All of the following will need to be added to the `web.xml` file.

Secure and HTTP only cookies to prevent XSS attacks

(paste before the </web-app> tag)

<cookie-config>
<http-only>true</http-only>
<secure>true</secure>
</cookie-config>

Custom Error pages

(paste before the </web-app> tag) (No error page will be displayed as Tomcat won't be able to locate the error page file, but it does get the job done by hiding the version number. If this is an issue for you skip this step.)

<error-page>
<error-code>404</error-code>
<location>/error.jsp</location>
</error-page>

<error-page>
<error-code>403</error-code>
<location>/error.jsp</location>
</error-page>

<error-page>
<error-code>500</error-code>
<location>/error.jsp</location>
</error-page>

<error-page>
<exception-type>java.lang.Exception</exception-type>
<location>/error.jsp</location>
</error-page>

<error-page>
<exception-type>java.lang.Throwable</exception-type>
<location>/error.jsp</location>
</error-page>

Read-only resources and no listings

(paste within the DefaultServlet part of the file)

<init-param>
<param-name>readonly</param-name>
<param-value>true</param-value>
</init-param>

<init-param>
<param-name>listings</param-name>
<param-value>false</param-value>
</init-param>

server.xml

Removing server banner

If you add the parameter `Server=" "` to the parameter-list of a connector, this hides the version number in HTTP headers. It should look something like this:

<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" Server =" " address="<webserver IP address or localhost>" clientAuth="false" sslProtocol="TLS" keystoreFile="${user.home}/.keystore" keystorePass="<your keystore pass>" />

Changing shutdown command

To prevent attackers from being able to shut-down your server you should consider changing the shutdown port as well as the shutdown command. This you can do in the beginning of your server.xml file within the <Server> tag:

<Server port="any unused port" shutdown="<long and secure string>">

AJP connector

If OC is the only webapp that will be deployed in Tomcat, you can safely comment out the AJP connector line, as OC is not using this connector.

Possibly good practices, but caused problems for me

Disabling autodeployment

To prevent someone from auto-deploying their own, malicious, webapps in your Tomcat instance, you can turn auto-deployment off within the server.xml file by setting the following parameters to false within the <Host> tag.

autoDeploy="false" deployOnStartup="false" deployXML="false"

This way if you restart your server you manually have to deploy OC, which you will have to figure out how to do.

Secure container

With tomcat you have the option to start your instance in a container which ensures that if anyone compromises the server, even in the worst-case scenario, they can only access resources and files within this container. It is, however, something you will have to spend time on to make work for OC, as it is known to (and have, for me) break an installation. It is probably worthwhile to do if you have the time and knowledge to make it work.

The usage is otherwise really simple, when starting the webserver add the -secure flag:

/usr/local/tomcat/bin/startup.sh -secure

That's it for Tomcat.

PostgreSQL

There is not a lot you can confgire for PostgreSQL in terms of security, but a little bit of access control can be done here as well.

Change the ownership of the pgsql folder:

sudo chown -R postgres:postgres /usr/local/pgsql

Change priviliges:

sudo chmod -R 700 /usr/local/pgsql

You also should be carefully configuring the pg_hba.conf file at `/usr/local/pgsql/data/` . This file controls the different access rights to your database. You need to give access to the openclinica database for the clinica user connecting locally, but block all other connection attempts. The authentication method should be set to `md5`, never used `password` as it sends these in clear text form. If you will be accessing the database directly from remote machines you will have to add the appropriate line here for that.

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# allow user clinica to connect to openclinica locally, using encrypted password based authentication (needed to automate backups)
local   openclinica clinica                           md5
# IPv4 local connections:
host    openclinica clinica     127.0.0.1/32          md5
# "local" is for Unix domain socket connections only
local   all         all                               reject
# IPv6 local connections:
host    all         all         ::1/128               reject

This configuration is also important for the automated backups, which will be described below.

Firewall

Setting up a firewall is a must for any webserver. This allows you to limit and monitor all traffic happening between your server and the outside world. The main goal here is to leave only those ports open which are necessary for your webapp to function. In this case it means:

• port 80/tcp, where incoming http traffic is expected,
• port 443/tcp, where incoming https traffic is expected,
• port 22/tcp, where ssh traffic is taking place.

You will might have to do some trial-and-error to see if closing a certain port breaks anything, but for me allowing only these 3 open did the trick.

FirewallD

If you already are comfortable using FirewallD, you can skip to the next section (Setting up FirewallD).

CentOS comes with FirewallD preinstalled which is a powerful tool to manage these connections. There are plenty of resources online to learn more about it, such as this.

(Update: one phenomenon described in this article is outdated. Packets which could not be handled by their assigned zone are not "kicked-up" to the next zone (the notion of "zone-drifting"). It is considered insecure and will be removed in future releases of FirewallD. If you really would like to have this behaviour you can enable zone drifting in FirewallD-s config file)

First make sure that firewalld is installed and enabled on your system. The default zone enabled is `Public` which you can keep this way for our purposes. You can check which services are enabled by default in this zone:

sudo firewall-cmd --zone=public --list-all

Services are not some special entities, they are merely representing the ports the given service needs open to be able to function. These are defined under `/usr/lib/firewalld/services` as self-explanatory xml files. If you would like to define your own services, you can do this by copying one of these xml files to `/etc/firewalld/services` and customising it to your own needs. You can add any service to your firewall zone by:

sudo firewall-cmd --zone=public --add-service=<your service> --permanent

And you can remove any unnecessary services by:

sudo firewall-cmd --zone=public --remove-service=<unnecessary service> --permanent

You can also add/remove directly the ports you need:

firewall-cmd --zone=public --add-port=<your_custom_portnumber>/tcp --permanent

Without the permanent flag the changes will be reset once FirewallD is reloaded or restarted. You can experiment with rules without the flag and once you found out what works for you, you can finalise these rules by adding the --permanent flag.

Setting up FirewallD

The below configuration assumes that you are using ssh to access your remote server, however, if this is not the case, you can skip setting up an internal zone.

There will be three zones set up for the firewall: public, internal and trusted. The public should handle all http/s requests from any requesting IP address. The internal zone should have a source added with either your IP or MAC address, therefore (in theory) only when your machine is communicating with the server should this zone handle the data transfer. This is also the zone with the ssh service enabled (=having port 22 open).

Before you start with the configuration of FirewallD you should make sure that iptables is disabled and stop it from ever starting by masking it. Otherwise it might interfere with FirewallD and lead to some odd behaviour.

systemctl mask iptables

systemctl disable iptables

First start by removing any unnecessary services from the public and internal zones. Use the --list-all flag to see what is allowed at the moment. You should only leave (or add, if necessary) http/https there. In the internal zone also add the ssh service.

You should assign your outward facing network interface to the public zone (so all request arriving from outside will be handled by this zone). You can list all the available interfaces by:

ip link show

You can add an interface by using the --add-interface=<interface> flag.

Unfortunately the naming of these interfaces is non-trivial (at least for me, mine is called `ens192`) so you'll might have to do some research to see what's what.

You should also add your own IP/MAC to the sources of the internal zone, so only request coming from this IP/MAC address will be handled by this zone, e.g.:

sudo firewall-cmd --zone=internal --permanent --add-source=154.112.12.18

You can also make this IP more general if you want ssh to be available e.g. from a certain VPN:

sudo firewall-cmd --zone=internal --permanent --add-source=154.112.0.0/16

Which will allow anyone from 154.112.0.0-154.112.255.255 to use ssh. (Note that zones are only active if there is at least 1 interface or source assigned to them!)

The target of the zone public should be set to DROP, which returns no message upon a request arriving to an invalid port, instead of transmitting a reject message. This is considered more secure.

sudo firewall-cmd --permanent --zone=public --set-target=DROP

Next, create port-forwarding rules to limit communication to https as well as to accommodate to Tomcat's port conventions (it uses 8080 instead of 80 and 8443 instead of 443):

sudo firewall-cmd --zone=public --permanent --add-forward-port=port=443:proto=tcp:toport=8443
sudo firewall-cmd --zone=public --permanent --add-forward-port=port=80:proto=tcp:toport=8443

sudo firewall-cmd --zone=internal --permanent --add-forward-port=port=443:proto=tcp:toport=8443
sudo firewall-cmd --zone=internal --permanent --add-forward-port=port=80:proto=tcp:toport=8443

For port-forwarding to work you need to also enable masquerading (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-port_forwarding):

sudo firewall-cmd --zone=public --add-masquerade --permanent

sudo firewall-cmd --zone=internal --add-masquerade --permanent

Now this step is something I am less sure about as I could not find a lot of info about it. I assigned the lo (loopback) interface to the trusted zone. This is the interface for communicating with your webserver from the same machine (localhost, 127.0.0.1). Ideally this is inaccessible from the outside world, therefore I assumed that the trusted zone is appropriate for it, but again, I might be wrong here.

If you need pinging to work, run this (replace zone by the zone in which you need pinging to work):

sudo firewall-cmd --permanent --zone=<YOUR_PINGING_ZONE> --add-rich-rule='rule protocol value="icmp" accept'

You have to reload firewalld for the changes to take effect:

sudo firewall-cmd --reload

Now you have FirewallD configured to only allow http/s communication with the outside world, but still allow ssh connections from your personal machine / your VPN network.

You can test your firewall using the nmap package to see which ports are open.

OpenClinica

You have done most of what you can do to make OC secure already, but there are a few things you can still do within OC itself. If you sign in as root to the web interface, navigate to `Tasks/Administartion/Users/Configure Password Requirements`, there will be a few options which you can turn on/off and tweak to suit your needs. You should also definitely enable user lockout after x number of tries (`Tasks/Administartion/Users/Lockout`). As root you can unlock these accounts on the web interface if they get locked. Another measure I myself took, but unsure if it is a necessity: asked the users of my OpenClinica instance to set their password challenge question to a long (unguessable) string. This I did, as I am unsure how OC sends out the password reset e-mail and if this e-mail could be hijacked by a potential attacker. A much safer (although admittedly more inconvenient, even impractical if you have too many users) solution is to, as root, reset their password from the web interface and send them yourself.

Backups

This also really should be a must. The solution I share here is just one way of doing it, if you know better, don't hesitate to do it your way.

Here is a guide is a guide on how backups should be implemented in OC. Basically you want to periodically save your OC configuration file, your study directory ($TOMCAT_ROOT/openclinica.data) and your PostgreSQL database.

I have attached a script which, if executed, should backup all of these, wrap them up in a tarball file and save it to `/usr/local/OpenClinica_backups` . You can (and should) change the backup location to be outside of your server by editing the BCKP_path variable in the script. Additionally you will have to create a .pgpass file for it to work, more details below.

My addition to the above linked guide is to make these backups happen automatically, e.g. on a daily basis. For this I used the `cron` service, which comes preinstalled on CentOS 8.

To make your script execute daily, place it under `/etc/cron.daily/` . Normally, to be able to get a pg_dump you would need the password for your openclinica database. To allow automatic updates you need to create the .pgpass file under `/var/lib/pgsql/` . The content of it should be as follows:

localhost:5432:openclinica:clinica:<YOUR_DATABASE_PASSWORD>

For Postgres to be able to use this file it HAS to be owned by postgres:

sudo chown postgres:postgres .pgpass

and the permissions on it HAVE to be set to 0600:

sudo chmod 0600 .pgpass

Should you have any problems with authentication, see the documentation of pgpass .

To be able to create the pg_dump, postgres needs to be able to write into your backup folder. Change the group of the folder to be postgres and give group rwx permissions:

sudo chown root:postgres <YOUR_BACKUP_FOLDER>

sudo chmod g+rwx <YOUR_BACKUP_FOLDER>

You should practice restoring your database to see how it is actually done. The above mentioned guide helps you with this.

Antivirus

There is this excellent GitHub repo which lets you run daily scans to detect various Malware, including Trojans, viruses or rootkits. If you place the script from the cloned repo under /etc/cron.daily. It will be executed on a daily basis alongside with the auto-backup script.

One Last Crumble

Just to make a clear job, it's a good idea to disable logins for the tomcat, clinica and postgres users:

usermode -L tomcat
usermode -L clinica
usermode -L postgres

That's it

Well done, you just made your OpenClinica installation a lot more secure! Unfortunately, you have to keep in mind that the software we have been hardening is outdated software (very outdated), so you can only do so much and hope for the best. The best advise I can give is to, if possible, do not include information of your subject which allows the identification of them. This way, even if everything fails, at least your subjects do not have to worry over personal identity thefts.

Auto backup script

#!

BCKP_path='/usr/local/OpenClinica_backups'
DATE=`date +"%Y-%m-%d"`

cd $BCKP_path
# create a database dump with pg_dump
sudo -u postgres /usr/local/pgsql/bin/pg_dump -U clinica openclinica -w > pg_dump
# backup the data directory of OC with CRF, XML, etc.. data
sudo tar -cf oc_data.tar /usr/local/tomcat/openclinica.data
# backup OC configuration
sudo cp /usr/local/tomcat/openclinica.config/datainfo.properties datainfo.properties

# tar all the above created files and assign a date
sudo tar -czf ${DATE}_openclinica_backup.tar.gz datainfo.properties  oc_data.tar  pg_dump

# remove created temporary files
sudo rm datainfo.properties
sudo rm oc_data.tar
sudo rm pg_dump

# add a bit of security
sudo chmod 400 *.tar.gz