Lab 5 - Apache Web Services lab¶

Welcome to the Web Services lab. Here's a quick overview of this week's tasks:

1. Validation of previous week's tasks
2. Introduction to web services
3. HTTP protocol
   • HTTP Flow
   • HTTP Messages
   • HTTP Headers
4. Setting up a personal web server
   • Understanding DNS CNAME records
   • Virtual Web Hosts Using Apache Web Server
   • Utilizing a web server as a proxy
5. WordPress setup
6. Configuring apache modules
   • Forensic logging
   • Security-hardening our webserver using mod_security
7. Ansible tips
   • Tags
   • Recommended modules
   • Handlers

1. Validation of previous week's tasks¶

Please make sure the following tasks have been completed before starting this lab, as we are using services from the previous labs:

• Personal domains are configured. Machine is accessible over <machine_name>.sa.cs.ut.ee inside the University network.
• All the tests on scoring.sa.cs.ut.ee are green.

For helping you to better debug and view the web services you'll be setting up this week, it's also ideal to validate your personal machine's DNS servers.

2. Introduction to web services¶

Before we can host our own web services, we need to understand how the browsers work. As most of the web works by utilizing the HTTP protocol, we'll first have a look into that.

2.1 HTTP protocol¶

HTTP is a protocol for fetching resources such as HTML documents. It is the foundation of any data exchange on the Web and it is a client-server protocol, which means requests are initiated by the recipient, usually the Web browser.

Each individual request is sent to a server, which handles it and provides an answer called the response. Between the client and the server there are numerous entities, collectively called proxies, which perform different operations and act as gateways or caches, for example.

In reality, there are more computers between a browser and the server handling the request: there are routers, modems, and more. Thanks to the layered design of the Web (OSI layers), these are hidden in the network and transport layers. HTTP is on top, as the application layer. Although important for diagnosing network problems, the underlying layers are mostly irrelevant to the description of HTTP.

To display a Web page, the browser sends an original request to fetch the HTML document that represents the page. It then parses this file, making additional requests corresponding to execution scripts, layout information (CSS) to display, and sub-resources contained within the page (usually images and videos). The Web browser then combines these resources to present the complete document, the Web page. Scripts executed by the browser can fetch more resources in later phases and the browser updates the Web page accordingly.

A Web page is a hypertext document. This means some parts of the displayed content are links, which can be activated (usually by a click of the mouse) to fetch a new Web page, allowing the user to direct their user-agent and navigate through the Web. The browser translates these directions into HTTP requests and further interprets the HTTP responses to present the user with a clear response.

Something to also mind is that HTTP is stateless (but not sessionless). This means that it's up to the client to track the success of requests. For doing this, every response given by the servers also contains a response code, which is an indication of whether something went wrong and what.

Examples of status codes: - 200 - OK

• 401 - Unauthorized (the client must authenticate itself to get the requested response)

• 403 - Forbidden (The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource.)

• 404 - Not found (Requested resource is not found on the server)

• 405 - Method Not Allowed (HTTP Method used is not allowed by the server)

• 418 - I'm a teapot (The server refuses the attempt to brew coffee with a teapot.)

2.2 HTTP Flow¶

When a client wants to communicate with a server, either the final server or an intermediate proxy, it performs the following steps:

1. Open a TCP connection: The TCP connection is used to send a request or several, and receive an answer. The client may open a new connection, reuse an existing connection, or open several TCP connections to the servers.

2. Send an HTTP message: HTTP messages (before HTTP/2) are human-readable. With HTTP/2, these simple messages are encapsulated in frames, making them impossible to read directly, but the principle remains the same. For example:

   GET / HTTP/1.1\ Host: developer.mozilla.org\ Accept-Language: fr

3. Read the response sent by the server, such as:

   HTTP/1.1 200 OK\\
   Date: Sat, 09 Oct 2010 14:28:02 GMT\\
   Server: Apache\\
   Last-Modified: Tue, 01 Dec 2009 20:18:22 GMT\\
   ETag: "51142bc1-7449-479b075b2891b"\\
   Accept-Ranges: bytes\\
   Content-Length: 29769\\
   Content-Type: text/html\\
   \\
   <!DOCTYPE html... (here come the 29769 bytes of the requested web page)
   

4. Close or reuse the connection for further requests.

HTTP Messages¶

There are two types of HTTP messages, requests, and responses, each with its own format.

An example HTTP request

Requests consist of the following elements:

• An HTTP method, usually a verb like GET, POST, or a noun like OPTIONS or HEAD that defines the operation the client wants to perform. Typically, a client wants to fetch a resource (using GET) or post the value of an HTML form (using POST), though more operations may be needed in other cases.

• The path of the resource to fetch; the URL of the resource stripped from elements that are obvious from the context, for example without the protocol (http://), the domain (here, developer.mozilla.org), or the TCP port (here, 80).

• The version of the HTTP protocol.

• Optional headers that convey additional information for the servers.

• A body, for some methods like POST, similar to those in responses, which contain the resource sent.

An example HTTP response

Responses consist of the following elements:

• The version of the HTTP protocol they follow.

• A status code, indicating if the request was successful or not, and why.

• A status message, a non-authoritative short description of the status code.

• HTTP headers, like those for requests.

• Optionally, a body containing the fetched resource.

2.4 HTTP Headers¶

HTTP headers let the client and the server pass additional information with an HTTP request or response, on top of the actual body. An HTTP header consists of its case-insensitive name followed by a colon (:), then by its value. Whitespace before the value is ignored.

Headers can be grouped according to their contexts:

• Request headers contain more information about the resource to be fetched, or about the client requesting the resource.

• Response headers hold additional information about the response, like its location or about the server providing it.

• Representation headers contain information about the body of the resource, like its MIME type, or encoding/compression applied.

• Payload headers contain representation-independent information about payload data, including content length and the encoding used for transport.

Examples of more important headers:

• Authorization: Basic <credentials> - Allows to send authorization information with headers.

• Wiewport-Width: <number> - Upon a request to a site, tell it how wide the screen is for appropriate site rendering in the browser.

• Accept: <MIME_type>/<MIME_subtype> - In a request, tell the server which kind of data is allowed to be returned.

• Set-Cookie: name=value - This HTTP response header is used to send a cookie from the server to the user agent so that the user agent can send it back to the server later. Very useful for persistent sessions.

• Content-Type: text/html; charset=UTF-8 - Indicates the type of content which is sent in the message body.

There's a large amount of other headers, that are being widely used. Which headers browsers or web servers accept depends on the software used, and even though most headers are agreed upon nowadays, there are still differences between the different browsers. This makes web developer's job a nightmare at times.

3. Setting up a personal web server¶

3.1 Understanding DNS CNAME records¶

In some situations there is a need for one system to have multiple DNS names, all pointing to the same IP. This could be the case when hosting multiple websites or running multiple services on the same IP using service-specific host names, like:

• www.<vm_name>.sa.cs.ut.ee

• ftp.<vm_name>.sa.cs.ut.ee

• myservice.<vm_name>.sa.cs.ut.ee

As we have a single machine, that's exactly what we will do - for every service, add a new record. For our main domain, we already have something like this.

• student-test.sa.cs.ut.ee IN A 193.40.154.247

• 247.154.40.193 IN PTR student-test.sa.cs.ut.ee

... and for the rest of the hostnames which we are going to use, we will do the following.

• www.student-test.sa.cs.ut.ee IN CNAME student-test.sa.cs.ut.ee

• ftp.student-test.sa.cs.ut.ee IN CNAME student-test.sa.cs.ut.ee

• myservice.student-test.sa.cs.ut.ee IN CNAME student-test.sa.cs.ut.ee

NB! Make sure you think through where you put a . in the end!

A CNAME record maps a domain name (alias) to another domain name (real name). Resolving CNAME record takes two steps - first, the alias is resolved to the resulting domain name, then that domain name is resolved to the IP address (using A record).

Something like this: Alias -> Domain Name -> IP

If the name has multiple CNAME aliases pointing to it, one just needs to change the real name's type A record to "move" all the names to a new IP address.

There can be however no reverse record for CNAME aliases, the corresponding IP will be always resolved to the actual A record. Additionally, the MX records and NS records can be only in use with A or AAAA records, we cannot assign them pointing to CNAME records.

More details about Canonical Name Record (CNAME) with examples you can read here

Mind the dot in the end. That is important, because if you do not add it, DNS server will add another "<vm_name>.sa.cs.ut.ee" part in the end, so your CNAME becomes:

<vm_name>.sa.cs.ut.ee.<vm_name>.sa.cs.ut.ee

3.2 Virtual Web Hosts Using Apache Web Server¶

Apache is the most popular web server software in use (as of February 2021). Apache supports a wide variety of features and can be extended with modules. We will use the ''virtual host'' functionality to set up multiple web hosts on the same web server.

In CentOS 8, the main configuration file for Apache is /etc/httpd/conf/httpd.conf. Although it is not really important which config file you use, it is recommended to keep different parts of the setup in different .conf files. The additional site and module configuration files are kept in /etc/httpd/conf.d directory, i.e. ports.conf.

This kind of separation offers better flexibility when adding and removing different web applications, modules and domains. When some parts of the configuration are kept in separate /etc/httpd/conf.d/*.conf files it is easier to navigate the setup and enable/disable blocks of instructions.

For example, directive Include mods-enabled/*.conf will load every .conf file in this directory and their instructions will be automatically included in the Apache configuration, everything else will be ignored. This way, adding or removing a bit of functionality can be just a matter of creating an appropriate my-functionality.conf file or changing the file extension to something other than .conf. For example - to my-functionality.conf.disabled without editing the actual .conf files

Additionally, adding appropriate files into place with Ansible is much easier, than adding/removing parts to an existing file.

The log files for the Apache webserver are located in the /var/log/httpd/ directory, where query (access_log) logs and error logs (error_log) are kept separately. It is recommended to have a separate log file for each virtual host as well.

Extra reading:

• The Apache HTTP server project

• Apache version 2.4 Documentation

• Red Hat specifics (Apache configuration guide)

During this lab, we will configure Apache to use name-based virtual hosts. With name-based virtual hosting, the server relies on the client to report the hostname as part of the HTTP headers. Using this technique, many different hosts can share the same IP address

First, configure the actual virtual host. We will create a virtual host for www.<vm_name>.sa.cs.ut.ee.

Apache documentation for Virtual Host and more specifically Name-based Virtual Host

You may have noticed that your page is empty. The next step is to create some content for the virtual hosts. To do that you need to be familiar with basic HTML tags and how a web page is constructed. If needed, refer to very good public tutorials here: https://www.w3schools.com/html/html_intro.asp and https://www.w3schools.com/html/html_basic.asp .

Now it is time to restart the Apache httpd server.

There are situations when you want to restart your Apache server, but can't interrupt its work. Imagine that you have a few hundred clients currently downloading files from your server and you need to avoid disconnecting them. In such situations you can use the following command:

# apachectl graceful

This will ''gracefully'' restart your Apache server with a new configuration without affecting your client's connections.

3.3 Utilizing a web server as a proxy¶

Very often, a web server is not utilized to serve files directly but used for forwarding requests to another service that is a web server itself, but is either less secure or does not support settings a proper webserver does. A good example is Python (Flask, Tornado), NodeJS, or Go web applications. This is called proxying.

This allows a systems administrator to have a single point of control over security settings, without having to delve into the application configuration or code.

In this example, we will use Python Flask, but you can use anything capable of serving the web. We will start a service on localhost port 5000, and proxy that to the internet.

Now you have the necessary libraries to run a flask program.

#!/bin/env python3

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run(port=5000)

Once you have run the python program, test if it works by doing curl localhost:5000. It should answer with whatever the program is programmed to do.

Now, to set up the proxy, do the following steps:

Now, because you cannot keep a terminal open constantly to keep this web service up, nor is it a good practice to run things like this, we are going to use our service manager, to keep our python service up for us. This also allows the service to start again when the machine reboots.

4. Let's add a default WordPress setup to our current setup¶

Let's install the prerequisite package necessary for a default WordPress setup.

Startup the MariaDB service

Now let's download the wordpress tarball, unpack it and move it to a proper webroot.

Let's grant user apache the right to work in /var/www/html/wordpress directory. And change the file SELinux security context.

Now let's consolidate our WordPress PHP logs to one place.

Now repeat the steps we did earlier for www.<vm_name>.sa.cs.ut.ee.

The final part of the WordPress setup is setting up the page, what we have done so far has been laying down the foundation, so to speak.

5. Configuring Apache modules¶

While apache2 on its own is a very powerful tool, its functionality can be expanded by loading extra modules. Currently, we will be looking at loading some simple logging modules, but apache modules are used to security-harden your websites, add TLS support, rewrite URL paths, handle downloads, caching, etc.

Each of these modules usually starts with a prefix mod_ and we can see that we are already loading and using some modules, for example, our proxy website already uses a directive named ProxyPass that is loaded from /etc/httpd/conf.modules.d/00-proxy.conf. Using a modular structure like this allows developers and administrators to configure extra functionality without rebuilding whole environments.

5.1 Forensic logging¶

Default Apache2 logging is done by mod_log_config and we are already familiar with this by the usage of the ErrorLog and CustomLog directives we configured for all of our virtual hosts. Looking at our access logs defined in CustomLog we notice some pretty simple info - request origin IP, datetime of the request, and some basic information about the request and response. While this is fine in most cases when working with developers chasing an elusive bug they might require extra information about the nature of the request. This is where mod_log_forensic comes in, as forensic logs offer a substantial increase in the amount of information apache provides, mainly the ability to determine which requests are part of one session.

+YDvAdxfNz6bRQrk@IWUKUQAAAMA|GET / HTTP/1.1|Host:localhost|User-Agent:Mozilla/5.0 (X11; Linux x86_64; rv%3a84.0) Gecko/20100101 Firefox/84.0|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8|Accept-Language:en-US,en;q=0.5|Accept-Encoding:gzip, deflate|DNT:1|Connection:keep-alive|Cookie:utbslang=et|Upgrade-Insecure-Requests:1|Cache-Control:max-age=0
-YDvAdxfNz6bRQrk@IWUKUQAAAMA

5.2 Security-hardening our webserver using mod_security¶

While basic website security should come from the developers' side, system administrators have a few tricks to make sure malicious requests are filtered at the webserver level. For this mod_security is used in apache2 to filter requests that might not always inherently be malicious but can often be exploited.

ModSecurity is a Web Application Firewall (WAF) that is a list of rules that get matched against each HTTP request. These rules try to determine whether the request is malicious in nature, for example trying to access system files like /etc/shadow, and if they do, then the request gets blocked.

Not all websites are always able to work behind a WAF. For example courses.cs.ut.ee needed heavy WAF custom configuration, because every edit to a page seemed like a bunch of code for a WAF, and it denied all requests.

# default action when matching rules
SecDefaultAction "phase:2,deny,log,status:406"

# [etc/passwd] is included in request URI
SecRule REQUEST_URI "etc/passwd" "id:'500001'"

# [../] is included in request URI
SecRule REQUEST_URI "\.\./" "id:'500002'"

# [<SCRIPT] is included in arguments
SecRule ARGS "<[Ss][Cc][Rr][Ii][Pp][Tt]" "id:'500003'"

• We will use the tool netcat to manually write our request

  First, let's craft the request itself, we will be using some printf magic to make sure it's correctly formatted. The request is similar to the one shown at the beginning of the lab guide

  GET / HTTP/1.1
  User-Agent: nc/0.0.1
  Host: www.<vmname>.sa.cs.ut.ee
  Accept: */*
  

  Notice the 2 extra empty lines at the end - these are necessary for netcat to understand where the end of the request is.
  • Next we need to craft this into a format printf can understand, for this we will replace all of the line breaks with \r\n
    • "GET / HTTP/1.1\r\nUser-Agent: nc/0.0.1\r\nHost: www.<vmname>.sa.cs.ut.ee\r\nAccept: */*\r\n\r\n"

  Now we pass this string onto printf, a command-line tool to format strings and pipe the output to netcat

  • printf "GET / HTTP/1.1\r\nUser-Agent: nc/0.0.1\r\nHost: www.<vmname>.sa.cs.ut.ee\r\nAccept: */*\r\n\r\n" | nc localhost 80

    • We should see a response from the server starting with HTTP/1.1 200 OK, followed by the HTML content of our indexpage

  • Now replace the query path / with /etc/passwd

    • We now see that the webserver has rejected our request with the header HTTP/1.1 406 Not Acceptable

Thankfully, with nowadays default webserver configuration, most of these kinds of attacks are impossible if you have not made a severe configuration error, but it is still good to have at least two layers of configuration to prevent these kinds of problems.

6. Ansible tips¶

6.1 Tags¶

Previously, we have asked you to define your roles with tags like - role { role: <role name>, tags: <tag name> }. Tags are used for running specific parts of your playbook, instead of the whole thing.

For example, you have edited some zone files under your DNS role and want to apply them. Running ansible-playbook playbook.yml will run all of the roles that you have set up there, this will get cumbersome if you have tens of huge roles that will be executed. Running these extra roles won't break anything if your playbook is well written, but in some cases, it will take a lot of time to run a whole playbook for a single configuration change. In this instance, it can be avoided by using ansible-playbook --tags=dns playbook.yml, which will run only the roles that have a DNS tag. The main purpose is to avoid the hassle of creating multiple playbooks for singular roles, editing the main playbook to comment something out, and all-in-all making using Ansible a better experience. More information about tags can be found at the Ansible documentation tags page

6.2 Ansible modules¶

Ansible modules are discreet sets of code that give ansible the functionality that it has. You have already used a few modules in your playbook tasks, like user: for user management, file: for creating files and modifying their permissions dnf: for installing packages, templates, etc. Ansible has a lot of modules built in that can greatly improve your playbook execution.

Ansible documentation always shows specific examples on every module usage, most of them even match with what we are trying to do in this lab. For example, we can take them from seboolean and sefcontext to get:

- name: Seboolean | this equates to 'setsebool -P httpd_can_network_connect=1'
  seboolean:
    name: httpd_can_network_connect
    state: yes
    persistent: yes

- name: Sefcontext | this equates to 'chcon -t httpd_sys_rw_content_t /var/www/html/wordpress -R'
  sefcontext:
    target: '/var/www/html/wordpress(/.*)?'
    setype: httpd_sys_rw_content_t
    state: present

6.3 Handlers¶

Ansible tasks have three main states of completion:

• ok - task ran successfully and nothing was changed

• changed - task ran successfully and something was changed

• failed - executing the task failed

There are some tasks that you may only want to run when a change is made. Mostly this is used to restart services only when a configuration file has been updated, not on every run. Just as it was said in the guide above, you want to avoid unnecessary restarts to apache as much as you can.

Handlers have their own subdirectory roles/<rolename>/handlers/main.yml

An example use of a handler in a task declaration

- name: Template | website config
  template:
    src: vhost.conf.j2
    dest: /etc/httpd/conf.d/www.{{ hostname }}.conf
    notify:
       - restart httpd

An example of an httpd handler in handlers/main.yml:

- name: restart httpd
  systemd:
    daemon_reload: yes
    name: httpd
    state: restarted
    enabled: yes

The ansible documentation for handlers has many great examples on how to run handlers. Now when a handler is notified, it will run at the end of your playbook to flush all of the changes you have done all at one time.

7. Automating lab 5¶

8. Keeping your Ansible repository safe in Gitlab¶

And as always push your stuff to our Gitlab.