Bump version

* Expanded installation instructions

Installation was moved to a new file, GUIDE.md, where you can include all documented information about Shynet.

I also included SSL without a reverse proxy instructions and a shell script for SSL through Gunicorn.

GUIDE.md

@@ -0,0 +1,199 @@
+# Getting Started
+
+## Table of Contents
+
+* [Installation](#installation)
+  * [Basic Installation](#basic-installation)
+  * [Installation with SSL](#installation-with-ssl)
+<!--
+* Usage
+  * Adding Shynet tracking to your first website
+  * Adding a new website
+  * Adding a new administrator
+* Setting up a reverse proxy
+  * Cloudflare
+  * nginx
+-->
+
+## Installation
+
+Installation of Shynet is easy! Follow the [Basic Installation](#basic-installation) guide below if you'd like to run Shynet over HTTP or if you are going to be running it over HTTPS through a reverse proxy. If you'd like to run Shynet over HTTPS without a reverse proxy, skip ahead to [Installation with SSL](#installation-with-ssl) instead.
+
+> **These commands assume Ubuntu.** If you're installing Shynet on a different platform, the process will be different.
+
+Before continuing, please be sure to have the latest version of Docker installed.
+
+### Basic Installation
+
+1. Pull the latest version of Shynet using `docker pull milesmcc/shynet:latest`. If you don't have Docker installed, [install it](https://docs.docker.com/get-docker/).
+
+2. Have a PostgreSQL server ready to go. This can be on the same machine as the deployment, or elsewhere. You'll just need a username, password, host, and port (default is `5432`). (For info on how to setup a PostgreSQL server on Ubuntu, follow [this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04)).
+
+3. Configure an environment file for Shynet. (For example, create a file called `.env`.) Be sure to swap out the variables below with the correct values for your setup. (The comments refer to the lines that follow. Note that Docker is weird with quotes, so it tends to be better to omit them from your env file.)
+
+```
+# Database
+DB_NAME=<your db name>
+DB_USER=<your db user>
+DB_PASSWORD=<your db user password>
+DB_HOST=<your db host>
+DB_PORT=<your db port>
+
+# General Django settings
+DJANGO_SECRET_KEY=<your Django secret key; just a random string>
+# Don't leak error details to visitors, very important
+DEBUG=False
+# Unless you are using an external Celery task queue, make sure this
+# is set to True.
+CELERY_TASK_ALWAYS_EAGER=True 
+# For better security, set this to your deployment's domain. Comma separated.
+ALLOWED_HOSTS=*
+# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
+SIGNUPS_ENABLED=False
+# Change as required
+TIME_ZONE=America/New_York
+# Set to "False" if you will not be serving content over HTTPS
+SCRIPT_USE_HTTPS=True
+```
+
+For more advanced deployments, you may consider adding the following settings to your environment file. **The following settings are optional, and not required for simple deployments.**
+
+```env
+# Email settings
+EMAIL_HOST_USER=<your SMTP email user>
+EMAIL_HOST_PASSWORD=<your SMTP email password>
+EMAIL_HOST=<your SMTP email hostname>
+SERVER_EMAIL=Shynet <noreply@shynet.example.com>
+
+# Redis and queue settings; not necessary for single-instance deployments
+REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
+# If set, make sure CELERY_TASK_ALWAYS_EAGER is False
+CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1
+
+# Other Shynet settings 
+# How frequently should the monitoring script "phone home" (in ms)?
+SCRIPT_HEARTBEAT_FREQUENCY=5000
+# Should only superusers (admins) be able to create services? This is helpful
+# when you'd like to invite others to your Shynet instance but don't want
+# them to be able to create services of their own.
+ONLY_SUPERUSERS_CREATE=False
+```
+
+4. Setup the Shynet database by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py migrate`.
+
+5. Create your admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
+
+6. Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
+
+7. Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
+
+8. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 80 (http); this can be done using the flag `-p 80:8080` after `run`.
+
+9. Visit your service's homepage, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 5. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
+
+10. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
+
+11. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
+
+### Installation with SSL
+
+If you are going to be running Shynet through a reverse proxy, please use the [Basic Installation](#basic-installation) guide instead.
+
+0. We'll be cloning this into the home directory to make this installation easier, so run `cd ~/` if you need to.
+
+1. Instead of pulling from Docker, we will be pulling from GitHub and building in Docker so that we may easily add SSL certificates. You will want to run `git clone https://github.com/milesmcc/shynet.git` to clone the GitHub repo to your computer.
+
+2. To install `certbot` follow [the guide here](https://certbot.eff.org/instructions) or follow along below
+   * Ubuntu 18.04
+     * `sudo apt-get update`
+     * `sudo apt-get install software-properties-common`
+     * `sudo add-apt-repository universe`
+     * `sudo add-apt-repository ppa:certbot/certbot`
+     * `sudo apt-get update`
+     * `sudo apt-get install certbot`
+
+3. Run `sudo certbot certonly --standalone` and follow the instructions to generate your SSL certificate.
+   * If you registering it to a domain name like `example.com`, please be sure to point your DNS records to your server before running `certbot`.
+
+4. We are going to move the SSL certificates to the Shynet repo with with command below. Replace `<domain>` with the domain name you used in step 3.
+   * `cp /etc/letsencrypt/live/<domain>/{cert,privkey}.pem ~/shynet/shynet/`
+
+5. With that, we are going to replace the `webserver.sh` with `ssl.webserver.sh` to enable the use of SSL certificates. The original `webserver.sh` will be backed up to `backup.webserver.sh`
+   * `mv ~/shynet/shynet/webserver.sh ~/shynet/shynet/backup.webserver.sh`
+   * `mv ~/shynet/shynet/ssl.webserver.sh ~/shynet/shynet/webserver.sh`
+
+6. Now we build the image!
+   * `docker image build shynet -t milesmcc/shynet:latest-ssl`
+
+7. Have a PostgreSQL server ready to go. This can be on the same machine as the deployment, or elsewhere. You'll just need a username, password, host, and port (default is `5432`). (For info on how to setup a PostgreSQL server on Ubuntu, follow [this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04)).
+
+8. Configure an environment file for Shynet. (For example, create a file called `.env`.) Be sure to swap out the variables below with the correct values for your setup. (The comments refer to the lines that follow. Note that Docker is weird with quotes, so it tends to be better to omit them from your env file.)
+
+```
+# Database
+DB_NAME=<your db name>
+DB_USER=<your db user>
+DB_PASSWORD=<your db user password>
+DB_HOST=<your db host>
+DB_PORT=<your db port>
+
+# General Django settings
+DJANGO_SECRET_KEY=<your Django secret key; just a random string>
+# Don't leak error details to visitors, very important
+DEBUG=False
+# Unless you are using an external Celery task queue, make sure this
+# is set to True.
+CELERY_TASK_ALWAYS_EAGER=True 
+# For better security, set this to your deployment's domain. Comma separated.
+ALLOWED_HOSTS=*
+# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
+SIGNUPS_ENABLED=False
+# Change as required
+TIME_ZONE=America/New_York
+# Set to "False" if you will not be serving content over HTTPS
+SCRIPT_USE_HTTPS=True
+```
+
+For more advanced deployments, you may consider adding the following settings to your environment file. **The following settings are optional, and not required for simple deployments.**
+
+```env
+# Email settings
+EMAIL_HOST_USER=<your SMTP email user>
+EMAIL_HOST_PASSWORD=<your SMTP email password>
+EMAIL_HOST=<your SMTP email hostname>
+SERVER_EMAIL=Shynet <noreply@shynet.example.com>
+
+# Redis and queue settings; not necessary for single-instance deployments
+REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
+# If set, make sure CELERY_TASK_ALWAYS_EAGER is False
+CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1
+
+
+# Other Shynet settings 
+# How frequently should the monitoring script "phone home" (in ms)?
+SCRIPT_HEARTBEAT_FREQUENCY=5000
+# Should only superusers (admins) be able to create services? This is helpful
+# when you'd like to invite others to your Shynet instance but don't want
+# them to be able to create services of their own.
+ONLY_SUPERUSERS_CREATE=False
+```
+
+9. Setup the Shynet database by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py migrate`.
+
+10. Create your admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
+
+11. Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
+
+12. Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
+
+13. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl`. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 443 (https); this can be done using the flag `-p 443:8080` after `run`.
+
+14. Visit your service's homepage using `https://`, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 10. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
+
+15. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
+
+16. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
+
+---
+
+**Next steps:** while out of the scope of this short guide, next steps include setting up Shynet behind a reverse proxy (be it your own [Nginx server](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) or [Cloudflare](https://cloudflare.com)), making it run in the background, and integrating it on your sites. Integration instructions are available on each service's management page.

README.md

@@ -1,15 +1,18 @@
+
 <p align="center">
-  <h3 align="center">🔭 Shynet 🔭</h3>
+  <img align="center" src="images/logo.png" height="50" alt="Shynet logo">
+  <br>
 
   <p align="center">
-     Web analytics that's self hosted, cookie free, privacy friendly, and useful(?) 
-    <br>
-    <br>
-    <a href="#installation"><strong>Getting started »</strong></a>
+    Modern, privacy-friendly, and cookie-free web analytics.
     <br>
+    <strong><a href="#installation">Getting started »</a></strong>
   </p>
+  <p align="center"><a href="#screenshots">Screenshots</a> &bull; <a href="#features">Features</a> &bull; <a href="https://github.com/milesmcc/a17t">Design</a></p>
 </p>
 
+<br>
+
 ## Motivation
 
 There are a _lot_ of web analytics tools. Unfortunately, most of them come with the following caveats:
@@ -31,16 +34,21 @@ _Note: These screenshots have been edited to hide sensitive data. The "real" Shy
 ![Shynet's homepage](images/homepage.png)
 _Shynet's homepage, where you can see all of your services at a glance._
 
-Not shown: service view, management view, session view, full service view. (You'll need to install Shynet to see those!)
+![A service page](images/service.png)
+_A real service page, where you can see higher-level details about a site._
+
+Not shown: management view, session view, full service view. (You'll need to install Shynet for yourself to see those!)
+
+> **Shynet is built using [a17t](https://github.com/milesmcc/a17t),** an atomic design library. Customization and extension is simple; [learn more about a17t](https://github.com/milesmcc/a17t).
 
 ## Features
 
 #### Architecture
 
-* **Runs on a single machine** &mdash; Because it's so small, Shynet can easily run as a single docker container on a single small VPS.
-* **...or across a giant Kubernetes cluster** &mdash; For higher traffic installations, Shynet can be deployed with as many parallelized ingress nodes as needed, with Redis caching and separate backend workers for database IO.
-* **Built using Django** &mdash; Shynet is built using Django, so deploying, updating, and migrating can be done without headaches.
-* **Multiple users and sites** &mdash; A single Shynet instance can support multiple users, each tracking multiple different sites.
+* **Runs on a single machine** &mdash; Because it's so small, Shynet can easily run as a single docker container on a single small VPS
+* **...or across a giant Kubernetes cluster** &mdash; For higher traffic installations, Shynet can be deployed with as many parallelized ingress nodes as needed, with Redis caching and separate backend workers for database IO
+* **Built using Django** &mdash; Shynet is built using Django, so deploying, updating, and migrating can be done without headaches
+* **Multiple users and sites** &mdash; A single Shynet instance can support multiple users, each tracking multiple different sites
 
 #### Tracking
 
@@ -67,7 +75,7 @@ Here's the information Shynet can give you about your visitors:
 
 #### Workflow
 * **Collaboration built-in** &mdash; Administrators can easily share services with other users, as well
-* **Accounts (or not)** &mdash; Shynet has a fully featured account management workflow (powered by [Django Allauth](https://github.com/pennersr/django-allauth/)).
+* **Accounts (or not)** &mdash; Shynet has a fully featured account management workflow (powered by [Django Allauth](https://github.com/pennersr/django-allauth/))
 
 ## Recommendations
 
@@ -85,77 +93,13 @@ Shynet is pretty simple, but there are a few key terms you need to know in order
 
 ## Installation
 
-To install Shynet using the simplest possible setup, follow these instructions. Instructions for multi-machine deployments will be available soon.
-
-> **These commands assume Ubuntu.** If you're installing Shynet on a different platform, the process will be different.
-
-1. Pull the latest version of Shynet using `docker pull milesmcc/shynet:latest`. If you don't have Docker installed, [install it](https://docs.docker.com/get-docker/).
-
-2. Have a PostgreSQL server ready to go. This can be on the same machine as the deployment, or elsewhere. You'll just need a username, password, and host. (For info on how to setup a PostgreSQL server on Ubuntu, follow [this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04)).
-
-3. Configure an environment file for Shynet. (For example, create a file called `.env`.) Be sure to swap out the variables below with the correct values for your setup. (The comments refer to the lines that follow. Note that Docker is weird with quotes, so it tends to be better to omit them from your env file.)
-
-```
-# Database
-DB_NAME=<your db name>
-DB_USER=<your db user>
-DB_PASSWORD=<your db user password>
-DB_HOST=<your db host>
-
-# General Django settings
-DJANGO_SECRET_KEY=<your Django secret key; just a random string>
-# Don't leak error details to visitors, very important
-DEBUG=False
-CELERY_TASK_ALWAYS_EAGER=True 
-# For better security, set this to your deployment's domain. Comma separated.
-ALLOWED_HOSTS=*
-# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
-SIGNUPS_ENABLED=False
-# Change as required
-TIME_ZONE=America/New_York
-# Set to "False" if you will not be serving content over HTTPS
-SCRIPT_USE_HTTPS=True
-```
-
-For more advanced deployments, you may consider adding the following settings to your environment file. **The following settings are optional, and not required for simple deployments.**
-
-```env
-# Email settings
-EMAIL_HOST_USER=<your SMTP email user>
-EMAIL_HOST_PASSWORD=<your SMTP email password>
-EMAIL_HOST=<your SMTP email hostname>
-SERVER_EMAIL=Shynet <noreply@shynet.example.com>
-
-# Redis and queue settings; not necessary for single-instance deployments
-REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
-# If set, make sure CELERY_TASK_ALWAYS_EAGER is False
-CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1
-```
-
-4. Setup the Shynet database by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py migrate`.
-
-5. Create your admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
-
-6. Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
-
-7. Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
-
-8. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. You may need to bind Docker's port 8000 (where Shynet runs) to your local port 8000; this can be done using the flag `-p 8080:8080`.
-
-9. Visit your service's homepage, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 5. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
-
-10. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
-
-11. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
-
-**Next steps:** while out of the scope of this short guide, next steps include setting up Shynet behind a reverse proxy (be it your own [Nginx server](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) or [Cloudflare](https://cloudflare.com)), making it run in the background, and integrating it on your sites. Integration instructions are available on each service's management page.
-
+You can find installation instructions in the [Getting Started Guide](GUIDE.md#installation).
 
 ## FAQ
 
 **Does Shynet respond to Do Not Track (DNT) signals?** Yes. While there isn't any standardized way to handle DNT requests, Shynet allows you to specify whether you want to collect any data from users with DNT enabled on a per-service basis. (By default, Shynet will _not_ collect any data from users who specify DNT.)
 
-**Is this GDPR compliant?** I think so, but it also depends on how you use it. If you're worried about GDPR, you should talk to a lawyer about your particular data collection practices. I'm not a lawyer. (And this isn't legal advice.)
+**Is this GDPR compliant?** It depends on how you use it. If you're worried about GDPR, you should talk to a lawyer about your particular data collection practices. I'm not a lawyer. (And this isn't legal advice.)
 
 ## Roadmap
 
@@ -182,4 +126,4 @@ Shynet is made available under the [Apache License, version 2.0](LICENSE).
 
 ---
 
-a17t was created by [Miles McCain](https://miles.land) at the [Recurse Center](https://recurse.com) using [a17t](https://a17t.miles.land).
+Shynet was created by [Miles McCain](https://miles.land) ([@MilesMcCain](https://twitter.com/MilesMcCain)) at the [Recurse Center](https://recurse.com) using [a17t](https://a17t.miles.land).

shynet/a17t/templates/a17t/head.html

@@ -1,3 +1,10 @@
 <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/a17t@0.1.3/dist/a17t.css">
 <script async src="https://use.fontawesome.com/releases/v5.3.1/js/all.js"></script>
 <link href="https://cdn.jsdelivr.net/npm/tailwindcss@^1.0/dist/tailwind.min.css" rel="stylesheet">
+<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&display=swap" rel="stylesheet">
+<style>
+    :root {
+        --family-primary: "Inter", system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, "Noto Sans", sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
+        --family-secondary: var(--family-primary);
+    }
+</style>

shynet/analytics/tasks.py

@@ -42,7 +42,15 @@ def _geoip2_lookup(ip):
 
 @shared_task
 def ingress_request(
-    service_uuid, tracker, time, payload, ip, location, user_agent, dnt=False, identifier=""
+    service_uuid,
+    tracker,
+    time,
+    payload,
+    ip,
+    location,
+    user_agent,
+    dnt=False,
+    identifier="",
 ):
     try:
         service = Service.objects.get(pk=service_uuid, status=Service.ACTIVE)
@@ -78,7 +86,8 @@ def ingress_request(
             if (
                 ua.is_bot
                 or (ua.browser.family or "").strip().lower() == "googlebot"
-                or (ua.device.family or ua.device.model or "").strip().lower() == "spider"
+                or (ua.device.family or ua.device.model or "").strip().lower()
+                == "spider"
             ):
                 device_type = "ROBOT"
             elif ua.is_mobile:

shynet/analytics/templates/analytics/scripts/page.js

@@ -4,6 +4,9 @@ window.onload = function () {
     Math.random().toString(36).substring(2, 15);
   function sendUpdate() {
     try {
+      if (document.hidden) {
+        return;
+      }
       var xhr = new XMLHttpRequest();
       xhr.open(
         "POST",
@@ -21,8 +24,8 @@ window.onload = function () {
             window.performance.timing.navigationStart,
         })
       );
-    } catch {}
+    } catch { }
   }
-  setInterval(sendUpdate, 5000);
+  setInterval(sendUpdate, parseInt("{{heartbeat_frequency}}"));
   sendUpdate();
 };

shynet/analytics/views/ingress.py

@@ -2,6 +2,7 @@ import base64
 import json
 
 from django.conf import settings
+from django.core.cache import cache
 from django.http import HttpResponse
 from django.shortcuts import render, reverse
 from django.utils import timezone
@@ -10,6 +11,8 @@ from django.views.decorators.csrf import csrf_exempt
 from django.views.generic import TemplateView, View
 from ipware import get_client_ip
 
+from core.models import Service
+
 from ..tasks import ingress_request
 
 
@@ -58,8 +61,15 @@ class PixelView(View):
 @method_decorator(csrf_exempt, name="dispatch")
 class ScriptView(View):
     def dispatch(self, request, *args, **kwargs):
+        service_uuid = self.kwargs.get("service_uuid")
+        origins = cache.get(f"service_origins_{service_uuid}")
+        if origins is None:
+            service = Service.objects.get(uuid=service_uuid)
+            origins = service.origins
+            cache.set(f"service_origins_{service_uuid}", origins, timeout=3600)
+
         resp = super().dispatch(request, *args, **kwargs)
-        resp["Access-Control-Allow-Origin"] = "*"
+        resp["Access-Control-Allow-Origin"] = origins
         resp["Access-Control-Allow-Methods"] = "GET,HEAD,OPTIONS,POST"
         resp[
             "Access-Control-Allow-Headers"
@@ -82,10 +92,15 @@ class ScriptView(View):
                 },
             )
         )
+        heartbeat_frequency = settings.SCRIPT_HEARTBEAT_FREQUENCY
         return render(
             self.request,
             "analytics/scripts/page.js",
-            context={"endpoint": endpoint, "protocol": protocol},
+            context={
+                "endpoint": endpoint,
+                "protocol": protocol,
+                "heartbeat_frequency": heartbeat_frequency,
+            },
             content_type="application/javascript",
         )
 

shynet/core/management/commands/hostname.py

@@ -1,10 +1,12 @@
-from django.core.management.base import BaseCommand, CommandError
+import traceback
+import uuid
+
 from django.conf import settings
 from django.contrib.sites.models import Site
-from core.models import User
+from django.core.management.base import BaseCommand, CommandError
 from django.utils.crypto import get_random_string
-import uuid
-import traceback
+
+from core.models import User
 
 
 class Command(BaseCommand):
@@ -12,8 +14,7 @@ class Command(BaseCommand):
 
     def add_arguments(self, parser):
         parser.add_argument(
-            "hostname",
-            type=str,
+            "hostname", type=str,
         )
 
     def handle(self, *args, **options):
@@ -24,4 +25,4 @@ class Command(BaseCommand):
             self.style.SUCCESS(
                 f"Successfully set the hostname to '{options.get('hostname')}'"
             )
-        )
+        )

shynet/core/management/commands/registeradmin.py

@@ -1,10 +1,12 @@
-from django.core.management.base import BaseCommand, CommandError
+import traceback
+import uuid
+
 from django.conf import settings
 from django.contrib.sites.models import Site
-from core.models import User
+from django.core.management.base import BaseCommand, CommandError
 from django.utils.crypto import get_random_string
-import uuid
-import traceback
+
+from core.models import User
 
 
 class Command(BaseCommand):
@@ -12,18 +14,13 @@ class Command(BaseCommand):
 
     def add_arguments(self, parser):
         parser.add_argument(
-            "email",
-            type=str,
+            "email", type=str,
         )
 
     def handle(self, *args, **options):
         email = options.get("email")
         password = get_random_string()
-        User.objects.create_superuser(
-            str(uuid.uuid4()), email=email, password=password
-        )
-        self.stdout.write(
-            self.style.SUCCESS("Successfully created a Shynet superuser")
-        )
+        User.objects.create_superuser(str(uuid.uuid4()), email=email, password=password)
+        self.stdout.write(self.style.SUCCESS("Successfully created a Shynet superuser"))
         self.stdout.write(f"Email address: {email}")
         self.stdout.write(f"Password: {password}")

shynet/core/management/commands/whitelabel.py

@@ -1,10 +1,12 @@
-from django.core.management.base import BaseCommand, CommandError
+import traceback
+import uuid
+
 from django.conf import settings
 from django.contrib.sites.models import Site
-from core.models import User
+from django.core.management.base import BaseCommand, CommandError
 from django.utils.crypto import get_random_string
-import uuid
-import traceback
+
+from core.models import User
 
 
 class Command(BaseCommand):
@@ -12,8 +14,7 @@ class Command(BaseCommand):
 
     def add_arguments(self, parser):
         parser.add_argument(
-            "name",
-            type=str,
+            "name", type=str,
         )
 
     def handle(self, *args, **options):
@@ -24,4 +25,4 @@ class Command(BaseCommand):
             self.style.SUCCESS(
                 f"Successfully set the whitelabel to '{options.get('name')}'"
             )
-        )
+        )

shynet/core/migrations/0003_service_respect_dnt.py

@@ -6,13 +6,13 @@ from django.db import migrations, models
 class Migration(migrations.Migration):
 
     dependencies = [
-        ('core', '0002_auto_20200415_1742'),
+        ("core", "0002_auto_20200415_1742"),
     ]
 
     operations = [
         migrations.AddField(
-            model_name='service',
-            name='respect_dnt',
+            model_name="service",
+            name="respect_dnt",
             field=models.BooleanField(default=True),
         ),
     ]

shynet/dashboard/forms.py

@@ -1,14 +1,14 @@
+from allauth.account.admin import EmailAddress
 from django import forms
 from django.utils.translation import gettext_lazy as _
 
 from core.models import Service, User
-from allauth.account.admin import EmailAddress
 
 
 class ServiceForm(forms.ModelForm):
     class Meta:
         model = Service
-        fields = ["name", "link", "respect_dnt", "collaborators"]
+        fields = ["name", "link", "respect_dnt", "origins", "collaborators"]
         widgets = {
             "name": forms.TextInput(),
             "origins": forms.TextInput(),
@@ -27,8 +27,10 @@ class ServiceForm(forms.ModelForm):
             "respect_dnt": "Should visitors who have enabled <a href='https://en.wikipedia.org/wiki/Do_Not_Track'>Do Not Track</a> be excluded from all data?",
         }
 
-    collaborators = forms.CharField(help_text="Which users should have read-only access to this service? (Comma separated list of emails.)", required=False)
-
+    collaborators = forms.CharField(
+        help_text="Which users should have read-only access to this service? (Comma separated list of emails.)",
+        required=False,
+    )
 
     def clean_collaborators(self):
         collaborators = []
@@ -36,7 +38,9 @@ class ServiceForm(forms.ModelForm):
             email = collaborator_email.strip()
             if email == "":
                 continue
-            collaborator_email_linked = EmailAddress.objects.filter(email__iexact=email).first()
+            collaborator_email_linked = EmailAddress.objects.filter(
+                email__iexact=email
+            ).first()
             if collaborator_email_linked is None:
                 raise forms.ValidationError(f"Email '{email}' is not registered")
             collaborators.append(collaborator_email_linked.user)

shynet/dashboard/mixins.py

@@ -1,6 +1,6 @@
+from datetime import datetime, time
 from urllib.parse import urlparse
 
-from datetime import time, datetime
 from django.utils import timezone
 
 

shynet/dashboard/templates/base.html

@@ -1,4 +1,4 @@
-{% load static rules %}
+{% load static rules helpers %}
 <!DOCTYPE html>
 
 <html>
@@ -20,8 +20,8 @@
 <body class="bg-gray-200 min-h-full">
   {% block body %}
 
-  <section class="max-w-screen-xl mx-auto px-4 md:px-0 py-4 md:py-12 md:flex">
-    <aside class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center">
+  <section class="max-w-screen-xl mx-auto px-4 py-4 md:py-12 md:flex">
+    <aside class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center overflow-x-hidden">
       <a class="icon ~urge ml-2 md:ml-6 md:mb-8 md:mt-3" href="{% url 'dashboard:dashboard' %}">
         <i class="fas fa-binoculars fa-3x text-purple-600 hidden md:block"></i>
         <i class="fas fa-binoculars fa-2x text-purple-600 md:hidden"></i>
@@ -93,6 +93,10 @@
         {% include 'dashboard/includes/sidebar_portal.html' with label="Sign Up" url=url %}
 
         {% endif %}
+
+        <hr class="sep h-8">
+
+        {% sidebar_footer %}
       </div>
     </aside>
     <div class="md:w-10/12">

shynet/dashboard/templates/dashboard/includes/service_form.html

@@ -0,0 +1,12 @@
+{% load a17t_tags %}
+
+{{form.name|a17t}}
+{{form.link|a17t}}
+{{form.collaborators|a17t}}
+
+<details class="p-4 border rounded">
+    <summary class="cursor-pointer text-sm">Advanced settings</summary>
+    <hr class="sep h-4">
+    {{form.respect_dnt|a17t}}
+    {{form.origins|a17t}}
+</details>

shynet/dashboard/templates/dashboard/includes/sidebar_footer.html

@@ -0,0 +1,5 @@
+<div class="pl-2">
+  <p class="support text-gray-600">
+    <a href="https://github.com/milesmcc/shynet">Shynet {{version}}</a>
+  </p>
+</div>

shynet/dashboard/templates/dashboard/pages/service_create.html

@@ -10,7 +10,7 @@
 <form class="card ~neutral !low p-0 max-w-xl" method="POST">
     {% csrf_token %}
     <div class="p-4">
-        {{form|a17t}}
+        {% include 'dashboard/includes/service_form.html' %}
     </div>
     <div class="section ~urge !normal p-4">
         <button type="submit" class="button ~urge !high">Create</button>

shynet/dashboard/templates/dashboard/pages/service_session.html

@@ -59,8 +59,8 @@
             </p>
         </div>
         <div>
-            <p>Time Zone</p>
-            <p class="label">{{session.time_zone|default:"Unknown"}}</p>
+            <p>IP</p>
+            <p class="label" title="{{session.ip}}">{{session.ip|truncatechars:"16"}}</p>
         </div>
     </div>
 </article>

shynet/dashboard/templates/dashboard/pages/service_update.html

@@ -23,7 +23,7 @@
     <form class="card ~neutral !low p-0" method="POST">
         {% csrf_token %}
         <div class="p-4">
-            {{form|a17t}}
+            {% include 'dashboard/includes/service_form.html' %}
         </div>
         <div class="section ~neutral !normal p-4 flex justify-between">
             <div>

shynet/dashboard/templatetags/helpers.py

@@ -1,7 +1,9 @@
 from urllib.parse import urlparse
-import pycountry
+
 import flag
+import pycountry
 from django import template
+from django.conf import settings
 from django.utils import timezone
 from django.utils.html import escape
 from django.utils.safestring import SafeString
@@ -41,17 +43,12 @@ def country_name(isocode):
 
 @register.simple_tag
 def relative_stat_tone(
-    start,
-    end,
-    good="UP",
-    good_classes=None,
-    bad_classes=None,
-    neutral_classes=None,
+    start, end, good="UP", good_classes=None, bad_classes=None, neutral_classes=None,
 ):
     good_classes = good_classes or "~positive"
     bad_classes = bad_classes or "~critical"
     neutral_classes = neutral_classes or "~neutral"
-    
+
     if start == None or end == None or start == end:
         return neutral_classes
     if good == "UP":
@@ -84,6 +81,11 @@ def percent_change_display(start, end):
 
     return SafeString(direction + pct_change)
 
+@register.inclusion_tag("dashboard/includes/sidebar_footer.html")
+def sidebar_footer():
+    return {
+        "version": settings.VERSION
+    }
 
 @register.inclusion_tag("dashboard/includes/stat_comparison.html")
 def compare(

shynet/dashboard/views.py

@@ -1,5 +1,6 @@
 from django.contrib.auth.mixins import LoginRequiredMixin
 from django.contrib.messages.views import SuccessMessageMixin
+from django.core.cache import cache
 from django.db.models import Q
 from django.shortcuts import get_object_or_404, reverse
 from django.utils import timezone
@@ -77,6 +78,13 @@ class ServiceUpdateView(
     def get_success_url(self):
         return reverse("dashboard:service", kwargs={"pk": self.object.uuid})
 
+    def form_valid(self, *args, **kwargs):
+        resp = super().form_valid(*args, **kwargs)
+        cache.set(
+            f"service_origins_{self.object.uuid}", self.object.origins, timeout=3600
+        )
+        return resp
+
 
 class ServiceDeleteView(
     LoginRequiredMixin, PermissionRequiredMixin, SuccessMessageMixin, DeleteView

shynet/shynet/settings.py

@@ -1,7 +1,5 @@
 """
-Django settings for shynet project.
-
-Generated by 'django-admin startproject' using Django 2.2.5.
+Django settings for Shynet.
 
 For more information on this file, see
 https://docs.djangoproject.com/en/2.2/topics/settings/
@@ -15,6 +13,9 @@ import os
 # Messages
 from django.contrib.messages import constants as messages
 
+# Increment on new releases
+VERSION = "v0.2.2"
+
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 
@@ -256,3 +257,7 @@ ONLY_SUPERUSERS_CREATE = os.getenv("ONLY_SUPERUSERS_CREATE", "True") == "True"
 # Should the script use HTTPS to send the POST requests? The hostname is from
 # the django SITE default. (Edit it using the admin panel.)
 SCRIPT_USE_HTTPS = os.getenv("SCRIPT_USE_HTTPS", "True") == "True"
+
+# How frequently should the tracking script "phone home" with a heartbeat, in
+# milliseconds?
+SCRIPT_HEARTBEAT_FREQUENCY = int(os.getenv("SCRIPT_HEARTBEAT_FREQUENCY", "5000"))

shynet/ssl.webserver.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# Start Gunicorn processes
+echo Launching Shynet web server...
+exec gunicorn shynet.wsgi:application \
+    --bind 0.0.0.0:8080 \
+    --workers 3 \
+    --timeout 100 \
+    --certfile=cert.pem \
+    --keyfile=privkey.pem

tests/pixel.html

@@ -7,8 +7,8 @@
 </head>
 
 <body>
-    <noscript><img src="//localhost:8000/ingress/211410e6-b401-4fe8-8740-7926368590be/pixel.gif"></noscript>
-    <script src="//localhost:8000/ingress/211410e6-b401-4fe8-8740-7926368590be/script.js"></script>
+    <noscript><img src="//localhost:8000/ingress/test_uuid/pixel.gif"></noscript>
+    <script src="//localhost:8000/ingress/test_uuid/script.js"></script>
 </body>
 
 </html>