Update Kubernetes to use default entrypoint

Make all scripts executable

Disable debug mode by default

Use eager tasks by default

Fix typo in settings

Refactoring

.gitignore

@@ -130,4 +130,6 @@ dmypy.json
 
 # Secrets & env
 secrets.yml
-.vscode
+.vscode
+.DS_Store
+compiledstatic/

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at shynet@sendmiles.email. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

Dockerfile

@@ -30,4 +30,4 @@ USER appuser
 
 EXPOSE 8080
 
-CMD [ "./webserver.sh" ]
+ENTRYPOINT [ "./entrypoint.sh" ]

GUIDE.md

@@ -2,18 +2,15 @@
 
 ## Table of Contents
 
-* [Installation](#installation)
-  * [Basic Installation](#basic-installation)
+- [Installation](#installation)
+- [Updating Your Configuration](#updating-your-configuration)
+- [Enhancements](#enhancements)
   * [Installation with SSL](#installation-with-ssl)
-<!--
-* Usage
-  * Adding Shynet tracking to your first website
-  * Adding a new website
-  * Adding a new administrator
-* Setting up a reverse proxy
-  * Cloudflare
-  * nginx
--->
+  * [Configuring a Reverse Proxy](#configuring-a-reverse-proxy)
+    + [Cloudflare](#cloudflare)
+    + [Nginx](#nginx)
+
+---
 
 ## Installation
 
@@ -27,81 +24,40 @@ Before continuing, please be sure to have the latest version of Docker installed
 
 1. Pull the latest version of Shynet using `docker pull milesmcc/shynet:latest`. If you don't have Docker installed, [install it](https://docs.docker.com/get-docker/).
 
-2. Have a PostgreSQL server ready to go. This can be on the same machine as the deployment, or elsewhere. You'll just need a username, password, host, and port (default is `5432`). (For info on how to setup a PostgreSQL server on Ubuntu, follow [this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04)).
+2. Have a PostgreSQL server ready to go. This can be on the same machine as the deployment, or elsewhere. You'll just need a username, password, host, and port. (For info on how to setup a PostgreSQL server on Ubuntu, follow [this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04)).
 
-3. Configure an environment file for Shynet. (For example, create a file called `.env`.) Be sure to swap out the variables below with the correct values for your setup. (The comments refer to the lines that follow. Note that Docker is weird with quotes, so it tends to be better to omit them from your env file.)
+3. Configure an environment file for Shynet, using [this file](/TEMPLATE.env) as a template. (This file is typically named `.env`.) Make sure you set the database settings, or Shynet won't be able to run.
 
-```
-# Database
-DB_NAME=<your db name>
-DB_USER=<your db user>
-DB_PASSWORD=<your db user password>
-DB_HOST=<your db host>
-DB_PORT=<your db port>
+4. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. Watch the output of the script; if it's the first run, you'll see a temporary password printed that you can use to log in. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 80 (http); this can be done using the flag `-p 80:8080` after `run`.
 
-# General Django settings
-DJANGO_SECRET_KEY=<your Django secret key; just a random string>
-# Don't leak error details to visitors, very important
-DEBUG=False
-# Unless you are using an external Celery task queue, make sure this
-# is set to True.
-CELERY_TASK_ALWAYS_EAGER=True 
-# For better security, set this to your deployment's domain. Comma separated.
-ALLOWED_HOSTS=*
-# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
-SIGNUPS_ENABLED=False
-# Change as required
-TIME_ZONE=America/New_York
-# Set to "False" if you will not be serving content over HTTPS
-SCRIPT_USE_HTTPS=True
-```
+5. Visit your service's homepage, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 4. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
 
-For more advanced deployments, you may consider adding the following settings to your environment file. **The following settings are optional, and not required for simple deployments.**
+6. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
 
-```env
-# Email settings
-EMAIL_HOST_USER=<your SMTP email user>
-EMAIL_HOST_PASSWORD=<your SMTP email password>
-EMAIL_HOST=<your SMTP email hostname>
-SERVER_EMAIL=Shynet <noreply@shynet.example.com>
+7. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
 
-# Redis and queue settings; not necessary for single-instance deployments
-REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
-# If set, make sure CELERY_TASK_ALWAYS_EAGER is False
-CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1
+## Updating Your Configuration
 
-# Other Shynet settings 
-# How frequently should the monitoring script "phone home" (in ms)?
-SCRIPT_HEARTBEAT_FREQUENCY=5000
-# Should only superusers (admins) be able to create services? This is helpful
-# when you'd like to invite others to your Shynet instance but don't want
-# them to be able to create services of their own.
-ONLY_SUPERUSERS_CREATE=False
-```
+When you first setup Shynet, you set a number of environment variables that determine first-run initialization settings (these variables start with `SHYNET_`). Once they're first set, though, changing them won't have any effect. Here's how to update their values:
 
-4. Setup the Shynet database by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py migrate`.
+* Create an admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
 
-5. Create your admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
+* Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
 
-6. Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
+* Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
 
-7. Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
+---
 
-8. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 80 (http); this can be done using the flag `-p 80:8080` after `run`.
+## Enhancements
 
-9. Visit your service's homepage, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 5. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
-
-10. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
-
-11. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
 
 ### Installation with SSL
 
-If you are going to be running Shynet through a reverse proxy, please use the [Basic Installation](#basic-installation) guide instead.
+If you are going to be running Shynet through a reverse proxy, please see [Configuring a Reverse Proxy](#configuring-a-reverse-proxy) instead.
 
 0. We'll be cloning this into the home directory to make this installation easier, so run `cd ~/` if you need to.
 
-1. Instead of pulling from Docker, we will be pulling from GitHub and building in Docker so that we may easily add SSL certificates. You will want to run `git clone https://github.com/milesmcc/shynet.git` to clone the GitHub repo to your computer.
+1. Instead of pulling from Docker, we will be pulling from GitHub and building using Docker in order to easily add SSL certificates. You will want to run `git clone https://github.com/milesmcc/shynet.git` to clone the GitHub repo to your current working directory.
 
 2. To install `certbot` follow [the guide here](https://certbot.eff.org/instructions) or follow along below
    * Ubuntu 18.04
@@ -113,7 +69,7 @@ If you are going to be running Shynet through a reverse proxy, please use the [B
      * `sudo apt-get install certbot`
 
 3. Run `sudo certbot certonly --standalone` and follow the instructions to generate your SSL certificate.
-   * If you registering it to a domain name like `example.com`, please be sure to point your DNS records to your server before running `certbot`.
+   * If you registering the certificate to a domain name like `example.com`, please be sure to point your DNS records to your current server before running `certbot`.
 
 4. We are going to move the SSL certificates to the Shynet repo with with command below. Replace `<domain>` with the domain name you used in step 3.
    * `cp /etc/letsencrypt/live/<domain>/{cert,privkey}.pem ~/shynet/shynet/`
@@ -123,77 +79,88 @@ If you are going to be running Shynet through a reverse proxy, please use the [B
    * `mv ~/shynet/shynet/ssl.webserver.sh ~/shynet/shynet/webserver.sh`
 
 6. Now we build the image!
-   * `docker image build shynet -t milesmcc/shynet:latest-ssl`
+   * `docker image build shynet -t shynet-ssl:latest`
 
 7. Have a PostgreSQL server ready to go. This can be on the same machine as the deployment, or elsewhere. You'll just need a username, password, host, and port (default is `5432`). (For info on how to setup a PostgreSQL server on Ubuntu, follow [this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04)).
 
-8. Configure an environment file for Shynet. (For example, create a file called `.env`.) Be sure to swap out the variables below with the correct values for your setup. (The comments refer to the lines that follow. Note that Docker is weird with quotes, so it tends to be better to omit them from your env file.)
+8. Follow the [Basic Installation](#basic-installation) guide with just one modification: in step #4, change the local bind port from `80` to `443`, and use `shynet-ssl:latest` as your Docker image instead of `milesmcc/shynet:latest`.
 
-```
-# Database
-DB_NAME=<your db name>
-DB_USER=<your db user>
-DB_PASSWORD=<your db user password>
-DB_HOST=<your db host>
-DB_PORT=<your db port>
+### Configuring a Reverse Proxy
 
-# General Django settings
-DJANGO_SECRET_KEY=<your Django secret key; just a random string>
-# Don't leak error details to visitors, very important
-DEBUG=False
-# Unless you are using an external Celery task queue, make sure this
-# is set to True.
-CELERY_TASK_ALWAYS_EAGER=True 
-# For better security, set this to your deployment's domain. Comma separated.
-ALLOWED_HOSTS=*
-# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
-SIGNUPS_ENABLED=False
-# Change as required
-TIME_ZONE=America/New_York
-# Set to "False" if you will not be serving content over HTTPS
-SCRIPT_USE_HTTPS=True
-```
+A reverse proxy has many benefits. It can be used for DDoS protection, caching files to reduce server load, routing HTTPS and/or HTTP connections, hosting multiple services on a single server, [and more](https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/)!
 
-For more advanced deployments, you may consider adding the following settings to your environment file. **The following settings are optional, and not required for simple deployments.**
+#### Cloudflare
 
-```env
-# Email settings
-EMAIL_HOST_USER=<your SMTP email user>
-EMAIL_HOST_PASSWORD=<your SMTP email password>
-EMAIL_HOST=<your SMTP email hostname>
-SERVER_EMAIL=Shynet <noreply@shynet.example.com>
+[Cloudflare](https://www.cloudflare.com/) is a great reverse proxy option. It's free, automatically configures HTTPs, offers out-of-the-box security features, provides DNS, and requires minimal setup.
 
-# Redis and queue settings; not necessary for single-instance deployments
-REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
-# If set, make sure CELERY_TASK_ALWAYS_EAGER is False
-CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1
+1. Follow Cloudflare's [getting started guide](https://support.cloudflare.com/hc/en-us/articles/201720164-Creating-a-Cloudflare-account-and-adding-a-website).
 
+2. After setting up Cloudflare, here are a few things you should consider doing:
+   * Under the `SSL` Tab > `Overview` > Change your `SSL/TLS Encryption Mode` to `Flexible`
+   * The following will block your admin panel from anyone who isn't on your IP address. This is optional, but great for security.
+     * Under the `Firewall` tab > `Overview` > `+ Create Firewall Rule`:
+     * Name: `Admin Panel Restriction`
+     * Field: `URI Path`
+     * Operator: `equals`
+     * Value: `/admin`
+     * Click `AND`
+     * Field: `IP Address`
+     * Operator: `does not equal`
+     * Value: `<your public IP address>`
+     * Then: `Block`
 
-# Other Shynet settings 
-# How frequently should the monitoring script "phone home" (in ms)?
-SCRIPT_HEARTBEAT_FREQUENCY=5000
-# Should only superusers (admins) be able to create services? This is helpful
-# when you'd like to invite others to your Shynet instance but don't want
-# them to be able to create services of their own.
-ONLY_SUPERUSERS_CREATE=False
-```
+#### Nginx
 
-9. Setup the Shynet database by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py migrate`.
+Nginx is a self hosted, highly configurable webserver. Nginx can be configured to run as a reverse proxy on either the same machine or a remote machine.
 
-10. Create your admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
+##### Set up
 
-11. Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
+> **These commands assume Ubuntu.** If you're installing Nginx on a different platform, the process will be different.
 
-12. Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
+0. Before starting, shut down your Docker containers (if any are running)
+   * Run `docker container ls` to find the container ID
+   * Run `docker stop <container id from the last step>`
 
-13. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest-ssl`. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 443 (https); this can be done using the flag `-p 443:8080` after `run`.
+1. Update your packages and install Nginx
+   * `sudo apt-get update`
+   * `sudo apt-get install nginx`
 
-14. Visit your service's homepage using `https://`, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 10. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
+2. Disable the default Nginx placeholder
+   * `sudo unlink /etc/nginx/sites-enabled/default`
 
-15. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
+3. Create the Nginx reverse proxy config file
+   * `cd /etc/nginx/sites-available/`
+   * `vi reverse-proxy.conf` or `nano reverse-proxy.conf`
+   * Paste the following configuration into that file:
 
-16. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
+   ```nginx
+   # Know what you're pasting! Read the Reference!
+   # Reference: https://nginx.org/en/docs/
+   server {
+       listen 80;
+       location / {
+           proxy_pass http://127.0.0.1:8080;
+       }
+   }
+   ```
 
----
+   * Save and exit the text editor
+     * `:wq` for vi
+     * `ctrl+x` then `y` for nano
+   * Link Nginx's `sites-enabled` to read the new config
+     * `sudo ln -s /etc/nginx/sites-available/reverse-proxy.conf /etc/nginx/sites-enabled/reverse-proxy.conf`
+   * Make sure the config is working
+     * `service nginx configtest`
+     * `service nginx restart`
 
-**Next steps:** while out of the scope of this short guide, next steps include setting up Shynet behind a reverse proxy (be it your own [Nginx server](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) or [Cloudflare](https://cloudflare.com)), making it run in the background, and integrating it on your sites. Integration instructions are available on each service's management page.
+4. Restart your Docker image, but this time use `8080` as the local bind port, as that's where we configured Nginx to look
+   * `cd ~/`
+   * `docker run -p 8080:8080 --env-file=<your env file> milesmcc/shynet:latest`
+
+5. Finally, time to test!
+   * Go to `http://<your site>/admin`
+
+6. If everything is working as expected, please read through some of the following links below to customize Nginx
+   * [How to add SSL/HTTPS to Nginx (Ubuntu 18.04)](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04)
+   * [How to add SSL/HTTPS to Nginx (Ubuntu 16.04)](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04)
+   * [Nginx Documentation](https://nginx.org/en/docs/)

README.md

@@ -8,7 +8,7 @@
     <br>
     <strong><a href="#installation">Getting started »</a></strong>
   </p>
-  <p align="center"><a href="#screenshots">Screenshots</a> &bull; <a href="#features">Features</a> &bull; <a href="#roadmap">Roadmap</a></p>
+  <p align="center"><a href="#screenshots">Screenshots</a> &bull; <a href="#features">Features</a> &bull; <a href="https://github.com/milesmcc/a17t">Design</a></p>
 </p>
 
 <br>
@@ -39,6 +39,8 @@ _A real service page, where you can see higher-level details about a site._
 
 Not shown: management view, session view, full service view. (You'll need to install Shynet for yourself to see those!)
 
+> **Shynet is built using [a17t](https://github.com/milesmcc/a17t),** an atomic design library. Customization and extension is simple; [learn more about a17t](https://github.com/milesmcc/a17t).
+
 ## Features
 
 #### Architecture
@@ -91,7 +93,8 @@ Shynet is pretty simple, but there are a few key terms you need to know in order
 
 ## Installation
 
-You can find installation instructions in the [Getting Started Guide](GUIDE.md#installation).
+You can find installation instructions in the [Getting Started Guide](GUIDE.md#installation). Out of the box, we support deploying via a simple
+Docker container, docker-compose, or Kubernetes (see [kubernetes](/kubernetes)).
 
 ## FAQ
 
@@ -101,14 +104,7 @@ You can find installation instructions in the [Getting Started Guide](GUIDE.md#i
 
 ## Roadmap
 
-The following features are planned:
-
-* **Rollups** (aggregate old data to save space)
-* **Anomaly detection** (get email alerts when you get a traffic spike or dip)
-* **Interactive traffic heatmap** (see where in the world your visitors are coming from)
-* **Better collaboration interface** (the current interface is... a draft)
-* **Data deletion tool** (easily prune user data by specifying an ID or IP)
-* **Differential privacy** (explore and share your data without revealing any personal information)
+To see the upcoming planned features, check out the repository's [roadmap project](https://github.com/milesmcc/shynet/projects/1). Upcoming features include data aggregation through rollups, anomaly detection, detailed data exports, two-factor authentication, and a data deletion tool.
 
 ## In the Wild
 
@@ -116,7 +112,7 @@ These sites use Shynet to monitor usage without violating visitors' privacy: [Po
 
 ## Contributing
 
-Are you interested in contributing to Shynet? Just send a pull request! Maybe once the project matures there will be more detailed contribution guidelines, but for now just send the code this way. Just know that by contributing, you agree to share all of your contributions under the same license as the project (see [LICENSE](LICENSE)).
+Are you interested in contributing to Shynet? Just send a pull request! Maybe once the project matures there will be more detailed contribution guidelines, but for now just send the code this way and we'll make sure it meets our standards together. Just know that by contributing, you agree to share all of your contributions under the same license as the project (see [LICENSE](LICENSE)). And always be sure to follow the [Code of Conduct](https://github.com/milesmcc/shynet/blob/master/CODE_OF_CONDUCT.md).
 
 ## License
 

TEMPLATE.env

@@ -0,0 +1,64 @@
+# This file shows all of the environment variables you can
+# set to configure Shynet, as well as information about their
+# effects. Make a copy of this file to configure your deployment.
+
+# Database settings (PostgreSQL)
+DB_NAME=shynet_db
+DB_USER=shynet_db_user
+DB_PASSWORD=shynet_db_user_password
+DB_HOST=db
+DB_PORT=5432
+
+# Email settings (optional)
+EMAIL_HOST_USER=example
+EMAIL_HOST_PASSWORD=example_password
+EMAIL_HOST=smtp.example.com
+SERVER_EMAIL=<Shynet> noreply@shynet.example.com
+
+# General Django settings
+DJANGO_SECRET_KEY=random_string
+
+# For better security, set this to your deployment's domain. Comma separated.
+ALLOWED_HOSTS=*
+
+# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
+SIGNUPS_ENABLED=False
+
+# The timezone of the admin panel. Affects how dates are displayed.
+TIME_ZONE=America/New_York
+
+# Set to "False" if you will not be serving content over HTTPS
+SCRIPT_USE_HTTPS=True
+
+# How frequently should the monitoring script "phone home" (in ms)?
+SCRIPT_HEARTBEAT_FREQUENCY=5000
+
+# How much time can elapse between requests from the same user before a new
+# session is created, in seconds?
+SESSION_MEMORY_TIMEOUT=1800
+
+# Should only superusers (admins) be able to create services? This is helpful
+# when you'd like to invite others to your Shynet instance but don't want
+# them to be able to create services of their own.
+ONLY_SUPERUSERS_CREATE=True
+
+# Whether to perform checks and setup at startup, including applying unapplied
+# migrations. For most setups, the recommended value is True. Defaults to True.
+# Will skip only if value is False.
+PERFORM_CHECKS_AND_SETUP=True
+
+# Your admin user's email. A temporary password will be printed
+# to the console on first run.
+SHYNET_ADMIN_EMAIL=you@example.com
+
+# The domain on which you'll be hosting Shynet. 
+SHYNET_HOST=shynet.example.com
+
+# What you'd like to call your Shynet instance.
+SHYNET_WHITELABEL=My Shynet Instance
+
+# Redis and queue settings; not necessary for single-instance deployments.
+# Don't uncomment these unless you know what you are doing!
+# REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
+# If CELERY_BROKER_URL is set, make sure CELERY_TASK_ALWAYS_EAGER is False
+# CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1

docker-compose.yml

@@ -0,0 +1,32 @@
+version: '3'
+services:
+  shynet:
+    image: milesmcc/shynet:latest
+    restart: unless-stopped
+    expose:
+      - 8080
+    env_file:
+      # Create a file called '.env' if it doesn't already exist.
+      # You can use `TEMPLATE.env` as a guide.
+      - .env
+    environment:
+      - DB_HOST=db
+    networks:
+      - internal
+    depends_on:
+      - db
+  db:
+    image: postgres
+    restart: always
+    environment:
+      - "POSTGRES_USER=${DB_USER}"
+      - "POSTGRES_PASSWORD=${DB_PASSWORD}"
+      - "POSTGRES_DB=${DB_NAME}"
+    volumes:
+      - shynet_db:/var/lib/postgresql/data
+    networks:
+      - internal
+volumes:
+  shynet_db:
+networks:
+  internal:

kubernetes/deployments.yml

@@ -18,7 +18,6 @@ spec:
       containers:
       - name: "covideo-webserver"
         image: "milesmcc/shynet:latest"
-        command: ["./webserver.sh"]
         imagePullPolicy: Always
         envFrom:
         - secretRef:

shynet/analytics/migrations/0003_auto_20200502_1227.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.0.5 on 2020-05-02 16:27
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('analytics', '0002_auto_20200415_1742'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='session',
+            name='ip',
+            field=models.GenericIPAddressField(db_index=True, null=True),
+        ),
+    ]

shynet/analytics/models.py

@@ -39,7 +39,7 @@ class Session(models.Model):
         default="OTHER",
     )
     os = models.TextField()
-    ip = models.GenericIPAddressField(db_index=True)
+    ip = models.GenericIPAddressField(db_index=True, null=True)
 
     # GeoIP data
     asn = models.TextField(blank=True)

shynet/analytics/tasks.py

@@ -3,6 +3,7 @@ import logging
 
 import geoip2.database
 import user_agents
+from hashlib import sha1
 from celery import shared_task
 from django.conf import settings
 from django.core.cache import cache
@@ -59,29 +60,33 @@ def ingress_request(
         if dnt and service.respect_dnt:
             return
 
-        ip_data = _geoip2_lookup(ip)
-        log.debug(f"Found geoip2 data")
-
         # Validate payload
         if payload.get("loadTime", 1) <= 0:
             payload["loadTime"] = None
 
-        # Create or update session
-        session = (
-            Session.objects.filter(
-                service=service,
-                last_seen__gt=timezone.now() - timezone.timedelta(minutes=10),
-                ip=ip,
-                user_agent=user_agent,
-            ).first()
-            # We used to check for identifiers, but that can cause issues when people
-            # re-open the page in a new tab, for example. It's better to match sessions
-            # solely based on IP and user agent.
+        association_id_hash = sha1()
+        association_id_hash.update(str(ip).encode("utf-8"))
+        association_id_hash.update(str(user_agent).encode("utf-8"))
+        session_cache_path = (
+            f"session_association_{service.pk}_{association_id_hash.hexdigest()}"
         )
+
+        # Create or update session
+        session = None
+        if cache.get(session_cache_path) is not None:
+            cache.touch(session_cache_path, settings.SESSION_MEMORY_TIMEOUT)
+            session = Session.objects.filter(
+                pk=cache.get(session_cache_path), service=service
+            ).first()
         if session is None:
-            log.debug("Cannot link to existing session; creating a new one...")
-            ua = user_agents.parse(user_agent)
             initial = True
+
+            log.debug("Cannot link to existing session; creating a new one...")
+
+            ip_data = _geoip2_lookup(ip)
+            log.debug(f"Found geoip2 data...")
+
+            ua = user_agents.parse(user_agent)
             device_type = "OTHER"
             if (
                 ua.is_bot
@@ -98,7 +103,7 @@ def ingress_request(
                 device_type = "DESKTOP"
             session = Session.objects.create(
                 service=service,
-                ip=ip,
+                ip=ip if service.collect_ips else None,
                 user_agent=user_agent,
                 identifier=identifier.strip(),
                 browser=ua.browser.family or "",
@@ -111,9 +116,14 @@ def ingress_request(
                 latitude=ip_data.get("latitude"),
                 time_zone=ip_data.get("time_zone", ""),
             )
+            cache.set(
+                session_cache_path, session.pk, timeout=settings.SESSION_MEMORY_TIMEOUT
+            )
         else:
-            log.debug("Updating old session with new data...")
             initial = False
+
+            log.debug("Updating old session with new data...")
+
             # Update last seen time
             session.last_seen = timezone.now()
             if session.identifier == "" and identifier.strip() != "":
@@ -124,9 +134,10 @@ def ingress_request(
         idempotency = payload.get("idempotency")
         idempotency_path = f"hit_idempotency_{idempotency}"
         hit = None
+
         if idempotency is not None:
             if cache.get(idempotency_path) is not None:
-                cache.touch(idempotency_path, 10 * 60)
+                cache.touch(idempotency_path, settings.SESSION_MEMORY_TIMEOUT)
                 hit = Hit.objects.filter(
                     pk=cache.get(idempotency_path), session=session
                 ).first()
@@ -137,6 +148,7 @@ def ingress_request(
                     hit.heartbeats += 1
                     hit.last_seen = timezone.now()
                     hit.save()
+
         if hit is None:
             log.debug("Hit is a page load; creating new hit...")
             # There is no existing hit; create a new one
@@ -153,7 +165,9 @@ def ingress_request(
             )
             # Set idempotency (if applicable)
             if idempotency is not None:
-                cache.set(idempotency_path, hit.pk, timeout=10 * 60)
+                cache.set(
+                    idempotency_path, hit.pk, timeout=settings.SESSION_MEMORY_TIMEOUT
+                )
     except Exception as e:
         log.exception(e)
         raise e

shynet/analytics/views/ingress.py

@@ -3,7 +3,8 @@ import json
 
 from django.conf import settings
 from django.core.cache import cache
-from django.http import HttpResponse
+from django.core.exceptions import ValidationError
+from django.http import HttpResponse, Http404, HttpResponseBadRequest
 from django.shortcuts import render, reverse
 from django.utils import timezone
 from django.utils.decorators import method_decorator
@@ -35,14 +36,37 @@ def ingress(request, service_uuid, identifier, tracker, payload):
         identifier=identifier,
     )
 
+class ValidateServiceOriginsMixin:
+    def dispatch(self, request, *args, **kwargs):
+        try:
+            service_uuid = self.kwargs.get("service_uuid")
+            origins = cache.get(f"service_origins_{service_uuid}")
 
-class PixelView(View):
+            if origins is None:
+                service = Service.objects.get(uuid=service_uuid)
+                origins = service.origins
+                cache.set(f"service_origins_{service_uuid}", origins, timeout=3600)
+
+            resp = super().dispatch(request, *args, **kwargs)
+            resp["Access-Control-Allow-Origin"] = origins
+            resp["Access-Control-Allow-Methods"] = "GET,HEAD,OPTIONS,POST"
+            resp[
+                "Access-Control-Allow-Headers"
+            ] = "Origin, X-Requested-With, Content-Type, Accept, Authorization, Referer"
+            return resp
+        except Service.DoesNotExist:
+            raise Http404()
+        except ValidationError:
+            return HttpResponseBadRequest()
+
+
+class PixelView(ValidateServiceOriginsMixin, View):
     # Fallback view to serve an unobtrusive 1x1 transparent tracking pixel for browsers with
     # JavaScript disabled.
-    def dispatch(self, request, *args, **kwargs):
+    def get(self, *args, **kwargs):
         # Extract primary data
         ingress(
-            request,
+            self.request,
             self.kwargs.get("service_uuid"),
             self.kwargs.get("identifier", ""),
             "PIXEL",
@@ -59,23 +83,7 @@ class PixelView(View):
 
 
 @method_decorator(csrf_exempt, name="dispatch")
-class ScriptView(View):
-    def dispatch(self, request, *args, **kwargs):
-        service_uuid = self.kwargs.get("service_uuid")
-        origins = cache.get(f"service_origins_{service_uuid}")
-        if origins is None:
-            service = Service.objects.get(uuid=service_uuid)
-            origins = service.origins
-            cache.set(f"service_origins_{service_uuid}", origins, timeout=3600)
-
-        resp = super().dispatch(request, *args, **kwargs)
-        resp["Access-Control-Allow-Origin"] = origins
-        resp["Access-Control-Allow-Methods"] = "GET,HEAD,OPTIONS,POST"
-        resp[
-            "Access-Control-Allow-Headers"
-        ] = "Origin, X-Requested-With, Content-Type, Accept, Authorization, Referer"
-        return resp
-
+class ScriptView(ValidateServiceOriginsMixin, View):
     def get(self, *args, **kwargs):
         protocol = "https" if settings.SCRIPT_USE_HTTPS else "http"
         endpoint = (

shynet/core/management/commands/startup_checks.py

@@ -0,0 +1,49 @@
+import traceback
+import uuid
+
+from django.conf import settings
+from django.contrib.sites.models import Site
+from django.core.management.base import BaseCommand, CommandError
+from django.utils.crypto import get_random_string
+
+from django.db import DEFAULT_DB_ALIAS, connections
+from django.db.utils import OperationalError, ConnectionHandler
+from django.core.exceptions import ImproperlyConfigured
+
+from core.models import User
+
+
+class Command(BaseCommand):
+    help = "Internal command to perform startup sanity checks."
+
+    def check_migrations(self):
+        from django.db.migrations.executor import MigrationExecutor
+        try:
+            executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
+        except OperationalError:
+            # DB_NAME database not found?
+            return True
+        except ImproperlyConfigured:
+            # No databases are configured (or the dummy one)
+            return True
+        
+        if executor.migration_plan(executor.loader.graph.leaf_nodes()):
+            return True
+        
+        return False
+    
+    
+    def handle(self, *args, **options):
+        migration = self.check_migrations()
+
+        admin, hostname, whitelabel = [True] * 3
+        if not migration:
+            admin = not User.objects.all().exists()
+            hostname = not Site.objects.filter(domain__isnull=False).exclude(domain__exact="").exclude(domain__exact="example.com").exists()
+            whitelabel = not Site.objects.filter(name__isnull=False).exclude(name__exact="").exclude(name__exact="example.com").exists()
+
+        self.stdout.write(
+            self.style.SUCCESS(
+                f"{migration} {admin} {hostname} {whitelabel}"
+            )
+        )

shynet/core/migrations/0004_service_collect_ips.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.0.5 on 2020-05-02 16:22
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0003_service_respect_dnt'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='service',
+            name='collect_ips',
+            field=models.BooleanField(default=True),
+        ),
+    ]

shynet/core/models.py

@@ -42,6 +42,7 @@ class Service(models.Model):
         max_length=2, choices=SERVICE_STATUSES, default=ACTIVE, db_index=True
     )
     respect_dnt = models.BooleanField(default=True)
+    collect_ips = models.BooleanField(default=True)
 
     class Meta:
         ordering = ["name", "uuid"]

shynet/dashboard/forms.py

@@ -8,15 +8,17 @@ from core.models import Service, User
 class ServiceForm(forms.ModelForm):
     class Meta:
         model = Service
-        fields = ["name", "link", "respect_dnt", "origins", "collaborators"]
+        fields = ["name", "link", "respect_dnt", "collect_ips", "origins", "collaborators"]
         widgets = {
             "name": forms.TextInput(),
             "origins": forms.TextInput(),
             "respect_dnt": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
+            "collect_ips": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
         }
         labels = {
             "origins": "Allowed Hostnames",
             "respect_dnt": "Respect DNT",
+            "collect_ips": "Collect IP addresses"
         }
         help_texts = {
             "name": _("What should the service be called?"),
@@ -25,6 +27,7 @@ class ServiceForm(forms.ModelForm):
                 "At what hostnames does the service operate? This sets CORS headers, so use '*' if you're not sure (or don't care)."
             ),
             "respect_dnt": "Should visitors who have enabled <a href='https://en.wikipedia.org/wiki/Do_Not_Track'>Do Not Track</a> be excluded from all data?",
+            "collect_ips": "Should individual IP addresses be collected? IP metadata (location, host, etc) will still be collected."
         }
 
     collaborators = forms.CharField(

shynet/dashboard/templates/base.html

@@ -20,8 +20,8 @@
 <body class="bg-gray-200 min-h-full">
   {% block body %}
 
-  <section class="max-w-screen-xl mx-auto px-4 md:px-0 py-4 md:py-12 md:flex">
-    <aside class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center">
+  <section class="max-w-screen-xl mx-auto px-4 py-4 md:py-12 md:flex">
+    <aside class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center overflow-x-hidden">
       <a class="icon ~urge ml-2 md:ml-6 md:mb-8 md:mt-3" href="{% url 'dashboard:dashboard' %}">
         <i class="fas fa-binoculars fa-3x text-purple-600 hidden md:block"></i>
         <i class="fas fa-binoculars fa-2x text-purple-600 md:hidden"></i>

shynet/dashboard/templates/dashboard/includes/service_form.html

@@ -8,5 +8,6 @@
     <summary class="cursor-pointer text-sm">Advanced settings</summary>
     <hr class="sep h-4">
     {{form.respect_dnt|a17t}}
+    {{form.collect_ips|a17t}}
     {{form.origins|a17t}}
 </details>

shynet/dashboard/templates/dashboard/pages/service_session.html

@@ -60,7 +60,7 @@
         </div>
         <div>
             <p>IP</p>
-            <p class="label" title="{{session.ip}}">{{session.ip|truncatechars:"16"}}</p>
+            <p class="label" title="{{session.ip}}">{{session.ip|default:"Not Collected"|truncatechars:"16"}}</p>
         </div>
     </div>
 </article>

shynet/entrypoint.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+if [[ ! $PERFORM_CHECKS_AND_SETUP == False ]]; then
+  ./startup_checks.sh
+fi
+
+./webserver.sh

shynet/shynet/settings.py

@@ -14,7 +14,7 @@ import os
 from django.contrib.messages import constants as messages
 
 # Increment on new releases
-VERSION = "v0.2.1"
+VERSION = "v0.3.0"
 
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -27,7 +27,7 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 SECRET_KEY = os.getenv("DJANGO_SECRET_KEY", "onlyusethisindev")
 
 # SECURITY WARNING: don't run with debug turned on in production!
-DEBUG = os.getenv("DEBUG", "True") == "True"
+DEBUG = os.getenv("DEBUG", "False") == "True"
 
 ALLOWED_HOSTS = os.getenv("ALLOWED_HOSTS", "*").split(",")
 
@@ -214,7 +214,7 @@ SITE_ID = 1
 
 # Celery
 
-CELERY_TASK_ALWAYS_EAGER = os.getenv("CELERY_TASK_ALWAYS_EAGER", "False") == "True"
+CELERY_TASK_ALWAYS_EAGER = os.getenv("CELERY_TASK_ALWAYS_EAGER", "True") == "True"
 
 CELERY_BROKER_URL = os.getenv("CELERY_BROKER_URL")
 CELERY_REDIS_SOCKET_TIMEOUT = 15
@@ -261,3 +261,7 @@ SCRIPT_USE_HTTPS = os.getenv("SCRIPT_USE_HTTPS", "True") == "True"
 # How frequently should the tracking script "phone home" with a heartbeat, in
 # milliseconds?
 SCRIPT_HEARTBEAT_FREQUENCY = int(os.getenv("SCRIPT_HEARTBEAT_FREQUENCY", "5000"))
+
+# How much time can elapse between requests from the same user before a new
+# session is created, in seconds?
+SESSION_MEMORY_TIMEOUT = int(os.getenv("SESSION_MEMORY_TIMEOUT", "1800"))

shynet/startup_checks.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Check if setup is necessary, do setup as needed
+echo "Performing startup checks..."
+sanity_results=( $(./manage.py startup_checks) )
+if [[ ${sanity_results[0]} == True ]]; then
+  echo "Running migrations (setting up DB)..."
+  {
+    ./manage.py migrate && echo "Migrations complete!"
+  } || {
+    echo "Migrations failed, exiting" && exit 1
+  }
+  else
+  echo "Database is ready to go."
+fi
+if [[ -n $SHYNET_ADMIN_EMAIL && ${sanity_results[1]} == True ]]; then
+  echo "Creating an admin user..."
+  {
+    temppwd=$( ./manage.py registeradmin $SHYNET_ADMIN_EMAIL ) && echo "Admin user ($SHYNET_ADMIN_EMAIL) created! Password: $temppwd"
+  } || {
+    echo "Failed to create admin, exiting" & exit 1
+  }
+  else
+  echo "Making no changes to admin user."
+fi
+if [[ -n $SHYNET_HOST && ${sanity_results[2]} == True ]]; then
+  echo "Setting hostname..."
+  {
+    ./manage.py hostname $SHYNET_HOST && echo "Hostname set to $SHYNET_HOST!"
+  } || {
+    echo "Failed setting hostname, exiting" & exit 1
+  }
+  else
+  echo "Making no changes to hostname."
+fi
+if [[ -n $SHYNET_WHITELABEL && ${sanity_results[3]} == True ]]; then
+  echo "Setting whitelabel..."
+  {
+    ./manage.py whitelabel $SHYNET_WHITELABEL && echo "Whitelabel set! Whitelabel: $SHYNET_WHITELABEL"
+  } || {
+    echo "Failed to set whitelabel, exiting" & exit 1
+  }
+  else
+  echo "Making no changes to whitelabel."
+fi
+echo "Startup checks complete!"

shynet/webserver.sh

@@ -1,8 +1,7 @@
 #!/bin/bash
-
 # Start Gunicorn processes
 echo Launching Shynet web server...
 exec gunicorn shynet.wsgi:application \
     --bind 0.0.0.0:8080 \
     --workers 3 \
-    --timeout 100
+    --timeout 100

tests/pixel.html

@@ -8,7 +8,7 @@
 
 <body>
     <noscript><img src="//localhost:8000/ingress/test_uuid/pixel.gif"></noscript>
-    <script src="//localhost:8000/ingress/test_uuid/script.js"></script>
+    <script src="//localhost:8000/ingress/66015ce4-c69d-40fb-be8f-5535538d795e/script.js"></script>
 </body>
 
-</html>
+</html>