Bump version

Add option to ignore robots
Remove erroneous app.json default
2020-06-15 20:14:56 +00:00 · 2020-06-15 19:07:13 +02:00 · 2020-06-14 02:05:11 +00:00 · 2020-06-14 01:51:44 +00:00 · 2020-06-14 01:49:24 +00:00 · 2020-06-14 01:49:18 +00:00
34 changed files with 539 additions and 189 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -109,6 +109,9 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+Vagrantfile
+.vagrant 
+ubuntu-xenial-16.04-cloudimg-console.log

 # Spyder project settings
 .spyderproject
--- a/2
+++ b/2
@@ -32,4 +32,4 @@ RUN python manage.py collectstatic --noinput && \
 # Launch
 USER appuser
 EXPOSE 8080
-ENTRYPOINT [ "./entrypoint.sh" ]
+CMD [ "./entrypoint.sh" ]
--- a/GUIDE.md
+++ b/GUIDE.md
@@ -3,6 +3,7 @@
 ## Table of Contents

 - [Installation](#installation)
+- [Heroku](#heroku)
 - [Updating Your Configuration](#updating-your-configuration)
 - [Enhancements](#enhancements)
  * [Installation with SSL](#installation-with-ssl)
@@ -10,9 +11,11 @@
    + [Cloudflare](#cloudflare)
    + [Nginx](#nginx)
 + [Troubleshooting](#troubleshooting)
-
 ---

+## Staying Updated
+**If you install Shynet, you should strongly consider enabling notifications when new versions are released.** You can do this under the "Watch" tab on GitHub (above). This will ensure that you are notified when new versions are available, some of which may be security updates. (Shynet will never automatically update itself.)
+
 ## Installation

 Installation of Shynet is easy! Follow the [Basic Installation](#basic-installation) guide below if you'd like to run Shynet over HTTP or if you are going to be running it over HTTPS through a reverse proxy. If you'd like to run Shynet over HTTPS without a reverse proxy, skip ahead to [Installation with SSL](#installation-with-ssl) instead.
@@ -29,29 +32,36 @@ Before continuing, please be sure to have the latest version of Docker installed

 3. Configure an environment file for Shynet, using [this file](/TEMPLATE.env) as a template. (This file is typically named `.env`.) Make sure you set the database settings, or Shynet won't be able to run.

-4. Launch the Shynet server by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. Watch the output of the script; if it's the first run, you'll see a temporary password printed that you can use to log in. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 80 (http); this can be done using the flag `-p 80:8080` after `run`.
+4. Launch the Shynet server for the first time by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. Provided you're using the default environment information (i.e., `PERFORM_CHECKS_AND_SETUP` is `True`), you'll see a few warnings about not having an admin user or host setup; these are normal. Don't worry — we'll do this in the next step. You only need to stop if you see a stacktrace about being unable to connect to the database.

-5. Visit your service's homepage, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 4. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.
+5. Create an admin user by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py registeradmin <your email>`. A temporary password will be printed to the console.

-6. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.
+6. Set the hostname of your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py hostname <your public hostname>`, where `<your public hostname>` is the _publicly accessible hostname_ of your instance, including port. This setting affects the URL that the tracking script sends its results to, so make sure it's correct. (Example hostnames: `shynet.rmrm.io` or `example.com:8000`.)

-7. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
+7. Set the whitelabel of your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py whitelabel <whitelabel>`. While this setting doesn't affect any core operations of Shynet, it lets you rename Shynet to whatever you want. (Example whitelabels: `"My Shynet Instance"` or `"Acme Analytics"`.)

-## Updating Your Configuration
+8. Launch your webserver by running `docker run --env-file=<your env file> milesmcc/shynet:latest`. You may need to bind Docker's port 8080 (where Shynet runs) to your local port 80 (http); this can be done using the flag `-p 80:8080` after `run`. Visit your service's homepage, and verify everything looks right! You should see a login prompt. Log in with the credentials from step 5. You'll probably be prompted to "confirm your email"—if you haven't set up an email server, the confirmation email will be printed to the console instead.

-When you first setup Shynet, you set a number of environment variables that determine first-run initialization settings (these variables start with `SHYNET_`). Once they're first set, though, changing them won't have any effect. Be sure to run the following commands in the same way that you deploy Shynet (i.e., linked to the same database).
+9. Create a service by clicking "+ Create Service" in the top right hand corner. Fill out the options as appropriate. Once you're done, press "create" and you'll be redirected to your new service's analytics page.

-* Create an admin account by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py registeradmin <your email>`. The command will print a temporary password that you'll be able to use to log in.
+10. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.

-* Configure Shynet's hostname (e.g. `shynet.example.com` or `localhost:8000`) by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py hostname "<your hostname>"`. This doesn't affect Shynet's bind port; instead, it determines what hostname to inject into the tracking script. (So you'll want to use the "user-facing" hostname here.)
+## Heroku

-* Name your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest python manage.py whitelabel "<your instance name>"`. This could be something like "My Shynet Server" or "Acme Analytics"—whatever suits you.
+You may wish to deploy Shynet on Heroku. Note that Heroku's free offerings (namely the free Postgres addon) are unlikely to support running any Shynet instance that records more than a few hundred requests per day &mdash; the database will quickly fill up. In most cases, the more cost-effective option for running Shynet is renting a VPS from a full cloud service provider. However, if you're sure Heroku is the right option for you, or you just want to try Shynet out, you can use the Quick Deploy button then follow the steps below. 
+
+[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/milesmcc/shynet/tree/master)
+
+Once you deploy, you'll need to setup an admin user, whitelabel, and hostname before you can use Shynet. Do that with the following commands:
+
+1. `heroku run --app=<your app> ./manage.py registeradmin <your email>`
+2. `heroku run --app=<your app> ./manage.py hostname <the hostname where you will run Shynet>`
+3. `heroku run --app=<your app> ./manage.py whitelabel "<your Shynet instance's name>"`

 ---

 ## Enhancements

-
 ### Installation with SSL

 If you are going to be running Shynet through a reverse proxy, please see [Configuring a Reverse Proxy](#configuring-a-reverse-proxy) instead.
@@ -114,8 +124,6 @@ A reverse proxy has many benefits. It can be used for DDoS protection, caching f

 Nginx is a self hosted, highly configurable webserver. Nginx can be configured to run as a reverse proxy on either the same machine or a remote machine.

-##### Set up
-
 > **These commands assume Ubuntu.** If you're installing Nginx on a different platform, the process will be different.

 0. Before starting, shut down your Docker containers (if any are running)
@@ -140,6 +148,7 @@ Nginx is a self hosted, highly configurable webserver. Nginx can be configured t
   server {
       listen 80;
       location / {
+           proxy_set_header X-Real-IP $remote_addr;
           proxy_pass http://127.0.0.1:8080;
       }
   }
@@ -181,7 +190,12 @@ Here are solutions for some common issues. If your situation isn't described her
 #### Shynet isn't linking different pageviews from the same visitor into a single session!

 * Verify that your cache is properly configured. (See #2 above.) In multi-instance deployments, it's critical that all webservers are using the _same_ cache—so make sure you configure a Redis cache if you're using a non-default installation.
+* This can happen between Shynet restarts if you're not using an external cache provider (like Redis).

 #### I changed the `SHYNET_WHITELABEL`/`SHYNET_HOST` environment variable, but nothing happened!

-* Those values only affect how your Shynet instance is setup on first run; once it's configured, they have no effect. See [updating your configuration](#updating-your-configuration) for help on how to update your configuration.
+* Those values only affect how your Shynet instance is setup on first run; once it's configured, they have no effect. See [updating your configuration](#updating-your-configuration) for help on how to update your configuration. (Note: these environment variables are not present in newer Shynet versions; they have been removed from the guide.)
+
+#### Shynet can't connect to my database running on `localhost`/`127.0.0.1`
+
+* The problem is likely that to Shynet, `localhost` points to the local network in the container itself, not on the host machine. Try adding the `--network='host'` option when you run Docker.
--- a/3
+++ b/3
@@ -23,6 +23,9 @@ psycopg2-binary = "*"
 redis = "*"
 django-redis-cache = "*"
 pycountry = "*"
+ipaddress = "*"
+html2text = "*"
+django-health-check = "*"

 [pipenv]
 allow_prereleases = true
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -1,7 +1,7 @@
 {
    "_meta": {
        "hash": {
-            "sha256": "2cd3e33ea333a40476d2030b6be66826e93c3a4de67032655061725835c92f09"
+            "sha256": "327b897f359bad486c08fc88fb70a1f9d2edaf1aadafcb1d31e5b3e144125ff7"
        },
        "pipfile-spec": 6,
        "requires": {},
@@ -16,10 +16,10 @@
    "default": {
        "amqp": {
            "hashes": [
-                "sha256:6e649ca13a7df3faacdc8bbb280aa9a6602d22fd9d545336077e573a1f4ff3b8",
-                "sha256:77f1aef9410698d20eaeac5b73a87817365f457a507d82edf292e12cbb83b08d"
+                "sha256:24dbaff8ce4f30566bb88976b398e8c4e77637171af3af6f1b9650f48890e60b",
+                "sha256:bb68f8d2bced8f93ccfd07d96c689b716b3227720add971be980accfc2952139"
            ],
-            "version": "==2.5.2"
+            "version": "==2.6.0"
        },
        "asgiref": {
            "hashes": [
@@ -37,18 +37,18 @@
        },
        "celery": {
            "hashes": [
-                "sha256:108a0bf9018a871620936c33a3ee9f6336a89f8ef0a0f567a9001f4aa361415f",
-                "sha256:5b4b37e276033fe47575107a2775469f0b721646a08c96ec2c61531e4fe45f2a"
+                "sha256:c3f4173f83ceb5a5c986c5fdaefb9456de3b0729a72a5776e46bd405fda7b647",
+                "sha256:d1762d6065522879f341c3d67c2b9fe4615eb79756d59acb1434601d4aca474b"
            ],
            "index": "pypi",
-            "version": "==4.4.2"
+            "version": "==4.4.5"
        },
        "certifi": {
            "hashes": [
-                "sha256:1d987a998c75633c40847cc966fcf5904906c920a7f17ef374f5aa4282abd304",
-                "sha256:51fcb31174be6e6664c5f69e3e1691a2d72a1a12e90f872cbdb1567eb47b6519"
+                "sha256:5ad7e9a056d25ffa5082862e36f119f7f7cec6457fa07ee2f8c339814b80c9b1",
+                "sha256:9cd41137dc19af6a5e03b630eefe7d1f458d964d406342dd3edf625839b944cc"
            ],
-            "version": "==2020.4.5.1"
+            "version": "==2020.4.5.2"
        },
        "chardet": {
            "hashes": [
@@ -59,25 +59,33 @@
        },
        "defusedxml": {
            "hashes": [
-                "sha256:6687150770438374ab581bb7a1b327a847dd9c5749e396102de3fad4e8a3ef93",
-                "sha256:f684034d135af4c6cbb949b8a4d2ed61634515257a67299e5f940fbaa34377f5"
+                "sha256:8ede8ba04cf5bf7999e1492fa77df545db83717f52c5eab625f97228ebd539bf",
+                "sha256:aa621655d72cdd30f57073893b96cd0c3831a85b08b8e4954531bdac47e3e8c8"
            ],
-            "version": "==0.6.0"
+            "version": "==0.7.0rc1"
        },
        "django": {
            "hashes": [
-                "sha256:642d8eceab321ca743ae71e0f985ff8fdca59f07aab3a9fb362c617d23e33a76",
-                "sha256:d4666c2edefa38c5ede0ec1655424c56dc47ceb04b6d8d62a7eac09db89545c1"
+                "sha256:db4c9b29615d17f808f2b1914d5cd73cd457c9fd90581195172c0888c210d944",
+                "sha256:dd96f98ec1c3e60877d45cea7350215f16de409848d23cced8443db1b188bd9b"
            ],
            "index": "pypi",
-            "version": "==3.0.5"
+            "version": "==3.1a1"
        },
        "django-allauth": {
            "hashes": [
-                "sha256:7ab91485b80d231da191d5c7999ba93170ef1bf14ab6487d886598a1ad03e1d8"
+                "sha256:f17209410b7f87da0a84639fd79d3771b596a6d3fc1a8e48ce50dabc7f441d30"
            ],
            "index": "pypi",
-            "version": "==0.41.0"
+            "version": "==0.42.0"
+        },
+        "django-health-check": {
+            "hashes": [
+                "sha256:0563827e003d25fd4d9ebbd7467dea5f390435628d645aaa63f8889deaded73a",
+                "sha256:9e6b7d93d4902901474efd4e25d31b5aaea7563b570c0260adce52cd3c3a9e36"
+            ],
+            "index": "pypi",
+            "version": "==3.12.1"
        },
        "django-ipware": {
            "hashes": [
@@ -102,6 +110,12 @@
            "index": "pypi",
            "version": "==1.2.1"
        },
+        "future": {
+            "hashes": [
+                "sha256:b1bead90b70cf6ec3f0710ae53a525360fa360d306a86583adc6bf83a4db537d"
+            ],
+            "version": "==0.18.2"
+        },
        "geoip2": {
            "hashes": [
                "sha256:5869e987bc54c0d707264fec4710661332cc38d2dca5a7f9bb5362d0308e2ce0",
@@ -118,6 +132,14 @@
            "index": "pypi",
            "version": "==20.0.4"
        },
+        "html2text": {
+            "hashes": [
+                "sha256:c7c629882da0cf377d66f073329ccf34a12ed2adf0169b9285ae4e63ef54c82b",
+                "sha256:e296318e16b059ddb97f7a8a1d6a5c1d7af4544049a01e261731d2d5cc277bbb"
+            ],
+            "index": "pypi",
+            "version": "==2020.1.16"
+        },
        "idna": {
            "hashes": [
                "sha256:7588d1c14ae4c77d74036e8c22ff447b26d0fde8f007354fd48a7814db15b7cb",
@@ -125,18 +147,26 @@
            ],
            "version": "==2.9"
        },
+        "ipaddress": {
+            "hashes": [
+                "sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc",
+                "sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2"
+            ],
+            "index": "pypi",
+            "version": "==1.0.23"
+        },
        "kombu": {
            "hashes": [
-                "sha256:2d1cda774126a044d91a7ff5fa6d09edf99f46924ab332a810760fe6740e9b76",
-                "sha256:598e7e749d6ab54f646b74b2d2df67755dee13894f73ab02a2a9feb8870c7cb2"
+                "sha256:437b9cdea193cc2ed0b8044c85fd0f126bb3615ca2f4d4a35b39de7cacfa3c1a",
+                "sha256:dc282bb277197d723bccda1a9ba30a27a28c9672d0ab93e9e51bb05a37bd29c3"
            ],
-            "version": "==4.6.8"
+            "version": "==4.6.10"
        },
        "maxminddb": {
            "hashes": [
-                "sha256:d0ce131d901eb11669996b49a59f410efd3da2c6dbe2c0094fe2fef8d85b6336"
+                "sha256:f4d28823d9ca23323d113dc7af8db2087aa4f657fafc64ff8f7a8afda871425b"
            ],
-            "version": "==1.5.2"
+            "version": "==1.5.4"
        },
        "oauthlib": {
            "hashes": [
@@ -197,10 +227,10 @@
        },
        "pytz": {
            "hashes": [
-                "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d",
-                "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"
+                "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed",
+                "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"
            ],
-            "version": "==2019.3"
+            "version": "==2020.1"
        },
        "pyyaml": {
            "hashes": [
@@ -221,11 +251,11 @@
        },
        "redis": {
            "hashes": [
-                "sha256:0dcfb335921b88a850d461dc255ff4708294943322bd55de6cfd68972490ca1f",
-                "sha256:b205cffd05ebfd0a468db74f0eedbff8df1a7bfc47521516ade4692991bb0833"
+                "sha256:0e7e0cfca8660dea8b7d5cd8c4f6c5e29e11f31158c0b0ae91a397f00e5a05a2",
+                "sha256:432b788c4530cfe16d8d943a09d40ca6c16149727e4afe8c2c9d5580c59d9f24"
            ],
            "index": "pypi",
-            "version": "==3.4.1"
+            "version": "==3.5.3"
        },
        "requests": {
            "hashes": [
@@ -250,10 +280,10 @@
        },
        "six": {
            "hashes": [
-                "sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
-                "sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
+                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
+                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
            ],
-            "version": "==1.14.0"
+            "version": "==1.15.0"
        },
        "sqlparse": {
            "hashes": [
@@ -294,20 +324,20 @@
        },
        "whitenoise": {
            "hashes": [
-                "sha256:0f9137f74bd95fa54329ace88d8dc695fbe895369a632e35f7a136e003e41d73",
-                "sha256:62556265ec1011bd87113fb81b7516f52688887b7a010ee899ff1fd18fd22700"
+                "sha256:60154b976a13901414a25b0273a841145f77eb34a141f9ae032a0ace3e4d5b27",
+                "sha256:6dd26bfda3af29177d8ab7333a0c7b7642eb615ce83764f4d15a9aecda3201c4"
            ],
            "index": "pypi",
-            "version": "==5.0.1"
+            "version": "==5.1.0"
        }
    },
    "develop": {
        "appdirs": {
            "hashes": [
-                "sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92",
-                "sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e"
+                "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41",
+                "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"
            ],
-            "version": "==1.4.3"
+            "version": "==1.4.4"
        },
        "attrs": {
            "hashes": [
@@ -326,10 +356,10 @@
        },
        "click": {
            "hashes": [
-                "sha256:8a18b4ea89d8820c5d0c7da8a64b2c324b4dabb695804dbfea19b9be9d88c0cc",
-                "sha256:e345d143d80bf5ee7534056164e5e112ea5e22716bbb1ce727941f4c8b471b9a"
+                "sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a",
+                "sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc"
            ],
-            "version": "==7.1.1"
+            "version": "==7.1.2"
        },
        "pathspec": {
            "hashes": [
@@ -340,36 +370,36 @@
        },
        "regex": {
            "hashes": [
-                "sha256:08119f707f0ebf2da60d2f24c2f39ca616277bb67ef6c92b72cbf90cbe3a556b",
-                "sha256:0ce9537396d8f556bcfc317c65b6a0705320701e5ce511f05fc04421ba05b8a8",
-                "sha256:1cbe0fa0b7f673400eb29e9ef41d4f53638f65f9a2143854de6b1ce2899185c3",
-                "sha256:2294f8b70e058a2553cd009df003a20802ef75b3c629506be20687df0908177e",
-                "sha256:23069d9c07e115537f37270d1d5faea3e0bdded8279081c4d4d607a2ad393683",
-                "sha256:24f4f4062eb16c5bbfff6a22312e8eab92c2c99c51a02e39b4eae54ce8255cd1",
-                "sha256:295badf61a51add2d428a46b8580309c520d8b26e769868b922750cf3ce67142",
-                "sha256:2a3bf8b48f8e37c3a40bb3f854bf0121c194e69a650b209628d951190b862de3",
-                "sha256:4385f12aa289d79419fede43f979e372f527892ac44a541b5446617e4406c468",
-                "sha256:5635cd1ed0a12b4c42cce18a8d2fb53ff13ff537f09de5fd791e97de27b6400e",
-                "sha256:5bfed051dbff32fd8945eccca70f5e22b55e4148d2a8a45141a3b053d6455ae3",
-                "sha256:7e1037073b1b7053ee74c3c6c0ada80f3501ec29d5f46e42669378eae6d4405a",
-                "sha256:90742c6ff121a9c5b261b9b215cb476eea97df98ea82037ec8ac95d1be7a034f",
-                "sha256:a58dd45cb865be0ce1d5ecc4cfc85cd8c6867bea66733623e54bd95131f473b6",
-                "sha256:c087bff162158536387c53647411db09b6ee3f9603c334c90943e97b1052a156",
-                "sha256:c162a21e0da33eb3d31a3ac17a51db5e634fc347f650d271f0305d96601dc15b",
-                "sha256:c9423a150d3a4fc0f3f2aae897a59919acd293f4cb397429b120a5fcd96ea3db",
-                "sha256:ccccdd84912875e34c5ad2d06e1989d890d43af6c2242c6fcfa51556997af6cd",
-                "sha256:e91ba11da11cf770f389e47c3f5c30473e6d85e06d7fd9dcba0017d2867aab4a",
-                "sha256:ea4adf02d23b437684cd388d557bf76e3afa72f7fed5bbc013482cc00c816948",
-                "sha256:fb95debbd1a824b2c4376932f2216cc186912e389bdb0e27147778cf6acb3f89"
+                "sha256:08997a37b221a3e27d68ffb601e45abfb0093d39ee770e4257bd2f5115e8cb0a",
+                "sha256:112e34adf95e45158c597feea65d06a8124898bdeac975c9087fe71b572bd938",
+                "sha256:1700419d8a18c26ff396b3b06ace315b5f2a6e780dad387e4c48717a12a22c29",
+                "sha256:2f6f211633ee8d3f7706953e9d3edc7ce63a1d6aad0be5dcee1ece127eea13ae",
+                "sha256:52e1b4bef02f4040b2fd547357a170fc1146e60ab310cdbdd098db86e929b387",
+                "sha256:55b4c25cbb3b29f8d5e63aeed27b49fa0f8476b0d4e1b3171d85db891938cc3a",
+                "sha256:5aaa5928b039ae440d775acea11d01e42ff26e1561c0ffcd3d805750973c6baf",
+                "sha256:654cb773b2792e50151f0e22be0f2b6e1c3a04c5328ff1d9d59c0398d37ef610",
+                "sha256:690f858d9a94d903cf5cada62ce069b5d93b313d7d05456dbcd99420856562d9",
+                "sha256:6ad8663c17db4c5ef438141f99e291c4d4edfeaacc0ce28b5bba2b0bf273d9b5",
+                "sha256:89cda1a5d3e33ec9e231ece7307afc101b5217523d55ef4dc7fb2abd6de71ba3",
+                "sha256:92d8a043a4241a710c1cf7593f5577fbb832cf6c3a00ff3fc1ff2052aff5dd89",
+                "sha256:95fa7726d073c87141f7bbfb04c284901f8328e2d430eeb71b8ffdd5742a5ded",
+                "sha256:97712e0d0af05febd8ab63d2ef0ab2d0cd9deddf4476f7aa153f76feef4b2754",
+                "sha256:b2ba0f78b3ef375114856cbdaa30559914d081c416b431f2437f83ce4f8b7f2f",
+                "sha256:bae83f2a56ab30d5353b47f9b2a33e4aac4de9401fb582b55c42b132a8ac3868",
+                "sha256:c78e66a922de1c95a208e4ec02e2e5cf0bb83a36ceececc10a72841e53fbf2bd",
+                "sha256:cf59bbf282b627130f5ba68b7fa3abdb96372b24b66bdf72a4920e8153fc7910",
+                "sha256:e3cdc9423808f7e1bb9c2e0bdb1c9dc37b0607b30d646ff6faf0d4e41ee8fee3",
+                "sha256:e9b64e609d37438f7d6e68c2546d2cb8062f3adb27e6336bc129b51be20773ac",
+                "sha256:fbff901c54c22425a5b809b914a3bfaf4b9570eee0e5ce8186ac71eb2025191c"
            ],
-            "version": "==2020.4.4"
+            "version": "==2020.6.8"
        },
        "toml": {
            "hashes": [
-                "sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c",
-                "sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e"
+                "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
+                "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
            ],
-            "version": "==0.10.0"
+            "version": "==0.10.1"
        },
        "typed-ast": {
            "hashes": [
--- a/README.md
+++ b/README.md
@@ -79,7 +79,7 @@ Here's the information Shynet can give you about your visitors:

 ## Recommendations

-Shynet isn't for everyone. It's great for personal projects and small to medium size websites, but hasn't been tested with ultra-high traffic sites. It's also requires a fair amount of technical know-how to deploy and maintain, so if you need a one-click solution, you're best served with other tools. 
+Shynet isn't for everyone. It's great for personal projects and small to medium size websites, but hasn't been tested with ultra-high traffic sites. It also requires a fair amount of technical know-how to deploy and maintain, so if you need a one-click solution, you're best served with other tools. 

 ## Concepts

@@ -93,8 +93,7 @@ Shynet is pretty simple, but there are a few key terms you need to know in order

 ## Installation

-You can find intructions on getting started and usage in the [Usage Guide](GUIDE.md#installation). Out of the box, we support deploying via a simple
-Docker container, docker-compose, or Kubernetes (see [kubernetes](/kubernetes)).
+You can find intructions on getting started and usage in the [Usage Guide](GUIDE.md#installation). Out of the box, we support deploying via a simple Docker container, docker-compose, Heroku, or Kubernetes (see [kubernetes](/kubernetes)).

 ## FAQ

--- a/TEMPLATE.env
+++ b/TEMPLATE.env
@@ -13,6 +13,7 @@ DB_PORT=5432
 EMAIL_HOST_USER=example
 EMAIL_HOST_PASSWORD=example_password
 EMAIL_HOST=smtp.example.com
+EMAIL_PORT=465
 SERVER_EMAIL=<Shynet> noreply@shynet.example.com

 # General Django settings
@@ -24,6 +25,10 @@ ALLOWED_HOSTS=*
 # Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
 SIGNUPS_ENABLED=False

+# Should user email addresses be verified? Only set this to `required` if you've setup the email settings and allow
+# public sign-ups; otherwise, it's unnecessary.
+ACCOUNT_EMAIL_VERIFICATION=none
+
 # The timezone of the admin panel. Affects how dates are displayed.
 TIME_ZONE=America/New_York

@@ -47,18 +52,13 @@ ONLY_SUPERUSERS_CREATE=True
 # Will skip only if value is False.
 PERFORM_CHECKS_AND_SETUP=True

-# Your admin user's email. A temporary password will be printed
-# to the console on first run.
-SHYNET_ADMIN_EMAIL=you@example.com
+# The port that Shynet should bind to. Don't set this if you're deploying on Heroku.
+PORT=8080

-# The domain on which you'll be hosting Shynet. 
-SHYNET_HOST=shynet.example.com
-
-# What you'd like to call your Shynet instance.
-SHYNET_WHITELABEL=My Shynet Instance
-
-# Redis and queue settings; not necessary for single-instance deployments.
+# Redis, queue, and parellization settings; not necessary for single-instance deployments.
 # Don't uncomment these unless you know what you are doing!
+# NUM_WORKERS=1
+# Make sure you set a REDIS_CACHE_LOCATION if you have more than one frontend worker/instance.
 # REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
 # If CELERY_BROKER_URL is set, make sure CELERY_TASK_ALWAYS_EAGER is False and
 # that you have a separate queue consumer running somewhere via `celeryworker.sh`.
--- a/app.json
+++ b/app.json
@@ -0,0 +1,122 @@
+{
+  "name": "Shynet",
+  "description":"Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.",
+    "keywords":[
+      "app.json",
+      "shynet",
+      "heroku",
+      "analytics",
+      "privacy",
+      "friendly"
+  ],
+  "website": "https://github.com/milesmcc/shynet",
+  "repository": "https://github.com/milesmcc/shynet",
+  "logo": "https://github.com/milesmcc/shynet/raw/master/images/slogo.png",
+  "success_url": "/",
+  "stack": "container",
+  "addons": [
+    "heroku-postgresql:hobby-dev"
+  ],
+  "formation": {
+    "web": {
+      "quantity": 1,
+      "size": "free"
+    }
+  },
+  "env": {
+    "DB_NAME": {
+      "description": "Postgres database name (not required if using Postgres addon)",
+      "value": "shynet",
+      "required": false
+    },
+    "DB_USER": {
+      "description": "Postgres database username (not required if using Postgres addon)",
+      "value": "",
+      "required": false
+    },
+    "DB_PASSWORD": {
+      "description": "Postgres database password (not required if using Postgres addon)",
+      "value": "",
+      "required": false
+    },
+    "DB_HOST": {
+      "description": "Postgres database hostname (not required if using Postgres addon)",
+      "value": "",
+      "required": false
+    },
+    "DB_PORT": {
+      "description": "Postgres database port (not required if using Postgres addon)",
+      "value": "5432",
+      "required": false
+    },
+    "EMAIL_HOST": {
+      "description": "SMTP server hostname (for sending emails)",
+      "value": "smtp.gmail.com",
+      "required": false
+    },
+    "EMAIL_PORT": {
+      "description": "SMTP server port (for sending emails)",
+      "value": "465",
+      "required": false
+    },
+    "EMAIL_HOST_USER": {
+      "description": "SMTP server username (for sending emails)",
+      "value": "",
+      "required": false
+    },
+    "EMAIL_HOST_PASSWORD": {
+      "description": "SMTP server password (for sending emails)",
+      "value": "",
+      "required": false
+    },
+    "SERVER_EMAIL": {
+      "description": "Email address (for sending emails)",
+      "value": "<Shynet> noreply@shynet.example.com",
+      "required": false
+    },
+    "DJANGO_SECRET_KEY": {
+      "description": "Django secret key",
+      "generator": "secret"
+    },
+    "ALLOWED_HOSTS": {
+      "description": "For better security, set this to your deployment's domain. (Where you will actually host, not embed, Shynet.) Set to '*' to allow serving all domains.",
+      "value": "*",
+      "required": false
+    },
+    "SIGNUPS_ENABLED": {
+      "description": "Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended).",
+      "value": "False",
+      "required": false
+    },
+    "TIME_ZONE": {
+      "description": "The timezone of the admin panel. Affects how dates are displayed.",
+      "value": "America/New_York",
+      "required": false
+    },
+    "SCRIPT_USE_HTTPS": {
+      "description": "Set to 'False' if you will not be serving Shynet over HTTPS.",
+      "value": "True",
+      "required": false
+    },
+    "SCRIPT_HEARTBEAT_FREQUENCY": {
+      "description": "How frequently should the monitoring script 'phone home' (in ms)?",
+      "value": "5000",
+      "required": false
+    },
+    "SESSION_MEMORY_TIMEOUT": {
+      "description": "How much time can elapse between requests from the same user before a new session is created, in seconds?",
+      "value": "1800",
+      "required": false
+    },
+    "ONLY_SUPERUSERS_CREATE": {
+      "description": "Should only superusers (admins) be able to create tracked services?",
+      "value": "True",
+      "required": false
+    },
+    "PERFORM_CHECKS_AND_SETUP": {
+      "description": "Whether to perform checks and setup at startup. Recommended value is 'True' for Heroku users.",
+      "value": "True",
+      "required": false
+    }
+  }
+}
--- a/heroku.yml
+++ b/heroku.yml
@@ -0,0 +1,3 @@
+build:
+  docker:
+    web: Dockerfile
--- a/images/slogo.png
+++ b/images/slogo.png
--- a/kubernetes/deployments.yml
+++ b/kubernetes/deployments.yml
@@ -16,12 +16,12 @@ spec:
        app: "shynet-webserver"
    spec:
      containers:
-      - name: "covideo-webserver"
+      - name: "shynet-webserver"
        image: "milesmcc/shynet:latest"
        imagePullPolicy: Always
        envFrom:
        - secretRef:
-            name: django-settings
+            name: shynet-settings
 ---
 apiVersion: "apps/v1"
 kind: "Deployment"
@@ -41,43 +41,43 @@ spec:
        app: "shynet-celeryworker"
    spec:
      containers:
-      - name: "covideo-celeryworker"
+      - name: "shynet-celeryworker"
        image: "milesmcc/shynet:latest"
        command: ["./celeryworker.sh"]
        imagePullPolicy: Always
        envFrom:
        - secretRef:
-            name: django-settings
+            name: shynet-settings
 ---
 apiVersion: v1
 kind: Service
 metadata:
-  name: redis
+  name: shynet-redis
 spec:
  ports:
    - port: 6379
      name: redis
  clusterIP: None
  selector:
-    app: redis
+    app: shynet-redis
 ---
 apiVersion: apps/v1beta2
 kind: StatefulSet
 metadata:
-  name: redis
+  name: shynet-redis
 spec:
  selector:
    matchLabels:
-      app: redis
-  serviceName: redis
+      app: shynet-redis
+  serviceName: shynet-redis
  replicas: 1
  template:
    metadata:
      labels:
-        app: redis
+        app: shynet-redis
    spec:
      containers:
-        - name: redis
+        - name: shynet-redis
          image: redis:latest
          imagePullPolicy: Always
          ports:
--- a/kubernetes/secrets_template.yml
+++ b/kubernetes/secrets_template.yml
@@ -1,7 +1,7 @@
 apiVersion: v1
 kind: Secret
 metadata:
-    name: django-settings
+    name: shynet-settings
 type: Opaque
 stringData:
    # Django settings
@@ -12,8 +12,8 @@ stringData:
    TIME_ZONE: "America/New_York"

    # Redis configuration (if you use the default Kubernetes config, this will work)
-    REDIS_CACHE_LOCATION: "redis://redis.default.svc.cluster.local/0"
-    CELERY_BROKER_URL: "redis://redis.default.svc.cluster.local/1"
+    REDIS_CACHE_LOCATION: "redis://shynet-redis.default.svc.cluster.local/0"
+    CELERY_BROKER_URL: "redis://shynet-redis.default.svc.cluster.local/1"

    # PostgreSQL settings
    DB_NAME: ""
--- a/shynet/analytics/migrations/0003_auto_20200502_1227.py
+++ b/shynet/analytics/migrations/0003_auto_20200502_1227.py
@@ -6,13 +6,13 @@ from django.db import migrations, models
 class Migration(migrations.Migration):

    dependencies = [
-        ('analytics', '0002_auto_20200415_1742'),
+        ("analytics", "0002_auto_20200415_1742"),
    ]

    operations = [
        migrations.AlterField(
-            model_name='session',
-            name='ip',
+            model_name="session",
+            name="ip",
            field=models.GenericIPAddressField(db_index=True, null=True),
        ),
    ]
--- a/shynet/analytics/tasks.py
+++ b/shynet/analytics/tasks.py
@@ -1,9 +1,10 @@
+import ipaddress
 import json
 import logging
+from hashlib import sha1

 import geoip2.database
 import user_agents
-from hashlib import sha1
 from celery import shared_task
 from django.conf import settings
 from django.core.cache import cache
@@ -60,6 +61,14 @@ def ingress_request(
        if dnt and service.respect_dnt:
            return

+        try:
+            remote_ip = ipaddress.ip_network(ip)
+            for ignored_network in service.get_ignored_networks():
+                if ignored_network.supernet_of(remote_ip):
+                    return
+        except ValueError as e:
+            log.exception(e)
+
        # Validate payload
        if payload.get("loadTime", 1) <= 0:
            payload["loadTime"] = None
@@ -101,6 +110,8 @@ def ingress_request(
                device_type = "TABLET"
            elif ua.is_pc:
                device_type = "DESKTOP"
+            if device_type == "ROBOT" and service.ignore_robots:
+                return
            session = Session.objects.create(
                service=service,
                ip=ip if service.collect_ips else None,
--- a/shynet/analytics/views/ingress.py
+++ b/shynet/analytics/views/ingress.py
@@ -4,7 +4,7 @@ import json
 from django.conf import settings
 from django.core.cache import cache
 from django.core.exceptions import ValidationError
-from django.http import HttpResponse, Http404, HttpResponseBadRequest
+from django.http import Http404, HttpResponse, HttpResponseBadRequest
 from django.shortcuts import render, reverse
 from django.utils import timezone
 from django.utils.decorators import method_decorator
@@ -36,6 +36,7 @@ def ingress(request, service_uuid, identifier, tracker, payload):
        identifier=identifier,
    )

+
 class ValidateServiceOriginsMixin:
    def dispatch(self, request, *args, **kwargs):
        try:
--- a/shynet/core/management/commands/startup_checks.py
+++ b/shynet/core/management/commands/startup_checks.py
@@ -3,12 +3,11 @@ import uuid

 from django.conf import settings
 from django.contrib.sites.models import Site
-from django.core.management.base import BaseCommand, CommandError
-from django.utils.crypto import get_random_string
-
-from django.db import DEFAULT_DB_ALIAS, connections
-from django.db.utils import OperationalError, ConnectionHandler
 from django.core.exceptions import ImproperlyConfigured
+from django.core.management.base import BaseCommand, CommandError
+from django.db import DEFAULT_DB_ALIAS, connections
+from django.db.utils import ConnectionHandler, OperationalError
+from django.utils.crypto import get_random_string

 from core.models import User

@@ -18,6 +17,7 @@ class Command(BaseCommand):

    def check_migrations(self):
        from django.db.migrations.executor import MigrationExecutor
+
        try:
            executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
        except OperationalError:
@@ -32,18 +32,25 @@ class Command(BaseCommand):

        return False

-    
    def handle(self, *args, **options):
        migration = self.check_migrations()

        admin, hostname, whitelabel = [True] * 3
        if not migration:
            admin = not User.objects.all().exists()
-            hostname = not Site.objects.filter(domain__isnull=False).exclude(domain__exact="").exclude(domain__exact="example.com").exists()
-            whitelabel = not Site.objects.filter(name__isnull=False).exclude(name__exact="").exclude(name__exact="example.com").exists()
+            hostname = (
+                not Site.objects.filter(domain__isnull=False)
+                .exclude(domain__exact="")
+                .exclude(domain__exact="example.com")
+                .exists()
+            )
+            whitelabel = (
+                not Site.objects.filter(name__isnull=False)
+                .exclude(name__exact="")
+                .exclude(name__exact="example.com")
+                .exists()
+            )

        self.stdout.write(
-            self.style.SUCCESS(
-                f"{migration} {admin} {hostname} {whitelabel}"
-            )
+            self.style.SUCCESS(f"{migration} {admin} {hostname} {whitelabel}")
        )
--- a/shynet/core/migrations/0004_service_collect_ips.py
+++ b/shynet/core/migrations/0004_service_collect_ips.py
@@ -6,13 +6,13 @@ from django.db import migrations, models
 class Migration(migrations.Migration):

    dependencies = [
-        ('core', '0003_service_respect_dnt'),
+        ("core", "0003_service_respect_dnt"),
    ]

    operations = [
        migrations.AddField(
-            model_name='service',
-            name='collect_ips',
+            model_name="service",
+            name="collect_ips",
            field=models.BooleanField(default=True),
        ),
    ]
--- a/shynet/core/migrations/0005_service_ignored_ips.py
+++ b/shynet/core/migrations/0005_service_ignored_ips.py
@@ -0,0 +1,22 @@
+# Generated by Django 3.0.6 on 2020-05-07 20:28
+
+from django.db import migrations, models
+
+import core.models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("core", "0004_service_collect_ips"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="service",
+            name="ignored_ips",
+            field=models.TextField(
+                blank=True, default="", validators=[core.models._validate_network_list]
+            ),
+        ),
+    ]
--- a/shynet/core/migrations/0006_service_hide_referrer_regex.py
+++ b/shynet/core/migrations/0006_service_hide_referrer_regex.py
@@ -0,0 +1,22 @@
+# Generated by Django 3.0.6 on 2020-05-07 21:23
+
+from django.db import migrations, models
+
+import core.models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("core", "0005_service_ignored_ips"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="service",
+            name="hide_referrer_regex",
+            field=models.TextField(
+                blank=True, default="", validators=[core.models._validate_regex]
+            ),
+        ),
+    ]
--- a/shynet/core/migrations/0007_service_ignore_robots.py
+++ b/shynet/core/migrations/0007_service_ignore_robots.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.0.6 on 2020-06-15 16:16
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0006_service_hide_referrer_regex'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='service',
+            name='ignore_robots',
+            field=models.BooleanField(default=False),
+        )
+    ]
--- a/shynet/core/models.py
+++ b/shynet/core/models.py
@@ -1,8 +1,11 @@
+import ipaddress
 import json
+import re
 import uuid

 from django.apps import apps
 from django.contrib.auth.models import AbstractUser
+from django.core.exceptions import ValidationError
 from django.db import models
 from django.db.models.functions import TruncDate
 from django.db.utils import NotSupportedError
@@ -14,6 +17,26 @@ def _default_uuid():
    return str(uuid.uuid4())


+def _validate_network_list(networks: str):
+    try:
+        _parse_network_list(networks)
+    except ValueError as e:
+        raise ValidationError(str(e))
+
+
+def _validate_regex(regex: str):
+    try:
+        re.compile(regex)
+    except re.error:
+        raise ValidationError(f"'{regex}' is not valid RegEx")
+
+
+def _parse_network_list(networks: str):
+    if len(networks.strip()) == 0:
+        return []
+    return [ipaddress.ip_network(network.strip()) for network in networks.split(",")]
+
+
 class User(AbstractUser):
    username = models.TextField(default=_default_uuid, unique=True)
    email = models.EmailField(unique=True)
@@ -42,7 +65,14 @@ class Service(models.Model):
        max_length=2, choices=SERVICE_STATUSES, default=ACTIVE, db_index=True
    )
    respect_dnt = models.BooleanField(default=True)
+    ignore_robots = models.BooleanField(default=False)
    collect_ips = models.BooleanField(default=True)
+    ignored_ips = models.TextField(
+        default="", blank=True, validators=[_validate_network_list]
+    )
+    hide_referrer_regex = models.TextField(
+        default="", blank=True, validators=[_validate_regex]
+    )

    class Meta:
        ordering = ["name", "uuid"]
@@ -50,6 +80,21 @@ class Service(models.Model):
    def __str__(self):
        return self.name

+    def get_ignored_networks(self):
+        return _parse_network_list(self.ignored_ips)
+
+    def get_ignored_referrer_regex(self):
+        if len(self.hide_referrer_regex.strip()) == 0:
+            return re.compile(r".^")  # matches nothing
+        else:
+            try:
+                return re.compile(self.hide_referrer_regex)
+            except re.error:
+                # Regexes are validated in the form, but this is an important
+                # fallback to prevent form validation and malformed source
+                # data from causing all service pages to error
+                return re.compile(r".^")
+
    def get_daily_stats(self):
        return self.get_core_stats(
            start_time=timezone.now() - timezone.timedelta(days=1)
@@ -96,12 +141,17 @@ class Service(models.Model):
            .order_by("-count")
        )

-        referrers = (
+        referrer_ignore = self.get_ignored_referrer_regex()
+        referrers = [
+            referrer
+            for referrer in (
                hits.filter(initial=True)
                .values("referrer")
                .annotate(count=models.Count("referrer"))
                .order_by("-count")
            )
+            if not referrer_ignore.match(referrer["referrer"])
+        ]

        countries = (
            sessions.values("country")
@@ -131,12 +181,6 @@ class Service(models.Model):
            .order_by("-count")
        )

-        device_types = (
-            sessions.values("device_type")
-            .annotate(count=models.Count("device_type"))
-            .order_by("-count")
-        )
-
        avg_load_time = hits.aggregate(load_time__avg=models.Avg("load_time"))[
            "load_time__avg"
        ]
--- a/shynet/dashboard/forms.py
+++ b/shynet/dashboard/forms.py
@@ -8,17 +8,33 @@ from core.models import Service, User
 class ServiceForm(forms.ModelForm):
    class Meta:
        model = Service
-        fields = ["name", "link", "respect_dnt", "collect_ips", "origins", "collaborators"]
+        fields = [
+            "name",
+            "link",
+            "respect_dnt",
+            "collect_ips",
+            "ignored_ips",
+            "ignore_robots",
+            "hide_referrer_regex",
+            "origins",
+            "collaborators",
+        ]
        widgets = {
            "name": forms.TextInput(),
            "origins": forms.TextInput(),
+            "ignored_ips": forms.TextInput(),
            "respect_dnt": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
            "collect_ips": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
+            "ignore_robots": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
+            "hide_referrer_regex": forms.TextInput(),
        }
        labels = {
            "origins": "Allowed Hostnames",
            "respect_dnt": "Respect DNT",
-            "collect_ips": "Collect IP addresses"
+            "collect_ips": "Collect IP addresses",
+            "ignored_ips": "Ignored IP addresses",
+            "ignore_robots": "Ignore robots",
+            "hide_referrer_regex": "Hide specific referrers",
        }
        help_texts = {
            "name": _("What should the service be called?"),
@@ -27,7 +43,10 @@ class ServiceForm(forms.ModelForm):
                "At what hostnames does the service operate? This sets CORS headers, so use '*' if you're not sure (or don't care)."
            ),
            "respect_dnt": "Should visitors who have enabled <a href='https://en.wikipedia.org/wiki/Do_Not_Track'>Do Not Track</a> be excluded from all data?",
-            "collect_ips": "Should individual IP addresses be collected? IP metadata (location, host, etc) will still be collected."
+            "collect_ips": "Should individual IP addresses be collected? IP metadata (location, host, etc) will still be collected.",
+            "ignored_ips": "A comma-separated list of IP addresses or IP ranges (IPv4 and IPv6) to exclude from tracking (e.g., '192.168.0.2, 127.0.0.1/32').",
+            "ignore_robots": "Should sessions generated by bots be excluded from tracking?",
+            "hide_referrer_regex": "Any referrers that match this <a href='https://regexr.com/'>RegEx</a> will not be listed in the referrer summary. Sessions will still be tracked normally. No effect if left blank.",
        }

    collaborators = forms.CharField(
--- a/shynet/dashboard/tasks.py
+++ b/shynet/dashboard/tasks.py
@@ -0,0 +1,16 @@
+from celery import shared_task
+from django.core import mail
+from django.conf import settings
+import html2text
+
+
+@shared_task
+def send_email(to: [str], subject: str, content: str, from_email: str = None):
+    text_content = html2text.html2text(content)
+    mail.send_mail(
+        subject,
+        text_content,
+        from_email or settings.DEFAULT_FROM_EMAIL,
+        to,
+        html_message=content,
+    )
--- a/shynet/dashboard/templates/dashboard/includes/service_form.html
+++ b/shynet/dashboard/templates/dashboard/includes/service_form.html
@@ -4,10 +4,13 @@
 {{form.link|a17t}}
 {{form.collaborators|a17t}}

-<details class="p-4 border rounded">
+<details {% if form.errors %}open{% endif %}>
    <summary class="cursor-pointer text-sm">Advanced settings</summary>
    <hr class="sep h-4">
    {{form.respect_dnt|a17t}}
    {{form.collect_ips|a17t}}
+    {{form.ignored_ips|a17t}}
+    {{form.ignore_robots|a17t}}
+    {{form.hide_referrer_regex|a17t}}
    {{form.origins|a17t}}
 </details>
--- a/shynet/dashboard/templates/dashboard/pages/service_update.html
+++ b/shynet/dashboard/templates/dashboard/pages/service_update.html
@@ -14,8 +14,8 @@
    <p>Place the following snippet at the end of the <code>&lt;body&gt;</code> tag on any page you'd like to track.</p>
    <div class="card ~neutral !high font-mono text-sm">
        {% filter force_escape %}<noscript><img
-                src="//{{request.site.domain}}{% url 'ingress:endpoint_pixel' object.uuid %}"></noscript>
-        <script src="//{{request.site.domain}}{% url 'ingress:endpoint_script' object.uuid %}"></script>
+                src="{{script_protocol}}{{request.site.domain}}{% url 'ingress:endpoint_pixel' object.uuid %}"></noscript>
+        <script src="{{script_protocol}}{{request.site.domain}}{% url 'ingress:endpoint_script' object.uuid %}"></script>
        {% endfilter %}
    </div>
    <hr class="sep h-4">
--- a/shynet/dashboard/templatetags/helpers.py
+++ b/shynet/dashboard/templatetags/helpers.py
@@ -81,11 +81,11 @@ def percent_change_display(start, end):

    return SafeString(direction + pct_change)

+
@register.inclusion_tag("dashboard/includes/sidebar_footer.html")
 def sidebar_footer():
-    return {
-        "version": settings.VERSION
-    }
+    return {"version": settings.VERSION}
+

@register.inclusion_tag("dashboard/includes/stat_comparison.html")
 def compare(
--- a/shynet/dashboard/views.py
+++ b/shynet/dashboard/views.py
@@ -1,3 +1,4 @@
+from django.conf import settings
 from django.contrib.auth.mixins import LoginRequiredMixin
 from django.contrib.messages.views import SuccessMessageMixin
 from django.core.cache import cache
@@ -85,6 +86,11 @@ class ServiceUpdateView(
        )
        return resp

+    def get_context_data(self, *args, **kwargs):
+        data = super().get_context_data(*args, **kwargs)
+        data["script_protocol"] = "https://" if settings.SCRIPT_USE_HTTPS else "http://"
+        return data
+

 class ServiceDeleteView(
    LoginRequiredMixin, PermissionRequiredMixin, SuccessMessageMixin, DeleteView
--- a/shynet/entrypoint.sh
+++ b/shynet/entrypoint.sh
@@ -1,7 +1,7 @@
 #!/bin/bash

 if [[ ! $PERFORM_CHECKS_AND_SETUP == False ]]; then
-  ./startup_checks.sh
+  ./startup_checks.sh && exec ./webserver.sh
+  else
+  exec ./webserver.sh
 fi
-
-./webserver.sh
--- a/shynet/manage.py
+++ b/shynet/manage.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 """Django's command-line utility for administrative tasks."""
 import os
 import sys
--- a/shynet/shynet/settings.py
+++ b/shynet/shynet/settings.py
@@ -9,12 +9,15 @@ https://docs.djangoproject.com/en/2.2/ref/settings/
 """

 import os
+# import module sys to get the type of exception
+import sys
+import urllib.parse as urlparse

 # Messages
 from django.contrib.messages import constants as messages

 # Increment on new releases
-VERSION = "v0.3.1"
+VERSION = "v0.5.2"

 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -43,6 +46,9 @@ INSTALLED_APPS = [
    "django.contrib.staticfiles",
    "django.contrib.sites",
    "django.contrib.humanize",
+    "health_check",
+    "health_check.db",
+    "health_check.cache",
    "rules.apps.AutodiscoverRulesConfig",
    "a17t",
    "core",
@@ -105,9 +111,29 @@ else:
            "PASSWORD": os.environ.get("DB_PASSWORD"),
            "HOST": os.environ.get("DB_HOST"),
            "PORT": os.environ.get("DB_PORT"),
+            "OPTIONS": {"connect_timeout": 5},
        }
    }

+# Solution to removal of Heroku DB Injection
+if 'DATABASE_URL' in os.environ:
+    if 'DATABASES' not in locals():
+        DATABASES = {}
+    url = urlparse.urlparse(os.environ['DATABASE_URL'])
+
+    # Ensure default database exists.
+    DATABASES['default'] = DATABASES.get('default', {})
+
+    # Update with environment configuration.
+    DATABASES['default'].update({
+        'NAME': url.path[1:],
+        'USER': url.username,
+        'PASSWORD': url.password,
+        'HOST': url.hostname,
+        'PORT': url.port,
+    })
+    if url.scheme == 'postgres':
+      DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'

 # Password validation
 # https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators
@@ -207,6 +233,7 @@ ACCOUNT_EMAIL_VERIFICATION = "mandatory"
 ACCOUNT_EMAIL_SUBJECT_PREFIX = ""
 ACCOUNT_USER_DISPLAY = lambda k: k.email
 ACCOUNT_SIGNUPS_ENABLED = os.getenv("ACCOUNT_SIGNUPS_ENABLED", "False") == "True"
+ACCOUNT_EMAIL_VERIFICATION = os.getenv("ACCOUNT_EMAIL_VERIFICATION", "none")

 LOGIN_REDIRECT_URL = "/"

--- a/shynet/shynet/urls.py
+++ b/shynet/shynet/urls.py
@@ -21,5 +21,6 @@ urlpatterns = [
    path("accounts/", include("allauth.urls")),
    path("ingress/", include(("analytics.ingress_urls", "ingress")), name="ingress"),
    path("dashboard/", include(("dashboard.urls", "dashboard"), namespace="dashboard")),
+    path("healthz/", include('health_check.urls')),
    path("", include(("core.urls", "core"), namespace="core")),
 ]
--- a/shynet/ssl.webserver.sh
+++ b/shynet/ssl.webserver.sh
@@ -3,8 +3,8 @@
 # Start Gunicorn processes
 echo Launching Shynet web server...
 exec gunicorn shynet.wsgi:application \
-    --bind 0.0.0.0:8080 \
-    --workers 3 \
+    --bind 0.0.0.0:${PORT:-8080} \
+    --workers ${NUM_WORKERS:-1} \
    --timeout 100 \
    --certfile=cert.pem \
    --keyfile=privkey.pem
--- a/shynet/startup_checks.sh
+++ b/shynet/startup_checks.sh
@@ -12,34 +12,13 @@ if [[ ${sanity_results[0]} == True ]]; then
  else
  echo "Database is ready to go."
 fi
-if [[ -n $SHYNET_ADMIN_EMAIL && ${sanity_results[1]} == True ]]; then
-  echo "Creating an admin user..."
-  {
-    temppwd=$( ./manage.py registeradmin $SHYNET_ADMIN_EMAIL ) && echo "Admin user ($SHYNET_ADMIN_EMAIL) created! Password: $temppwd"
-  } || {
-    echo "Failed to create admin, exiting" & exit 1
-  }
-  else
-  echo "Making no changes to admin user."
+if [[ ${sanity_results[1]} == True ]]; then
+  echo "Warning: no admin user available. Consult docs for instructions."
 fi
-if [[ -n $SHYNET_HOST && ${sanity_results[2]} == True ]]; then
-  echo "Setting hostname..."
-  {
-    ./manage.py hostname $SHYNET_HOST && echo "Hostname set to $SHYNET_HOST!"
-  } || {
-    echo "Failed setting hostname, exiting" & exit 1
-  }
-  else
-  echo "Making no changes to hostname."
+if [[ ${sanity_results[2]} == True ]]; then
+  echo "Warning: Shynet's hostname is not set. The script won't work correctly. Consult docs for instructions."
 fi
-if [[ -n $SHYNET_WHITELABEL && ${sanity_results[3]} == True ]]; then
-  echo "Setting whitelabel..."
-  {
-    ./manage.py whitelabel $SHYNET_WHITELABEL && echo "Whitelabel set! Whitelabel: $SHYNET_WHITELABEL"
-  } || {
-    echo "Failed to set whitelabel, exiting" & exit 1
-  }
-  else
-  echo "Making no changes to whitelabel."
+if [[ ${sanity_results[3]} == True ]]; then
+  echo "Warning: Shynet's whitelabel is not set. Consult docs for instructions."
 fi
 echo "Startup checks complete!"
--- a/shynet/webserver.sh
+++ b/shynet/webserver.sh
@@ -2,6 +2,6 @@
 # Start Gunicorn processes
 echo Launching Shynet web server...
 exec gunicorn shynet.wsgi:application \
-    --bind 0.0.0.0:8080 \
-    --workers 3 \
+    --bind 0.0.0.0:${PORT:-8080} \
+    --workers ${NUM_WORKERS:-1} \
    --timeout 100
Author	SHA1	Message	Date
R. Miles McCain	4ced1365d4	Bump version	2020-06-15 20:14:56 +00:00
Ruben van Erk	dcdbb7cd45	Add option to ignore robots	2020-06-15 19:07:13 +02:00
R. Miles McCain	b3102f5f32	Remove erroneous app.json default	2020-06-14 02:05:11 +00:00
R. Miles McCain	2a61cf1b51	Update GUIDE structure	2020-06-14 01:51:44 +00:00
R. Miles McCain	0d7c9c4c33	Bump version	2020-06-14 01:49:24 +00:00
R. Miles McCain	6649aeaaf0	Update dependencies	2020-06-14 01:49:18 +00:00
R. Miles McCain	cb11dc0c4e	Add mention of Heroku to docs	2020-06-14 00:12:31 +00:00
R. Miles McCain	4a4f2645df	Add Heroku setup instructions	2020-06-14 00:05:39 +00:00
R. Miles McCain	81a836df53	Improve startup behavior	2020-06-13 23:59:36 +00:00
Thomas Letsch Groch	919ca52ca1	👌 Refine code review changes	2020-06-06 06:02:33 -03:00
Thomas Letsch Groch	f7ecb88659	👌 Code review changes commented out mysql logic and refine GUIDE.md	2020-06-02 20:34:02 -03:00
Thomas Letsch Groch	1a0dcf7579	📝 Improving heroku form	2020-06-01 19:31:19 -03:00
Thomas Letsch Groch	0f3037b315	📝 update template on heroku button	2020-06-01 18:47:30 -03:00
Thomas Letsch Groch	b234ef2917	✨ Add heroku compatibility Recognize heroku with the environment variable DATABASE_URL, parse it and replace the others based on it.	2020-06-01 18:41:39 -03:00
R. Miles McCain	1b344fb90c	Bump version	2020-05-28 22:02:50 +00:00
R. Miles McCain	d164306f8b	Make email verification optional (fixes #30 )	2020-05-28 22:01:22 +00:00
R. Miles McCain	c61d23caf1	Add health check endpoint (fixes #31 )	2020-05-28 22:00:49 +00:00
R. Miles McCain	fcfbbe8809	Remove confusing setup variables; migrate to commands.	2020-05-28 21:47:17 +00:00
R. Miles McCain	1bb4aac32f	Use $PORT env variable	2020-05-28 21:40:43 +00:00
R. Miles McCain	d895eac14d	Add note about watching the repo	2020-05-24 01:47:01 +00:00
R. Miles McCain	5cce890ff6	Add localhost help to guide	2020-05-21 21:06:55 -04:00
R. Miles McCain	387c1e375d	Add database connection timeout	2020-05-21 20:59:58 -04:00
R. Miles McCain	4e13842334	Bump version	2020-05-19 17:58:21 -04:00
R. Miles McCain	62c3a87cda	Add remote address to nginx conf	2020-05-19 17:57:02 -04:00
Santiago Alessandri	cac6d44166	Quote whitelabel to allow whitespace (#33 ) The SHYNET_WHITELABEL variable was being used without quotes, which breaks the commnand if it contains whitespaces	2020-05-17 00:23:38 -04:00
R. Miles McCain	1a5f68e353	Fix typo in README	2020-05-09 12:35:59 -04:00
R. Miles McCain	4569744726	Add email sending task	2020-05-09 12:35:50 -04:00
R. Miles McCain	6978bbd03e	Bump version, cleanup	2020-05-07 17:49:18 -04:00
R. Miles McCain	d88f61b281	Add better tracking script protocol support	2020-05-07 17:49:02 -04:00
R. Miles McCain	c84dac6b01	Add referrer hiding support (closes #26 )	2020-05-07 17:44:39 -04:00
R. Miles McCain	abe37800ec	Small GUIDE expansions	2020-05-07 17:10:31 -04:00
R. Miles McCain	8aef1f0dc7	Update Kubernetes defaults	2020-05-07 17:06:04 -04:00
R. Miles McCain	1c01c27326	Merge #27 (duplication fix)	2020-05-07 16:54:49 -04:00
R. Miles McCain	a766c1eaa2	Add ip address exclusion support (closes #22 ) Co-authored-by: Anthony Abeo <anthonyabeo@gmail.com>	2020-05-07 16:53:03 -04:00
Abeo Anthony, A	a457c2be7b	remove duplicated device_types value	2020-05-07 19:59:09 +00:00
Abeo Anthony, A	6a5ce6ddb9	ignore vagrant config files	2020-05-07 19:58:31 +00:00
R. Miles McCain	bd88617dc5	Update Kubernetes settings	2020-05-05 14:42:26 -04:00
R. Miles McCain	77f1fbc2cc	Fix faulty parallelization	2020-05-04 14:20:34 -04:00