Bump version

Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.3.1 to 5.4.
- [Release notes](https://github.com/yaml/pyyaml/releases)
- [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES)
- [Commits](https://github.com/yaml/pyyaml/compare/5.3.1...5.4)

Signed-off-by: dependabot[bot] <support@github.com>

.gitignore

@@ -139,3 +139,6 @@ secrets.yml
 .vscode
 .DS_Store
 compiledstatic/
+
+# Pycharm
+.idea

GUIDE.md

@@ -4,6 +4,7 @@
 
 - [Installation](#installation)
 - [Heroku](#heroku)
+- [Render](#render)
 - [Updating Your Configuration](#updating-your-configuration)
 - [Advanced Usage](#advanced-usage)
   * [Installation with SSL](#installation-with-ssl)
@@ -12,15 +13,17 @@
     + [Nginx](#nginx)
   * [Health Checks](#health-checks)
   * [Primary Key Integration](#primary-key-integration)
+  * [Usage with Single-Page Applications](#usage-with-single-page-applications)
 + [Troubleshooting](#troubleshooting)
 ---
 
 ## Staying Updated
+
 **If you install Shynet, you should strongly consider enabling notifications when new versions are released.** You can do this under the "Watch" tab on GitHub (above). This will ensure that you are notified when new versions are available, some of which may be security updates. (Shynet will never automatically update itself.)
 
 ## Installation
 
-Installation of Shynet is easy! Follow the [Basic Installation](#basic-installation) guide below if you'd like to run Shynet over HTTP or if you are going to be running it over HTTPS through a reverse proxy. If you'd like to run Shynet over HTTPS without a reverse proxy, skip ahead to [Installation with SSL](#installation-with-ssl) instead.
+Installation of Shynet is easy! Follow the [Basic Installation](#basic-installation) guide or the [Basic Installation with Docker Compose](#basic-installation-with-docker-compose) below for a minimal installation, or if you are going to be running Shynet over HTTPS through a reverse proxy. If you'd like to run Shynet over HTTPS without a reverse proxy, skip ahead to [Installation with SSL](#installation-with-ssl) instead.
 
 > **These commands assume Ubuntu.** If you're installing Shynet on a different platform, the process will be different.
 
@@ -48,6 +51,27 @@ Before continuing, please be sure to have the latest version of Docker installed
 
 10. Finally, click on "Manage" in the top right of the service's page to get the tracking script code. Inject this script on all pages you'd like the service to track.
 
+
+### Basic Installation with Docker Compose
+
+> Make sure you have `docker-compose` installed. If not, [install it](https://docs.docker.com/compose/install/) 
+
+1. Clone the repository.
+
+2. Using [TEMPLATE.env](/TEMPLATE.env) as a template, confiure the environment for your Shynet instance and place the modified config in a file called `.env` in the root of the repository. Do _not_ change the port number at the end; you can set the public facing port in the next step. 
+
+3. On line 2 of the `nginx.conf` file located in the root of the repository, replace `example.com` with your hostname. Then, in the `docker-compose.yml` file, set the port number by replacing `8080` in line 38 ( `- 8080:80` ) with whatever local port you want to bind it to. For example, set the port number to `- 80:80` if you want your site will be available via HTTP (port 80) at `http://<your hostname>`.
+
+4. Launch the Shynet server for the first time by running `docker-compose up -d`. If you get an error like "permission denied" or "Couldn't connect to Docker daemon", either prefix the command with `sudo` or add your user to the `docker` group.
+
+5. Create an admin user by running `docker exec -it shynet_main ./manage.py registeradmin <your email>`. A temporary password will be printed to the console.
+
+6. Set the hostname of your Shynet instance by running `docker exec -it shynet_main ./manage.py hostname <your public hostname>`, where `<your public hostname>` is the same as the hostname you set in step 3. This setting affects the URL that the tracking script sends its results to, so make sure it's correct. (Example hostnames: shynet.example.com or example.com:8000.)
+
+7. Set the whitelabel of your Shynet instance by running `docker exec -it shynet_main ./manage.py whitelabel <whitelabel>`. While this setting doesn't affect any core operations of Shynet, it lets you rename Shynet to whatever you want. (Example whitelabels: "My Shynet Instance" or "Acme Analytics".)
+
+Your site should now be accessible at `http://hostname:port`. Now you can follow steps 9-10 of the [Basic Installation](#basic-installation) guide above to get Shynet integrated on your sites.
+
 ## Heroku
 
 You may wish to deploy Shynet on Heroku. Note that Heroku's free offerings (namely the free Postgres addon) are unlikely to support running any Shynet instance that records more than a few hundred requests per day &mdash; the database will quickly fill up. In most cases, the more cost-effective option for running Shynet is renting a VPS from a full cloud service provider. However, if you're sure Heroku is the right option for you, or you just want to try Shynet out, you can use the Quick Deploy button then follow the steps below. 
@@ -60,6 +84,20 @@ Once you deploy, you'll need to setup an admin user, whitelabel, and hostname be
 2. `heroku run --app=<your app> ./manage.py hostname <the hostname where you will run Shynet>`
 3. `heroku run --app=<your app> ./manage.py whitelabel "<your Shynet instance's name>"`
 
+## Render
+
+[Render](https://render.com) is a modern cloud platform to build and run all your apps and websites with free SSL, a global CDN, private networks and auto deploys from Git. To deploy Shynet, click the `Deploy to Render` button and follow the steps below.
+
+[![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/render-examples/shynet)
+
+Once your deploy has completed, use the **Render Shell** to configure your app:
+
+1. Set your email: `./manage.py registeradmin your-email@example.com`
+1. Add your onrender.com domain: `./manage.py hostname your-shynet-domain.onrender.com`
+1. Set your whitelabel: `./manage.py whitelabel "Your Shynet Instance Name"`
+
+See the [Render docs](https://render.com/docs/deploy-shynet) for more information on deploying your application on Render.
+
 ---
 
 ## Advanced Usage

Pipfile

@@ -3,17 +3,14 @@ name = "pypi"
 url = "https://pypi.org/simple"
 verify_ssl = true
 
-[pipenv]
-allow_prereleases = true
-
 [packages]
-django = "~=3.0"
+django = "~=3.1"
 django-allauth = "~=0.42.0"
 geoip2 = "~=3.0.0"
 whitenoise = "~=5.1.0"
 celery = "~=4.4.6"
 django-ipware = "~=2.1.0"
-pyyaml = "~=5.3.1"
+pyyaml = "~=5.4"
 ua-parser = "~=0.10.0"
 user-agents = "~=2.1"
 emoji-country-flag = "~=1.2.1"
@@ -21,11 +18,9 @@ rules = "~=2.2"
 gunicorn = "~=20.0.4"
 psycopg2-binary = "~=2.8.5"
 redis = "~=3.5.3"
-django-redis-cache = "~=2.1.1"
+django-redis-cache = "~=3.0.0"
 pycountry = "~=19.8.18"
 html2text = "~=2020.1.16"
 django-health-check = "~=3.12.1"
 django-npm = "~=1.0.0"
-
+django-debug-toolbar = "*"
-[dev-packages]
-black = "*"

Pipfile.lock

@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "c18d6dc7c78d5f0634e38bb81bc1cf2cd4a0c128d70ca667fe765a66b294e66e"
+            "sha256": "f8c76565a776f1bd36364077a86d6c16fccc522d9d2024bb9b51be5cb9f8b4b5"
         },
         "pipfile-spec": 6,
         "requires": {},
@@ -16,17 +16,19 @@
     "default": {
         "amqp": {
             "hashes": [
-                "sha256:24dbaff8ce4f30566bb88976b398e8c4e77637171af3af6f1b9650f48890e60b",
+                "sha256:70cdb10628468ff14e57ec2f751c7aa9e48e7e3651cfd62d431213c0c4e58f21",
-                "sha256:bb68f8d2bced8f93ccfd07d96c689b716b3227720add971be980accfc2952139"
+                "sha256:aa7f313fb887c91f15474c1229907a04dac0b8135822d6603437803424c0aa59"
             ],
-            "version": "==2.6.0"
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
+            "version": "==2.6.1"
         },
         "asgiref": {
             "hashes": [
-                "sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a",
+                "sha256:5ee950735509d04eb673bd7f7120f8fa1c9e2df495394992c73234d526907e17",
-                "sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed"
+                "sha256:7162a3cb30ab0609f1a4c95938fd73e8604f63bdba516a7f7d64b83ff09478f0"
             ],
-            "version": "==3.2.10"
+            "markers": "python_version >= '3.5'",
+            "version": "==3.3.1"
         },
         "billiard": {
             "hashes": [
@@ -37,40 +39,42 @@
         },
         "celery": {
             "hashes": [
-                "sha256:ef17d7dffde7fc73ecab3a3b6389d93d3213bac53fa7f28e68e33647ad50b916",
+                "sha256:a92e1d56e650781fb747032a3997d16236d037c8199eacd5217d1a72893bca45",
-                "sha256:fd77e4248bb1b7af5f7922dd8e81156f540306e3a5c4b1c24167c1f5f06025da"
+                "sha256:d220b13a8ed57c78149acf82c006785356071844afe0b27012a4991d44026f9f"
             ],
             "index": "pypi",
-            "version": "==4.4.6"
+            "version": "==4.4.7"
         },
         "certifi": {
             "hashes": [
-                "sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3",
+                "sha256:1a4995114262bffbc2413b159f2a1a480c969de6e6eb13ee966d470af86af59c",
-                "sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41"
+                "sha256:719a74fb9e33b9bd44cc7f3a8d94bc35e4049deebe19ba7d8e108280cfd59830"
             ],
-            "version": "==2020.6.20"
+            "version": "==2020.12.5"
         },
         "chardet": {
             "hashes": [
-                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
+                "sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa",
-                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
+                "sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5"
             ],
-            "version": "==3.0.4"
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
+            "version": "==4.0.0"
         },
         "defusedxml": {
             "hashes": [
-                "sha256:8ede8ba04cf5bf7999e1492fa77df545db83717f52c5eab625f97228ebd539bf",
+                "sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69",
-                "sha256:aa621655d72cdd30f57073893b96cd0c3831a85b08b8e4954531bdac47e3e8c8"
+                "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61"
             ],
-            "version": "==0.7.0rc1"
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
+            "version": "==0.7.1"
         },
         "django": {
             "hashes": [
-                "sha256:045be31d68dfed684831e39ab1d9e77a595f1a393935cb43b6c5451d2e78c8a4",
+                "sha256:32ce792ee9b6a0cbbec340123e229ac9f765dff8c2a4ae9247a14b2ba3a365a7",
-                "sha256:ccf6c208424c0e1b0eaffd36efe12618a9ab4d0037e26f6ffceaa5277af985d7"
+                "sha256:baf099db36ad31f970775d0be5587cc58a6256a6771a44eb795b554d45f211b8"
             ],
             "index": "pypi",
-            "version": "==3.1b1"
+            "version": "==3.1.7"
         },
         "django-allauth": {
             "hashes": [
@@ -79,13 +83,21 @@
             "index": "pypi",
             "version": "==0.42.0"
         },
-        "django-health-check": {
+        "django-debug-toolbar": {
             "hashes": [
-                "sha256:0563827e003d25fd4d9ebbd7467dea5f390435628d645aaa63f8889deaded73a",
+                "sha256:84e2607d900dbd571df0a2acf380b47c088efb787dce9805aefeb407341961d2",
-                "sha256:9e6b7d93d4902901474efd4e25d31b5aaea7563b570c0260adce52cd3c3a9e36"
+                "sha256:9e5a25d0c965f7e686f6a8ba23613ca9ca30184daa26487706d4829f5cfb697a"
             ],
             "index": "pypi",
-            "version": "==3.12.1"
+            "version": "==3.2"
+        },
+        "django-health-check": {
+            "hashes": [
+                "sha256:2667b89b8f85ad9b2a24c90581b376016d22ea912fedf37f9866413a3c2e0a5d",
+                "sha256:894738bd7e461b2405c005927403ad5ee8048bbaf5934cf30b2c81a4e047d4b0"
+            ],
+            "index": "pypi",
+            "version": "==3.12.3"
         },
         "django-ipware": {
             "hashes": [
@@ -103,25 +115,18 @@
         },
         "django-redis-cache": {
             "hashes": [
-                "sha256:06d4e48545243883f88dc9263dda6c8a0012cb7d0cee2d8758d8917eca92cece",
+                "sha256:9a2eebef421d996a82098a19d17ff6b321265cd73178fa398913019764e8394a"
-                "sha256:b19ee6654cc2f2c89078c99255e07e19dc2dba8792351d76ba7ea899d465fbb0"
             ],
             "index": "pypi",
-            "version": "==2.1.1"
+            "version": "==3.0.0"
         },
         "emoji-country-flag": {
             "hashes": [
-                "sha256:67c0cb6a3765fb53f31b34160d6b1c8a5f44b297bc278d1835c6f2e5b0a9a592",
+                "sha256:338f5e374119dcde093cfeaa8ca3af372d4b8d984d89a7fb2fb0db0011662560",
-                "sha256:ae7edb38077b0840210fa9e37673f481f2b9c032446e13ad6dab2b1108cd7ad6"
+                "sha256:a3a068191294294143d8ef294fdfe9792c5c243753eac130798bf2fa5de38185"
             ],
             "index": "pypi",
-            "version": "==1.2.1"
+            "version": "==1.2.4"
-        },
-        "future": {
-            "hashes": [
-                "sha256:b1bead90b70cf6ec3f0710ae53a525360fa360d306a86583adc6bf83a4db537d"
-            ],
-            "version": "==0.18.2"
         },
         "geoip2": {
             "hashes": [
@@ -152,6 +157,7 @@
                 "sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6",
                 "sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==2.10"
         },
         "kombu": {
@@ -159,56 +165,64 @@
                 "sha256:be48cdffb54a2194d93ad6533d73f69408486483d189fe9f5990ee24255b0e0a",
                 "sha256:ca1b45faac8c0b18493d02a8571792f3c40291cf2bcf1f55afed3d8f3aa7ba74"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
             "version": "==4.6.11"
         },
         "maxminddb": {
             "hashes": [
-                "sha256:f4d28823d9ca23323d113dc7af8db2087aa4f657fafc64ff8f7a8afda871425b"
+                "sha256:47e86a084dd814fac88c99ea34ba3278a74bc9de5a25f4b815b608798747c7dc"
             ],
-            "version": "==1.5.4"
+            "markers": "python_version >= '3.6'",
+            "version": "==2.0.3"
         },
         "oauthlib": {
             "hashes": [
                 "sha256:bee41cc35fcca6e988463cacc3bcb8a96224f470ca547e697b604cc697b2f889",
                 "sha256:df884cd6cbe20e32633f1db1072e9356f53638e4361bef4e8b03c9127c9328ea"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==3.1.0"
         },
         "psycopg2-binary": {
             "hashes": [
-                "sha256:008da3ab51adc70a5f1cfbbe5db3a22607ab030eb44bcecf517ad11a0c2b3cac",
+                "sha256:0deac2af1a587ae12836aa07970f5cb91964f05a7c6cdb69d8425ff4c15d4e2c",
-                "sha256:07cf82c870ec2d2ce94d18e70c13323c89f2f2a2628cbf1feee700630be2519a",
+                "sha256:0e4dc3d5996760104746e6cfcdb519d9d2cd27c738296525d5867ea695774e67",
-                "sha256:08507efbe532029adee21b8d4c999170a83760d38249936038bd0602327029b5",
+                "sha256:11b9c0ebce097180129e422379b824ae21c8f2a6596b159c7659e2e5a00e1aa0",
-                "sha256:107d9be3b614e52a192719c6bf32e8813030020ea1d1215daa86ded9a24d8b04",
+                "sha256:15978a1fbd225583dd8cdaf37e67ccc278b5abecb4caf6b2d6b8e2b948e953f6",
-                "sha256:17a0ea0b0eabf07035e5e0d520dabc7950aeb15a17c6d36128ba99b2721b25b1",
+                "sha256:1fabed9ea2acc4efe4671b92c669a213db744d2af8a9fc5d69a8e9bc14b7a9db",
-                "sha256:3286541b9d85a340ee4ed42732d15fc1bb441dc500c97243a768154ab8505bb5",
+                "sha256:2dac98e85565d5688e8ab7bdea5446674a83a3945a8f416ad0110018d1501b94",
-                "sha256:3939cf75fc89c5e9ed836e228c4a63604dff95ad19aed2bbf71d5d04c15ed5ce",
+                "sha256:42ec1035841b389e8cc3692277a0bd81cdfe0b65d575a2c8862cec7a80e62e52",
-                "sha256:40abc319f7f26c042a11658bf3dd3b0b3bceccf883ec1c565d5c909a90204434",
+                "sha256:6422f2ff0919fd720195f64ffd8f924c1395d30f9a495f31e2392c2efafb5056",
-                "sha256:51f7823f1b087d2020d8e8c9e6687473d3d239ba9afc162d9b2ab6e80b53f9f9",
+                "sha256:6a32f3a4cb2f6e1a0b15215f448e8ce2da192fd4ff35084d80d5e39da683e79b",
-                "sha256:6bb2dd006a46a4a4ce95201f836194eb6a1e863f69ee5bab506673e0ca767057",
+                "sha256:7312e931b90fe14f925729cde58022f5d034241918a5c4f9797cac62f6b3a9dd",
-                "sha256:702f09d8f77dc4794651f650828791af82f7c2efd8c91ae79e3d9fe4bb7d4c98",
+                "sha256:7d92a09b788cbb1aec325af5fcba9fed7203897bbd9269d5691bb1e3bce29550",
-                "sha256:7036ccf715925251fac969f4da9ad37e4b7e211b1e920860148a10c0de963522",
+                "sha256:833709a5c66ca52f1d21d41865a637223b368c0ee76ea54ca5bad6f2526c7679",
-                "sha256:7b832d76cc65c092abd9505cc670c4e3421fd136fb6ea5b94efbe4c146572505",
+                "sha256:89705f45ce07b2dfa806ee84439ec67c5d9a0ef20154e0e475e2b2ed392a5b83",
-                "sha256:8f74e631b67482d504d7e9cf364071fc5d54c28e79a093ff402d5f8f81e23bfa",
+                "sha256:8cd0fb36c7412996859cb4606a35969dd01f4ea34d9812a141cd920c3b18be77",
-                "sha256:930315ac53dc65cbf52ab6b6d27422611f5fb461d763c531db229c7e1af6c0b3",
+                "sha256:950bc22bb56ee6ff142a2cb9ee980b571dd0912b0334aa3fe0fe3788d860bea2",
-                "sha256:96d3038f5bd061401996614f65d27a4ecb62d843eb4f48e212e6d129171a721f",
+                "sha256:a0c50db33c32594305b0ef9abc0cb7db13de7621d2cadf8392a1d9b3c437ef77",
-                "sha256:a20299ee0ea2f9cca494396ac472d6e636745652a64a418b39522c120fd0a0a4",
+                "sha256:a0eb43a07386c3f1f1ebb4dc7aafb13f67188eab896e7397aa1ee95a9c884eb2",
-                "sha256:a34826d6465c2e2bbe9d0605f944f19d2480589f89863ed5f091943be27c9de4",
+                "sha256:aaa4213c862f0ef00022751161df35804127b78adf4a2755b9f991a507e425fd",
-                "sha256:a69970ee896e21db4c57e398646af9edc71c003bc52a3cc77fb150240fefd266",
+                "sha256:ac0c682111fbf404525dfc0f18a8b5f11be52657d4f96e9fcb75daf4f3984859",
-                "sha256:b9a8b391c2b0321e0cd7ec6b4cfcc3dd6349347bd1207d48bcb752aa6c553a66",
+                "sha256:ad20d2eb875aaa1ea6d0f2916949f5c08a19c74d05b16ce6ebf6d24f2c9f75d1",
-                "sha256:ba13346ff6d3eb2dca0b6fa0d8a9d999eff3dcd9b55f3a890f12b0b6362b2b38",
+                "sha256:b4afc542c0ac0db720cf516dd20c0846f71c248d2b3d21013aa0d4ef9c71ca25",
-                "sha256:bb0608694a91db1e230b4a314e8ed00ad07ed0c518f9a69b83af2717e31291a3",
+                "sha256:b8a3715b3c4e604bcc94c90a825cd7f5635417453b253499664f784fc4da0152",
-                "sha256:c8830b7d5f16fd79d39b21e3d94f247219036b29b30c8270314c46bf8b732389",
+                "sha256:ba28584e6bca48c59eecbf7efb1576ca214b47f05194646b081717fa628dfddf",
-                "sha256:cac918cd7c4c498a60f5d2a61d4f0a6091c2c9490d81bc805c963444032d0dab",
+                "sha256:ba381aec3a5dc29634f20692349d73f2d21f17653bda1decf0b52b11d694541f",
-                "sha256:cc30cb900f42c8a246e2cb76539d9726f407330bc244ca7729c41a44e8d807fb",
+                "sha256:bd1be66dde2b82f80afb9459fc618216753f67109b859a361cf7def5c7968729",
-                "sha256:ccdc6a87f32b491129ada4b87a43b1895cf2c20fdb7f98ad979647506ffc41b6",
+                "sha256:c2507d796fca339c8fb03216364cca68d87e037c1f774977c8fc377627d01c71",
-                "sha256:d1a8b01f6a964fec702d6b6dac1f91f2b9f9fe41b310cbb16c7ef1fac82df06d",
+                "sha256:cec7e622ebc545dbb4564e483dd20e4e404da17ae07e06f3e780b2dacd5cee66",
-                "sha256:e004db88e5a75e5fdab1620fb9f90c9598c2a195a594225ac4ed2a6f1c23e162",
+                "sha256:d14b140a4439d816e3b1229a4a525df917d6ea22a0771a2a78332273fd9528a4",
-                "sha256:eb2f43ae3037f1ef5e19339c41cf56947021ac892f668765cd65f8ab9814192e",
+                "sha256:d1b4ab59e02d9008efe10ceabd0b31e79519da6fb67f7d8e8977118832d0f449",
-                "sha256:fa466306fcf6b39b8a61d003123d442b23707d635a5cb05ac4e1b62cc79105cd"
+                "sha256:d5227b229005a696cc67676e24c214740efd90b148de5733419ac9aaba3773da",
+                "sha256:e1f57aa70d3f7cc6947fd88636a481638263ba04a742b4a37dd25c373e41491a",
+                "sha256:e74a55f6bad0e7d3968399deb50f61f4db1926acf4a6d83beaaa7df986f48b1c",
+                "sha256:e82aba2188b9ba309fd8e271702bd0d0fc9148ae3150532bbb474f4590039ffb",
+                "sha256:ee69dad2c7155756ad114c02db06002f4cded41132cc51378e57aad79cc8e4f4",
+                "sha256:f5ab93a2cb2d8338b1674be43b442a7f544a0971da062a5da774ed40587f18f5"
             ],
             "index": "pypi",
-            "version": "==2.8.5"
+            "version": "==2.8.6"
         },
         "pycountry": {
             "hashes": [
@@ -226,27 +240,45 @@
         },
         "pytz": {
             "hashes": [
-                "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed",
+                "sha256:83a4a90894bf38e243cf052c8b58f381bfe9a7a483f6a9cab140bc7f702ac4da",
-                "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"
+                "sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798"
             ],
-            "version": "==2020.1"
+            "version": "==2021.1"
         },
         "pyyaml": {
             "hashes": [
-                "sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97",
+                "sha256:08682f6b72c722394747bddaf0aa62277e02557c0fd1c42cb853016a38f8dedf",
-                "sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76",
+                "sha256:0f5f5786c0e09baddcd8b4b45f20a7b5d61a7e7e99846e3c799b05c7c53fa696",
-                "sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2",
+                "sha256:129def1b7c1bf22faffd67b8f3724645203b79d8f4cc81f674654d9902cb4393",
-                "sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648",
+                "sha256:294db365efa064d00b8d1ef65d8ea2c3426ac366c0c4368d930bf1c5fb497f77",
-                "sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf",
+                "sha256:3b2b1824fe7112845700f815ff6a489360226a5609b96ec2190a45e62a9fc922",
-                "sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f",
+                "sha256:3bd0e463264cf257d1ffd2e40223b197271046d09dadf73a0fe82b9c1fc385a5",
-                "sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2",
+                "sha256:4465124ef1b18d9ace298060f4eccc64b0850899ac4ac53294547536533800c8",
-                "sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee",
+                "sha256:49d4cdd9065b9b6e206d0595fee27a96b5dd22618e7520c33204a4a3239d5b10",
-                "sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d",
+                "sha256:4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc",
-                "sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c",
+                "sha256:5accb17103e43963b80e6f837831f38d314a0495500067cb25afab2e8d7a4018",
-                "sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a"
+                "sha256:607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e",
+                "sha256:6c78645d400265a062508ae399b60b8c167bf003db364ecb26dcab2bda048253",
+                "sha256:72a01f726a9c7851ca9bfad6fd09ca4e090a023c00945ea05ba1638c09dc3347",
+                "sha256:74c1485f7707cf707a7aef42ef6322b8f97921bd89be2ab6317fd782c2d53183",
+                "sha256:895f61ef02e8fed38159bb70f7e100e00f471eae2bc838cd0f4ebb21e28f8541",
+                "sha256:8c1be557ee92a20f184922c7b6424e8ab6691788e6d86137c5d93c1a6ec1b8fb",
+                "sha256:bb4191dfc9306777bc594117aee052446b3fa88737cd13b7188d0e7aa8162185",
+                "sha256:bfb51918d4ff3d77c1c856a9699f8492c612cde32fd3bcd344af9be34999bfdc",
+                "sha256:c20cfa2d49991c8b4147af39859b167664f2ad4561704ee74c1de03318e898db",
+                "sha256:cb333c16912324fd5f769fff6bc5de372e9e7a202247b48870bc251ed40239aa",
+                "sha256:d2d9808ea7b4af864f35ea216be506ecec180628aced0704e34aca0b040ffe46",
+                "sha256:d483ad4e639292c90170eb6f7783ad19490e7a8defb3e46f97dfe4bacae89122",
+                "sha256:dd5de0646207f053eb0d6c74ae45ba98c3395a571a2891858e87df7c9b9bd51b",
+                "sha256:e1d4970ea66be07ae37a3c2e48b5ec63f7ba6804bdddfdbd3cfd954d25a82e63",
+                "sha256:e4fac90784481d221a8e4b1162afa7c47ed953be40d31ab4629ae917510051df",
+                "sha256:fa5ae20527d8e831e8230cbffd9f8fe952815b2b7dae6ffec25318803a7528fc",
+                "sha256:fd7f6999a8070df521b6384004ef42833b9bd62cfee11a09bda1079b4b704247",
+                "sha256:fdc842473cd33f45ff6bce46aea678a54e3d21f1b61a7750ce3c498eedfe25d6",
+                "sha256:fe69978f3f768926cfa37b867e3843918e012cf83f680806599ddce33c2c68b0"
             ],
             "index": "pypi",
-            "version": "==5.3.1"
+            "version": "==5.4.1"
         },
         "redis": {
             "hashes": [
@@ -258,15 +290,17 @@
         },
         "requests": {
             "hashes": [
-                "sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b",
+                "sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804",
-                "sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898"
+                "sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e"
             ],
-            "version": "==2.24.0"
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
+            "version": "==2.25.1"
         },
         "requests-oauthlib": {
             "hashes": [
                 "sha256:7f71572defaecd16372f9006f33c2ec8c077c3cfa6f5911a9a90202beb513f3d",
-                "sha256:b4261601a71fd721a8bd6d7aa1cc1d6a8a93b4a9f5e96626f8e4d91e8beeaa6a"
+                "sha256:b4261601a71fd721a8bd6d7aa1cc1d6a8a93b4a9f5e96626f8e4d91e8beeaa6a",
+                "sha256:fa6c47b933f01060936d87ae9327fead68768b69c6c9ea2109c48be30f2d4dbc"
             ],
             "version": "==1.3.0"
         },
@@ -277,19 +311,13 @@
             "index": "pypi",
             "version": "==2.2"
         },
-        "six": {
-            "hashes": [
-                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
-                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
-            ],
-            "version": "==1.15.0"
-        },
         "sqlparse": {
             "hashes": [
-                "sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e",
+                "sha256:017cde379adbd6a1f15a61873f43e8274179378e95ef3fede90b5aa64d304ed0",
-                "sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548"
+                "sha256:0f91fd2e829c44362cbcfab3e9ae12e22badaa8a29ad5ff599f9ec109f0454e8"
             ],
-            "version": "==0.3.1"
+            "markers": "python_version >= '3.5'",
+            "version": "==0.4.1"
         },
         "ua-parser": {
             "hashes": [
@@ -301,24 +329,26 @@
         },
         "urllib3": {
             "hashes": [
-                "sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527",
+                "sha256:2f4da4594db7e1e110a944bb1b551fdf4e6c136ad42e4234131391e21eb5b0df",
-                "sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115"
+                "sha256:e7b021f7241115872f92f43c6508082facffbd1c048e3c6e2bb9c2a157e28937"
             ],
-            "version": "==1.25.9"
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'",
+            "version": "==1.26.4"
         },
         "user-agents": {
             "hashes": [
-                "sha256:da54371d856c35d8ead0622da24ad5ef6d667eda3629a750e3373a3e847a054b",
+                "sha256:a98c4dc72ecbc64812c4534108806fb0a0b3a11ec3fd1eafe807cee5b0a942e7",
-                "sha256:e727ab6f169e829bc25d41dbd25b9ff679b4631bd81959bcf7de1e246da67194"
+                "sha256:d36d25178db65308d1458c5fa4ab39c9b2619377010130329f3955e7626ead26"
             ],
             "index": "pypi",
-            "version": "==2.1"
+            "version": "==2.2.0"
         },
         "vine": {
             "hashes": [
                 "sha256:133ee6d7a9016f177ddeaf191c1f58421a1dcc6ee9a42c58b34bed40e1d2cd87",
                 "sha256:ea4947cc56d1fd6f2095c8d543ee25dad966f78692528e68b4fada11ba3f98af"
             ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
             "version": "==1.3.0"
         },
         "whitenoise": {
@@ -330,101 +360,5 @@
             "version": "==5.1.0"
         }
     },
-    "develop": {
+    "develop": {}
-        "appdirs": {
-            "hashes": [
-                "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41",
-                "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"
-            ],
-            "version": "==1.4.4"
-        },
-        "attrs": {
-            "hashes": [
-                "sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
-                "sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
-            ],
-            "version": "==19.3.0"
-        },
-        "black": {
-            "hashes": [
-                "sha256:1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b",
-                "sha256:c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539"
-            ],
-            "index": "pypi",
-            "version": "==19.10b0"
-        },
-        "click": {
-            "hashes": [
-                "sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a",
-                "sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc"
-            ],
-            "version": "==7.1.2"
-        },
-        "pathspec": {
-            "hashes": [
-                "sha256:7d91249d21749788d07a2d0f94147accd8f845507400749ea19c1ec9054a12b0",
-                "sha256:da45173eb3a6f2a5a487efba21f050af2b41948be6ab52b6a1e3ff22bb8b7061"
-            ],
-            "version": "==0.8.0"
-        },
-        "regex": {
-            "hashes": [
-                "sha256:08997a37b221a3e27d68ffb601e45abfb0093d39ee770e4257bd2f5115e8cb0a",
-                "sha256:112e34adf95e45158c597feea65d06a8124898bdeac975c9087fe71b572bd938",
-                "sha256:1700419d8a18c26ff396b3b06ace315b5f2a6e780dad387e4c48717a12a22c29",
-                "sha256:2f6f211633ee8d3f7706953e9d3edc7ce63a1d6aad0be5dcee1ece127eea13ae",
-                "sha256:52e1b4bef02f4040b2fd547357a170fc1146e60ab310cdbdd098db86e929b387",
-                "sha256:55b4c25cbb3b29f8d5e63aeed27b49fa0f8476b0d4e1b3171d85db891938cc3a",
-                "sha256:5aaa5928b039ae440d775acea11d01e42ff26e1561c0ffcd3d805750973c6baf",
-                "sha256:654cb773b2792e50151f0e22be0f2b6e1c3a04c5328ff1d9d59c0398d37ef610",
-                "sha256:690f858d9a94d903cf5cada62ce069b5d93b313d7d05456dbcd99420856562d9",
-                "sha256:6ad8663c17db4c5ef438141f99e291c4d4edfeaacc0ce28b5bba2b0bf273d9b5",
-                "sha256:89cda1a5d3e33ec9e231ece7307afc101b5217523d55ef4dc7fb2abd6de71ba3",
-                "sha256:92d8a043a4241a710c1cf7593f5577fbb832cf6c3a00ff3fc1ff2052aff5dd89",
-                "sha256:95fa7726d073c87141f7bbfb04c284901f8328e2d430eeb71b8ffdd5742a5ded",
-                "sha256:97712e0d0af05febd8ab63d2ef0ab2d0cd9deddf4476f7aa153f76feef4b2754",
-                "sha256:b2ba0f78b3ef375114856cbdaa30559914d081c416b431f2437f83ce4f8b7f2f",
-                "sha256:bae83f2a56ab30d5353b47f9b2a33e4aac4de9401fb582b55c42b132a8ac3868",
-                "sha256:c78e66a922de1c95a208e4ec02e2e5cf0bb83a36ceececc10a72841e53fbf2bd",
-                "sha256:cf59bbf282b627130f5ba68b7fa3abdb96372b24b66bdf72a4920e8153fc7910",
-                "sha256:e3cdc9423808f7e1bb9c2e0bdb1c9dc37b0607b30d646ff6faf0d4e41ee8fee3",
-                "sha256:e9b64e609d37438f7d6e68c2546d2cb8062f3adb27e6336bc129b51be20773ac",
-                "sha256:fbff901c54c22425a5b809b914a3bfaf4b9570eee0e5ce8186ac71eb2025191c"
-            ],
-            "version": "==2020.6.8"
-        },
-        "toml": {
-            "hashes": [
-                "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
-                "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
-            ],
-            "version": "==0.10.1"
-        },
-        "typed-ast": {
-            "hashes": [
-                "sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355",
-                "sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919",
-                "sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa",
-                "sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652",
-                "sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75",
-                "sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01",
-                "sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d",
-                "sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1",
-                "sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907",
-                "sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c",
-                "sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3",
-                "sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b",
-                "sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614",
-                "sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb",
-                "sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b",
-                "sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41",
-                "sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6",
-                "sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34",
-                "sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe",
-                "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
-                "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
-            ],
-            "version": "==1.4.1"
-        }
-    }
 }

TEMPLATE.env

@@ -14,7 +14,10 @@ EMAIL_HOST_USER=example
 EMAIL_HOST_PASSWORD=example_password
 EMAIL_HOST=smtp.example.com
 EMAIL_PORT=465
-SERVER_EMAIL=<Shynet> noreply@shynet.example.com
+EMAIL_USE_SSL=True
+# Comment out EMAIL_USE_SSL & uncomment EMAIL_USE_TLS if your SMTP server uses TLS.
+# EMAIL_USE_TLS=True
+SERVER_EMAIL=Shynet <noreply@shynet.example.com>
 
 # General Django settings
 DJANGO_SECRET_KEY=random_string
@@ -55,6 +58,9 @@ PERFORM_CHECKS_AND_SETUP=True
 # The port that Shynet should bind to. Don't set this if you're deploying on Heroku.
 PORT=8080
 
+# Set to "False" if you do not want the version to be displayed on the frontend.
+SHOW_SHYNET_VERSION=True
+
 # Redis, queue, and parellization settings; not necessary for single-instance deployments.
 # Don't uncomment these unless you know what you are doing!
 # NUM_WORKERS=1
@@ -64,3 +70,16 @@ PORT=8080
 # that you have a separate queue consumer running somewhere via `celeryworker.sh`.
 # CELERY_TASK_ALWAYS_EAGER=False
 # CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1
+
+# Should Shynet show third-party icons in the dashboard?
+SHOW_THIRD_PARTY_ICONS=True
+
+# Should Shynet block collection of IP addresses globally?
+BLOCK_ALL_IPS=False
+
+# Should Shynet include the date and site ID when hashing users?
+# This will prevent any possibility of cross-site tracking provided
+# that IP collection is also disabled, and external keys (primary
+# keys) aren't supplied. It will also prevent sessions from spanning
+# one day to another.
+AGGRESSIVE_HASH_SALTING=True

app.json

@@ -1,7 +1,7 @@
 {
   "name": "Shynet",
-  "description":"Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.",
+  "description": "Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.",
-    "keywords":[
+  "keywords": [
     "app.json",
     "shynet",
     "heroku",
@@ -117,6 +117,11 @@
       "description": "Whether to perform checks and setup at startup. Recommended value is 'True' for Heroku users.",
       "value": "True",
       "required": false
+    },
+    "SHOW_SHYNET_VERSION": {
+      "description": "Set to 'False' if you do not want the version to be displayed on the frontend.",
+      "value": "True",
+      "required": false
     }
   }
 }

docker-compose.yml

@@ -1,6 +1,7 @@
 version: '3'
 services:
   shynet:
+    container_name: shynet_main
     image: milesmcc/shynet:latest
     restart: unless-stopped
     expose:
@@ -16,6 +17,7 @@ services:
     depends_on:
       - db
   db:
+    container_name: shynet_database
     image: postgres
     restart: always
     environment:
@@ -26,6 +28,18 @@ services:
       - shynet_db:/var/lib/postgresql/data
     networks:
       - internal
+  webserver:
+    container_name: shynet_webserver
+    image: nginx
+    restart: always
+    volumes:
+      - ./nginx.conf:/etc/nginx/conf.d/default.conf    
+    ports:
+      - 8080:80
+    depends_on:
+      - shynet
+    networks:
+      - internal
 volumes:
   shynet_db:
 networks:

kubernetes/deployments.yml

@@ -17,7 +17,7 @@ spec:
     spec:
       containers:
       - name: "shynet-webserver"
-        image: "milesmcc/shynet:latest"
+        image: "milesmcc/shynet:dev"
         imagePullPolicy: Always
         envFrom:
         - secretRef:
@@ -42,7 +42,7 @@ spec:
     spec:
       containers:
       - name: "shynet-celeryworker"
-        image: "milesmcc/shynet:latest"
+        image: "milesmcc/shynet:dev"
         command: ["./celeryworker.sh"]
         imagePullPolicy: Always
         envFrom:
@@ -61,7 +61,7 @@ spec:
   selector:
     app: shynet-redis
 ---
-apiVersion: apps/v1beta2
+apiVersion: apps/v1
 kind: StatefulSet
 metadata:
   name: shynet-redis
@@ -83,3 +83,37 @@ spec:
           ports:
             - containerPort: 6379
               name: redis
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: shynet-webserver-service  
+spec:
+  type: ClusterIP
+  ports:
+  - port: 8080
+  selector:
+    app: shynet-webserver
+---
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: shynet-webserver-ingress
+  annotations:
+    kubernetes.io/ingress.class: addon-http-application-routing
+spec:
+  rules:
+  - host: shynet.rmrm.io
+    http:
+      paths:
+      - backend:
+          serviceName: shynet-webserver-service
+          servicePort: 8080
+        path: /
+  - host: shynet-beta.rmrm.io
+    http:
+      paths:
+      - backend:
+          serviceName: shynet-webserver-service
+          servicePort: 8080
+        path: /

nginx.conf

@@ -0,0 +1,19 @@
+server {
+    server_name example.com;
+    access_log /var/log/nginx/bin.access.log;
+    error_log /var/log/nginx/bin.error.log error;
+
+
+    location / {
+        proxy_pass http://shynet:8080;
+        proxy_redirect off;
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Protocol $scheme;
+        proxy_set_header X-Url-Scheme $scheme;
+    }
+    listen 80; 
+
+}

package-lock.json

@@ -4,19 +4,19 @@
   "lockfileVersion": 1,
   "dependencies": {
     "@fortawesome/fontawesome-free": {
-      "version": "5.13.1",
+      "version": "5.15.1",
-      "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-free/-/fontawesome-free-5.13.1.tgz",
+      "resolved": "https://registry.npmjs.org/@fortawesome/fontawesome-free/-/fontawesome-free-5.15.1.tgz",
-      "integrity": "sha512-D819f34FLHeBN/4xvw0HR0u7U2G7RqjPSggXqf7LktsxWQ48VAfGwvMrhcVuaZV2fF069c/619RdgCCms0DHhw=="
+      "integrity": "sha512-OEdH7SyC1suTdhBGW91/zBfR6qaIhThbcN8PUXtXilY4GYnSBbVqOntdHbC1vXwsDnX0Qix2m2+DSU1J51ybOQ=="
     },
     "a17t": {
-      "version": "0.2.2",
+      "version": "0.5.1",
-      "resolved": "https://registry.npmjs.org/a17t/-/a17t-0.2.2.tgz",
+      "resolved": "https://registry.npmjs.org/a17t/-/a17t-0.5.1.tgz",
-      "integrity": "sha512-/hUtRe5KTwPpfy62jtOsFm35Sq/W0PtuDp/ltbSU+3j4Disop5g85YuuQ6mfc6jRjDgIa6XRs8PdJZVkKe1Y2A=="
+      "integrity": "sha512-peIPrH9eDiu49LLzLlSTFFrXj6WLlEX3TRsUkqyyOHi/i58ilJ/eERnu7AcswXhuCBx+/2W9EUuHM+8iAq4ipg=="
     },
     "apexcharts": {
-      "version": "3.19.3",
+      "version": "3.24.0",
-      "resolved": "https://registry.npmjs.org/apexcharts/-/apexcharts-3.19.3.tgz",
+      "resolved": "https://registry.npmjs.org/apexcharts/-/apexcharts-3.24.0.tgz",
-      "integrity": "sha512-pECgHHNR/etDW2SLUTA58ElrrEyUrhQsEgSiBJCLTwgJ8GMPHA/uSiI5pUJ2jy9+v2FY8Tj+8suH4CCCl3T/pQ==",
+      "integrity": "sha512-iT6czJCIVrmAtrcO90MZTQCvC+xi6R6Acf0jNH/d40FVTtCfcqECuKIh5iAMyOTtgUb7+fQ8rbadH2bm1kbL9Q==",
       "requires": {
         "svg.draggable.js": "^2.2.2",
         "svg.easing.js": "^2.0.0",
@@ -27,9 +27,9 @@
       }
     },
     "inter-ui": {
-      "version": "3.13.1",
+      "version": "3.15.0",
-      "resolved": "https://registry.npmjs.org/inter-ui/-/inter-ui-3.13.1.tgz",
+      "resolved": "https://registry.npmjs.org/inter-ui/-/inter-ui-3.15.0.tgz",
-      "integrity": "sha512-A+gHBm9WXZZmIYHdQci9ZoIrsPkzwYvWqG2+DyrwOuxjZVnRyz3b73ridPUWI/JvZ1nGf2j0VdJ+vxh0/bKBwg=="
+      "integrity": "sha512-6v0WK8FHkVYbNQZ7L9O5tP8280pgTBR9ydxqYwssMuUH6SZO70ZFK/NQ1Ob8nNmOOzpUJAzT0WE73ty96z1tAQ=="
     },
     "litepicker": {
       "version": "1.5.7",
@@ -99,11 +99,6 @@
       "requires": {
         "svg.js": "^2.6.5"
       }
-    },
-    "turbolinks": {
-      "version": "5.2.0",
-      "resolved": "https://registry.npmjs.org/turbolinks/-/turbolinks-5.2.0.tgz",
-      "integrity": "sha512-pMiez3tyBo6uRHFNNZoYMmrES/IaGgMhQQM+VFF36keryjb5ms0XkVpmKHkfW/4Vy96qiGW3K9bz0tF5sK9bBw=="
     }
   }
 }

package.json

@@ -17,11 +17,10 @@
   },
   "homepage": "https://github.com/milesmcc/shynet#readme",
   "dependencies": {
-    "@fortawesome/fontawesome-free": "^5.13.1",
+    "@fortawesome/fontawesome-free": "^5.15.1",
-    "a17t": "^0.2.2",
+    "a17t": "^0.5.1",
-    "apexcharts": "^3.19.3",
+    "apexcharts": "^3.24.0",
-    "inter-ui": "^3.13.1",
+    "inter-ui": "^3.15.0",
-    "litepicker": "^1.5.7",
+    "litepicker": "^1.5.7"
-    "turbolinks": "^5.2.0"
   }
 }

shynet/a17t/templates/a17t/includes/pagination.html

@@ -1,14 +1,14 @@
 <nav class="flex w-full flex-wrap items-center justify-between" role="navigation" aria-label="pagination">
     <div class="w-full md:w-auto mb-2">
         {% if page.has_previous %}
-        <a href="?page={{ page.previous_page_number }}{{url_parameters}}" class="button field w-auto mr-1">Previous</a>
+        <a href="?page={{ page.previous_page_number }}&{{url_parameters}}" class="button field bg-neutral-000 w-auto mr-1">Previous</a>
         {% else %}
-        <a class="button field w-auto mr-1" disabled>Previous</a>
+        <a class="button field bg-neutral-000 w-auto mr-1" disabled>Previous</a>
         {% endif %}
         {% if page.has_next %}
-        <a href="?page={{ page.next_page_number }}{{url_parameters}}" class="button field w-auto">Next</a>
+        <a href="?page={{ page.next_page_number }}&{{url_parameters}}" class="button field bg-neutral-000 w-auto">Next</a>
         {% else %}
-        <a class="button field w-auto" disabled>Next</a>
+        <a class="button field bg-neutral-000 w-auto" disabled>Next</a>
         {% endif %}
     </div>
 
@@ -17,7 +17,7 @@
         {% ifequal page.number pnum %}
         <li><a class="button field w-auto mx-1 text-white bg-neutral-700">{{ pnum }}</a></li>
         {% else %}
-        <li><a class="button field w-auto mx-1" href="?page={{ pnum }}{{url_parameters}}">{{ pnum }}</a></li>
+        <li><a class="button field bg-neutral-000 w-auto mx-1" href="?page={{ pnum }}&{{url_parameters}}">{{ pnum }}</a></li>
         {% endifequal %}
         {% endfor %}
 
@@ -27,7 +27,7 @@
         {% ifequal page.number pnum %}
         <li><a class="button field w-auto mx-1 text-white bg-neutral-700">{{ pnum }}</a></li>
         {% else %}
-        <li><a class="button field w-auto mx-1" href="?page={{ pnum }}{{url_parameters}}">{{ pnum }}</a></li>
+        <li><a class="button field bg-neutral-000 w-auto mx-1" href="?page={{ pnum }}&{{url_parameters}}">{{ pnum }}</a></li>
         {% endifequal %}
         {% endfor %}
         {% endif %}
@@ -38,7 +38,7 @@
         {% ifequal page.number pnum %}
         <li><a class="button field w-auto mx-1 text-white bg-neutral-700">{{ pnum }}</a></li>
         {% else %}
-        <li><a class="button field w-auto mx-1" href="?page={{ pnum }}{{url_parameters}}">{{ pnum }}</a></li>
+        <li><a class="button field bg-neutral-000 w-auto mx-1" href="?page={{ pnum }}&{{url_parameters}}">{{ pnum }}</a></li>
         {% endifequal %}
         {% endfor %}
         {% endif %}

shynet/a17t/templatetags/pagination.py

@@ -15,9 +15,9 @@ def pagination(
     before_current_pages=4,
     after_current_pages=4,
 ):
-    url_parameters = "".join(
+    url_parameters = urlencode(
         [
-            f"&{urlencode(key)}={urlencode(value)}"
+            (key, value)
             for key, value in request.GET.items()
             if key != "page"
         ]

shynet/analytics/migrations/0004_auto_20210328_1514.py

@@ -0,0 +1,38 @@
+# Generated by Django 3.1.7 on 2021-03-28 19:14
+
+from django.db import migrations, models
+import django.utils.timezone
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('analytics', '0003_auto_20200502_1227'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='hit',
+            name='last_seen',
+            field=models.DateTimeField(default=django.utils.timezone.now),
+        ),
+        migrations.AlterField(
+            model_name='hit',
+            name='start_time',
+            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now),
+        ),
+        migrations.AlterField(
+            model_name='session',
+            name='last_seen',
+            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now),
+        ),
+        migrations.AlterField(
+            model_name='session',
+            name='start_time',
+            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now),
+        ),
+        migrations.AddIndex(
+            model_name='session',
+            index=models.Index(fields=['service', '-last_seen'], name='analytics_s_service_10bb96_idx'),
+        ),
+    ]

shynet/analytics/migrations/0005_auto_20210328_1518.py

@@ -0,0 +1,24 @@
+# Generated by Django 3.1.7 on 2021-03-28 19:18
+
+from django.db import migrations, models
+import django.utils.timezone
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('analytics', '0004_auto_20210328_1514'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='hit',
+            name='last_seen',
+            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now),
+        ),
+        migrations.AlterField(
+            model_name='hit',
+            name='load_time',
+            field=models.FloatField(db_index=True, null=True),
+        ),
+    ]

shynet/analytics/migrations/0006_hit_service.py

@@ -0,0 +1,38 @@
+# Generated by Django 3.1.7 on 2021-03-28 19:36
+
+from ..models import Hit, Session
+from django.db import migrations, models
+import django.db.models.deletion
+from django.db.models import Subquery, OuterRef
+
+def add_service_to_hits(_a, _b):
+    service = Session.objects.filter(
+        pk=OuterRef('session')
+    ).values_list(
+        'service'
+    )[:1]
+
+    Hit.objects.update(
+        service=Subquery(service)
+    )
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0008_auto_20200628_1403'),
+        ('analytics', '0005_auto_20210328_1518'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='hit',
+            name='service',
+            field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, to='core.service'),
+        ),
+        migrations.RunPython(add_service_to_hits, lambda: ()),
+        migrations.AlterField(
+            model_name='hit',
+            name='service',
+            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.service'),
+        ),
+    ]

shynet/analytics/migrations/0007_auto_20210328_1634.py

@@ -0,0 +1,17 @@
+# Generated by Django 3.1.7 on 2021-03-28 20:34
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('analytics', '0006_hit_service'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='hit',
+            index=models.Index(fields=['service', '-start_time'], name='analytics_h_service_f4f41e_idx'),
+        ),
+    ]

shynet/analytics/migrations/0008_session_is_bounce.py

@@ -0,0 +1,24 @@
+# Generated by Django 3.1.7 on 2021-03-28 21:38
+
+from django.db.models.expressions import F
+from ..models import Session, Hit
+from django.db import migrations, models
+from django.db.models import Subquery, OuterRef
+
+def update_bounce_stats(_a, _b):
+    Session.objects.all().annotate(hit_count=models.Count("hit")).filter(hit_count__gt=1).update(is_bounce=False)
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('analytics', '0007_auto_20210328_1634'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='session',
+            name='is_bounce',
+            field=models.BooleanField(db_index=True, default=True),
+        ),
+        migrations.RunPython(update_bounce_stats, lambda: ()),
+    ]

shynet/analytics/models.py

@@ -1,6 +1,7 @@
 import json
 import uuid
 
+from django.conf import settings
 from django.db import models
 from django.shortcuts import reverse
 from django.utils import timezone
@@ -20,8 +21,8 @@ class Session(models.Model):
     identifier = models.TextField(blank=True, db_index=True)
 
     # Time
-    start_time = models.DateTimeField(auto_now_add=True, db_index=True)
+    start_time = models.DateTimeField(default=timezone.now, db_index=True)
-    last_seen = models.DateTimeField(auto_now_add=True)
+    last_seen = models.DateTimeField(default=timezone.now, db_index=True)
 
     # Core request information
     user_agent = models.TextField()
@@ -48,16 +49,21 @@ class Session(models.Model):
     latitude = models.FloatField(null=True)
     time_zone = models.TextField(blank=True)
 
+    is_bounce = models.BooleanField(default=True, db_index=True)
+
     class Meta:
         ordering = ["-start_time"]
         indexes = [
             models.Index(fields=["service", "-start_time"]),
+            models.Index(fields=["service", "-last_seen"]),
             models.Index(fields=["service", "identifier"]),
         ]
 
     @property
     def is_currently_active(self):
-        return timezone.now() - self.last_seen < timezone.timedelta(seconds=10)
+        return timezone.now() - self.last_seen < timezone.timedelta(
+            milliseconds=settings.SCRIPT_HEARTBEAT_FREQUENCY * 2
+        )
 
     @property
     def duration(self):
@@ -72,14 +78,20 @@ class Session(models.Model):
             kwargs={"pk": self.service.pk, "session_pk": self.uuid},
         )
 
+    def recalculate_bounce(self):
+        bounce = self.hit_set.count() == 1
+        if bounce != self.is_bounce:
+            self.is_bounce = bounce
+            self.save()
+
 
 class Hit(models.Model):
     session = models.ForeignKey(Session, on_delete=models.CASCADE, db_index=True)
     initial = models.BooleanField(default=True, db_index=True)
 
     # Base request information
-    start_time = models.DateTimeField(auto_now_add=True, db_index=True)
+    start_time = models.DateTimeField(default=timezone.now, db_index=True)
-    last_seen = models.DateTimeField(auto_now_add=True)
+    last_seen = models.DateTimeField(default=timezone.now, db_index=True)
     heartbeats = models.IntegerField(default=0)
     tracker = models.TextField(
         choices=[("JS", "JavaScript"), ("PIXEL", "Pixel (noscript)")]
@@ -88,12 +100,17 @@ class Hit(models.Model):
     # Advanced page information
     location = models.TextField(blank=True, db_index=True)
     referrer = models.TextField(blank=True, db_index=True)
-    load_time = models.FloatField(null=True)
+    load_time = models.FloatField(null=True, db_index=True)
+
+    # While not necessary, we store the root service directly for performance.
+    # It makes querying much easier; no need for inner joins.
+    service = models.ForeignKey(Service, on_delete=models.CASCADE, db_index=True)
 
     class Meta:
         ordering = ["-start_time"]
         indexes = [
             models.Index(fields=["session", "-start_time"]),
+            models.Index(fields=["service", "-start_time"]),
             models.Index(fields=["session", "location"]),
             models.Index(fields=["session", "referrer"]),
         ]
@@ -105,5 +122,5 @@ class Hit(models.Model):
     def get_absolute_url(self):
         return reverse(
             "dashboard:service_session",
-            kwargs={"pk": self.session.service.pk, "session_pk": self.session.pk},
+            kwargs={"pk": self.service.pk, "session_pk": self.session.pk},
         )

shynet/analytics/tasks.py

@@ -1,5 +1,4 @@
 import ipaddress
-import json
 import logging
 from hashlib import sha256
 
@@ -79,6 +78,11 @@ def ingress_request(
         association_id_hash = sha256()
         association_id_hash.update(str(ip).encode("utf-8"))
         association_id_hash.update(str(user_agent).encode("utf-8"))
+        if settings.AGGRESSIVE_HASH_SALTING:
+            association_id_hash.update(str(service.pk).encode("utf-8"))
+            association_id_hash.update(
+                str(timezone.now().date().isoformat()).encode("utf-8")
+            )
         session_cache_path = (
             f"session_association_{service.pk}_{association_id_hash.hexdigest()}"
         )
@@ -117,12 +121,14 @@ def ingress_request(
                 return
             session = Session.objects.create(
                 service=service,
-                ip=ip if service.collect_ips else None,
+                ip=ip if service.collect_ips and not settings.BLOCK_ALL_IPS else None,
                 user_agent=user_agent,
                 identifier=identifier.strip(),
                 browser=ua.browser.family or "",
                 device=ua.device.family or ua.device.model or "",
                 device_type=device_type,
+                start_time=time,
+                last_seen=time,
                 os=ua.os.family or "",
                 asn=ip_data.get("asn") or "",
                 country=ip_data.get("country") or "",
@@ -139,7 +145,7 @@ def ingress_request(
             log.debug("Updating old session with new data...")
 
             # Update last seen time
-            session.last_seen = timezone.now()
+            session.last_seen = time
             if session.identifier == "" and identifier.strip() != "":
                 session.identifier = identifier.strip()
             session.save()
@@ -160,7 +166,7 @@ def ingress_request(
                     # this is a heartbeat.
                     log.debug("Hit is a heartbeat; updating old hit with new data...")
                     hit.heartbeats += 1
-                    hit.last_seen = timezone.now()
+                    hit.last_seen = time
                     hit.save()
 
         if hit is None:
@@ -176,7 +182,14 @@ def ingress_request(
                 location=payload.get("location", location),
                 referrer=payload.get("referrer", ""),
                 load_time=payload.get("loadTime"),
+                start_time=time,
+                last_seen=time,
+                service=service
             )
+
+            # Recalculate whether the session is a bounce
+            session.recalculate_bounce()
+
             # Set idempotency (if applicable)
             if idempotency is not None:
                 cache.set(

shynet/analytics/views/ingress.py

@@ -5,7 +5,12 @@ from urllib.parse import urlparse
 from django.conf import settings
 from django.core.cache import cache
 from django.core.exceptions import ValidationError
-from django.http import Http404, HttpResponse, HttpResponseBadRequest
+from django.http import (
+    Http404,
+    HttpResponse,
+    HttpResponseBadRequest,
+    HttpResponseForbidden,
+)
 from django.shortcuts import render, reverse
 from django.utils import timezone
 from django.utils.decorators import method_decorator
@@ -53,9 +58,17 @@ class ValidateServiceOriginsMixin:
 
             if origins != "*":
                 remote_origin = request.META.get("HTTP_ORIGIN")
-                origins = [origin.strip() for origin in origins.split(",")]
+                if (
+                    remote_origin is None
+                    and request.META.get("HTTP_REFERER") is not None
+                ):
+                    parsed = urlparse(request.META.get("HTTP_REFERER"))
+                    remote_origin = f"{parsed.scheme}://{parsed.netloc}".lower()
+                origins = [origin.strip().lower() for origin in origins.split(",")]
                 if remote_origin in origins:
                     resp["Access-Control-Allow-Origin"] = remote_origin
+                else:
+                    return HttpResponseForbidden()
             else:
                 resp["Access-Control-Allow-Origin"] = "*"
 
@@ -87,7 +100,7 @@ class PixelView(ValidateServiceOriginsMixin, View):
             "R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw=="
         )
         resp = HttpResponse(data, content_type="image/gif")
-        resp["Cache-Control"] = "no-cache"
+        resp["Cache-Control"] = "no-cache, no-store, must-revalidate"
         resp["Access-Control-Allow-Origin"] = "*"
         return resp
 

shynet/core/management/commands/demo.py

@@ -0,0 +1,114 @@
+import traceback
+from django.utils.timezone import now
+from django.utils.timezone import timedelta
+import random
+import uuid
+
+from django.conf import settings
+from django.contrib.sites.models import Site
+from django.core.management.base import BaseCommand, CommandError
+from django.utils.crypto import get_random_string
+import user_agents
+from logging import info
+
+from core.models import User, Service
+from analytics.models import Session, Hit
+from analytics.tasks import ingress_request
+
+LOCATIONS = [
+    "/",
+    "/post/{rand}",
+    "/login",
+    "/me",
+]
+
+REFERRERS = [
+    "https://news.ycombinator.com/item?id=11116274",
+    "https://news.ycombinator.com/item?id=24872911",
+    "https://reddit.com",
+    "https://facebook.com",
+    "https://twitter.com/milesmccain",
+    "https://twitter.com",
+    "https://stanford.edu/~mccain/",
+    "https://tiktok.com",
+    "https://io.stanford.edu",
+    "https://en.wikipedia.org",
+    "https://stackoverflow.com",
+    "",
+    "",
+    "",
+    "",
+]
+
+USER_AGENTS = [
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 Firefox/43.4",
+    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
+    "Mozilla/5.0 (iPhone; CPU iPhone OS 11_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko)",
+    "Version/10.0 Mobile/14E304 Safari/602.1",
+]
+
+
+class Command(BaseCommand):
+    help = "Configures a Shynet demo service"
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "name", type=str,
+        )
+        parser.add_argument("owner_email", type=str)
+        parser.add_argument(
+            "avg", type=int,
+        )
+        parser.add_argument("deviation", type=float, default=0.4)
+        parser.add_argument(
+            "days", type=int,
+        )
+        parser.add_argument("load_time", type=float, default=1000)
+
+    def handle(self, *args, **options):
+        owner = User.objects.get(email=options.get("owner_email"))
+        service = Service.objects.create(name=options.get("name"), owner=owner)
+
+        print(
+            f"Created demo service `{service.name}` (uuid: `{service.uuid}`, owner: {owner})"
+        )
+
+        # Go through each day requested, creating sessions and hits
+        for days in range(options.get("days")):
+            day = (now() - timedelta(days=days)).replace(hour=0, minute=0, second=0)
+            print(f"Populating info for {day}...")
+            avg = options.get("avg")
+            deviation = options.get("deviation")
+            ips = [
+                ".".join(map(str, (random.randint(0, 255) for _ in range(4))))
+                for _ in range(avg)
+            ]
+
+            n = avg + random.randrange(-1 * deviation * avg, deviation * avg)
+            for _ in range(n):
+                time = day + timedelta(
+                    hours=random.randrange(0, 23),
+                    minutes=random.randrange(0, 59),
+                    seconds=random.randrange(0, 59),
+                )
+                ip = random.choice(ips)
+                load_time = random.normalvariate(options.get("load_time"), 500)
+                referrer = random.choice(REFERRERS)
+                location = "https://example.com" + random.choice(LOCATIONS).replace(
+                    "{rand}", str(random.randint(0, n))
+                )
+                user_agent = random.choice(USER_AGENTS)
+                ingress_request(
+                    service.uuid,
+                    "JS",
+                    time,
+                    {"loadTime": load_time, "referrer": referrer},
+                    ip,
+                    location,
+                    user_agent,
+                )
+
+            print(f"Created {n} demo hits on {day}!")
+
+        self.stdout.write(self.style.SUCCESS(f"Successfully created demo data!"))

shynet/core/management/commands/hostname.py

@@ -20,6 +20,12 @@ class Command(BaseCommand):
     def handle(self, *args, **options):
         site = Site.objects.get(pk=settings.SITE_ID)
         site.domain = options.get("hostname")
+        if options.get("hostname").lower().startswith("http"):
+            self.stdout.write(
+                self.style.WARNING(
+                    f"Warning: the hostname '{options.get('hostname')}' starts with `http`. You almost certainly don't want this. The hostname is supposed to be the raw domain name of your Shynet instance, without `http://` or `https://`. For example, if your Shynet instance will eventually be hosted at `https://analytics.example.com`, the hostname should be `analytics.example.com`."
+                )
+            )
         site.save()
         self.stdout.write(
             self.style.SUCCESS(

shynet/core/management/commands/startup_checks.py

@@ -13,7 +13,7 @@ from core.models import User
 
 
 class Command(BaseCommand):
-    help = "Internal command to perform startup sanity checks."
+    help = "Internal command to perform startup checks."
 
     def check_migrations(self):
         from django.db.migrations.executor import MigrationExecutor

shynet/core/migrations/0008_auto_20200628_1403.py

@@ -6,18 +6,20 @@ from django.db import migrations, models
 class Migration(migrations.Migration):
 
     dependencies = [
-        ('core', '0007_service_ignore_robots'),
+        ("core", "0007_service_ignore_robots"),
     ]
 
     operations = [
         migrations.AddField(
-            model_name='service',
+            model_name="service",
-            name='script_inject',
+            name="script_inject",
-            field=models.TextField(blank=True, default=''),
+            field=models.TextField(blank=True, default=""),
         ),
         migrations.AlterField(
-            model_name='user',
+            model_name="user",
-            name='first_name',
+            name="first_name",
-            field=models.CharField(blank=True, max_length=150, verbose_name='first name'),
+            field=models.CharField(
+                blank=True, max_length=150, verbose_name="first name"
+            ),
         ),
     ]

shynet/core/models.py

@@ -129,11 +129,11 @@ class Service(models.Model):
         session_count = sessions.count()
 
         hits = Hit.objects.filter(
-            session__service=self, start_time__lt=end_time, start_time__gt=start_time
+            service=self, start_time__lt=end_time, start_time__gt=start_time
         )
         hit_count = hits.count()
 
-        bounces = sessions.annotate(hit_count=models.Count("hit")).filter(hit_count=1)
+        bounces = sessions.filter(is_bounce=True)
         bounce_count = bounces.count()
 
         locations = (

shynet/dashboard/forms.py

@@ -1,5 +1,6 @@
 from allauth.account.admin import EmailAddress
 from django import forms
+from django.conf import settings
 from django.utils.translation import gettext_lazy as _
 
 from core.models import Service, User
@@ -18,7 +19,7 @@ class ServiceForm(forms.ModelForm):
             "hide_referrer_regex",
             "origins",
             "collaborators",
-            "script_inject"
+            "script_inject",
         ]
         widgets = {
             "name": forms.TextInput(),
@@ -28,12 +29,11 @@ class ServiceForm(forms.ModelForm):
             "collect_ips": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
             "ignore_robots": forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
             "hide_referrer_regex": forms.TextInput(),
-            "script_inject": forms.Textarea(attrs={'class':'font-mono', 'rows': 5})
+            "script_inject": forms.Textarea(attrs={"class": "font-mono", "rows": 5}),
         }
         labels = {
             "origins": "Allowed origins",
             "respect_dnt": "Respect DNT",
-            "collect_ips": "Collect IP addresses",
             "ignored_ips": "Ignored IP addresses",
             "ignore_robots": "Ignore robots",
             "hide_referrer_regex": "Hide specific referrers",
@@ -46,13 +46,27 @@ class ServiceForm(forms.ModelForm):
                 "At what origins does the service operate? Use commas to separate multiple values. This sets CORS headers, so use '*' if you're not sure (or don't care)."
             ),
             "respect_dnt": "Should visitors who have enabled <a href='https://en.wikipedia.org/wiki/Do_Not_Track'>Do Not Track</a> be excluded from all data?",
-            "collect_ips": "Should individual IP addresses be collected? IP metadata (location, host, etc) will still be collected.",
             "ignored_ips": "A comma-separated list of IP addresses or IP ranges (IPv4 and IPv6) to exclude from tracking (e.g., '192.168.0.2, 127.0.0.1/32').",
             "ignore_robots": "Should sessions generated by bots be excluded from tracking?",
             "hide_referrer_regex": "Any referrers that match this <a href='https://regexr.com/'>RegEx</a> will not be listed in the referrer summary. Sessions will still be tracked normally. No effect if left blank.",
             "script_inject": "Optional additional JavaScript to inject at the end of the Shynet script. This code will be injected on every page where this service is installed.",
         }
 
+    collect_ips = forms.BooleanField(
+        help_text="IP address collection is disabled globally by your administrator."
+        if settings.BLOCK_ALL_IPS
+        else "Should individual IP addresses be collected? IP metadata (location, host, etc) will still be collected.",
+        widget=forms.RadioSelect(choices=[(True, "Yes"), (False, "No")]),
+        initial=False if settings.BLOCK_ALL_IPS else True,
+        required=False,
+        disabled=settings.BLOCK_ALL_IPS,
+    )
+
+    def clean_collect_ips(self):
+        collect_ips = self.cleaned_data["collect_ips"]
+        # Forces collect IPs to be false if it is disabled globally
+        return False if settings.BLOCK_ALL_IPS else collect_ips
+
     collaborators = forms.CharField(
         help_text="Which users on this Shynet instance should have read-only access to this service? (Comma separated list of emails.)",
         required=False,
@@ -60,6 +74,9 @@ class ServiceForm(forms.ModelForm):
 
     def clean_collaborators(self):
         collaborators = []
+        users_to_emails = (
+            {}
+        )  # maps users to the email they are listed under as a collaborator
         for collaborator_email in self.cleaned_data["collaborators"].split(","):
             email = collaborator_email.strip()
             if email == "":
@@ -69,6 +86,12 @@ class ServiceForm(forms.ModelForm):
             ).first()
             if collaborator_email_linked is None:
                 raise forms.ValidationError(f"Email '{email}' is not registered")
+            user = collaborator_email_linked.user
+            if user in collaborators:
+                raise forms.ValidationError(
+                    f"The emails '{email}' and '{users_to_emails[user]}' both correspond to the same user"
+                )
+            users_to_emails[user] = email
             collaborators.append(collaborator_email_linked.user)
         return collaborators
 

shynet/dashboard/static/dashboard/css/global.css

@@ -14,3 +14,17 @@
 .rf {
     text-align: right !important;
 }
+
+:root {
+    --color-neutral-000: white;
+    --color-neutral-50: #F8FAFC;
+    --color-neutral-100: #F1F5F9;
+    --color-neutral-200: #E2E8F0;
+    --color-neutral-300: #CBD5E1;
+    --color-neutral-400: #94A3B8;
+    --color-neutral-500: #64748B;
+    --color-neutral-600: #475569;
+    --color-neutral-700: #334155;
+    --color-neutral-800: #1E293B;
+    --color-neutral-900: #0F172A;
+}

shynet/dashboard/templates/base.html

@@ -8,20 +8,21 @@
   <meta name="robots" content="noindex">
   <meta name="viewport" content="width=device-width, initial-scale=1.0">
   {% include 'a17t/includes/head.html' %}
+  <link rel="icon" type="image/png" href="{% static 'dashboard/images/icon.png' %}">
   <script src="{% static 'apexcharts/dist/apexcharts.min.js'%}"></script>
   <script src="{% static 'litepicker/dist/js/main.js' %}"></script>
-  <script src="{% static 'turbolinks/dist/turbolinks.js' %}"></script>
   <script src="{% static 'dashboard/js/base.js' %}"></script>
   <link rel="stylesheet" href="{% static 'dashboard/css/global.css' %}">
   {% block extra_head %}
   {% endblock %}
 </head>
 
-<body class="bg-neutral-200 min-h-full">
+<body class="bg-neutral-100 min-h-full">
   {% block body %}
 
   <section class="max-w-screen-xl mx-auto px-4 py-4 md:py-12 md:flex">
-    <aside class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center overflow-x-hidden">
+    <aside
+      class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center overflow-x-hidden">
       <a class="icon ~urge ml-2 md:ml-6 md:mb-8 md:mt-3" href="{% url 'dashboard:dashboard' %}">
         <i class="fas fa-binoculars fa-3x text-urge-600 hidden md:block"></i>
         <i class="fas fa-binoculars fa-2x text-urge-600 md:hidden"></i>
@@ -41,7 +42,7 @@
 
         {% for service in user.owning_services.all %}
         {% url 'dashboard:service' service.uuid as url %}
-        {% include 'dashboard/includes/sidebar_portal.html' with label=service.name|truncatechars:20 url=url %}
+        {% include 'dashboard/includes/sidebar_portal.html' with label=service.name|truncatechars:16 url=url icon=service.link|iconify %}
         {% endfor %}
 
         {% endif %}

shynet/dashboard/templates/dashboard/includes/service_overview.html

@@ -4,8 +4,9 @@
     {% with stats=object.stats %}
     <div class="p-4 md:flex justify-between">
         <div class="flex items-center mb-4 md:mb-0">
-            <h3 class="heading text-xl md:text-2xl mr-2 mb-1 text-urge-600">
+            <h3 class="heading text-xl md:text-2xl mr-2 mb-1 text-urge-600 flex items-center">
-                {{object.name}}
+                {{object.link|iconify}}
+                <span>{{object.name}}</span>
             </h3>
             {% include 'dashboard/includes/stats_status_chip.html' %}
         </div>

shynet/dashboard/templates/dashboard/includes/sidebar_portal.html

@@ -1,6 +1,6 @@
 {% load helpers %}
 
 <div>
-    <a class="portal !low {% if request.get_full_path|startswith:url %}~urge active bg-neutral-100{% endif %}"
+    <a class="portal !low {% if request.get_full_path|startswith:url %}~urge active bg-neutral-100{% endif %} flex items-center"
-        {% if disable_turbolinks %}data-turbolinks="false"{% endif %} href="{{url}}">{{label}}</a>
+        {% if disable_turbolinks %}data-turbolinks="false"{% endif %} href="{{url}}">{{icon}} {{label}}</a>
 </div>

shynet/dashboard/templates/dashboard/pages/dashboard.html

@@ -1,6 +1,6 @@
 {% extends "base.html" %}
 
-{% load rules %}
+{% load rules pagination %}
 
 {% block content %}
 <div class="md:flex justify-between items-center">
@@ -18,9 +18,12 @@
     </div>
 </div>
 <hr class="sep">
-{% for object in services|dictsortreversed:"stats.session_count" %}
+{% for object in object_list|dictsortreversed:"stats.session_count" %}
 {% include 'dashboard/includes/service_overview.html' %}
 {% empty %}
 <p>You don't have any services on {{request.site.name|default:"Shynet"}} yet.</p>
 {% endfor %}
+
+{% pagination page_obj request %}
+
 {% endblock %}

shynet/dashboard/templates/dashboard/pages/service.html

@@ -166,7 +166,7 @@
             <tbody>
                 {% for os in stats.operating_systems %}
                 <tr>
-                    <td>{{os.os|default:"Unknown"}}</td>
+                    <td class="flex items-center">{{os.os|iconify}}<span>{{os.os|default:"Unknown"}}</span></td>
                     <td class="rf">{{os.count|intcomma}}</td>
                 </tr>
                 {% empty %}
@@ -188,7 +188,8 @@
             <tbody>
                 {% for browser in stats.browsers %}
                 <tr>
-                    <td>{{browser.browser|default:"Unknown"}}</td>
+                    <td class="flex items-center">
+                        {{browser.browser|iconify}}<span>{{browser.browser|default:"Unknown"}}</span></td>
                     <td class="rf">{{browser.count|intcomma}}</td>
                 </tr>
                 {% empty %}
@@ -225,7 +226,8 @@
 <div class="card ~neutral !low limited-height py-2">
     {% include 'dashboard/includes/session_list.html' %}
     <hr class="sep h-8">
-    <a href="{% url 'dashboard:service_session_list' service.uuid %}" class="button ~neutral w-auto mb-2">View more sessions
+    <a href="{% url 'dashboard:service_session_list' service.uuid %}" class="button ~neutral w-auto mb-2">View more
+        sessions
         &rarr;</a>
 </div>
 {% endblock %}

shynet/dashboard/templates/dashboard/pages/service_session_list.html

@@ -6,7 +6,7 @@
 
 {% block service_actions %}
 <div class="mr-2">{% include 'dashboard/includes/date_range.html' %}</div>
-<a href="{% url 'dashboard:service' object.uuid %}" class="button field ~neutral w-auto">Analytics &rarr;</a>
+<a href="{% url 'dashboard:service' object.uuid %}" class="button field ~neutral bg-neutral-000 w-auto">Analytics &rarr;</a>
 {% endblock %}
 
 {% block service_content %}

shynet/dashboard/templates/dashboard/pages/service_update.html

@@ -15,7 +15,7 @@
     <div class="card ~neutral !high font-mono text-sm">
         {% filter force_escape %}<noscript><img
                 src="{{script_protocol}}{{request.site.domain}}{% url 'ingress:endpoint_pixel' object.uuid %}"></noscript>
-        <script src="{{script_protocol}}{{request.site.domain}}{% url 'ingress:endpoint_script' object.uuid %}"></script>
+        <script defer src="{{script_protocol}}{{request.site.domain}}{% url 'ingress:endpoint_script' object.uuid %}"></script>
         {% endfilter %}
     </div>
     <hr class="sep h-4">

shynet/dashboard/templates/dashboard/service_base.html

@@ -8,6 +8,7 @@
 <div class="md:flex justify-between items-center" id="heading">
     <a class="flex items-center mb-4 md:mb-0" href="{% url 'dashboard:service' object.uuid %}">
         <h3 class="heading leading-none mr-4">
+            {{object.link|iconify}}
             {{object.name}}
         </h3>
         <div class='text-3xl'>

shynet/dashboard/templatetags/helpers.py

@@ -1,3 +1,4 @@
+from datetime import timedelta
 from urllib.parse import urlparse
 
 import flag
@@ -73,7 +74,7 @@ def percent_change_display(start, end):
     elif start == 0:
         pct_change = "0%"
     else:
-        change = int(round(100 * abs(end - start) / start))
+        change = int(round(100 * abs(end - start) / max(start, 1)))
         if change > 999:
             return "> 999%"
         else:
@@ -84,7 +85,7 @@ def percent_change_display(start, end):
 
 @register.inclusion_tag("dashboard/includes/sidebar_footer.html")
 def sidebar_footer():
-    return {"version": settings.VERSION}
+    return {"version": settings.VERSION if settings.SHOW_SHYNET_VERSION else ""}
 
 
 @register.inclusion_tag("dashboard/includes/stat_comparison.html")
@@ -97,6 +98,12 @@ def compare(
     bad_classes=None,
     neutral_classes=None,
 ):
+    if isinstance(start, timedelta):
+        start = start.seconds
+
+    if isinstance(end, timedelta):
+        end = end.seconds
+
     return {
         "start": start,
         "end": end,
@@ -115,12 +122,60 @@ def startswith(text, starts):
     return False
 
 
+@register.filter
+def iconify(text):
+    if not settings.SHOW_THIRD_PARTY_ICONS:
+        return ""
+
+    text = text.lower()
+    icons = {
+        "chrome": "chrome.com",
+        "safari": "www.apple.com",
+        "windows": "windows.com",
+        "edge": "microsoft.com",
+        "firefox": "firefox.com",
+        "opera": "opera.com",
+        "unknown": "example.com",
+        "linux": "kernel.org",
+        "ios": "www.apple.com",
+        "mac": "www.apple.com",
+        "macos": "www.apple.com",
+        "mac os x": "www.apple.com",
+        "android": "android.com",
+        "chrome os": "chrome.com",
+        "ubuntu": "ubuntu.com",
+        "fedora": "getfedora.org",
+        "mobile safari": "www.apple.com",
+        "chrome mobile ios": "chrome.com",
+        "chrome mobile": "chrome.com",
+        "samsung internet": "samsung.com",
+        "google": "google.com",
+        "chrome mobile webview": "chrome.com",
+        "firefox mobile": "firefox.com",
+        "edge mobile": "microsoft.com",
+        "chromium": "chromium.org",
+    }
+
+    domain = None
+    if text.startswith("http"):
+        domain = urlparse(text).netloc
+    elif text in icons:
+        domain = icons[text]
+    else:
+        # This fallback works better than you'd think!
+        domain = text + ".com"
+
+    return SafeString(
+        f'<span class="icon mr-1"><img src="https://icons.duckduckgo.com/ip3/{domain}.ico"></span>'
+    )
+
+
 @register.filter
 def urldisplay(url):
     if url.startswith("http"):
         display_url = url.replace("http://", "").replace("https://", "")
         return SafeString(
-            f"<a href='{url}' title='{url}' rel='nofollow'>{escape(display_url if len(display_url) < 40 else display_url[:40] + '...')}</a>"
+            f"<a href='{url}' title='{url}' rel='nofollow' class='flex items-center'>{iconify(url)} {escape(display_url if len(display_url) < 40 else display_url[:40] + '...')}</a>"
         )
     else:
         return url

shynet/dashboard/views.py

@@ -22,16 +22,24 @@ from .forms import ServiceForm
 from .mixins import DateRangeMixin
 
 
-class DashboardView(LoginRequiredMixin, DateRangeMixin, TemplateView):
+class DashboardView(
+    LoginRequiredMixin, DateRangeMixin, ListView
+):
+    model = Service
     template_name = "dashboard/pages/dashboard.html"
+    paginate_by = 5
+
+    def get_queryset(self):
+        return Service.objects.filter(
+            Q(owner=self.request.user) | Q(collaborators__in=[self.request.user])
+        ).distinct()
 
     def get_context_data(self, **kwargs):
         data = super().get_context_data(**kwargs)
-        data["services"] = Service.objects.filter(
+
-            Q(owner=self.request.user) | Q(collaborators__in=[self.request.user])
+        for service in data["object_list"]:
-        )
+            service.stats = service.get_core_stats(self.get_start_date(), self.get_end_date())
-        for service in data["services"]:
+
-            service.stats = service.get_core_stats(data["start_date"], data["end_date"])
         return data
 
 
@@ -139,6 +147,9 @@ class ServiceSessionView(LoginRequiredMixin, PermissionRequiredMixin, DetailView
     context_object_name = "session"
     permission_required = "core.view_service"
 
+    def get_permission_object(self, **kwargs):
+        return self.get_object().service
+
     def get_context_data(self, **kwargs):
         data = super().get_context_data(**kwargs)
         data["object"] = get_object_or_404(Service, pk=self.kwargs.get("pk"))

shynet/shynet/settings.py

@@ -18,7 +18,7 @@ import urllib.parse as urlparse
 from django.contrib.messages import constants as messages
 
 # Increment on new releases
-VERSION = "v0.6.1"
+VERSION = "v0.8.0"
 
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -58,6 +58,7 @@ INSTALLED_APPS = [
     "allauth",
     "allauth.account",
     "allauth.socialaccount",
+    "debug_toolbar"
 ]
 
 MIDDLEWARE = [
@@ -70,6 +71,7 @@ MIDDLEWARE = [
     "django.contrib.sites.middleware.CurrentSiteMiddleware",
     "django.contrib.messages.middleware.MessageMiddleware",
     "django.middleware.clickjacking.XFrameOptionsMiddleware",
+    "debug_toolbar.middleware.DebugToolbarMiddleware",
 ]
 
 ROOT_URLCONF = "shynet.urls"
@@ -247,6 +249,10 @@ LOGIN_REDIRECT_URL = "/"
 
 SITE_ID = 1
 
+INTERNAL_IPS = [
+    '127.0.0.1',
+]
+
 # Celery
 
 CELERY_TASK_ALWAYS_EAGER = os.getenv("CELERY_TASK_ALWAYS_EAGER", "True") == "True"
@@ -279,20 +285,21 @@ else:
     EMAIL_PORT = int(os.environ.get("EMAIL_PORT", 465))
     EMAIL_HOST_USER = os.environ.get("EMAIL_HOST_USER")
     EMAIL_HOST_PASSWORD = os.environ.get("EMAIL_HOST_PASSWORD")
-    EMAIL_USE_SSL = True
+    EMAIL_USE_SSL = os.environ.get("EMAIL_USE_SSL")
+    EMAIL_USE_TLS = os.environ.get("EMAIL_USE_TLS")
 
 # NPM
 
 NPM_ROOT_PATH = "../"
 
 NPM_FILE_PATTERNS = {
-    "a17t": ["dist/a17t.css", "dist/tailwind.css"],
+    "a17t": [os.path.join("dist", "a17t.css"), os.path.join("dist", "tailwind.css")],
-    "@fortawesome/fontawesome-free": ["js/all.min.js"],
+    "apexcharts": [os.path.join("dist", "apexcharts.min.js")],
-    "apexcharts": ["dist/apexcharts.min.js"],
+    "litepicker": [os.path.join("dist", "js", "main.js")],
-    "litepicker": ["dist/js/main.js"],
+    "turbolinks": [os.path.join("dist", "turbolinks.js")],
-    "turbolinks": ["dist/turbolinks.js"],
+    "stimulus": [os.path.join("dist", "stimulus.umd.js")],
-    "stimulus": ["dist/stimulus.umd.js"],
+    "inter-ui": [os.path.join("Inter (web)", "*")],
-    "inter-ui": ["Inter (web)/*"],
+    "@fortawesome": [os.path.join("fontawesome-free", "js", "all.min.js")],
 }
 
 # Shynet
@@ -314,3 +321,15 @@ SCRIPT_HEARTBEAT_FREQUENCY = int(os.getenv("SCRIPT_HEARTBEAT_FREQUENCY", "5000")
 # How much time can elapse between requests from the same user before a new
 # session is created, in seconds?
 SESSION_MEMORY_TIMEOUT = int(os.getenv("SESSION_MEMORY_TIMEOUT", "1800"))
+
+# Should the Shynet version information be displayed?
+SHOW_SHYNET_VERSION = os.getenv("SHOW_SHYNET_VERSION", "True") == "True"
+
+# Should Shynet show third-party icons in the dashboard?
+SHOW_THIRD_PARTY_ICONS = os.getenv("SHOW_THIRD_PARTY_ICONS", "True") == "True"
+
+# Should Shynet never collect any IP?
+BLOCK_ALL_IPS = os.getenv("BLOCK_ALL_IPS", "False") == "True"
+
+# Include date and service ID in salt?
+AGGRESSIVE_HASH_SALTING = os.getenv("AGGRESSIVE_HASH_SALTING", "False") == True

shynet/shynet/urls.py

@@ -15,8 +15,10 @@ Including another URLconf
 """
 from django.contrib import admin
 from django.urls import include, path
+import debug_toolbar
 
 urlpatterns = [
+    path('__debug__/', include(debug_toolbar.urls)),
     path("admin/", admin.site.urls),
     path("accounts/", include("allauth.urls")),
     path("ingress/", include(("analytics.ingress_urls", "ingress")), name="ingress"),

tests/js.html

@@ -0,0 +1,13 @@
+<!DOCTYPE html>
+
+<html>
+
+<head>
+    <title>Pixel test</title>
+</head>
+
+<body>
+    <script src="http://localhost:8000/ingress/9b2c4e2f-8d29-4418-82d4-b68e06795025/script.js"></script>
+</body>
+
+</html>

tests/pixel.html

@@ -7,8 +7,7 @@
 </head>
 
 <body>
-    <noscript><img src="http://localhost:8000/ingress/9b2c4e2f-8d29-4418-82d4-b68e06795025/pixel.gif"></noscript>
+    <img src="http://localhost:8000/ingress/9b2c4e2f-8d29-4418-82d4-b68e06795025/pixel.gif">
-    <script src="http://localhost:8000/ingress/9b2c4e2f-8d29-4418-82d4-b68e06795025/script.js"></script>
 </body>
 
 </html>