Document SPA behavior

GUIDE.md

@@ -5,11 +5,13 @@
 - [Installation](#installation)
 - [Heroku](#heroku)
 - [Updating Your Configuration](#updating-your-configuration)
-- [Enhancements](#enhancements)
+- [Advanced Usage](#advanced-usage)
   * [Installation with SSL](#installation-with-ssl)
   * [Configuring a Reverse Proxy](#configuring-a-reverse-proxy)
     + [Cloudflare](#cloudflare)
     + [Nginx](#nginx)
+  * [Health Checks](#health-checks)
+  * [Primary Key Integration](#primary-key-integration)
 + [Troubleshooting](#troubleshooting)
 ---
 
@@ -36,7 +38,7 @@ Before continuing, please be sure to have the latest version of Docker installed
 
 5. Create an admin user by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py registeradmin <your email>`. A temporary password will be printed to the console.
 
-6. Set the hostname of your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py hostname <your public hostname>`, where `<your public hostname>` is the _publicly accessible hostname_ of your instance, including port. This setting affects the URL that the tracking script sends its results to, so make sure it's correct. (Example hostnames: `shynet.rmrm.io` or `example.com:8000`.)
+6. Set the hostname of your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py hostname <your public hostname>`, where `<your public hostname>` is the _publicly accessible hostname_ of your instance, including port. This setting affects the URL that the tracking script sends its results to, so make sure it's correct. (Example hostnames: `shynet.example.com` or `example.com:8000`.)
 
 7. Set the whitelabel of your Shynet instance by running `docker run --env-file=<your env file> milesmcc/shynet:latest ./manage.py whitelabel <whitelabel>`. While this setting doesn't affect any core operations of Shynet, it lets you rename Shynet to whatever you want. (Example whitelabels: `"My Shynet Instance"` or `"Acme Analytics"`.)
 
@@ -60,7 +62,7 @@ Once you deploy, you'll need to setup an admin user, whitelabel, and hostname be
 
 ---
 
-## Enhancements
+## Advanced Usage
 
 ### Installation with SSL
 
@@ -175,6 +177,40 @@ Nginx is a self hosted, highly configurable webserver. Nginx can be configured t
    * [How to add SSL/HTTPS to Nginx (Ubuntu 16.04)](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04)
    * [Nginx Documentation](https://nginx.org/en/docs/)
 
+### Health Checks
+
+By default, Shynet includes a default health check endpoint at `/healthz/`. If the instance is running normally, this endpoint will return an HTTP status code of 200; if something is wrong, it will have a non-200 status code. To view the health data as JSON, send your request to `/healthz/?format=json`.
+
+This feature is helpful when running Shynet with Kubernetes, as it allows you to setup [startup readiness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) that prevent traffic from being sent to your Shynet instances before they are ready.
+
+### Primary-Key Integration
+
+In some cases, it is useful to associate particular users on your platform with their sessions in Shynet. In Shynet, this is called _primary key integration_, and is done by adding an additional element to the Shynet script url for each particular user.
+
+If the Shynet script location (for either the pixel or the script) is, for example, `//shynet.example.com/ingress/your_service_uuid/pixel.gif` and `//shynet.example.com/ingress/your_service_uuid/script.js`, the URLs for primary-key enabled users would be `//shynet.example.com/ingress/your_service_uuid/USER_PRIMARY_KEY/pixel.gif` and `//shynet.example.com/ingress/your_service_uuid/USER_PRIMARY_KEY/script.js`.
+
+Adding this path can be done easily using server-side rendering. For example, here is a Django template that adds users' primary keys to the Shynet tracking script:
+
+```html
+{% if request.user.is_authenticated %}
+<noscript>
+   <img src="//shynet.example.com/ingress/service-uuid/{{request.user.email|urlencode:""}}/pixel.gif">
+</noscript>
+<script src="//shynet.example.com/ingress/service-uuid/{{request.user.email|urlencode:""}}/script.js"></script>
+{% else %}
+<noscript>
+   <img src="//shynet.example.com/ingress/service-uuid/pixel.gif">
+</noscript>
+<script src="//shynet.example.com/ingress/service-uuid/script.js"></script>
+{% endif %}
+```
+
+### Usage with Single-Page Applications
+
+In a single-page application, the page never reloads. (That's the entire point of single-page applications, after all!) Unfortunately, this also means that Shynet will not automatically recognize and track when the user navigates between pages _within_ your application.
+
+Fortunately, Shynet offers a simple method you can call from anywhere within your JavaScript to indicate that a new page has been loaded: `Shynet.newPageLoad()`. Add this method call to the code that handles routing in your app, and you'll be ready to go.
+
 ---
 
 ## Troubleshooting

Pipfile

@@ -23,7 +23,6 @@ psycopg2-binary = "~=2.8.5"
 redis = "~=3.5.3"
 django-redis-cache = "~=2.1.1"
 pycountry = "~=19.8.18"
-ipaddress = "~=1.0.23"
 html2text = "~=2020.1.16"
 django-health-check = "~=3.12.1"
 django-npm = "~=1.0.0"

Pipfile.lock

@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "ff989ac3413a6bd2253c9350c8f368a91942393221f1e47e5e39e60e457cc590"
+            "sha256": "c18d6dc7c78d5f0634e38bb81bc1cf2cd4a0c128d70ca667fe765a66b294e66e"
         },
         "pipfile-spec": 6,
         "requires": {},
@@ -154,14 +154,6 @@
             ],
             "version": "==2.10"
         },
-        "ipaddress": {
-            "hashes": [
-                "sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc",
-                "sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2"
-            ],
-            "index": "pypi",
-            "version": "==1.0.23"
-        },
         "kombu": {
             "hashes": [
                 "sha256:be48cdffb54a2194d93ad6533d73f69408486483d189fe9f5990ee24255b0e0a",
@@ -227,10 +219,10 @@
         },
         "python3-openid": {
             "hashes": [
-                "sha256:0086da6b6ef3161cfe50fb1ee5cceaf2cda1700019fda03c2c5c440ca6abe4fa",
-                "sha256:628d365d687e12da12d02c6691170f4451db28d6d68d050007e4a40065868502"
+                "sha256:33fbf6928f401e0b790151ed2b5290b02545e8775f982485205a066f874aaeaf",
+                "sha256:6626f771e0417486701e0b4daff762e7212e820ca5b29fcc0d05f6f8736dfa6b"
             ],
-            "version": "==3.1.0"
+            "version": "==3.2.0"
         },
         "pytz": {
             "hashes": [

package.json

@@ -18,11 +18,10 @@
   "homepage": "https://github.com/milesmcc/shynet#readme",
   "dependencies": {
     "@fortawesome/fontawesome-free": "^5.13.1",
-    "a17t": "^0.1.4",
+    "a17t": "^0.2.2",
     "apexcharts": "^3.19.3",
     "inter-ui": "^3.13.1",
     "litepicker": "^1.5.7",
-    "tailwindcss": "^1.4.6",
     "turbolinks": "^5.2.0"
   }
 }

shynet/a17t/templates/a17t/includes/head.html

@@ -2,7 +2,7 @@
 
 <link rel="stylesheet" href="{% static 'a17t/dist/a17t.css' %}">
 <script async src="{% static '@fortawesome/fontawesome-free/js/all.min.js' %}" data-mutate-approach="sync"></script>
-<link href="{% static 'tailwindcss/dist/tailwind.min.css' %}" rel="stylesheet">
+<link href="{% static 'a17t/dist/tailwind.css' %}" rel="stylesheet">
 <link href="{% static 'inter-ui/Inter (web)/inter.css' %}" rel="stylesheet">
 <style>
     :root {

shynet/a17t/templates/a17t/includes/pagination.html

@@ -15,7 +15,7 @@
     <ul class="pagination-list w-full md:w-auto mb-2 flex">
         {% for pnum in begin %}
         {% ifequal page.number pnum %}
-        <li><a class="button field w-auto mx-1 text-white bg-gray-700">{{ pnum }}</a></li>
+        <li><a class="button field w-auto mx-1 text-white bg-neutral-700">{{ pnum }}</a></li>
         {% else %}
         <li><a class="button field w-auto mx-1" href="?page={{ pnum }}{{url_parameters}}">{{ pnum }}</a></li>
         {% endifequal %}
@@ -25,7 +25,7 @@
         <li><span class="pagination-ellipsis">&hellip;</span></li>
         {% for pnum in middle %}
         {% ifequal page.number pnum %}
-        <li><a class="button field w-auto mx-1 text-white bg-gray-700">{{ pnum }}</a></li>
+        <li><a class="button field w-auto mx-1 text-white bg-neutral-700">{{ pnum }}</a></li>
         {% else %}
         <li><a class="button field w-auto mx-1" href="?page={{ pnum }}{{url_parameters}}">{{ pnum }}</a></li>
         {% endifequal %}
@@ -36,7 +36,7 @@
         <li><span class="pagination-ellipsis">&hellip;</span></li>
         {% for pnum in end %}
         {% ifequal page.number pnum %}
-        <li><a class="button field w-auto mx-1 text-white bg-gray-700">{{ pnum }}</a></li>
+        <li><a class="button field w-auto mx-1 text-white bg-neutral-700">{{ pnum }}</a></li>
         {% else %}
         <li><a class="button field w-auto mx-1" href="?page={{ pnum }}{{url_parameters}}">{{ pnum }}</a></li>
         {% endifequal %}

shynet/analytics/tasks.py

@@ -64,7 +64,10 @@ def ingress_request(
         try:
             remote_ip = ipaddress.ip_network(ip)
             for ignored_network in service.get_ignored_networks():
-                if ignored_network.supernet_of(remote_ip):
+                if (
+                    ignored_network.version == remote_ip.version
+                    and ignored_network.supernet_of(remote_ip)
+                ):
                     return
         except ValueError as e:
             log.exception(e)

shynet/dashboard/forms.py

@@ -31,7 +31,7 @@ class ServiceForm(forms.ModelForm):
             "script_inject": forms.Textarea(attrs={'class':'font-mono', 'rows': 5})
         }
         labels = {
-            "origins": "Allowed Hostnames",
+            "origins": "Allowed origins",
             "respect_dnt": "Respect DNT",
             "collect_ips": "Collect IP addresses",
             "ignored_ips": "Ignored IP addresses",
@@ -43,7 +43,7 @@ class ServiceForm(forms.ModelForm):
             "name": _("What should the service be called?"),
             "link": _("What's the service's primary URL?"),
             "origins": _(
-                "At what hostnames does the service operate? This sets CORS headers, so use '*' if you're not sure (or don't care)."
+                "At what origins does the service operate? Use commas to separate multiple values. This sets CORS headers, so use '*' if you're not sure (or don't care)."
             ),
             "respect_dnt": "Should visitors who have enabled <a href='https://en.wikipedia.org/wiki/Do_Not_Track'>Do Not Track</a> be excluded from all data?",
             "collect_ips": "Should individual IP addresses be collected? IP metadata (location, host, etc) will still be collected.",

shynet/dashboard/templates/base.html

@@ -7,7 +7,7 @@
   <title>{% block head_title %}Privacy-oriented analytics{% endblock %} | {{request.site.name}}</title>
   <meta name="robots" content="noindex">
   <meta name="viewport" content="width=device-width, initial-scale=1.0">
-  {% include 'a17t/head.html' %}
+  {% include 'a17t/includes/head.html' %}
   <script src="{% static 'apexcharts/dist/apexcharts.min.js'%}"></script>
   <script src="{% static 'litepicker/dist/js/main.js' %}"></script>
   <script src="{% static 'turbolinks/dist/turbolinks.js' %}"></script>
@@ -17,14 +17,14 @@
   {% endblock %}
 </head>
 
-<body class="bg-gray-200 min-h-full">
+<body class="bg-neutral-200 min-h-full">
   {% block body %}
 
   <section class="max-w-screen-xl mx-auto px-4 py-4 md:py-12 md:flex">
     <aside class="mb-8 md:w-2/12 md:pr-6 relative flex flex-wrap md:block justify-between items-center overflow-x-hidden">
       <a class="icon ~urge ml-2 md:ml-6 md:mb-8 md:mt-3" href="{% url 'dashboard:dashboard' %}">
-        <i class="fas fa-binoculars fa-3x text-purple-600 hidden md:block"></i>
-        <i class="fas fa-binoculars fa-2x text-purple-600 md:hidden"></i>
+        <i class="fas fa-binoculars fa-3x text-urge-600 hidden md:block"></i>
+        <i class="fas fa-binoculars fa-2x text-urge-600 md:hidden"></i>
       </a>
 
       <button class="button ~neutral !low md:hidden"
@@ -35,7 +35,7 @@
       </button>
       <hr class="sep h-4 md:h-8 w-full">
       <div id="navMenuExpanded"
-        class="bg-white shadow-lg md:shadow-none p-4 hidden rounded-lg md:block md:bg-transparent md:border-none md:p-0 w-full">
+        class="bg-neutral-000 shadow-lg md:shadow-none p-4 hidden rounded-lg md:block md:bg-transparent md:border-none md:p-0 w-full">
         {% if user.owning_services.all %}
         <p class="ml-2 mb-1 supra font-medium text-gray-500 pointer-events-none">Services</p>
 

shynet/dashboard/templates/dashboard/includes/date_range.html

@@ -2,7 +2,7 @@
     <input type="hidden" name="startDate" value="{{start_date.isoformat}}" id="startDate">
     <input type="hidden" name="endDate" value="{{end_date.isoformat}}" id="endDate">
 </form>
-<input type="input" id="rangePicker" placeholder="Date range" class="input ~neutral cursor-pointer" readonly>
+<input type="input" id="rangePicker" placeholder="Date range" class="input ~neutral bg-neutral-000 cursor-pointer" readonly>
 <style>
     :root {
         --litepickerMonthButtonHover: var(--color-urge);

shynet/dashboard/templates/dashboard/includes/service_overview.html

@@ -4,7 +4,7 @@
     {% with stats=object.stats %}
     <div class="p-4 md:flex justify-between">
         <div class="flex items-center mb-4 md:mb-0">
-            <h3 class="heading text-xl md:text-2xl mr-2 mb-1 text-purple-600">
+            <h3 class="heading text-xl md:text-2xl mr-2 mb-1 text-urge-600">
                 {{object.name}}
             </h3>
             {% include 'dashboard/includes/stats_status_chip.html' %}

shynet/dashboard/templates/dashboard/includes/session_list.html

@@ -13,7 +13,7 @@
         <tr>
             <td>
                 <a href="{% url 'dashboard:service_session' object.pk session.pk %}"
-                    class="font-medium text-purple-700">
+                    class="font-medium text-urge-700">
                     {{session.start_time|date:"M j Y, g:i a"|capfirst}}
                     {% if session.is_currently_active %}
                     <span class="badge ~positive">Online</span>

shynet/dashboard/templates/dashboard/includes/sidebar_portal.html

@@ -1,6 +1,6 @@
 {% load helpers %}
 
 <div>
-    <a class="portal !low {% if request.get_full_path|startswith:url %}~urge active bg-gray-100{% endif %}"
+    <a class="portal !low {% if request.get_full_path|startswith:url %}~urge active bg-neutral-100{% endif %}"
         {% if disable_turbolinks %}data-turbolinks="false"{% endif %} href="{{url}}">{{label}}</a>
 </div>

shynet/dashboard/templates/dashboard/pages/dashboard.html

@@ -13,7 +13,7 @@
         </div>
         {% has_perm "core.create_service" user as can_create %}
         {% if can_create %}
-        <a href="{% url 'dashboard:service_create' %}" class="button field w-auto">+ New Service</a>
+        <a href="{% url 'dashboard:service_create' %}" class="button field bg-neutral-000 w-auto">+ New Service</a>
         {% endif %}
     </div>
 </div>

shynet/dashboard/templates/dashboard/pages/service.html

@@ -6,13 +6,13 @@
 <div class="mr-2">{% include 'dashboard/includes/date_range.html' %}</div>
 {% has_perm 'core.change_service' user object as can_update %}
 {% if can_update %}
-<a href="{% url 'dashboard:service_update' service.uuid %}" class="button field ~neutral w-auto">Manage &rarr;</a>
+<a href="{% url 'dashboard:service_update' service.uuid %}" class="button field bg-neutral-000 w-auto">Manage &rarr;</a>
 {% endif %}
 {% endblock %}
 
 {% block service_content %}
 <div class="grid grid-cols-2 gap-6 md:flex justify-between mb-6 card ~neutral !high px-6" id="stats">
-    {% with classes="text-sm font-semibold" good_classes="text-green-400" bad_classes="text-red-400" neutral_classes="text-gray-400" %}
+    {% with classes="text-sm font-semibold" good_classes="text-positive-400" bad_classes="text-critical-400" neutral_classes="text-gray-400" %}
     <article class="">
         <p class="label text-gray-400">Sessions</p>
         <p class="heading">

shynet/dashboard/templates/dashboard/pages/service_session.html

@@ -5,7 +5,7 @@
 {% block head_title %}{{object.name}} Session{% endblock %}
 
 {% block service_actions %}
-<a href="{% url 'dashboard:service' object.uuid %}" class="button field ~neutral w-auto">Analytics &rarr;</a>
+<a href="{% url 'dashboard:service' object.uuid %}" class="button field bg-neutral-000 w-auto">Analytics &rarr;</a>
 {% endblock %}
 
 {% block service_content %}

shynet/dashboard/templates/dashboard/pages/service_update.html

@@ -5,7 +5,7 @@
 {% block head_title %}{{object.name}} Management{% endblock %}
 
 {% block service_actions %}
-<a href="{% url 'dashboard:service' object.uuid %}" class="button field ~neutral w-auto">View &rarr;</a>
+<a href="{% url 'dashboard:service' object.uuid %}" class="button field bg-neutral-000 w-auto">View &rarr;</a>
 {% endblock %}
 
 {% block service_content %}

shynet/shynet/settings.py

@@ -18,7 +18,7 @@ import urllib.parse as urlparse
 from django.contrib.messages import constants as messages
 
 # Increment on new releases
-VERSION = "v0.6.0"
+VERSION = "v0.6.1"
 
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -286,9 +286,8 @@ else:
 NPM_ROOT_PATH = "../"
 
 NPM_FILE_PATTERNS = {
-    "a17t": ["dist/a17t.css"],
+    "a17t": ["dist/a17t.css", "dist/tailwind.css"],
     "@fortawesome/fontawesome-free": ["js/all.min.js"],
-    "tailwindcss": ["dist/tailwind.min.css"],
     "apexcharts": ["dist/apexcharts.min.js"],
     "litepicker": ["dist/js/main.js"],
     "turbolinks": ["dist/turbolinks.js"],

shynet/startup_checks.sh

@@ -1,8 +1,8 @@
 #!/bin/bash
 # Check if setup is necessary, do setup as needed
 echo "Performing startup checks..."
-sanity_results=( $(./manage.py startup_checks) )
-if [[ ${sanity_results[0]} == True ]]; then
+startup_results=( $(./manage.py startup_checks) )
+if [[ ${startup_results[0]} == True ]]; then
   echo "Running migrations (setting up DB)..."
   {
     ./manage.py migrate && echo "Migrations complete!"
@@ -12,13 +12,13 @@ if [[ ${sanity_results[0]} == True ]]; then
   else
   echo "Database is ready to go."
 fi
-if [[ ${sanity_results[1]} == True ]]; then
+if [[ ${startup_results[1]} == True ]]; then
   echo "Warning: no admin user available. Consult docs for instructions."
 fi
-if [[ ${sanity_results[2]} == True ]]; then
+if [[ ${startup_results[2]} == True ]]; then
   echo "Warning: Shynet's hostname is not set. The script won't work correctly. Consult docs for instructions."
 fi
-if [[ ${sanity_results[3]} == True ]]; then
+if [[ ${startup_results[3]} == True ]]; then
   echo "Warning: Shynet's whitelabel is not set. Consult docs for instructions."
 fi
 echo "Startup checks complete!"