init

2025-09-02 14:49:16 +08:00
commit 38ba663466
2885 changed files with 391107 additions and 0 deletions
--- a/resources/extra-large-conference/README.md
+++ b/resources/extra-large-conference/README.md
@@ -0,0 +1,123 @@
+WARNING: This is still a Work In Progress
+
+The final implementation may diverge. Currently, only the participants after a
+configured threshold will be just viewers (visitors) and there is no promotion
+mechanism to become a main participant yet.
+
+TODO:
+* Polls
+* Speaker stats
+* call duration
+
+# Low-latency conference streaming to very large audiences
+
+To have a low-latency conference with a very large audience, the media and
+signaling load must be spread  beyond what can be handled by a typical Jitsi
+installation. A call with 10k participants requires around 50 bridges on decent
+vms (8+ cores). The main participants of a conference with a very large
+audience will share a main prosody, like with normal conferences, and
+additional prosody vms are needed to support signaling to the audience.
+
+In the example configuration we use a 16 core machine. Eight of the cores are
+used for the main prosody and other services (nginx, jicofo, etc) and the other
+eight cores are used to run prosody services for visitors, i.e., "visitor
+prosodies".
+
+We consider 2000 participants per visitor node a safe value. So eight visitor
+prosodies will be enough for one 10k participants meeting.
+
+<img src="imgs/visitors-prosody.svg" alt="diagram of a central prosody connected to several visitor prosodies" width="500"/>
+
+# Configuration
+If using older than Prosody 0.12.4 you need to apply the patch - s2sout_override1.patch and s2sout_override2.patch.
+Use the `pre-configure.sh` script to configure your system, passing it the
+number of visitor prosodies to set up.
+`./pre-configure.sh 8`
+
+The script will add for each visitor prosody:
+- folders in `/etc/`
+- a systemd unit file in `/lib/systemd/system/`
+- a user for jicofo
+- a config entry in jicofo.conf
+
+Setting up configuration for the main prosody is a manual process:
+- Add to the enabled modules list in the general part (e.g. [here](https://github.com/bjc/prosody/blob/76bf6d511f851c7cde8a81257afaaae0fb7a4160/prosody.cfg.lua.dist#L33)):
+```
+      "s2s_bidi";
+      "certs_s2soutinjection";
+      "s2sout_override";
+      "s2s_whitelist";
+```
+
+- Add the following config also in the general part (matching the number of prosodies you generated config for):
+```
+-- targets must be IPs, not hostnames
+s2sout_override = {
+    ["conference.v1.meet.jitsi"] = "tcp://127.0.0.1:52691";
+    ["v1.meet.jitsi"] = "tcp://127.0.0.1:52691"; -- needed for v1.meet.jitsi->visitors.jitmeet.example.com
+    ["conference.v2.meet.jitsi"] = "tcp://127.0.0.1:52692";
+    ["v2.meet.jitsi"] = "tcp://127.0.0.1:52692";
+    ["conference.v3.meet.jitsi"] = "tcp://127.0.0.1:52693";
+    ["v3.meet.jitsi"] = "tcp://127.0.0.1:52693";
+    ["conference.v4.meet.jitsi"] = "tcp://127.0.0.1:52694";
+    ["v4.meet.jitsi"] = "tcp://127.0.0.1:52694";
+    ["conference.v5.meet.jitsi"] = "tcp://127.0.0.1:52695";
+    ["v5.meet.jitsi"] = "tcp://127.0.0.1:52695";
+    ["conference.v6.meet.jitsi"] = "tcp://127.0.0.1:52696";
+    ["v6.meet.jitsi"] = "tcp://127.0.0.1:52696";
+    ["conference.v7.meet.jitsi"] = "tcp://127.0.0.1:52697";
+    ["v7.meet.jitsi"] = "tcp://127.0.0.1:52697";
+    ["conference.v8.meet.jitsi"] = "tcp://127.0.0.1:52698";
+    ["v8.meet.jitsi"] = "tcp://127.0.0.1:52698";
+}
+-- allowed list of server-2-server connections
+s2s_whitelist = {
+    "conference.v1.meet.jitsi", "conference.v2.meet.jitsi", "conference.v3.meet.jitsi", "conference.v4.meet.jitsi",
+    "conference.v5.meet.jitsi", "conference.v6.meet.jitsi", "conference.v7.meet.jitsi", "conference.v8.meet.jitsi"
+};
+```
+
+- Make sure s2s is not in modules_disabled
+- Enable `"visitors";` module under the main virtual host (e.g. [here](https://github.com/jitsi/jitsi-meet/blob/f42772ec5bcc87ff6de17423d36df9bcad6e770d/doc/debian/jitsi-meet-prosody/prosody.cfg.lua-jvb.example#L57))
+  You can add under main virtual host the config: `visitors_ignore_list = { "recorder.jitmeet.example.com" }` to ignore jibri and transcribers from the visitor logic and use them only in the main prosody conference.
+- Create the visitors component in /etc/prosody/conf.d/jitmeet.example.com.cfg.lua:
+```
+Component "visitors.jitmeet.example.com" "visitors_component"
+  auto_allow_visitor_promotion = true
+  admins = { "focus@auth.jitmeet.example.com" }
+```
+- Make sure you add the correct upstreams to nginx config
+```
+upstream v1 {
+    zone upstreams 64K;
+    server 127.0.0.1:52801;
+    keepalive 2;
+}
+upstream v2 {
+    zone upstreams 64K;
+    server 127.0.0.1:52802;
+    keepalive 2;
+}
+```
+
+After configuring you can set the maximum number of main participants, before
+redirecting to visitors.
+```
+hocon -f /etc/jitsi/jicofo/jicofo.conf set "jicofo.visitors.enabled" true
+hocon -f /etc/jitsi/jicofo/jicofo.conf set "jicofo.visitors.max-participants" 30
+```
+Now restart prosody and jicofo
+```
+service prosody restart
+service jicofo restart
+service nginx restart
+```
+
+Now after the main 30 participants join, the rest will be visitors using the
+visitor nodes.
+
+To enable promotion where visitors need to be approved by a moderator to join the meeting:
+  - you need to switch `auto_allow_visitor_promotion=false`.
+  - You need to enable http requests to jicofo by editing config.js and adding a nginx rule for it. 
+    - In /etc/jitsi/meet/jitmeet.example.com-config.js uncomment conferenceRequestUrl.
+    - In jitsi-meet nginx config make sure you have the conference-request location rules.
--- a/resources/extra-large-conference/imgs/visitors-prosody.svg
+++ b/resources/extra-large-conference/imgs/visitors-prosody.svg
--- a/resources/extra-large-conference/pre-configure.sh
+++ b/resources/extra-large-conference/pre-configure.sh
@@ -0,0 +1,61 @@
+#!/bin/bash
+
+SCRIPT_DIR=`dirname "$0"`
+cd $SCRIPT_DIR
+
+NUMBER_OF_INSTANCES=$1
+
+if ! [[ $NUMBER_OF_INSTANCES =~ ^[0-9]+([.][0-9]+)?$ ]] ; then
+   echo "error: Not a number param" >&2;
+   exit 1
+fi
+
+echo "Will configure $NUMBER_OF_INSTANCES number of visitor prosodies"
+set -e
+set -x
+
+JICOFO_HOSTNAME=$(echo get jitsi-videobridge/jvb-hostname | sudo debconf-communicate jicofo | cut -d' ' -f2-)
+
+# Configure prosody instances
+for (( i=1 ; i<=${NUMBER_OF_INSTANCES} ; i++ ));
+do
+    cp prosody-v.service.template /lib/systemd/system/prosody-v${i}.service
+    sed -i "s/vX/v${i}/g" /lib/systemd/system/prosody-v${i}.service
+    mkdir /etc/prosody-v${i}
+    ln -s /etc/prosody/certs /etc/prosody-v${i}/certs
+    cp prosody.cfg.lua.visitor.template /etc/prosody-v${i}/prosody.cfg.lua
+    sed -i "s/vX/v${i}/g" /etc/prosody-v${i}/prosody.cfg.lua
+    sed -i "s/jitmeet.example.com/$JICOFO_HOSTNAME/g" /etc/prosody-v${i}/prosody.cfg.lua
+    # fix the ports
+    sed -i "s/52691/5269${i}/g" /etc/prosody-v${i}/prosody.cfg.lua
+    sed -i "s/52221/5222${i}/g" /etc/prosody-v${i}/prosody.cfg.lua
+    sed -i "s/52801/5280${i}/g" /etc/prosody-v${i}/prosody.cfg.lua
+    sed -i "s/52811/5281${i}/g" /etc/prosody-v${i}/prosody.cfg.lua
+done
+
+# Configure jicofo
+HOCON_CONFIG="/etc/jitsi/jicofo/jicofo.conf"
+hocon -f $HOCON_CONFIG set "jicofo.bridge.selection-strategy" "VisitorSelectionStrategy"
+hocon -f $HOCON_CONFIG set "jicofo.bridge.visitor-selection-strategy" "RegionBasedBridgeSelectionStrategy"
+hocon -f $HOCON_CONFIG set "jicofo.bridge.participant-selection-strategy" "RegionBasedBridgeSelectionStrategy"
+hocon -f $HOCON_CONFIG set "jicofo.bridge.topology-strategy" "VisitorTopologyStrategy"
+
+PASS=$(hocon -f $HOCON_CONFIG get "jicofo.xmpp.client.password")
+for (( i=1 ; i<=${NUMBER_OF_INSTANCES} ; i++ ));
+do
+  prosodyctl --config /etc/prosody-v${i}/prosody.cfg.lua register focus auth.meet.jitsi $PASS
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.enabled" true
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.conference-service" "conference.v${i}.meet.jitsi"
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.hostname" 127.0.0.1
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.port" 5222${i}
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.domain" "auth.meet.jitsi"
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.xmpp-domain" "v${i}.meet.jitsi"
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.password" "${PASS}"
+  hocon -f $HOCON_CONFIG set "jicofo.xmpp.visitors.v${i}.disable-certificate-verification" true
+done
+
+for (( i=1 ; i<=${NUMBER_OF_INSTANCES} ; i++ ));
+do
+  service prosody-v${i} restart
+done
+service jicofo restart
--- a/resources/extra-large-conference/prosody-v.service.template
+++ b/resources/extra-large-conference/prosody-v.service.template
@@ -0,0 +1,46 @@
+[Unit]
+### see man systemd.unit
+Description=Prosody vX (visitor vX) JVB XMPP Server
+Documentation=https://prosody.im/doc
+
+Requires=network-online.target
+After=network-online.target network.target mariadb.service mysql.service postgresql.service
+Before=biboumi.service
+
+[Service]
+### see man systemd.service
+Type=simple
+
+# Start by executing the main executable
+# Note: -F option requires Prosody 0.11.5 or later
+ExecStart=/usr/bin/prosody --config /etc/prosody-vX/prosody.cfg.lua -F
+ExecReload=/bin/kill -HUP $MAINPID
+Restart=on-abnormal
+
+### see man systemd.exec
+User=prosody
+Group=prosody
+UMask=0027
+
+RuntimeDirectory=prosody-vX
+ConfigurationDirectory=prosody-vX
+StateDirectory=prosody-vX
+StateDirectoryMode=0750
+LogsDirectory=prosody-vX
+WorkingDirectory=~
+
+# Set stdin to /dev/null since Prosody does not need it
+StandardInput=null
+
+# Direct stdout/-err to journald for use with log = "*stdout"
+StandardOutput=journal
+StandardError=inherit
+
+# Allow binding low ports
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+
+[Install]
+### see man systemd.unit
+WantedBy=multi-user.target
+
+# vim: filetype=systemd
--- a/resources/extra-large-conference/prosody.cfg.lua.visitor.template
+++ b/resources/extra-large-conference/prosody.cfg.lua.visitor.template
@@ -0,0 +1,140 @@
+---------- Server-wide settings ----------
+s2s_ports = { 52691 };
+c2s_ports = { 52221 }
+http_ports = { 52801 }
+https_ports = { 52811 }
+
+daemonize = true;
+
+-- we use a common jid for jicofo
+admins = {
+    'focus@auth.meet.jitsi'
+}
+
+-- Enable use of native prosody 0.11 support for epoll over select
+network_backend = 'epoll';
+network_settings = {
+  tcp_backlog = 511;
+}
+
+modules_enabled = {
+      'saslauth';
+      'tls';
+      'disco';
+      'posix';
+
+      'secure_interfaces';
+
+    -- jitsi
+      'websocket';
+      'bosh';
+      's2s_bidi';
+      's2s_whitelist';
+      's2sout_override';
+      'certs_s2soutinjection';
+};
+
+s2s_whitelist = {
+    'conference.jitmeet.example.com', -- needed for visitors to send messages to main room
+    'visitors.jitmeet.example.com'; -- needed for sending promotion request to visitors.jitmeet.example.com component
+    'jitmeet.example.com'; -- unavailable presences back to main room
+};
+
+s2sout_override = {
+    ["conference.jitmeet.example.com"] = "tcp://127.0.0.1:5269"; -- needed for visitors to send messages to main room
+    ["jitmeet.example.com"] = "tcp://127.0.0.1:5269"; -- needed for the main room when connecting in to send main participants
+    ["visitors.jitmeet.example.com"] = "tcp://127.0.0.1:5269"; -- needed for sending promotion request to visitors.jitmeet.example.com component
+}
+
+external_service_secret = '__turnSecret__';
+
+external_services = {
+    { type = 'stun', host = 'jitmeet.example.com', port = 3478 },
+    { type = 'turn', host = 'jitmeet.example.com', port = 3478, transport = 'udp', secret = true, ttl = 86400, algorithm = 'turn' },
+    { type = 'turns', host = 'jitmeet.example.com', port = 5349, transport = 'tcp', secret = true, ttl = 86400, algorithm = 'turn' }
+};
+
+muc_mapper_domain_base = 'vX.meet.jitsi';
+main_domain = 'jitmeet.example.com';
+
+-- https://prosody.im/doc/modules/mod_smacks
+smacks_max_unacked_stanzas = 5;
+smacks_hibernation_time = 60;
+-- this is dropped in 0.12
+smacks_max_hibernated_sessions = 1;
+smacks_max_old_sessions = 1;
+
+unlimited_jids = { 'focus@auth.meet.jitsi' }
+limits = {
+    c2s = {
+        rate = '512kb/s';
+    };
+}
+
+modules_disabled = {
+    'offline';
+    'pubsub';
+    'register';
+};
+
+allow_registration = false;
+authentication = 'internal_hashed'
+storage = 'internal'
+log = {
+    -- Log files (change 'info' to 'debug' for debug logs):
+    info = '/var/log/prosody-vX/prosody.log';
+    error = '/var/log/prosody-vX/prosody.err';
+}
+
+consider_websocket_secure = true;
+consider_bosh_secure = true;
+bosh_max_inactivity = 60;
+
+plugin_paths = { '/usr/share/jitsi-meet/prosody-plugins/' }
+
+----------- Virtual hosts -----------
+VirtualHost 'vX.meet.jitsi'
+    authentication = 'jitsi-anonymous'
+    ssl = {
+        key = '/etc/prosody/certs/jitmeet.example.com.key';
+        certificate = '/etc/prosody/certs/jitmeet.example.com.crt';
+    }
+    modules_enabled = {
+      'bosh';
+      'ping';
+      'external_services';
+      'smacks';
+      'jiconop';
+      'conference_duration';
+    }
+    main_muc = 'conference.vX.meet.jitsi';
+
+VirtualHost 'auth.meet.jitsi'
+    modules_enabled = {
+          'limits_exception';
+          'ping';
+          'smacks';
+    }
+    authentication = 'internal_hashed'
+    smacks_hibernation_time = 15;
+
+Component 'conference.vX.meet.jitsi' 'muc'
+    storage = 'memory'
+    muc_room_cache_size = 10000
+    restrict_room_creation = true
+    modules_enabled = {
+        'muc_hide_all';
+        'muc_domain_mapper';
+        'muc_meeting_id';
+        'fmuc';
+        's2s_bidi';
+        's2s_whitelist';
+        's2sout_override';
+    }
+    muc_room_default_presence_broadcast = {
+        visitor = false;
+        participant = true;
+        moderator = true;
+    };
+    muc_room_locking = false
+    muc_room_default_public_jids = true