This article will demonstrate how to configure Fedora Server as an alert and notification server that can place calls using an Asterisk PBX and send SMS text messages using Twilio. The use of the SMS message feature is optional. By using the call_only endpoint of the caller_prometheus_webhook component, you can limit the alerts to voice calls only.
Please consider that interacting with the Asterisk PBX is not easy. But it isn’t too hard either. If this is your first time working with this kind of application, coming to understand the concepts may require some patience. Fortunately, Fedora Server can be configured with Ansible and the installation of the py-phone-caller containers will not be difficult.
Note: The py-phone-caller packages do not have any endorsement or relation with Twilio or Asterisk PBX/FreePBX. These services and products were chosen for their ease of use and their commitment to open source.
Note: This guide assumes that the server is on a local area network (LAN) that is behind a firewall configured to block all outside connections. Do not open the services’ ports to the internet without requiring authentication.
The big picture
The py-phone-caller components are represented by the blue boxes. The third party components or dependencies are the green boxes. And the yellow box represents the receiver of the calls and/or text messages.
Component overview
generate_audio
- Role: used to create and host the audio files player by the Asterisk PBX.
- Container repository: quay.io/py-phone-caller/generate_audio
- FROM: fedora:34 (base container image)
caller_sms
- Role: used to send the SMS messages through a service provider.
- Container repository: quay.io/py-phone-caller/caller_sms
- FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
caller_prometheus_webhook
- Role: start a call or send an SMS message when a Prometheus alert is received.
- Container repository: quay.io/py-phone-caller/caller_prometheus_webhook
- FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
- Note: This container has four endpoints that behave differently:
– call_only: used to originate a single call
– sms_only: used to send a single SMS
– sms_before_call: first, send an SMS and later originate a call (after the number of seconds configured in: sms_before_call_wait_seconds)
– call_and_sms: used to send the SMS and place the call at the same time.
call_register
- Role: used to register on the PostgreSQL DB the arriving calls with useful details.
- Container repository: quay.io/py-phone-caller/call_register
- FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
asterisk_ws_monitor
- Role: this component register the Stasis application against Asterisk and also log the events to the DB (table: asterisk_ws_events).
- Container repository: quay.io/py-phone-caller/asterisk_ws_monitor
- FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
asterisk_recall
- Role: reading from the database and considering the times_to_dial and seconds_to_forget configuration parameters, retries a failed or not acknowledged call (… press 4 to acknowledge…).
- Container repository: quay.io/py-phone-caller/asterisk_recall
- FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
asterisk_call
- Role: has the responsibility to place the calls against the Asterisk PBX through the REST interface.
- Container repository: quay.io/py-phone-caller/asterisk_call
- FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
Known limitations
Version 0.0.2 of the py-phone-caller application does not have a call traffic controller. Consequently, if multiple events are sent to asterisk_call at the same time, some notifications may be dropped. For reliability, be sure to send no more than one alert at a time from the caller_prometheus_webhook and send only one HTTP POST at a time when using a HTTP client such as curl. To avoid collisions between messages in asterisk_call, do not send events more frequently than the value configured in seconds_to_forget.
Optionally, support for concurrent calls can be enabled by adding more instances of asterisk_call and asterisk_ws_monitor behind an instance of HAProxy. Each additional instance of asterisk_call will require an additional entry in extensions_custom.conf.
To be clear, you need to add one more instance of each of the following for each additional concurrent request that you want to be able to handle:
- asterisk_call has the responsibility of the call initialization using a given custom extension. This extension calls the Stasis application.
- asterisk_ws_monitor registers new instances of the Stasis application. It is referenced in extensions_custom.conf. Be sure to use unique names.
- An entry in extensions_custom.conf is required for exclusively by the asterisk_call instance. This is also where you will reference the Stasis application name configured in asterisk_stasis_app.
Note: You can use the same configuration file and override the settings with environment variables. The variables will have priority over the values in the configuration file. The environmental variables use the same names as their corresponding settings, but they must be in upper case. For example, asterisk_stasis_app becomes ASTERISK_STASIS_APP.
Note: chan_pjsip (PJSIP channel) is not yet supported. Support will be added in a future release.
Prerequisites
- A Twillio account is required for sending SMS messages to cell phones (sending SMS messages is optional).
- A SIP trunk, a media gateway or a SIM card inside a device is required for placing calls to the landlines or cell phones.
- An internet connection.
Note: Twilio is the only SMS service provider currently available. Other providers will be added.
Note: If you choose not to send SMS (Short Message Service) messages and not to place calls to paid phone networks, you can still make calls over a SIP (Session Initiation Protocol) or IAX2 (Inter-Asterisk eXchange version 2) extension of the PBX (Private Branch eXchange) to a softphone (software phone) installed on your cell phone or to a physical phone that supports either of these protocols.
Note: You will need to pay a few cents to place calls and send SMS messages to landlines or cell phones. These services are rarely free.
Systems overview
This guide will use the FreePBX Asterisk distribution. It has a web interface that is more user-friendly. The components that initiate calls will be based on Fedora Server 34. Other Fedora Linux variants such as Fedora Workstation or Fedora Cloud Base might work. But all tests and working setups have been based on Fedora Server or CentOS 7 in Docker containers. I am also planning to test this deployment on RHEL 8 and OpenShift/Kubernetes.
- IP Address: 192.168.122.104
- SNG7-PBX-64bit-2104-1.iso
- IP Address: 192.168.122.234
Note: You can use different IP addresses for the Fedora Server and the Asterisk system.
Configuration of the Asterisk PBX
A SIP Trunk is required to place calls to cell phones or landlines. If you choose to use only local extensions and you do not intend to place calls to external phones, you can configure a SIP or IAX2 extension instead.
The py-phone-caller will also require a custom extension and an ARI (Asterisk REST Interface) user.
Configure the SIP Trunk
- First click Connectivity.
- Then click Trunks.
- Click + Add Trunk.
- Click + Add SIP (chan_sip) Trunk.
- Configure the Trunk Name. This name will form part of the value of asterisk_chan_type in the py-phone-caller configuration. For example, if sip-provider is chosen for Trunk Name, the corresponding py-phone-caller configuration value would be SIP/sip-provider.
- In the Outbound CallerID, you can use any preferred value.
- Click the sip Settings tab.
- Click the Outgoing tab.
- Configure the Trunk Name. Use the same name that you chose previously.
- On the PEER Details section, enter the configuration values required to reach the SIP provider.
- To save the configuration, click the Submit button.
An example PEER Details configuration:
type=peer auth=md5 username=your-username fromuser=your-username secret=your-password host=sip.provider.com port=5060 qualify=yes insecure=very
- Click Apply Config.
- Wait until the reloading process is done.
You should now have a trunk that can be used to place calls to cell and landline phones.
Note: This configuration has a cost depending on the provider or device used to access the public phone network (cell or landline).
Define a custom dial plan
This extension will initiate the call and then pass control to py-phone-caller. The message will be the description of a Prometheus alert. It will be sent from the Alertmanager to the caller_prometheus_webhook.
- Click Admin.
- Click Config Edit.
- From the panel on the left, select extensions_custom.conf.
- Place the text from the custom dial plan below in the Working on extensions_custom.conf text area on the right.
- Click again on Working on extensions_custom.conf to validate the new settings.
- Click Save.
- Click Apply Config and wait until the configuration reloading process has finished.
An example dial plan:
[py-phone-caller]
exten => 3216,1,Noop()
; Greeting message
same => n,Playback(greeting-message)
; Give the control of the ongoing call to the 'py-phone-caller' Stasis application
same => n,Stasis(py-phone-caller)
; Do a get HTTP request against the 'call_register' when the message was played
same => n,Set(RES=${CURL(http://192.168.122.104:8083/heard?asterisk_chan=${CHANNEL(uniqueid)})})
; Play an audio message in order to get the acknowledgement
; in our case we wait for '4'. "see line xx: {IF($[ ${get} = 4 ..."
same => n,Playback(press-4-for-acknowledgement)
same => n,Playback(beep)
same => n,Read(get,"silence/1",,,,2)
; If not digit is provided go to the priority 20 in order to say 'goodbye'
same => n,Set(gotdigit=${ISNULL(${get})})
same => n,GotoIf(${gotdigit}=1?20)
; When '4' is pressed go to the priority 30, update the DB record
; and say Goodbye.
same => e,Playback(vm-goodbye) ; If there's an error, say goodbye
same => n,Set(NOTIFYACK=${IF($[ ${get} = 4]?3:0)})
same => n,Wait(1)
same => n,GotoIf(${NOTIFYACK}=3?30)
same => 20,Set(NOTIFYACK=2)
same => 21,Playback(vm-goodbye)
same => 22,Wait(1)
same => 23,Hangup()
; Do a get HTTP request against the 'call_register' to update the DB record
; when the call was acknowledged.
same => 30,Set(RES=${CURL(http://192.168.122.104:8083/ack?asterisk_chan=${CHANNEL(uniqueid)})})
same => 31,Playback(vm-goodbye)
same => 32,Wait(1)
same => 33,Hangup()
same => n,Playback(vm-goodbye)
same => n,Hangup()
A step-by-step breakdown of the above dial plan:
- The extension name ([py-phone-caller])
- The extension number (3216). You can choose a different number. If you use a different number, be sure to use the same number when configuring py-phone-caller.
- The PBX plays the audio file greeting-message.wav. How to create this file and copy it to the PBX is explained later in this guide.
- The Stasis(py-phone-caller) line transfers control of the call to py-phone-caller.
- After py-phone-caller returns, an HTTP GET request is sent to call_register to record that a message was sent.
- The PBX plays the audio file press-4-for-acknowledgement.wav.
- The PBX plays a beep to signal to the callee that it is ready to receive input.
- The Read(get,"silence/1",,,,2) function waits for the callee’s input.
- If the callee didn’t press the number 4 the call is terminated.
Creating a standard Asterisk extension
A SIP or IAX2 extension can be used instead of a SIP Trunk. However, these protocols will only be able to dial out to softphones or VoIP phones. You will not be able to call a PSTN or cell phone using SIP or IAX2. There are no additional costs when using these extensions. They can be configured directly within the PBX. They can be used on the same local network the PBX is connected to. Or they can be routed over the internet. If you choose to route these protocols over the internet, please use a security layer such as TLS and use secure passwords. If you use a softphone installed on your cell phone, be sure that your internet connection has sufficient bandwidth to provide high-quality audio.
Note: When using these alternative protocols, the value of asterisk_chan_type will be SIP instead of SIP/sip-provider. When using these protocols, a SIP Trunk or Media Gateway is not required.
Note: This endpoint can be viewed as the callee. It is the extension or phone number that will be called when a new Prometheus alert or HTTP POST request is sent to asterisk_call. It is also is possible to initiate calls directly from cron jobs and other scripts.
- Click Applications.
- Click Extensions.
- Click + Add Extension.
- Click Add New SIP (Legacy) [chan_sip] Extension.
- Configure the User Extension. This is usually a number. This example uses 1614. This number will be used as the value for prometheus_webhook_receivers in the caller_prometheus_webhook configuration block.
- Configure the Display Name. This text will appear on the display of the callee’s phone.
- Provide a Secret. This is the password for your softphone or SIP phone. FreePBX will suggest a strong password automatically.
- Click Submit.
- Click Apply Config.
- Wait for Reloading to finish.
Configure the ARI user
The ARI (Asterisk REST Interface) user will be used in asterisk_ws_monitor to register the Stasis application (a permanent WebSocket connection). This account will also be used by asterisk_call to make calls to the Asterisk PBX (or FreePBX).
- Click Settings.
- Click Asterisk REST Interface Users.
- Click + Add User.
- Set the REST Interface User Name. This example uses py-phone-caller.
- Set the REST Interface User Password. You may choose something different from the random value that is suggested.
- Set the Password Type. During development and testing, it is OK to use Plain Text. In production use Crypt. When using crypt, the password will need to be provided in encrypted form using the sha-512 hash algorithm. See here for more information.
- Set the Read Only option to No.
- Click Submit.
- Click Apply Config.
- Wait for Reloading to finish.
Create the audio recordings
- Greeting message: “Hello. This is a recorded message from the alerting system.“
- Request acknowledgement: “Please press the number four to acknowledge this call.”
Install espeak.
$ sudo dnf -y install espeak
Generate the audio files in wave format.
$ espeak -s 140 -g 4 -w /tmp/greeting-message_22050.wav "Hello. This is a recorded message from the alerting system." $ espeak -s 140 -g 4 -w /tmp/press-4-for-acknowledgement_22050.wav "After the beep, please press the number four to acknowledge this call."
The espeak program records the audio files with a sampling frequency of 22050 Hz. However, Asterisk requires 8000 Hz. Install sox and use it to convert the files to the required format.
$ sudo dnf -y install sox $ sox /tmp/greeting-message_22050.wav -r 8000 -c 1 /tmp/greeting-message.wav $ sox /tmp/press-4-for-acknowledgement_22050.wav -r 8000 -c 1 /tmp/press-4-for-acknowledgement.wav
You can read more about espeak in its man pages and on Fedora Magazine. Alternatively, if you have a microphone and you know someone with a good voice you can record a real person.
The above example audio files are available here: assets/generic-audio-for-dialplan
Upload the recordings to FreePBX
Upload the audio files you created in the previous section to your Asterisk system.
$ scp /tmp/*.wav root@192.168.122.234:~ root@192.168.122.234's password: greeting-message.wav 100% 107KB 21.7MB/s 00:00 press-4-for-acknowledgement.wav 100% 77KB 21.9MB/s 00:00
Login to the Asterisk (FreePBX) system.
$ ssh root@192.168.122.234 root@192.168.122.234's password: Last failed login: Tue Aug 3 23:29:04 UTC 2021 from 192.168.122.1 on ssh:notty There was 1 failed login attempt since the last successful login. Last login: Tue Aug 3 23:33:50 2021 from 192.168.122.1 ______ ______ ______ __ __ | ___| | ___ \| ___ \\ \ / / | |_ _ __ ___ ___ | |_/ /| |_/ / \ V / | _| | '__| / _ \ / _ \| __/ | ___ \ / \ | | | | | __/| __/| | | |_/ // /^\ \ \_| |_| \___| \___|\_| \____/ \/ \/ NOTICE! You have 2 notifications! Please log into the UI to see them! Current Network Configuration +-----------+-------------------+-------------------------+ | Interface | MAC Address | IP Addresses | +-----------+-------------------+-------------------------+ | eth0 | 52:54:00:F1:1C:F6 | 192.168.122.234 | | | | fe80::5054:ff:fef5:6cf1 | +-----------+-------------------+-------------------------+ [...] [root@freepbx ~]#
This guide uses Asterisk’s default language, English. For English, the audio files are stored in /var/lib/asterisk/sounds/en. Move the uploaded audio files to this directory. If you configured a different language, you would need to move the files to a different directory.
The audio files should be in root’s home directory (/root). Move them to Asterisk’s directory.
[root@freepbx ~]# mv greeting-message.wav press-4-for-acknowledgement.wav /var/lib/asterisk/sounds/en
Finally, set the ownership for the files to asterisk:asterisk.
[root@freepbx ~]# chown asterisk:asterisk /var/lib/asterisk/sounds/en/{greeting-message.wav,press-4-for-acknowledgement.wav}
The configuration of the Asterisk PBX should now be complete.
Install the required software packages
This guide is using Fedora Server edition. Login to your Fedora server and use the sudo command to change to root user.
$ sudo -i
Install Podman.
# dnf -y install podman podman-plugins podman-docker Last metadata expiration check: 0:42:59 ago on Wed 28 Jul 2021 23:31:23 PM CEST. Dependencies resolved. ============================================================================================================================================================================= Package Architecture Version Repository Size ============================================================================================================================================================================= Installing: podman x86_64 3:3.2.3-1.fc34 updates 12 M podman-docker noarch 3:3.2.3-1.fc34 updates 177 k podman-plugins x86_64 3:3.2.3-1.fc34 updates 2.6 M Installing dependencies: conmon x86_64 2:2.0.29-2.fc34 updates 53 k container-selinux noarch 2:2.164.1-1.git563ba3f.fc34 updates 48 k containernetworking-plugins x86_64 1.0.0-0.2.rc1.fc34 updates 8.9 M containers-common noarch 4:1-21.fc34 updates 61 k criu x86_64 3.15-3.fc34 fedora 521 k criu-libs x86_64 3.15-3.fc34 fedora 31 k crun x86_64 0.20.1-1.fc34 updates 172 k fuse-common x86_64 3.10.4-1.fc34 updates 8.5 k fuse3 x86_64 3.10.4-1.fc34 updates 54 k fuse3-libs x86_64 3.10.4-1.fc34 updates 91 k libbsd x86_64 0.10.0-7.fc34 fedora 106 k libnet x86_64 1.2-2.fc34 fedora 58 k libslirp x86_64 4.4.0-4.fc34 updates 68 k yajl x86_64 2.1.0-16.fc34 fedora 38 k Installing weak dependencies: catatonit x86_64 0.1.5-4.fc34 fedora 305 k fuse-overlayfs x86_64 1.5.0-1.fc34 fedora 75 k slirp4netns x86_64 1.1.9-1.fc34 fedora 57 k Transaction Summary ============================================================================================================================================================================= Install 20 Packages Total download size: 25 M Installed size: 123 M [...]
Install Ansible and PostgreSQL.
# dnf install -y ansible python3-psycopg2 postgresql Last metadata expiration check: 0:38:54 ago on Wed 28 Jul 2021 23:41:48 PM CEST. Dependencies resolved. ============================================================================================================================================================================= Package Architecture Version Repository Size ============================================================================================================================================================================= Installing: ansible noarch 2.9.23-1.fc34 updates 15 M python3-psycopg2 x86_64 2.8.6-3.fc34 fedora 183 k Installing dependencies: libpq x86_64 13.3-1.fc34 updates 202 k libsodium x86_64 1.0.18-7.fc34 fedora 165 k python3-babel noarch 2.9.1-1.fc34 updates 5.8 M python3-bcrypt x86_64 3.1.7-7.fc34 fedora 44 k python3-cffi x86_64 1.14.5-1.fc34 fedora 244 k python3-chardet noarch 4.0.0-1.fc34 fedora 214 k python3-cryptography x86_64 3.4.6-1.fc34 fedora 1.4 M python3-idna noarch 2.10-3.fc34 fedora 99 k python3-jinja2 noarch 2.11.3-1.fc34 fedora 493 k python3-jmespath noarch 0.10.0-1.fc34 updates 46 k python3-markupsafe x86_64 1.1.1-10.fc34 fedora 32 k python3-ntlm-auth noarch 1.5.0-2.fc34 fedora 53 k python3-ply noarch 3.11-11.fc34 fedora 103 k python3-pycparser noarch 2.20-3.fc34 fedora 126 k python3-pynacl x86_64 1.4.0-2.fc34 fedora 110 k python3-pysocks noarch 1.7.1-8.fc34 fedora 35 k python3-pytz noarch 2021.1-2.fc34 fedora 49 k python3-pyyaml x86_64 5.4.1-2.fc34 fedora 194 k python3-requests noarch 2.25.1-1.fc34 fedora 114 k python3-requests_ntlm noarch 1.1.0-14.fc34 fedora 18 k python3-urllib3 noarch 1.25.10-5.fc34 updates 174 k python3-xmltodict noarch 0.12.0-11.fc34 fedora 23 k sshpass x86_64 1.09-1.fc34 fedora 27 k Installing weak dependencies: python3-paramiko noarch 2.7.2-4.fc34 fedora 287 k python3-pyasn1 noarch 0.4.8-4.fc34 fedora 133 k python3-winrm noarch 0.4.1-2.fc34 fedora 79 k Transaction Summary ============================================================================================================================================================================= Install 28 Packages Total download size: 25 M Installed size: 144 M [...]
Exit the root shell.
# exit logout
Install the Ansible role for managing Podman.
$ ansible-galaxy collection install containers.podman Process install dependency map Starting collection install process Installing 'containers.podman:1.6.1' to '/home/fedora/.ansible/collections/ansible_collections/containers/podman'
Install the Ansible role for managing PostgreSQL.
$ ansible-galaxy collection install community.postgresql Process install dependency map Starting collection install process Installing 'community.postgresql:1.4.0' to '/home/fedora/.ansible/collections/ansible_collections/community/postgresql'
Verify the Podman installation by running a small test container.
$ podman run hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob b8dfde127a29 done
Copying config d1165f2212 done
Writing manifest to image destination
Storing signatures
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
The following files will be required to install py-phone-caller via Ansible.
- caller_config.toml.jinja2
- py-phone-caller-podman.yml
- py_phone_caller_vars_file.yml
Fisrt, create a directory to store the installation playbook and cd into it.
$ mkdir ansible_py-phone-caller $ cd ansible_py-phone-caller
Now download the files.
$ wget https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/caller_config.toml.jinja2 --2021-07-28 23:48:11-- https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/caller_config.toml.jinja2 Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 3730 (3.6K) [text/plain] Saving to: ‘caller_config.toml.jinja2’ caller_config.toml.jinja2 100%[========================================================================================>] 3.64K --.-KB/s in 0s 2021-07-28 23:48:11 (20.4 MB/s) - ‘caller_config.toml.jinja2’ saved [3730/3730] $ wget https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py-phone-caller-podman.yml --2021-07-28 23:49:55-- https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py-phone-caller-podman.yml Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10793 (11K) [text/plain] Saving to: ‘py-phone-caller-podman.yml’ py-phone-caller-podman.yml 100%[========================================================================================>] 10.54K --.-KB/s in 0s 2021-07-28 23:49:55 (25.7 MB/s) - ‘py-phone-caller-podman.yml’ saved [10793/10793] $ wget https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py_phone_caller_vars_file.yml --2021-07-28 23:51:20-- https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py_phone_caller_vars_file.yml Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7149 (7.0K) [text/plain] Saving to: ‘py_phone_caller_vars_file.yml’ py_phone_caller_vars_file.yml 100%[========================================================================================>] 6.98K --.-KB/s in 0s 2021-07-28 23:51:20 (14.6 MB/s) - ‘py_phone_caller_vars_file.yml’ saved [7149/7149]
The variables file py_phone_caller_vars_file.yml:
---
# Variables files for the 'py-phone-caller' installed through Podman
ansible_python_interpreter: /usr/bin/python3
# Podman / 'py-phone-caller' vars
container_host: 192.168.122.104
installation_user: fedora
installation_folder: "/home/{{ installation_user }}"
installation_folder_name: py-phone-caller
caller_config_toml_template_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/9c0bb97110cef988adfd23a6c0cb8e25168bbdfc/assets/ansible/rh/caller_config.toml.jinja2"
py_phone_caller_config_tmp_path: /tmp/caller_config.toml.jinja
py_phone_caller_config_path: "{{ installation_folder }}/{{ installation_folder_name }}/config/caller_config.toml"
config_mounted_in_container: /opt/py-phone-caller/config/caller_config.toml:Z
py_phone_caller_version: 0.0.2
container_registry_url: quay.io/py-phone-caller
py_phone_caller_network: py-phone-caller
py_phone_caller_subnet: 172.19.0.0/24
asterisk_ws_monitor_ip: 172.19.0.10
asterisk_recall_ip: 172.19.0.11
postgres_ip: 172.19.0.50
generate_audio_ip: 172.19.0.82
call_register_ip: 172.19.0.83
asterisk_call_ip: 172.19.0.81
caller_prometheus_webhook_ip: 172.19.0.84
caller_sms_ip: 172.19.0.85
# PostgreSQL container vars
db_schema_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/DB/db-schema.sql"
postgresql_container_image: "docker.io/library/postgres:13.3-alpine3.14"
postgresql_login_host: 127.0.0.1
postgresql_admin: postgres
postgresql_admin_pass: Use-A-Secure-Password-Here
# SystemD user integration
container_asterisk_call_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call.service"
container_asterisk_call_register_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call_register.service"
container_asterisk_recall_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_recall.service"
container_asterisk_ws_monitor_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_ws_monitor.service"
container_caller_prometheus_webhook_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_prometheus_webhook.service"
container_caller_sms_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_sms.service"
container_generate_audio_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-generate_audio.service"
container_postgres_13_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-postgres_13.service"
systemd_user_path: "/home/{{ installation_user }}/.config/systemd/user"
container_asterisk_call_service_unit: "container-asterisk_call.service"
container_asterisk_call_register_service_unit: "container-asterisk_call_register.service"
container_asterisk_recall_service_unit: "container-asterisk_recall.service"
container_asterisk_ws_monitor_service_unit: "container-asterisk_ws_monitor.service"
container_caller_prometheus_webhook_service_unit: "container-caller_prometheus_webhook.service"
container_caller_sms_service_unit: "container-caller_sms.service"
container_generate_audio_service_unit: "container-generate_audio.service"
container_postgres_13_service_unit: "container-postgres_13.service"
# py-phone-caller - 'caller_config.toml' vars
# [commons]
asterisk_user: "py-phone-caller"
asterisk_pass: "Use-A-Secure-Password-Here"
asterisk_host: "192.168.122.234"
asterisk_web_port: "8088"
asterisk_http_scheme: "http"
# [asterisk_call]
asterisk_ari_channels: "ari/channels"
asterisk_ari_play: "play?media=sound"
asterisk_context: "py-phone-caller"
asterisk_extension: "3216"
asterisk_chan_type: "SIP/sip-provider"
asterisk_callerid: "Py-Phone-Caller"
asterisk_call_http_scheme: "http"
asterisk_call_host: "{{ container_host }}"
asterisk_call_port: "8081"
asterisk_call_app_route_asterisk_init: "asterisk_init"
asterisk_call_app_route_play: "play"
seconds_to_forget: 300
client_timeout_total: 5 # For 'ClientTimeout(total=5)'
# [call_register]
call_register_http_scheme: "http"
call_register_host: "{{ container_host }}"
call_register_port: "8083"
call_register_app_route_register_call: "register_call"
call_register_app_route_voice_message: "msg"
call_register_app_route_acknowledge: "ack"
call_register_app_route_heard: "heard"
# [asterisk_ws_monitor]
asterisk_stasis_app: "py-phone-caller"
# [asterisk_recall]
times_to_dial: 3
# [generate_audio]
generate_audio_http_scheme: "http"
generate_audio_host: "{{ container_host }}"
generate_audio_port: "8082"
generate_audio_app_route: "make_audio"
gcloud_tts_language_code: "es"
serving_audio_folder: "audio"
num_of_cpus: 4
# [caller_prometheus_webhook]
prometheus_webhook_port: "8084"
prometheus_webhook_app_route_call_only: "call_only"
prometheus_webhook_app_route_sms_only: "sms_only"
prometheus_webhook_app_route_sms_before_call: "sms_before_call"
prometheus_webhook_app_route_call_and_sms: "call_and_sms"
prometheus_webhook_receivers: '[ "+123456789" ]'
# [caller_sms]
caller_sms_http_scheme: "http"
caller_sms_host: "{{ container_host }}"
caller_sms_port: "8085"
caller_sms_app_route: "send_sms"
sms_before_call_wait_seconds: 120
caller_sms_carrier: "twilio"
twilio_account_sid: "Your-Twilio-account-sid"
twilio_auth_token: "Your-Twilio-auth-token"
twilio_sms_from: "+1987654321"
# [database]
db_host: "{{ postgres_ip }}"
db_name: "py_phone_caller"
db_user: "py_phone_caller"
db_password: 'Use-A-Secure-Password-Here'
db_max_size: 50
db_max_inactive_connection_lifetime: 30.0
# [logger]
log_formatter: "%(asctime)s %(message)s"
acknowledge_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/ack?asterisk_chan=[The Asterisk Channel ID]"
heard_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/heard?asterisk_chan=[The Asterisk Channel ID]"
registercall_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/?phone=[Destination Phone Number]&messagge=[Alert Message Text]&asterisk_chan=[The Asterisk Channel ID]"
voice_message_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/msg?asterisk_chan=[The Asterisk Channel ID]"
asterisk_call_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/asterisk?phone=[Destination Phone Number]&messagge=[Alert Message Text]"
asterisk_play_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/play?asterisk_chan=[The Asterisk Channel ID]&msg_chk_sum=[The message cecksum]"
generate_audio_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/make_audio?messagge=[Alert Message Text]&msg_chk_sum=[The message cecksum]"
caller_sms_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/?phone=[Destination Phone Number]&messagge=[Alert Message Text]"
lost_directory_error: "The folder to serve the audio files was not found."
Install and configuring py-phone-caller using Ansible.
Ansible is a really cool configuration management and orchestration tool. Using Ansible, all the py-phone-caller components can be installed and configured automatically. After all the parameters are set in the assets/ansible/rh/py_phone_caller_vars_file.yml file, run ansible-playbook and in a few minutes you will have a fully configured and ready to be used py-phone-caller.
Variables that you might want to change in the assets/ansible/rh/py_phone_caller_vars_file.yml file:
[...] # Podman / 'py-phone-caller' vars [...] container_host: 192.168.122.104 # Here we need to configure the IP address of our Fedora Server instance installation_user: fedora # The user within the installation will be done. [...] # PostgreSQL container vars [...] postgresql_admin_pass: Use-A-Secure-Password-Here # The administrative password for PostgreSQL (user: postgres) [...] # py-phone-caller - 'caller_config.toml' vars # [commons] asterisk_pass: "Use-A-Secure-Password-Here" # The password for the 'py-phone-caller' Asterisk ARI user asterisk_host: "192.168.122.234" # The IP address of the Asterisk (in our case FreePBX) instance. [...] # [caller_sms] # Only if we want send SMS through Twilio [...] twilio_account_sid: "Your-Twilio-account-sid" # The Twilio SID twilio_auth_token: "Your-Twilio-auth-token" # The Twilio auth token twilio_sms_from: "+1987654321" # The Twilio number [...] # [database] [...] db_password: 'Use-A-Secure-Password-Here' # The password for the PostgreSQL 'py-phone-caller' user. [...]
To install py-phone-caller, run ansible-playbook.
$ ansible-playbook --connection=local --limit=127.0.0.1 --inventory=127.0.0.1, ansible_py-phone-caller/py-phone-caller-podman.yml
PLAY [Configuring 'py-phone-caller' installed through 'Podman'] *************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [127.0.0.1]
TASK [Creating the 'py-phone-caller' directory tree at '/home/fedora/py-phone-caller'] **************************************************************************************
changed: [127.0.0.1]
TASK [Check that the 'caller_config.toml' exists] ***************************************************************************************************************************
ok: [127.0.0.1]
TASK [Download the 'caller_config.toml' template file] **********************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the configuration file 'caller_config.toml' through a template] **********************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'py-phone-caller' container network] *********************************************************************************************************************
changed: [127.0.0.1]
TASK [Checking if the 'pgdata' volume exists] *******************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["podman", "volume", "inspect", "pgdata"], "delta": "0:00:00.059469", "end": "2021-07-28 22:04:44.174686", "msg": "non-zero return code", "rc": 125, "start": "2021-07-28 22:04:44.115217", "stderr": "Error: error inspecting object: no such volume pgdata", "stderr_lines": ["Error: error inspecting object: no such volume pgdata"], "stdout": "[]", "stdout_lines": ["[]"]}
TASK [Creating the 'pgdata' volume for the PostgreSQL container] ************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'postgres_13' container] *********************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'asterisk_call' container] *******************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'caller_prometheus_webhook' container] *******************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'caller_sms' container] **********************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'generate_audio' container] ******************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'py_phone_caller' PostgreSQL user] ***********************************************************************************************************************
changed: [127.0.0.1]
TASK [Gattering info about the 'py_phone_caller' DB (if exists)] ************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "unable to connect to database: FATAL: database \"py_phone_caller\" does not exist\n"}
TASK [Download the dumped 'py_phone_caller' DB schema (SQL format)] *********************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'py_phone_caller' DB] ************************************************************************************************************************************
changed: [127.0.0.1]
TASK [Restoring the 'py_phone_caller' DB by restoring a dump of the schema] *************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'asterisk_ws_monitor' container] *************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'asterisk_recall' container] *****************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the 'call_register' container] *******************************************************************************************************************************
changed: [127.0.0.1]
TASK [Creating the Systemd user folder] *************************************************************************************************************************************
changed: [127.0.0.1]
TASK [Download the Systemd Unit files] **************************************************************************************************************************************
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call.service', 'path': 'container-asterisk_call.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call_register.service', 'path': 'container-asterisk_call_register.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_recall.service', 'path': 'container-asterisk_recall.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_ws_monitor.service', 'path': 'container-asterisk_ws_monitor.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_prometheus_webhook.service', 'path': 'container-caller_prometheus_webhook.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_sms.service', 'path': 'container-caller_sms.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-generate_audio.service', 'path': 'container-generate_audio.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-postgres_13.service', 'path': 'container-postgres_13.service'})
TASK [Enable and run the user SystemD services for 'py-phone-caller'] *******************************************************************************************************
changed: [127.0.0.1] => (item=container-asterisk_call.service)
changed: [127.0.0.1] => (item=container-asterisk_call_register.service)
changed: [127.0.0.1] => (item=container-asterisk_recall.service)
changed: [127.0.0.1] => (item=container-asterisk_ws_monitor.service)
changed: [127.0.0.1] => (item=container-caller_prometheus_webhook.service)
changed: [127.0.0.1] => (item=container-caller_sms.service)
changed: [127.0.0.1] => (item=container-generate_audio.service)
changed: [127.0.0.1] => (item=container-postgres_13.service)
TASK [Deleting the '/tmp' files] ********************************************************************************************************************************************
changed: [127.0.0.1] => (item=/tmp/db-schema.sql)
changed: [127.0.0.1] => (item=/tmp/caller_config.toml.jinja)
PLAY RECAP ******************************************************************************************************************************************************************
127.0.0.1 : ok=23 changed=21 unreachable=0 failed=0 skipped=0 rescued=2 ignored=0
Run the following Firewalld commands to allow the necessary connections.
$ sudo firewall-cmd --add-source="192.168.122.0/24" --permanent success $ sudo firewall-cmd --add-source="172.19.0.0/24" --permanent success $ sudo firewall-cmd --add-port=8081/tcp --permanent success $ sudo firewall-cmd --add-port=8082/tcp --permanent success $ sudo firewall-cmd --add-port=8083/tcp --permanent success $ sudo firewall-cmd --add-port=8084/tcp --permanent success $ sudo firewall-cmd --reload success
Install and Configure the Prometheus Monitoring Stack
The final pieces to be installed are the Node Expoter, the Alertmanager and Prometheus. With these, you will be able to monitor, alert and collect metrics when some metric violates a condition defined in the alerting rules.
Packages to be installed:
- golang-github-prometheus
- golang-github-prometheus-alertmanager
- golang-github-prometheus-node-exporter
Enter the following command to install the packages.
$ sudo dnf -y install golang-github-prometheus golang-github-prometheus-alertmanager golang-github-prometheus-node-exporter Last metadata expiration check: 0:17:09 ago on Thu 12 Aug 2021 21:23:21 PM CEST. Dependencies resolved. ============================================================================================================================================================================= Package Architecture Version Repository Size ============================================================================================================================================================================= Installing: golang-github-prometheus x86_64 2.24.1-6.fc34 updates 31 M golang-github-prometheus-alertmanager x86_64 0.21.0-3.fc34 fedora 13 M golang-github-prometheus-node-exporter x86_64 1.1.1-2.fc34 fedora 4.5 M Transaction Summary ============================================================================================================================================================================= Install 3 Packages Total download size: 48 M Installed size: 212 M Downloading Packages: [...]
A quick illustration of how the components interact
Node Exporter <-- Prometheus --> Alertmanager --> caller_prometheus_webhook
The Node Exporter exposes the metrics of the Fedora Server system. Prometheus does a periodic pull of these metrics. If some alerting rules are defined and one or more metrics violates a condition then Prometheus will send a request to the Alertmanager. Alertmanager will then call caller_prometheus_webhook.
The Prometheus Alertmanager
The role of this component is to trigger the different notification systems (e-mail, PagerDuty, Slack, VictorOps, etc.). It can also make a POST request to a webhook receiver such as caller_prometheus_webhook to place a call or send an SMS message (or whatever else you have configured it to do).
One useful feature of this component is the ability to silence the alerts from the web interface. When someone takes an action to resolve the problem, they can silence it. Otherwise the default configuration would repeat the alert in three hours (repeat_interval: 3h).
A list of the package contents:
# dnf repoquery -l golang-github-prometheus-alertmanager Last metadata expiration check: 0:50:29 ago on Thu 12 Aug 2021 04:23:21 PM CEST. /usr/bin/alertmanager /usr/bin/amtool /usr/lib/.build-id /usr/lib/.build-id/0b /usr/lib/.build-id/0b/275747139b1a6f1b398166b54ee83f625e1d4e /usr/lib/.build-id/4d /usr/lib/.build-id/4d/be93dda0226df14625b47ea1a6747eacd1e0d1 /usr/share/doc/golang-github-prometheus-alertmanager /usr/share/doc/golang-github-prometheus-alertmanager/CHANGELOG.md /usr/share/doc/golang-github-prometheus-alertmanager/MAINTAINERS.md /usr/share/doc/golang-github-prometheus-alertmanager/README.md /usr/share/doc/golang-github-prometheus-alertmanager/doc /usr/share/doc/golang-github-prometheus-alertmanager/doc/arch.svg /usr/share/doc/golang-github-prometheus-alertmanager/doc/arch.xml /usr/share/doc/golang-github-prometheus-alertmanager/doc/design /usr/share/doc/golang-github-prometheus-alertmanager/doc/design/secure-cluster-traffic.md /usr/share/doc/golang-github-prometheus-alertmanager/doc/examples /usr/share/doc/golang-github-prometheus-alertmanager/doc/examples/simple.yml /usr/share/doc/golang-github-prometheus-alertmanager/examples /usr/share/doc/golang-github-prometheus-alertmanager/examples/ha /usr/share/doc/golang-github-prometheus-alertmanager/examples/ha/alertmanager.yml /usr/share/doc/golang-github-prometheus-alertmanager/examples/ha/send_alerts.sh /usr/share/doc/golang-github-prometheus-alertmanager/examples/webhook /usr/share/doc/golang-github-prometheus-alertmanager/examples/webhook/echo.go /usr/share/licenses/golang-github-prometheus-alertmanager /usr/share/licenses/golang-github-prometheus-alertmanager/COPYRIGHT.txt /usr/share/licenses/golang-github-prometheus-alertmanager/LICENSE /usr/share/licenses/golang-github-prometheus-alertmanager/NOTICE
From the previous code snippet you can see that there is no systemd service file, directory to store the data, or configuration directory. You will need to create them manually.
# mkdir -p /etc/alertmanager/template /var/lib/prometheus/alertmanager/data # chown -R prometheus. /var/lib/prometheus/alertmanager # cp /usr/share/doc/golang-github-prometheus-alertmanager/doc/examples/simple.yml /etc/alertmanager/alertmanager.yml
Next, add the following configuration blocks to /etc/alertmanager/alertmanager.yml.
In the routes section:
[...]
# py-phone-caller example
- match:
severity: disaster
receiver: py-phone-caller
[...]
In the receivers section:
[...]
- name: 'py-phone-caller'
webhook_configs:
# Onr of the endpoints of 'caller_prometheus_webhook'
- url: 'http://127.0.0.1:8084/sms_before_call'
send_resolved: false
[...]
The complete /etc/alertmanager/alertmanager.yml file:
global:
# The smarthost and SMTP sender used for mail notifications.
smtp_smarthost: 'localhost:25'
smtp_from: 'alertmanager@example.org'
smtp_auth_username: 'alertmanager'
smtp_auth_password: 'password'
# The directory from which notification templates are read.
templates:
- '/etc/alertmanager/template/*.tmpl'
# The root route on which each incoming alert enters.
route:
# The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in for cluster=A and alertname=LatencyHigh would
# be batched into a single group.
#
# To aggregate by all possible labels use '...' as the sole label name.
# This effectively disables aggregation entirely, passing through all
# alerts as-is. This is unlikely to be what you want, unless you have
# a very low alert volume or your upstream notification system performs
# its own grouping. Example: group_by: [...]
group_by: ['alertname', 'cluster', 'service']
# When a new group of alerts is created by an incoming alert, wait at
# least 'group_wait' to send the initial notification.
# This way ensures that you get multiple alerts for the same group that start
# firing shortly after another are batched together on the first
# notification.
group_wait: 30s
# When the first notification was sent, wait 'group_interval' to send a batch
# of new alerts that started firing for that group.
group_interval: 5m
# If an alert has successfully been sent, wait 'repeat_interval' to
# resend them.
repeat_interval: 3h
# A default receiver
receiver: team-X-mails
# All the above attributes are inherited by all child routes and can
# overwritten on each.
# The child route trees.
routes:
# This routes performs a regular expression match on alert labels to
# catch alerts that are related to a list of services.
- match_re:
service: ^(foo1|foo2|baz)$
receiver: team-X-mails
# The service has a sub-route for critical alerts, any alerts
# that do not match, i.e. severity != critical, fall-back to the
# parent node and are sent to 'team-X-mails'
routes:
- match:
severity: critical
receiver: team-X-pager
- match:
service: files
receiver: team-Y-mails
# py-phone-caller example
- match:
severity: disaster
receiver: py-phone-caller
routes:
- match:
severity: critical
receiver: team-Y-pager
# This route handles all alerts coming from a database service. If there's
# no team to handle it, it defaults to the DB team.
- match:
service: database
receiver: team-DB-pager
# Also group alerts by affected database.
group_by: [alertname, cluster, database]
routes:
- match:
owner: team-X
receiver: team-X-pager
continue: true
- match:
owner: team-Y
receiver: team-Y-pager
# Inhibition rules allow to mute a set of alerts given that another alert is
# firing.
# We use this to mute any warning-level notifications if the same alert is
# already critical.
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
# Apply inhibition if the alertname is the same.
# CAUTION:
# If all label names listed in `equal` are missing
# from both the source and target alerts,
# the inhibition rule will apply!
equal: ['alertname', 'cluster', 'service']
receivers:
- name: 'team-X-mails'
email_configs:
- to: 'team-X+alerts@example.org'
- name: 'team-X-pager'
email_configs:
- to: 'team-X+alerts-critical@example.org'
pagerduty_configs:
- service_key: <team-X-key>
- name: 'team-Y-mails'
email_configs:
- to: 'team-Y+alerts@example.org'
- name: 'team-Y-pager'
pagerduty_configs:
- service_key: <team-Y-key>
- name: 'team-DB-pager'
pagerduty_configs:
- service_key: <team-DB-key>
- name: 'py-phone-caller'
webhook_configs:
# Onr of the endpoints of 'caller_prometheus_webhook'
- url: 'http://127.0.0.1:8084/sms_before_call'
send_resolved: false
Note: This is an example file and several blocks can be removed because they aren’t used. For your use case it will work fine. The unused parts weren’t removed in order to show a complete landscape of this file.
Run systemctl cat prometheus.service to view the prometheus.service systemd unit file.
# systemctl cat prometheus.service
# /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus service monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Wants=network-online.target
After=network-online.target
[Service]
Restart=on-failure
EnvironmentFile=/etc/sysconfig/prometheus
User=prometheus
Group=prometheus
ExecStart=/usr/bin/prometheus \
--config.file=${CONFIG_FILE} \
--storage.tsdb.path=${STORAGE_TSDB_PATH} \
--web.console.libraries=${WEB_CONSOLE_LIBRARIES_PATH} \
--web.console.templates=${WEB_CONSOLE_TEMPLATES_PATH} \
--web.listen-address=${WEB_LISTEN_ADDRESS}
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
The /etc/systemd/system/alertmanager.service file:
[Unit] Description=Alertmanager for the Prometheus monitoring system Documentation=https://prometheus.io/docs/alerting/latest/alertmanager/ Wants=network-online.target After=network-online.target [Service] Restart=on-failure User=prometheus Group=prometheus ExecStart=/usr/bin/alertmanager --config.file="/etc/alertmanager/alertmanager.yml" --storage.path="/var/lib/prometheus/alertmanager/data" ExecReload=/bin/kill -HUP $MAINPID TimeoutStopSec=20s SendSIGKILL=no [Install] WantedBy=multi-user.target
Enable alertmanager.service.
# systemctl enable --now alertmanager.service Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager.service → /etc/systemd/system/alertmanager.service.
Use the following command to verify that the service is running.
# systemctl status alertmanager.service
● alertmanager.service - Alertmanager for the Prometheus monitoring system
Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2021-08-12 21:44:32 CEST; 1s ago
Docs: https://prometheus.io/docs/alerting/latest/alertmanager/
Main PID: 49281 (alertmanager)
Tasks: 11 (limit: 4647)
Memory: 16.2M
CPU: 86ms
CGroup: /system.slice/alertmanager.service
└─49281 /usr/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/prometheus/alertmanager/data
Aug 12 21:44:32 fedora systemd[1]: Started Alertmanager for the Prometheus monitoring system.
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.390Z caller=main.go:216 msg="Starting Alertmanager" version="(version=, branch=, revision=)"
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.390Z caller=main.go:217 build_context="(go=go1.16rc1, user=, date=)"
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.391Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=1>
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.392Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.425Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/>
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.426Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration>
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.430Z caller=main.go:485 msg=Listening address=:9093
The Node Exporter
This component exposes various metrics on the host where it is running. For more information about the metrics exposed by this component, see the README.md file in the Github repository.
Enable and start the node_exporter service.
# systemctl enable --now node_exporter.service Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /etc/systemd/system/node_exporter.service.
Check the status of the node_exporter service.
# systemctl status node_exporter.service
● node_exporter.service - Node Exporter
Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2021-08-12 21:55:48 CEST; 7s ago
Main PID: 49401 (node_exporter)
Tasks: 6 (limit: 4647)
Memory: 5.3M
CPU: 15ms
CGroup: /system.slice/node_exporter.service
└─49401 /usr/sbin/node_exporter --collector.textfile.directory /var/lib/node_exporter/textfile_collector
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=thermal_zone
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=time
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=timex
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=udp_queues
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=uname
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=vmstat
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=xfs
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=zfs
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:195 msg="Listening on" address=:9100
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.067Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
Prometheus
You will need to make some changes before starting the Prometheus service. You need to change the default listening port from 9090 to another value. Port 9090 is reserved by Cockpit on Fedora Server edition.
# ss -wenotulipas | grep :9090
tcp LISTEN 0 4096 *:9090 *:* users:(("systemd",pid=1,fd=139)) ino:18756 sk:e cgroup:/system.slice/cockpit.socket v6only:0 <-
The /etc/sysconfig/prometheus file:
CONFIG_FILE=/etc/prometheus/prometheus.yml STORAGE_TSDB_PATH=/var/lib/prometheus WEB_CONSOLE_LIBRARIES_PATH=/etc/prometheus/console_libraries WEB_CONSOLE_TEMPLATES_PATH=/etc/prometheus/consoles WEB_LISTEN_ADDRESS=127.0.0.1:9091
In the /etc/prometheus/prometheus.yml file, modify are the following blocks.
The alertmanagers block:
[...]
alerting:
alertmanagers:
- static_configs:
- targets:
- 127.0.0.1:9093
[...]
The scrape_configs block:
[...]
# The local node exporter
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
[...]
The rule_files block:
[...] # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: - "alert_rules.yml" [...]
The /etc/prometheus/prometheus.yml file:
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- 127.0.0.1:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "alert_rules.yml"
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9091']
# The local node exporter
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
The /etc/prometheus/alert_rules.yml file:
groups:
- name: py-phone-caller test alerts
rules:
- alert: NodeExporterDown
expr: up{job="node-exporter"} == 0
for: 5m
labels:
severity: disaster
annotations:
summary: "The Node Exporter instance is down"
description: "The Node Exporter instance is down, please check soon as possible"
Enable and start the prometheus service
# systemctl enable --now prometheus.service Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /usr/lib/systemd/system/prometheus.service.
Check the status of the prometheus service.
# systemctl status prometheus.service
● prometheus.service - Prometheus service monitoring system and time series database
Loaded: loaded (/usr/lib/systemd/system/prometheus.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2021-08-12 22:03:57 CEST; 6s ago
Docs: https://prometheus.io/docs/introduction/overview/
man:prometheus(1)
Main PID: 49560 (prometheus)
Tasks: 12 (limit: 4647)
Memory: 31.9M
CPU: 102ms
CGroup: /system.slice/prometheus.service
└─49560 /usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.libraries=/etc/prometheus/consol>
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.753Z caller=head.go:645 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.753Z caller=head.go:659 component=tsdb msg="On-disk memory mappable chunks replay completed" dur>
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.754Z caller=head.go:665 component=tsdb msg="Replaying WAL, this may take a while"
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.754Z caller=head.go:717 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.754Z caller=head.go:722 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.>
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.755Z caller=main.go:758 fs_type=XFS_SUPER_MAGIC
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.755Z caller=main.go:761 msg="TSDB started"
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.756Z caller=main.go:887 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Aug 12 22:03:58 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:58.005Z caller=main.go:918 msg="Completed loading of configuration file" filename=/etc/prometheus/p>
Aug 12 22:03:58 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:58.006Z caller=main.go:710 msg="Server is ready to receive web requests."
Add the following firewall rules for Prometeus and the Alertmanager.
$ sudo firewall-cmd --add-port=9091/tcp --permanent success $ sudo firewall-cmd --add-port=9093/tcp --permanent success $ sudo firewall-cmd --reload success
Behind the scenes
As always, the logs are useful in order to understand what’s happening with your programs.
- Logs of caller_prometheus_webhook
When an alerts is sent from Prometheus to the Alermanager, then the Alertmanager sends an HTTP POST request to the caller_prometheus_webhook. In this example we’ve used the /sms_before_call endpoint and as consequence an SMS will be sent soon as possible and later, the phone call.
2021-08-12 18:45:01,803 Call/Message 'The Node Exporter instance is down, please check soon as possible.' for '+393312345678' through the endpoint 'sms_before_call' 2021-08-12 18:45:01,807 - 172.19.0.84 [12/Aug/2021:16:45:01 +0000] "POST /sms_before_call HTTP/1.1" 200 180 "-" "Alertmanager/"
- Logs of caller_sms
When the caller_prometheus_webhook receives the POST request from the Alertmanager it does other POST requesto to the caller_sms to send the message to the contact configured in prometheus_webhook_receivers. Through the SMS service provider.
2021-08-12 18:45:01,812 Sending the SMS message 'The Node Exporter instance is down, please check soon as possible.' to '+393312345678'
2021-08-12 18:45:01,862 - POST Request: https://api.twilio.com/2010-04-01/Accounts/C3RTyd674gtalirS567q25687g57980gtr53/Messages.json
2021-08-12 18:45:01,862 - PAYLOAD: {'To': '+393312345678', 'From': '+15596123456', 'Body': 'The Node Exporter instance is down, please check soon as possible.'}
2021-08-12 18:45:02,543 - POST Response: 201 {"sid": "ABcDEFGhIJKLMNoPqRsTU1234565671z", "date_created": "Thu, 12 Aug 2021 18:45:02 +0000", "date_updated": "Thu, 12 Aug 2021 18:45:02 +0000", "date_sent": null,
"account_sid": "C3RTyd674gtalirS567q25687g57980gtr53", "to": "+393312345678", "from": "+15596123456", "messaging_service_sid": null, "body": "The Node Exporter instance is down, please check soon as possible.",
"status": "queued", "num_segments": "1", "num_media": "0", "direction": "outbound-api", "api_version": "2010-04-01", "price": null, "price_unit": "USD", "error_code": null, "error_message": null,
"uri": "/2010-04-01/Accounts/C3RTyd674gtalirS567q25687g57980gtr53/Messages/ABcDEFGhIJKLMNoPqRsTU1234565671z.json",
"subresource_uris": {"media": "/2010-04-01/Accounts/C3RTyd674gtalirS567q25687g57980gtr53/Messages/ABcDEFGhIJKLMNoPqRsTU1234565671z/Media.json"}}
2021-08-12 18:45:02,544 - 172.19.0.85 [12/Aug/2021:18:45:01 +0000] "POST /None?phone=%2B393312345678&message=The+Node+Exporter+instance+is+down,+please+check+soon+as+possible. HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0"
- Logs of the asterisk_asterisk_call
Some seconds or minutes after the SMS (configured in sms_before_call_wait_seconds) the asterisk_asterisk_call starts the calls round against the receiver number.
- Parameters from the config/caller_config.toml file
seconds_to_forget = 300 times_to_dial = 3
seconds_to_forget is the time window where asterisk_recall will try to recall the receiver. times_to_dial is the number of times to retry to call.
2021-08-12 18:47:02,655 - 172.19.0.81 [12/Aug/2021:18:47:02 +0000] "POST /asterisk_init?phone=00393312345678&message=The+Node+Exporter+instance+is+down,+please+check+soon+as+possible. HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0" 2021-08-12 18:47:22,690 Asterisk server 'http://192.168.122.234:8088' response: 201. Playing audio 'e23c1ebc.wav' to the channel '1628886822.1' 2021-08-12 18:47:22,694 Restoring the call control to the PBX on the channel '1628886822.1' 2021-08-12 18:47:22,694 - 172.19.0.81 [12/Aug/2021:18:47:22 +0000] "POST /play?asterisk_chan=1628886822.1&msg_chk_sum=e23c1ebc HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0" 2021-08-12 18:48:17,711 - 172.19.0.81 [12/Aug/2021:18:48:17 +0000] "POST /asterisk_init?phone=00393312345678&message=The+Node+Exporter+instance+is+down,+please+check+soon+as+possible. HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0" 2021-08-12 18:48:32,391 Asterisk server 'http://192.168.122.234:8088' response: 201. Playing audio 'e23c1ebc.wav' to the channel '1628886897.2' 2021-08-12 18:48:32,393 - 172.19.0.81 [12/Aug/2021:18:48:32 +0000] "POST /play?asterisk_chan=1628886897.2&msg_chk_sum=e23c1ebc HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0" 2021-08-12 18:48:32,393 Restoring the call control to the PBX on the channel '1628886897.2'
- Logs of the asterisk_recall
When the number 4 is not pressed by the receiver/callee, this component will recall again (times_to_dial).
2021-08-12 18:10:56,388 - Using the default path 'config/caller_config.toml' 2021-08-12 18:48:17,679 Retry to call phone number: '00393312345678' to play the message: 'The Node Exporter instance is down, please check soon as possible.' - Total retry period: '300' seconds
- Logs of the asterisk_ws_monitor
This component manages and records into the asterisk_ws_events table of the database the events of the Asterisk PBX when the py-phone-caller components are managing the call.
2021-08-12 18:47:22,695 Response for the playing audio 'e23c1ebc.wav' on the Asterisk channel '1628886822.1': '{"status": 201}'
2021-08-12 18:48:32,394 Response for the playing audio 'e23c1ebc.wav' on the Asterisk channel '1628886897.2': '{"status": 201}'
- Last but not least
The call_register component writes all the managed calls into the calls table. An example of table structure from the assets/DB/db-schema.sql file:
CREATE TABLE calls (
id integer NOT NULL,
phone character varying(64),
message character varying(1024),
asterisk_chan character varying(64),
msg_chk_sum character varying(64),
call_chk_sum character varying(64),
unique_chk_sum character varying(64),
times_to_dial smallint,
dialed_times smallint,
seconds_to_forget integer,
first_dial timestamp without time zone,
last_dial timestamp without time zone,
heard_at timestamp without time zone,
acknowledge_at timestamp without time zone,
cycle_done boolean DEFAULT false
);
Manual testing
You can send messages and place calls from the terminal. This can be useful if you don’t intend to use the Prometheus monitoring system. You can trigger the messages or calls from cron or other scripts, for example.
An example of the payload sent from the Prometheus Alert Manager
The test-alert-from-the-alertmanager.json file:
{
"receiver":"webhook",
"status":"firing",
"alerts":[
{
"status":"firing",
"labels":{
"alertname":"TestAlertFromAlertmanager",
"instance":"localhost:9090",
"job":"prometheus"
},
"annotations":{
"description":"This text will be an audio message in order to verify if the setup is working",
"summary":"our summary"
},
"startsAt":"2021-03-02T21:52:26.558311875+01:00",
"endsAt":"0001-01-01T00:00:00Z",
"generatorURL":"http://prometheus:9090/graph"
}
],
"groupLabels":{
"alertname":"TestAlertFromAlertmanager",
"job":"prometheus"
},
"commonLabels":{
"alertname":"TestAlertFromAlertmanager",
"instance":"localhost:9090",
"job":"prometheus"
},
"commonAnnotations":{
"description":"our description",
"summary":"our summary"
},
"externalURL":"http://alertmanager:9093",
"version":"4",
"groupKey":"{}:{alertname=\"TestAlertFromAlertmanager\", job=\"prometheus\"}"
}
Make an HTTP POST request to simulate a Prometheus Alert.
$ curl --header "Content-Type: application/json" -X POST -d @ansible_py-phone-caller/alert.json http://127.0.0.1:8084/sms_before_call
{"status": "200"}
Start a Single Call from the shell
You can start a call from the shell using the curl command. Please pay attention to format of the number that’s using the 00 and the country code 39 because we’re using the Asterisk trunk without dial rules to modify the number. For example if you want to place a call in the UK, the first four numbers will be 0044.
Make a HTTP POST request to place a call from the terminal.
$ curl -X POST "http://127.0.0.1:8081/asterisk_init?phone=00392235896425&message=New%20message%20from%20the%20Fedora%20Server%20shell"
{"status": 200}
Send a Single SMS from the shell
Something similar to the previous curl command happens when you want to send a single SMS message. Twilio wants the + symbol instead of the 00 prefix. You will have to use %2B to encod the + symbol.
Make a HTTP POST request to send a SMS message from the terminal.
$ curl -X POST "http://127.0.0.1:8085/send_sms?phone=%2B392235896425&message=New%20message%20from%20the%20Fedora%20Server%20shell"
{"status": 200}
In future versions I hope to make this more user-friendly.
Finally, the classic ‘Wrapping Up’
By following the steps in this guide you can create a mini pager system. py-phone-caller isn’t intended to be used in mission critical environments because it is not yet ready for that kind of workload.
Thanks for reading. I hope this article was useful to you.
Jorge Carlos Franco
Also valid for Fedora Server 35 😉
RG
Though I don’t get why advertise another distribution under the hood here but there’s also comparable 3CX https://www.3cx.de/
Jorge Carlos Franco
Hi RG,
FreePBX was used because is easy to install and manage through the web interface, and it’s also Open Source. Also it will work with an Asterisk without the web interface, but the configuration is harder those that haven’t to many experience with this kind of packages.
The idea is build a solution using Open Source software. Logically you’re free to choose the product that you prefer.
The product that you suggest fits your needs, that’s great.
Best regards.
Peter Boy
Very informative article. I had not even thought of setting up such a system before. Do you have the system in practical use yourself? If so, I would be happy if you would present such a use case as part of the Fedora Server documentation.
Jorge Carlos Franco
@Peter Boy,
For the component that sends the SMS messages, we can consider it in a stage phase. Works well when the alerts are posted through the Prometheus Alert Manager.
The component that places the phone calls also works well, but only managing a single call (to the callee) at once. In order to manage more calls to many destination numbers we need to deploy a new instance of the ‘asterisk_call’ by every single callee.
For example it can be configured in this way when you need notify more than one person at the same time, and in order to make it work we need to place the ‘asterisk_call’ instances behind a load balancer as HAProxy in order to do the HTTP Post request in a round robin way (by this way you can achieve acceptable results).
It will not work very well for high alert rates (for example more than 5 post requests that comes with a few milliseconds of distance). Additional work is needed in order to manage this use case. For example send first the alerts to a queue in order to be consumed a few seconds after the arrival and so on.
Automating Calls
Nice article!