Romancero bot: converting article text into voice with Amazon Polly and sending it to Telegram

The following Python script extracts the content of an article using Readibility (ported to Python), converts it to voice using the Amazon Polly service and finally sends the audio as a voice note to a given user in Telegram using Telethon (Telegram client for Python).

For running the script you will need to install the following Python packages:

Also, you will need to create a AWS account. If you already have an AWS account, make sure that you have a user created in the IAM Management Console with the following permissions:

User permissions for creating Polly jobs and accessing/writing files in a S3 bucket.

When creating this user, make sure you write down its ID access key and its secret access key. You will need them to configure the aws-cli client.

With this Amazon credentials, you can configure the AWS client by executing the following command:

In this step, you will need to fulfill the details with the user credentials you wrote down when creating it.

Now, you will need to create a Telegram API ID. For this, you can go to the Telegram section «Create an Application«. After following the steps described in the official documentation, you will obtain an API ID (it’s a number) and a API hash (it’s a string).

With these steps already completed, you can place all the needed details in the script and run it.

The user you specified will receive a message like this:

Parser in PHP using regular expressions

You can use regular expressions in PHP with the function preg_match ( string $pattern , string $subject [, array &$matches [, int $flags = 0 [, int $offset = 0 ]]] ) . Only the first two paremeters are mandatory and they are the regex and the string where you want to search respectively.

In case of finding a result, preg_match() returns an array where the item at index 0 is the whole match. From 1 onwards they are placed the different groups of your regular expressions (in case there is any). If no match is found, preg_match() returns null.

One of the details that must be taken into account when using regular expressions on PHP is that they must be enclosed by forward slashes (/), e.g. $multiline_meaning_re = '/^([A-za-z ,"().\';:]+)/'; . This regular expression matches any string with any set of letters, spaces, commas, double and single quotes, parenthesis points, colon and/or semicolon.

As a complete example, the following snippet opens a file, parses it to look for English idioms and uploads all of them a MySQL database.

You can find further information about the preg_match() in the PHP official documentation.

Introduction to D3 (Data-Driven Document)

One of the first questions you may ask yourself when getting introduced in D3 is: why are we using selectAll(‘html-tag-name’) method if there is no item to select of that type?

First, let’s see an example of the situation we are talking about:

In the previous example, the only existing HTML tag is <body> . We select body ( d3.select('body') ) and then we perform the .selectAll('h2') . On its own, it makes no sense as the method won’t return any value since no <h2>  tag exists. Nevertheless, it will make sense if we keep looking at the following code.

After the select .selectAll('h2')  we attach the existing dataset to the selected items ( .data(dataset) ). Then, we use the enter()  method, which gives meaning to the previous .selectAll('h2') . When using enter() , D3 looks for the number of selected items to bind them with the data. In case of having not enough items in the selection, the enter()  method will create them.

Therefore, as .selectAll('h2')  was empty and the dataset  variable contains 9 elements, it will iterate the code 9 times. In case of having already created some  <h2>  elements, it will simply fulfill the HTML code the necessary iteration to cover all the dataset  elements. Remember that who does this iteration is  the data() method.  It parses the data set, and any method that’s chained after data() is run once for each item in the data set.

You can find more information in the official documentation at the D3js.org website.

Scales

In D3 there exists the Scale function to change the value of the data set so that it can fit in the screen. Two important methods are range() and  domain(). The domain method covers the set of input values whereas the range function convers the set of output values. Let’s see an example:

From freeCodeCamp:

The domain()  method passes information to the scale about the raw data values for the plot. The range()  method gives it information about the actual space on the web page for the visualization.

Send audio from mobile phone (Android/iOS) to Raspberry

Platform used: Raspberry Pi 3 B+
Bluetooth module: built-in
First, update firmware to make sure you have latest version:

Configure the bluetooth in the Raspberry:

In your mobile phone, search for the raspberry bluetooth signal and pair to it:

Now, you can try to play some audio in your mobile and it should be reproduced in the the Raspberry.
In my case, audio was driven out to the HDMI. In case you want to switch it through the Jack 3.5 mm output you can run:

Then, select Advanced Options > Audio > Force 3.5 mm (‘headphone’) jack
With this all set, you can stream any audio such as the built-in music player, Spotify or YouTube to the Raspberry and from it to the connected speakers or HDMI display.

Finally, as the device has been stored as a trusted device, every time the Raspberry is booted, you won’t need to repeat this process. It will be so easy as connecting your mobile phone to the available raspberry bluetooth signal.

Handling with several devices

If you pair several devices, only first connected device will be able to stream audio. If I connect my computer to the raspberry and then I try to connect my mobile phone (both previously paired and trusted), phone connection will fail. First, I’ll need to disconenct my computer and only then I’ll be able to successfully connect to the raspberry with my mobile phone.

Web scraper with Scrapy

Basis in a vector space

A vector space basis is the skeleton from which a vector space is built. It allows to decompose any signal into a linear combination of simple building blocks, namely, the basis vectors. The Fourier Transform is just a change of basis.

A vector basis is the linear combination of a set of vector that can write any vector of the space.

\[ w^{(k)} \leftarrow \text{basis} \]

The canonical basis in \(\mathbb{R}^2\) are:

\(e^{(0)} = [1, 0]^T \ \ e^{(1) } = [0,1]^T \)

Nevertheless, there are more basis for \(\mathbb{R}^2\):

\(e^{(0)} = [1, 0]^T \ \ e^{(1) } = [1,1]^T \)

This former basis is not linearly independent as information of \(e^{(0)}\) is inside \(e^{(1)}\).

Formal definition

H is a vector space.

W is a set of vectors from H such that \(W = \left\{ w^{(k)}  \right\} \)

W is a basis of H if:

  1. We can write \( \forall x \in H\): \( x = \sum_{k=0}^{K-1} \alpha_k w^{(k)}, \ \ \alpha_k \in \mathbb{C} \)
  2. \( \alpha_k  \) are unique, namely, there is linear independence in the basis, as a given point can only be expressed in a unique combination of the basis.

Orthogonal basis are those which inner product returns 0:

\( \left \langle w^{(k)}, w^{(n)} \right \rangle = 0, \ \ \text{for } k \neq n \)

In addition, if the self inner product of every basis element return 1, the basis are orthonormal.

How to change the basis?

An element in the vector space can be represented with a new basis computing the projection of the current basis in the new basis. If \(x\) is a vector element and is represented with the vector basis \(w^{(K)}\) with the coefficients \(a_k\), it can also be represented as a linear combination of the basis \(v^{(k)}\) with the coefficients \( \beta_k\). In a mathematical notation:

\[ x = \sum_{k=0}^{K-1} \alpha_k w^{(k)} = \sum_{k=0}^{K-1} \beta_k v^{(k)} \]

If \(\left\{ v^{(k)} \right\}\) is orthonormal, the new coefficients \(\beta_k\) can be computed as a linear combination of the previous coefficients and the projection of the new basis over the original one:

\[\beta_h = \left \langle v^{(h)}, x \right \rangle = \left \langle v^{(h)}, \sum_{k=0}^{K-1} \alpha_k w^{(k)} \right \rangle = \sum_{k=0}^{K-1} \alpha_k \left\langle v^{(h)}, w^{(k)} \right \rangle \]

This operation can also be represented in a matrix form as follows:

\[ \beta_h = \begin{bmatrix}
c_{00} & c_{01} & \cdots & c_{0\left(K-1 \right )}\\
& & \vdots & \\
c_{\left(K-1 \right )0} & c_{\left(K-1 \right )01} & \cdots & c_{\left(K-1 \right )\left(K-1 \right )}
\end{bmatrix}\begin{bmatrix}
\alpha_0 \\
\vdots \\
\alpha_{K-1}
\end{bmatrix} \]

This operation is widely used in algebra. A well-known example of a change of basis could be the Discrete Fourier Transform (DFT).

Inner product in vector space

The inner product is an operation that measures the similarity between vectors.  In a general way, the inner product could be defined as an operation of 2 operands, which are elements of a vector space. The result is a scalar in the set of the complex numbers:

\[ \left \langle \cdot, \cdot \right \rangle : V \times V \rightarrow \mathbb{C}  \]

Formal properties

For \(x, y, z \in V\) and \(\alpha \in \mathbb{C}\), the inner product must fulfill the following rules:

To be distributive to vector addition:

\( \left \langle x+y, z \right \rangle = \left \langle x, z \right \rangle + \left \langle y, z \right \rangle \)

Conmutative with conjugate (applies when vectors are complex):

\( \left \langle x,y \right \rangle  = \left \langle y, x \right \rangle^* \)

Distributive respect scalar multiplication:

\(  \left \langle \alpha x, y \right \rangle =  \alpha^* \left \langle x, u \right \rangle \)

\(  \left \langle  x, \alpha y \right \rangle =  \alpha \left \langle x, u \right \rangle \)

The self inner product must be necessarily a real number:

\(  \left \langle  x, x \right \rangle \geq 0 \)

The self inner product can be zero only when the element is the null element:

\( \left \langle x,x \right \rangle = 0 \Leftrightarrow x = 0 \)

Inner product in \(\mathbb{R}^2 \)

The inner product in \( \mathbb{R}^2\) is defined as follows:

\( \left \langle x, y \right \rangle = x_0 y_0 + x_1 y_1 \)

In self inner product represents the squared norm of the vector:

\( \left \langle x, x \right \rangle = x^2_0 + y^2_0 = \left \| x \right \|^2 \)

Inner product in finite length signals

In this case, the inner product is defined as:

\[ \left \langle x ,y \right \rangle = \sum_{n= 0}^{N-1} x^*[n] y[n] \]

Properties of vector spaces

Vector spaces must meet the following rules:
Addition to be commutative:
\( x + y = y + x \)

Addition to be distributive:
\( (x+y)+z = x + (y + z) \)

Scalar multiplication to be distributive with respect to vector addition:
\( \alpha\left(x + y \right) = \alpha x + \alpha y\)

Scalar multiplication to be distributive with respect to vector the addition of field scalars:
\( \left( \alpha + \beta \right) x = \alpha x + \beta y \)

Scalar multiplication to be associative:
\( \alpha\left(\beta x \right) = \left(\alpha \beta \right) x \)

It must exist a null element:
\( \exists 0 \in V \ \ | \ \ x + 0 = 0 + x = x \)

It must exist an inverse element for every element in the vector space:
\( \forall x \in V \exists (-x)\ \ | \ \ x + (-x) = 0\)

The uvm_object class

The uvm_object class is the base class for all UVM classes. From it, all the rest of classes are extended. It provides basic functionalities such as print, compare, copy and similar methods.

This class can be used when defining reusable parts of a sequence items. For example, in a packet like uvm_sequence_item, we could define a uvm_object extended object for defining the header. This would be:

This packet_header could be included in a packet class for conforming the uvm_sequence_item (the transaction) which will compose the sequences:

 

Pseudo-random number generator with Fibonacci sequence

\[ s_k = (k\cdot A) \bmod B\]

\(s_k\) is the pseudo-random number and \(A\) and \(B\) are prime numbers. \(k\) is in the range \([0,B-1]\). If \(k\) is greater than \(B-1\), the results will be repeat as \(B\) is the period of the sequence.
For example, \(A = 7\) and \(B = 17\). This sequence written in MATLAB could be:

 

Pseudo-random values
Periodicity of the sequence when k > B