Examination in Google Search Console of its indexing activities showed many rejected pages that should have been indexed.
Here is a checklist of fixes
1) Support all redirects
According to my VPS manuals advertised below, I had set up a correct redirect of http to hhtps. But other redirects were missing.
Suppose your site name is of the form mysite.net
In the Apache website .htaccess file, which can be found in my recommended setup (see manuals below) at:
sudo vim /home/<directory for mysite>/www/.htaccess
These new redirect rules must be added:
RewriteCond %{HTTP_HOST} ^mysite\.net [NC]
RewriteRule ^(.*)$ https://www.mysite.net/$1 [L,R=301]
<Directory /home/goalpost/www>
AllowOverride All
</Directory>
This rule should already be present and at the top of the file:
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
Save and quit the file when the changes have been made.
You must also add a virtual host for the non-htttps domain. Do this in the main config file.
sudo vim /etc/apache2/apache2.conf
<VirtualHost 123.456.789.000:80> // use your server's public ip here
ServerName mysite.net
Redirect permanent / https://www.mysite.net/
</VirtualHost>
A similar change must be added to the file that handles secure connections on port 443, this being the equivalent virtual host, but for the https domain:
sudo vim /etc/apache2/sites-enabled/apache2-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost 123.456.789.000:443> // use your server's public ip here
ServerName mysite.net
Redirect permanent / https://www.mysite.net
DocumentRoot "/var/www/mysite"
SSLCertificateFile /etc/letsencrypt/live/DIRECTORY_USED_FOR_CERT/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/DIRECTORY_USED_FOR_CERT/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>
However, this last change is insufficient by itself. The reason is that a call to https://mysite.net will be refused because your existing security cert is for www. mysite.net and not mysite.net. One final change is required.
Do:
sudo certbot certificates
to get a list of certificate urls
then
sudo certbot certonly --cert-name domain_name_used_for_cert -v --expand -d site_url1 -d site_url2_etc -d www.mysite.net -d mysite.net
Make sure to include all the existing urls, including www.mysite.net but adding mysite.net
Of course, restart Apache with
sudo systemctl restart apache2
THis should solve all redirect problems for the tested web site. Make parallel additions for each of your other web sites.
2) Add canonical tags to every web page you want indexed
Put this once only in the <head> section:
<link rel="canonical" href="https://www.yoursitename.xxx/page-url" />
You only need one of these per file, and it should have the full https://www prefix
This will remove any refusals to index from Google considering the canonical format incorrect.
The instructions above are given in the context of the Apache web server setup described in full detail in my two VPS guides. These manuals are not expensive to buy and will save you hours of wasted time and grief trying to get your VPS servers to work reliably and securely.
