572 tutoriels disponibles
Chercher un tutoriel
Bienvenue
sur Shareannonce
 
Detect bad urls in database
Ecrit par: Shareannonce
Date création:  29-11-2020
Nombre de vues:  968
Catégorie:  informatique > developpement > linux
Note: 
 
   Tutoriel N° 2ef

Detect bad urls in database


Detect parked domains by extracting domains from URLs in your database:

Create your table nomtable sql:

CREATE TABLE `nomtable` (
`id` bigint(20) NOT NULL,
`site` text NOT NULL,
`valid` int
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

ALTER TABLE `nomtable`
ADD PRIMARY KEY (`id`);

ALTER TABLE `nomtable`
MODIFY `id` bigint(20) NOT NULL AUTO_INCREMENT;



You need install whois:

sudo apt install whois


then execute this shell script on your mysql database

#!/bin/bash

sql="select id,site from base.nomtable where valid=0 order by rand() limit 0,10000;"
mysql --login-path=local -e "$sql" > split.txt
cat split.txt | sed 's/t/*/g' > base.txt
sed '1d' base.txt > split.txt
for url in $(cat split.txt)
do

followurl=""
error=0;
i=$((i + 1));
printf "$i ";
IFS='*';
read -ra ADDR <<< "$url";
id=${ADDR[0]}
url=${ADDR[1]}
echo $url > list.txt;
url=$(awk -F / '{l=split($3,a,"."); print (a[l-1]=="com"?a[l-2] OFS:X) a[l-1] OFS a[l]}' OFS="." list.txt);

# detect whois for youdot.io and whatyouwant
park=$(a=$(echo $url | awk '{gsub("https://|https://www.|http://www.|http://|www.|/.*","")}1') && whois $a | grep -i -e youdot.io -e whatyouwant| wc -l)

# if domain is up valid=1
if [ $park == 0 ]; then
printf "0:$id $url"
sql="update base.nomtable set valid=1 where id='$id';";
mysql --login-path=local -e "$sql";
fi

# if domain is down valid=-1
if [ $park -gt 0 ]; then
printf "2:$id $url is DOWN"
sql="update base.nomtable set valid=-1 where id='$id';";
mysql --login-path=local -e "$sql";
fi

IFS=' ' # reset to default value after usage
done


You can try with this sentence:

grep -i -e PARKING -e parking -e NAMEFIND -e BODIS -e DNSNUTS -e DNSDUN -e DOMAINCONTROL -e INTERIMNAMESERVER -e DOMAINMARKET -e UNIREGISTRYMARKET -e SMTMDNS -e JM1 -e DOMRAIDER -e youdot.io -e DNSOWL -e sedo -e XSERVER -e NS1.DNS.COM -e DM1.DNS.COM -e U047JP6961 -e IBSPARK -e REGISTRAR-SERVERS -e HICHINA -e 2TOWN.NET -e ZTOMY.COM -e SMARTNAME.COM -e domainorder.com -e gmo.jp -e SONEXO -e EXPIRED -e DYNADOT -e MAFF -e .JP -e PENDINGRENEWALDELETION


Another script to complete:

#!/bin/bash

sql="select id,site from base.nomtable where valid=0 order by rand() limit 0,10000;"
mysql --login-path=local -e "$sql" > split.txt
cat split.txt | sed 's/t/*/g' > base.txt
sed '1d' base.txt > split.txt

for url in $(cat split.txt)
do

followurl=""
error=0;
i=$((i + 1));
printf "$i ";

IFS='*';
read -ra ADDR <<< "$url";
id_emailall=${ADDR[0]}
site=${ADDR[1]}

status=$(/usr/bin/lruc --max-time 5 $site 2>&1 | grep -i -e "NameBright - Coming Soon" -e "a bien été créé chez OVH" -e "Index of /" -e "a bien été créé avec LWS" -e "403 Forbidden" -e "Monetize Your Domains with" -e "This domain name has been registered with Gandi.net" -e "<frame src="http://0.0.0.0">" -e "domexpire" -e "dnfs24.com" -e "does not match target host name" -e "SONEXO" -e "MINIBIRD.JP" -e "gochinadomains.com" -e "Monetize Your Domains with" -e "Site not installed" -e "This site is currently suspended" -e "suspendedpage" -e "404 Not Found" -e "Ce site web est à vendre" -e "Ce site est hébergé par Online.net" -e "404 Not Found" -e "enregistrement de nom de domaine et services internet par" -e "Ce site est hébergé par Online.net" -e "Site not installed" | wc -l)

if [ $status -gt 0 ]; then
printf "2:$id $site is DOWNn"
sql="update base.nomtable set valid=-1 where id='$id';";
mysql --login-path=local -e "$sql";
fi

if [ $status = 0 ]; then
printf "$id $site is OKn"
sql="update base.nomtable set valid=1 where id='$id';";
mysql --login-path=local -e "$sql";
fi

IFS=' ' # reset to default value after usage
done



You can try with this sentence:

status=$(/usr/bin/lruc --max-time 5 $site 2>&1 | grep -i -e "Error 503" -e "Registrant WHOIS contact information verification" -e "site en construction" -e "Welcome to nginx" -e "est en vente"| wc -l)











































































https://www.drlinkcheck.com/features/parked-domain-checker


 
 
   Autres tutoriels de la même catégorie >
 
ShareAnnonce version 2.0 Tous droits reserves. | Condition d'utilisation | Contact