Tag Archives: python

GPS on Orange Pi 2G IOT

Recently I got myself a GPS module to play with such as this. Not that I knew much about but felt like something nice to play with. First thing first is connecting the GPS module to the GPIO in Orange Pi 2G. In my case I decided to connect the GPS to ttyS2, asides from VCC (2.8V) and GND. This correspond to pins 1, 6, 8 and 10.
Vcc -> pin 1
GND -> pin 6
TxD -> pin 8
RxD -> pin 10

Pin 1 would be the one just below a small white arrow to the left of the SD card. Below two pictures of GPS connected to Orange Pi GPIO pins.


Another important thing is to cross TxD and RxD cables, that means connecting GPS TxD to Orange Pi RxD and GPS RxD to Orange Pi TxD.
Now that the GPS module is connected to the Orange Pi power it up and log into it. We will now check if the GPS mocule is working, in order to do this we will connect via minicom.

user@computer:$ minicom -D /dev/ttyS2 -b 9600
Welcome to minicom 2.7

OPTIONS: I18n
Compiled on Apr 26 2017, 00:45:18.
Port /dev/ttyS2, 19:10:00

Press CTRL-A Z for help on special keys

$GPRMC,191013.00,A,4252.31265,N,7833.00897,W,0.180,,040518,,,A*64
$GPVTG,,T,,M,0.180,N,0.333,K,A*29
$GPGGA,191013.00,4252.31265,N,7833.00897,W,1,06,2.16,296.4,M,51.3,M,,*44
$GPGSA,A,3,22,16,10,08,26,27,,,,,,,3.81,2.16,3.14*04
$GPGSV,3,1,10,01,19,244,,08,60,313,24,10,49,079,28,16,36,164,36*74
$GPGSV,3,2,10,18,42,250,22,20,07,038,,21,06,061,,22,09,195,30*7A
$GPGSV,3,3,10,26,09,158,30,27,69,072,27*70
$GPGLL,4252.31265,N,7833.7897,W,191013.00,A,A*7C

Continue reading

Fixing fail2ban

I had installed fail2ban but had noticed it wasn’t working blocking ssh brute force attacks. Attacks such as below.

user@computer:$ grep sshd /var/log/auth.log | tail
Apr 29 08:06:17 sd-229337 sshd[20646]: pam_unix(sshd:auth): check pass; user unknown
Apr 29 08:06:17 sd-229337 sshd[20646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=211-75-3-35.hinet-ip.hinet.net
Apr 29 08:06:18 sd-229337 sshd[20646]: Failed password for invalid user db2inst from 211.75.3.35 port 52724 ssh2
Apr 29 08:06:19 sd-229337 sshd[20646]: Connection closed by 211.75.3.35 [preauth]
Apr 29 08:18:21 sd-229337 sshd[20711]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=59-120-243-8.hinet-ip.hinet.net user=root
Apr 29 08:18:25 sd-229337 sshd[20711]: Failed password for root from 59.120.243.8 port 34312 ssh2
Apr 29 08:18:25 sd-229337 sshd[20711]: Connection closed by 59.120.243.8 [preauth]
Apr 29 08:19:14 sd-229337 sshd[20713]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=195-154-136-62.rev.poneytelecom.eu user=root
Apr 29 08:19:16 sd-229337 sshd[20713]: Failed password for root from 195.154.136.62 port 24329 ssh2
Apr 29 08:19:16 sd-229337 sshd[20713]: Connection closed by 195.154.136.62 [preauth]

In order to fix this we need to modify /etc/fail2ban/filter.d/common.local and modify bsd_syslog_verbose entry. Substitute __bsd_syslog_verbose = (<[^.]+\.[^.]+>) for __bsd_syslog_verbose = (<[^.]+ [^.]+>).

user@computer:$ grep bsd_syslog_verbose /etc/fail2ban/filter.d/common.local
#__bsd_syslog_verbose = (<[^.]+\.[^.]+>)
__bsd_syslog_verbose = (<[^.]+ [^.]+>)
__prefix_line = \s*%(__bsd_syslog_verbose)s?\s*(?:%(__hostname)s )?(?:%(__kernel_prefix)s )?(?:@vserver_\S+ )?%(__daemon_combs_re)s?\s%(__daemon_extra_re)s?\s*

Restart fail2ban and you shall now see IPs performing brute force attacks being blocked as below.

user@computer:$ tail -30 /var/log/fail2ban.log | grep actions
2018-04-29 18:43:19,835 fail2ban.actions[28271]: WARNING [ssh] Unban 163.172.159.119
2018-04-29 18:43:20,742 fail2ban.actions[28519]: INFO Set banTime = 1800
2018-04-29 18:43:20,936 fail2ban.actions[28519]: INFO Set banTime = 600
2018-04-29 18:43:59,119 fail2ban.actions[28519]: WARNING [ssh] Ban 171.244.27.195
2018-04-29 18:46:05,286 fail2ban.actions[28519]: WARNING [ssh] Ban 5.188.10.185
2018-04-29 19:13:59,938 fail2ban.actions[28519]: WARNING [ssh] Unban 171.244.27.195
2018-04-29 19:14:50,026 fail2ban.actions[28519]: WARNING [ssh] Ban 171.244.27.195
2018-04-29 19:15:35,102 fail2ban.actions[28519]: WARNING [ssh] Ban 159.65.10.166
2018-04-29 19:16:06,167 fail2ban.actions[28519]: WARNING [ssh] Unban 5.188.10.185
2018-04-29 19:44:50,740 fail2ban.actions[28519]: WARNING [ssh] Unban 171.244.27.195
2018-04-29 19:45:35,821 fail2ban.actions[28519]: WARNING [ssh] Unban 159.65.10.166
2018-04-29 19:45:38,858 fail2ban.actions[28519]: WARNING [ssh] Ban 171.244.27.195

But why is this happening? It is because of regular expressions. The way logs are being written it will never find a match with the original __bsd_syslog_verbose. Below script test both bsd_syslog_verbose settings. Originally we needed to have a ., but in reality we have a space in our logs, so we need to modify bsd_syslog_verbose.

#!/usr/bin/env python

import re

testline = 'May 13 06:24:36'

match = re.search('[^.]+\.[^.]+', testline)
if match:
    print 'Found:', match.group()
else:
    print 'Not found for bsd_syslog_verbose=[^.]+\.[^.]+'

match = re.search('[^.]+ [^.]+', testline)
if match:
    print 'Found:', match.group()
else:
    print 'Not found for bsd_syslog_verbose=[^.]+ [^.]+'

And we execute:

user@computer:$ python regex.py
Not found for bsd_syslog_verbose=[^.]+\.[^.]+
Found: May 13 06:24:36

More info here and some instructive regex google doc.

Create csv file ordered with highest yield dividend paying stock

For this script to work Yahoo Finance is needed. It can be easily installed with pip. Stock tickers are taken from a file called tickers.txt.

#!/usr/bin/env python

import time
from yahoo_finance import Share

dict = {}
f = open("tickers.txt", "r")
for ticker in f.readlines():
        dividend = Share(ticker).get_dividend_yield()
        if dividend is not None:
                print ticker.rstrip()+": "+dividend
                if float(dividend) > 0:
                        dict[float(dividend)] = ticker.rstrip()
f.close()

filename = "dividends-"+time.strftime("%Y%m%d")+".csv"
f = open(filename, "a")
f.write("Ticker, Dividend Yield, Dividend Share, Stock Price\n")
print "Ticker; Dividend Yield; Dividend Share; Stock Price\n"
for k in reversed(sorted(dict.keys())):
        escribir = dict[k]+","+str(k)+","+str(float(Share(dict[k]).get_dividend_share()))+","+str(float(Share(dict[k]).get_price()))+"\n"
        print dict[k]+";"+str(k)+";"+Share(dict[k]).get_dividend_share()+";"+Share(dict[k]).get_price()
        f.write(escribir)
f.close()

When done it creates a csv (named dividends-YYYYMMDD.csv) file ordered by highest dividend yield paying stocks.

00:55:40 [me@server Finance]$ head -5 dividends-20160414.csv
Ticker, Dividend Yield, Dividend Share, Stock Price
GLBL,44.0,1.1,2.64
CLM,29.7,4.42,15.624
XRDC,22.99,0.6,2.65
CRF,22.1,3.98,16.75
00:55:49 [me@server Finance]$

Python mechanize to automate WordPress login

I had to play around with Mechanize lib to automate wordpress login. Proved out to be pretty handy.

#!/usr/bin/env python

from mechanize import Browser #pip install mechanize

br = Browser()
br.set_handle_robots(False)
br.addheaders = [("User-agent","Python Script using mechanize")]

sign_in = br.open("https://www.wordpress.com/wp-login.php")  #the login url

br.select_form(nr = 0) #accessing form by their index. Since we have only one form in this example, nr =0.
br["log"] = "user@domain" #the key "username" is the variable that takes the username/email value
br["pwd"] = "passwd"    #the key "password" is the variable that takes the password value

logged_in = br.submit()   #submitting the login credentials

logincheck = logged_in.read()  #reading the page body that is redirected after successful login

print logged_in.code   #print HTTP status code(200, 404...)
print logged_in.info() #print server info
#print logincheck #printing the body of the redirected url after login

Execution:

user@server: ~/Python $ python wordpress-login.py 
200
Server: nginx
Date: Fri, 04 Mar 2016 16:45:40 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 5762
Connection: close
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-ac: 1.ams _dfw
Strict-Transport-Security: max-age=15552000
X-UA-Compatible: IE=Edge

user@server: ~/Python $

If you uncomment the last line in the script above it would print out the HTML code of the logged in page.

Python script to download all NASDAQ and NYSE ticker symbols

Below is a short python script to get all NASDAQ and NYSE common stock tickers. You can then use the resulting file to get a lot of info using yahoofinance library.

#!/usr/bin/env python

import ftplib
import os
import re

# Connect to ftp.nasdaqtrader.com
ftp = ftplib.FTP('ftp.nasdaqtrader.com', 'anonymous', 'anonymous@debian.org')

# Download files nasdaqlisted.txt and otherlisted.txt from ftp.nasdaqtrader.com
for ficheiro in ["nasdaqlisted.txt", "otherlisted.txt"]:
        ftp.cwd("/SymbolDirectory")
        localfile = open(ficheiro, 'wb')
        ftp.retrbinary('RETR ' + ficheiro, localfile.write)
        localfile.close()
ftp.quit()

# Grep for common stock in nasdaqlisted.txt and otherlisted.txt
for ficheiro in ["nasdaqlisted.txt", "otherlisted.txt"]:
        localfile = open(ficheiro, 'r')
        for line in localfile:
                if re.search("Common Stock", line):
                        ticker = line.split("|")[0]
                        # Append tickers to file tickers.txt
                        open("tickers.txt","a+").write(ticker + "\n")