Python methods cheat sheet

Python list methods

append(item)

>>> lst = [1, 2, 2]
>>> lst.append(4)
[1, 2, 2, 4]

count(item)

>>> lst.count(2)
2

extend(list)

>>> lst.extend([5, 6])
[1, 2, 2, 4, 5, 6]

index(item)

>>> lst.index(5)
4

insert(position, item)

>>>lst.insert(1, 9)
[1, 9, 2, 2, 3, 5, 6]

sort()

>>> lst.sort()
[1, 2, 2, 3, 5, 6, 9]

pop(index)

>>> lst.pop()
9

reverse()

>>> lst.reverse()
[6, 5, 3, 2, 1, 1]

Python string methods

>>> s = 'Hello2Python'

count(sub, start, end)

>>> s.count('l', 1, 5)
2

find(sub, start, end)

>>> s.find('l', 0, 5)
2

index(sub, start, end)

>>> s.index('t', 0, 9)
8

isalnum()

>>> s.isalnum()
True

isalpha()

>>> s.isalpha()
False

isdigit()

>>> s.isdigit()
False

islower()

>>> s.islower()
False

isupper()

>>> s.isupper()
False

isspace()

>>> s.isspace()
False

join()

>>> s.join('___')
'_Hello2Python_Hello2Python_Hello2Python_'

lower()

>>> s.lower()
hello2python

partition(sep)

>>> s.partition('2')
('Hello', '2', 'Python')

split(sep)

>>> s.split('2')
['Hello',  'Python']

strip()

>>> s.strip('n')
'Hello2Pytho'

upper()

>>> s.upper()
'HELLO2PYTHON'

Read More

Common Linux command examples

Here is a collection of commonly used Linux command

Move files between servers

Move file from current server to another server (remote_server)

scp this_path/this_file username@remote_server:remote_path/

Move file from remote server to local server

scp username@remote_server:remote_path/remote_file current_path/

Check resource usage
top
free -h

Check server CPU information
lscpu
cat /proc/cpuinfo

Read More

Set up machine learning and deep learning on AWS

Here is the simple instructions to set up a EC2 instance to run machine learning and deep learning on AWS

1.  Run an EC2 instance from AWS
Login the AWS account and switch to Oregon. Click on the Community AMI on the left panel.
Search ami-813110b1 and lunch the instance
Choose t2.micro the free tier

2.  Install and configure Anaconda

Download Anaconda

$ wget https://repo.continuum.io/archive/Anaconda2-4.2.0-Linux-x86_64.sh

• Install with bash

$ bash Anaconda2-4.2.0-Linux-x86_64.sh

• Configure Anaconda

Start python

>>> from notebook.auth import passwd

>>> passwd()

Copy the passwrod here and paste to the configuration file

$ jupyter notebook --generate-config

Open ./jupyter/jupyter_notebook_config.py

c.NotebookApp.ip = '*'

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

3. Install tensorflow, keras
$ pip install tensorflow
$ pip install keras

4. Start jupyter notebook

$ jupyter notebook --no-browser --port=8888

Open the browser and enter DNS the address

http://ec2-35-161-215-37.us-west-2.compute.amazonaws.com:8888

Read More

Adjust the cell width of the Jupyter notebook

The default width of the notebook cell is sometimes not fit the purpose to display the content. In this case, we would like to adjust the cell size. There are few solutions to do this.

The permanent solution is to create a file in the home folder for IPython notebook.

~/.ipython/profile_default/static/custom/custom.css

or Jupyter notebook

~/.jupyter/custom/custom.css

with content

.container { width:100% !important; }

Then restart iPython/Jupyter notebooks. Note that this will affect all notebooks. 

If you don't want to change your default settings, and you only want to change the width of the current notebook you're working on, you can enter the following into a cell:

from IPython.core.display import display, HTML
display(HTML("<style>.container {width:90% !important;</style>"))

Read More

Python map and filter function

Python provides map and filter functions to process computations on the list of items. Here is a list of examples how to use map and filter.

Python map function examples

Compute the square of 1 to 10

def f(x):
    return x*x

map(f, range(1, 11))

or use lambda functions

map(lambda x: x*x, range(10)) 

Python filter function examples

Extract the event numbers from 1 to 10

def f(x):
    return x%2 == 0

filter(f, range(1, 11))

or use lambda functions

filter(lambda x: x%2 == 0, range(1, 11))

Read More

Python - list comprehension examples

List comprehension is a Python feature to process elements in the list without writing a function.

Square of numbers from 0 to n

Python script
for x in range(10):
    print(x**2)

List comprehension
[x**2 for x in range(n)]

Map function
map(lambda x: x**2, range(n))

Square of even numbers from 0 to n

Python script
for x in range(10):
    if x%2 == 0:
        print(x**2)

List comprehension with if clause
[x**2 for x in range(n) if x%2 == 0]


List comprehension with two items
[x + y for x in range(10) for y in range(10)]
[(x, y, x+y) for x in range(10) for y in range(10)]

List comprehension with two items with if clause
[(x, y, x+y) for x in range(10) for y in range(10) if (x%2==0) & * (y%2==0)]

Read More

Ordinal label encoder of categorical variables

Ordinal label encoder of categorical variables

During the machine learning model development, it is usually required to convert the categorical variables into numerical in order to fit the machine learning packages. This process is called categorical variable encoder. There are various ways to code the categorical variables into numerical, such as label encoder, one-hot encoder, weight of evidence encoder, etc. 

Label encoder is the most simple methodology, which convert the categorical values into numbers without any order requirement. Suppose we have a pandas data frame with a categorical variable - "cat" and the target variable - "target". The following code snippet will label each unique values into a number. The benefit comparing to the sklearn labelencoder is that the missing values have been considered as a new category, while the sklearn will prompt errors.

Simple label encoder convert the values of the categorical variables into labels

def ordinal_label(df, v, target):
    df_group = df.groupby([v])[target].mean().reset_index()
    values = df_group.sort_values([target])[v].values
    return dict(zip(values, range(len(values))))


df['cat_label'] = df['cat'].map(get_label(df, 'cat'))

What if we want the label to be in order? In this case, the categorical variable is truly converted into a numerical variable that are rank ordering with the dependent variable. One way is to compute the mean of the target variables for each value of the categorical variable. If we sort the pair of cat-target by target, the values of the categorical variables have been sorted in order.

def ordinal_label(df, v, target):
    df_group = df.groupby([v])[target].mean().reset_index()
    values = df_group.sort_values([target])[v].values
    return dict(zip(values, range(len(values))))

df['cat_label'] = df['cat'].map(get_label(df, 'cat'))

Or more compact way,

def ordinal_label(df, cat, target):
    values = df.groupby([cat])[target].mean().reset_index().\

        sort_values([target])[cat].values
    return dict(zip(values, range(len(values))

df['cat_label'] = df['cat'].map(get_label(df, 'cat'))

A better strategy is to do some binning of the values with similar 'mean of target', which can be tackled manually.

Read More