About Python
Good for:
- Machine learning
- Task automation
- Multifaceted projects
Why Python:
- It’s freely available
- Well supported libraries and documentation
- Used throughout academia and industry for many things
Recommended setup(s):
- Anaconda Python 3.x. If you prefer a graphical user interface, consider an IDE like PyCharm, Visual Studio Code, Spyder. PyCharm is free for students for a 1 year license. Visual Studio is also a staple of industry. Spyder ships built into Anaconda Python and is open source.
Best resource to learn quickly:
Python Data Analysis on LinkedIn Learning (access through SSO)
- A nice introduction to key features of Python, with good examples of how to work with datasets and extract insights. Login through SSO to get access to the exercise files.
- Average video length: 5+ min
- Total duration: 2h30m
Using Libraries
How to load/use a library in Python that isn't loaded by default
In [2]:
# to load libraries you've installed with pip or conda,
# you import them like so:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import sys
# note you can choose an alias to refer to a loaded library or module.
# This means we can refer to matplotlib.pyplot as plt when calling functions later.
Installing New Python Libraries
How to install and load new packages
In [3]:
# We highly recommend using Anaconda python 3.x and the conda environment for data analysis.
# To install a new library in Anaconda, open the Anaconda prompt and activate whatever
# environment you want to use (defualt is 'base'). Then simply call: conda install <library>
# where <library> is the name of the library you want to install.
#from the anaconda prompt, you only need to type:
# conda install wxPython
# OR
# pip install wxPython
# again, conda is preferred because it does some checking to make sure you won't break everything
# by installing something incompatible with whatever else you have installed.
# from inside a jupyter notebook, it's best to use the following syntax:
!conda install --yes --prefix {sys.prefix} wxPython
Using Multiple Python Scripts
How to call functions and classes from other files
In [4]:
# To access other python scripts, you can import them as you would a library.
# For example, if you had a script called other_stuff.py in the same folder,
# you can import it as follows:
import other_stuff
# if there was a function called, do_stuff(a,b) which returned a + (b/2),
# you could now call it like:
a = other_stuff.do_stuff(2,7)
print(a)
Importing Data From Excel Spreadsheet
How to open and work with Excel datasets in Python.
In [6]:
# The pandas library is a great package for dealing with raw data.
# It can import from csv or xlsx files directly, and handles
# a lot of the importing and data types for you.
import pandas as pd
data = pd.read_excel('ExampleDataClean.xlsx')
print(data.columns, data)
Importing Data From CSV
How to read in CSV files
In [7]:
# In Python, there are many ways to read in a csv file.
# One of the easiest is to use pandas like above.
# you would call read_csv() like so:
import pandas as pd
data = pd.read_csv('ExampleDataClean2.csv')
print(data.columns, data)
# If your data is dirtier than this and contains blank cells, use the keep_default_na parameter to compensate.
Calculating Statistics by Index
How to analyze parts of your data
In [14]:
# Use slicing to look at specific components of your data.
import pandas as pd
import numpy as np
data = pd.read_csv('ExampleDataClean2.csv')
print(data.columns, data)
# Now we'll calculate the min and max of Val2 and Val3. NOTE: Unlike R and Matlab, python is 0 indexed.
# 0 indexing means the first value starts at index 0, not 1.
minVal2, minVal3 = data[['Val2', 'Val3']].min()
maxVal2, maxVal3 = data[['Val2', 'Val3']].max()
print(f'min of Val2: {minVal2}, max of Val2: {maxVal2}\n' +
f'min of Val3: {minVal3}, max of Val3: {maxVal3}.')
# You can also apply your own function to the pandas dataframe like so:
# Let's take the inverse root of the values in the last column:
res = data['Val3'].apply(lambda x: 1/np.sqrt(x) if x !=0 else 0)
print(res)
# For column 3, we'll run into some negatives. we can filter those out in our function
res2 = data['Val2'].apply(lambda x: 1/np.sqrt(x) if x > 0 else -1/np.sqrt(x) if x > 0 else 0)
print(res2)
Plotting Data
How to plot data in Python
In [2]:
# There are a lot of great libraries for plotting (matplotlib, bokeh, seaborn, plot.ly, ...)
# Matplotlib is a staple though, and very extensible. We'll use it here.
%matplotlib notebook
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import pandas as pd
import numpy as np
import datetime
import dateutil
data = pd.read_csv('ExampleDataClean2.csv')
print(data.columns, data)
# we can print directly from pandas like so:
data.plot(use_index=True, y=['Val2', 'Val3'], style='o-')
# we can also extract that data and plot it separately:
xs = data['Timestamp'].tolist()
ys = data['Val3'].tolist()
# convert datetime strings to dates
xs = [dateutil.parser.parse(s)+datetime.timedelta(0,60-np.floor(float((xs[0].split(':')[-1])))) for s in xs]
# set up new plot
fig = plt.figure(2)
ax = fig.add_subplot()
# set x axis to datetime
ax.xaxis_date()
ax.xaxis.set_major_locator(mdates.SecondLocator(interval=5))
# format the date so we don't get super long strings
date_fmt = mdates.DateFormatter('%S.0')
ax.xaxis.set_major_formatter(date_fmt)
# plot the data
ax.plot(xs, ys, linestyle='-', linewidth=1, color='darkgrey')
# do some formatting to clean up the look of the plots
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.xaxis.set_tick_params(top='off', direction='out', width=1, labelsize=10)
ax.yaxis.set_tick_params(right='off', direction='out', width=1, labelsize=10)
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')
ax.set_xlabel(r'Time, $sec$', fontsize=20)
# display the plot
plt.show()
