Executed: Mon Mar 27 11:38:07 2017
Duration: 10 seconds.
usALEX-5samples - Template

This notebook is executed through 8-spots paper analysis.
For a direct execution, uncomment the cell below.

ph_sel_name = "AexAem"

data_id = "17d"

# ph_sel_name = "all-ph"
# data_id = "7d"

from fretbursts import *

init_notebook()
from IPython.display import display

import os
data_dir = os.path.abspath(data_dir) + '/'
assert os.path.exists(data_dir), "Path '%s' does not exist." % data_dir

from glob import glob

file_list = sorted(f for f in glob(data_dir + '*.hdf5') if '_BKG' not in f)
file_list

## Selection for POLIMI 2012-12-6 dataset
# file_list.pop(2)
# file_list = file_list[1:-2]
# display(file_list)
# labels = ['22d', '27d', '17d', '12d', '7d']

## Selection for P.E. 2012-12-6 dataset
# file_list.pop(1)
# file_list = file_list[:-1]
# display(file_list)
# labels = ['22d', '27d', '17d', '12d', '7d']

## Selection for POLIMI 2012-11-26 datatset
labels = ['17d', '27d', '7d', '12d', '22d']

files_dict = {lab: fname for lab, fname in zip(labels, file_list)}
files_dict

ph_sel_map = {'all-ph': Ph_sel('all'), 'AexAem': Ph_sel(Aex='Aem')}
ph_sel = ph_sel_map[ph_sel_name]

data_id, ph_sel_name

Laser alternation selection
At this point we have only the timestamps and the detector numbers:

We need to define some parameters: donor and acceptor ch, excitation period and donor and acceptor excitiations:

We should check if everithing is OK with an alternation histogram:

If the plot looks good we can apply the parameters with:

Measurements infos
All the measurement data is in the d variable. We can print it:

Compute background
Compute the background using automatic threshold:

d.calc_bg(bg.exp_fit, time_s=60, tail_min_us='auto', F_bg=1.7)

dplot(d, timetrace_bg)

d.rate_m, d.rate_dd, d.rate_ad, d.rate_aa

from mpl_toolkits.axes_grid1 import AxesGrid
import lmfit
print('lmfit version:', lmfit.__version__)

assert d.dir_ex == 0
assert d.leakage == 0

d.burst_search(m=10, F=6, ph_sel=ph_sel)

print(d.ph_sel, d.num_bursts)

ds_sa = d.select_bursts(select_bursts.naa, th1=30)
ds_sa.num_bursts

mask = (d.naa[0] - np.abs(d.na[0] + d.nd[0])) > 30
ds_saw = d.select_bursts_mask_apply([mask])

ds_sas0 = ds_sa.select_bursts(select_bursts.S, S2=0.10)
ds_sas = ds_sa.select_bursts(select_bursts.S, S2=0.15)
ds_sas2 = ds_sa.select_bursts(select_bursts.S, S2=0.20)
ds_sas3 = ds_sa.select_bursts(select_bursts.S, S2=0.25)

ds_st = d.select_bursts(select_bursts.size, add_naa=True, th1=30)
ds_sas.num_bursts

dx = ds_sas0
size = dx.na[0] + dx.nd[0]
s_hist, s_bins = np.histogram(size, bins=np.r_[-15 : 25 : 1], density=True)
s_ax = s_bins[:-1] + 0.5*(s_bins[1] - s_bins[0])
plot(s_ax, s_hist, '-o', alpha=0.5)

dx = ds_sas
size = dx.na[0] + dx.nd[0]
s_hist, s_bins = np.histogram(size, bins=np.r_[-15 : 25 : 1], density=True)
s_ax = s_bins[:-1] + 0.5*(s_bins[1] - s_bins[0])
plot(s_ax, s_hist, '-o', alpha=0.5)

dx = ds_sas2
size = dx.na[0] + dx.nd[0]
s_hist, s_bins = np.histogram(size, bins=np.r_[-15 : 25 : 1], density=True)
s_ax = s_bins[:-1] + 0.5*(s_bins[1] - s_bins[0])
plot(s_ax, s_hist, '-o', alpha=0.5)

dx = ds_sas3
size = dx.na[0] + dx.nd[0]
s_hist, s_bins = np.histogram(size, bins=np.r_[-15 : 25 : 1], density=True)
s_ax = s_bins[:-1] + 0.5*(s_bins[1] - s_bins[0])
plot(s_ax, s_hist, '-o', alpha=0.5)

plt.title('(nd + na) for A-only population using different S cutoff');

dx = ds_sa

alex_jointplot(dx);

dplot(ds_sa, hist_S)

A-direct excitation fitting
To extract the A-direct excitation coefficient we need to fit the
S values for the A-only population.
The S value for the A-only population is fitted with different methods:
- Histogram git with 2 Gaussians or with 2 asymmetric Gaussians
(an asymmetric Gaussian has right- and left-side of the peak
decreasing according to different sigmas).
- KDE maximum
In the following we apply these methods using different selection
or weighting schemes to reduce amount of FRET population and make
fitting of the A-only population easier.
Even selection
Here A-only and FRET population are evenly selected.

dx = ds_sa

bin_width = 0.03
bandwidth = 0.03
bins = np.r_[-0.2 : 1 : bin_width]
x_kde = np.arange(bins.min(), bins.max(), 0.0002)

## Weights
weights = None

## Histogram fit
fitter_g = mfit.MultiFitter(dx.S)
fitter_g.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter_g.fit_histogram(model = mfit.factory_two_gaussians(p1_center=0.1, p2_center=0.4))
S_hist_orig = fitter_g.hist_pdf

S_2peaks = fitter_g.params.loc[0, 'p1_center']
dir_ex_S2p = S_2peaks/(1 - S_2peaks)
print('Fitted direct excitation (na/naa) [2-Gauss]:', dir_ex_S2p)

## KDE
fitter_g.calc_kde(bandwidth=bandwidth)
fitter_g.find_kde_max(x_kde, xmin=0, xmax=0.15)

S_peak = fitter_g.kde_max_pos[0]
dir_ex_S_kde = S_peak/(1 - S_peak)
print('Fitted direct excitation (na/naa) [KDE]:    ', dir_ex_S_kde)

fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))

mfit.plot_mfit(fitter_g, ax=ax[0])
ax[0].set_title('2-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks*100))

mfit.plot_mfit(fitter_g, ax=ax[1], plot_model=False, plot_kde=True)
ax[1].set_title('KDE fit (S_fit = %.2f %%)' % (S_peak*100));

## 2-Asym-Gaussian
fitter_ag = mfit.MultiFitter(dx.S)
fitter_ag.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter_ag.fit_histogram(model = mfit.factory_two_asym_gaussians(p1_center=0.1, p2_center=0.4))
#print(fitter_ag.fit_obj[0].model.fit_report())

S_2peaks_a = fitter_ag.params.loc[0, 'p1_center']
dir_ex_S2pa = S_2peaks_a/(1 - S_2peaks_a)
print('Fitted direct excitation (na/naa) [2-Gauss]:', dir_ex_S2pa)

fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))

mfit.plot_mfit(fitter_g, ax=ax[0])
ax[0].set_title('2-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks*100))

mfit.plot_mfit(fitter_ag, ax=ax[1])
ax[1].set_title('2-Asym-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks_a*100));

Zero threshold on nd
Select bursts with:
$$n_d < 0$$.

dx = ds_sa.select_bursts(select_bursts.nd, th1=-100, th2=0)

fitter = bext.bursts_fitter(dx, 'S')
fitter.fit_histogram(model = mfit.factory_gaussian(center=0.1))
S_1peaks_th = fitter.params.loc[0, 'center']
dir_ex_S1p = S_1peaks_th/(1 - S_1peaks_th)
print('Fitted direct excitation (na/naa) [2-Gauss]:', dir_ex_S1p)

mfit.plot_mfit(fitter)
plt.xlim(-0.1, 0.6)

Selection 1
Bursts are weighted using $w = f(S)$, where the function $f(S)$ is a
Gaussian fitted to the $S$ histogram of the FRET population.

dx = ds_sa

## Weights
weights = 1 - mfit.gaussian(dx.S[0], fitter_g.params.loc[0, 'p2_center'], fitter_g.params.loc[0, 'p2_sigma'])
weights[dx.S[0] >= fitter_g.params.loc[0, 'p2_center']] = 0

## Histogram fit
fitter_w1 = mfit.MultiFitter(dx.S)
fitter_w1.weights = [weights]
fitter_w1.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter_w1.fit_histogram(model = mfit.factory_two_gaussians(p1_center=0.1, p2_center=0.4))
S_2peaks_w1 = fitter_w1.params.loc[0, 'p1_center']
dir_ex_S2p_w1 = S_2peaks_w1/(1 - S_2peaks_w1)
print('Fitted direct excitation (na/naa) [2-Gauss]:', dir_ex_S2p_w1)

## KDE
fitter_w1.calc_kde(bandwidth=bandwidth)
fitter_w1.find_kde_max(x_kde, xmin=0, xmax=0.15)
S_peak_w1 = fitter_w1.kde_max_pos[0]
dir_ex_S_kde_w1 = S_peak_w1/(1 - S_peak_w1)
print('Fitted direct excitation (na/naa) [KDE]:    ', dir_ex_S_kde_w1)

def plot_weights(x, weights, ax):
    ax2 = ax.twinx()
    x_sort = x.argsort()
    ax2.plot(x[x_sort], weights[x_sort], color='k', lw=4, alpha=0.4)
    ax2.set_ylabel('Weights');

fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))
mfit.plot_mfit(fitter_w1, ax=ax[0])
mfit.plot_mfit(fitter_g, ax=ax[0], plot_model=False, plot_kde=False)
plot_weights(dx.S[0], weights, ax=ax[0])
ax[0].set_title('2-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks_w1*100))

mfit.plot_mfit(fitter_w1, ax=ax[1], plot_model=False, plot_kde=True)
mfit.plot_mfit(fitter_g, ax=ax[1], plot_model=False, plot_kde=False)
plot_weights(dx.S[0], weights, ax=ax[1])
ax[1].set_title('KDE fit (S_fit = %.2f %%)' % (S_peak_w1*100));

Selection 2
Bursts are here weighted using weights $w$:
$$w = n_{aa} - |n_a + n_d|$$

## Weights
sizes = dx.nd[0] + dx.na[0] #- dir_ex_S_kde_w3*dx.naa[0]
weights = dx.naa[0] - abs(sizes)
weights[weights < 0] = 0

## Histogram
fitter_w4 = mfit.MultiFitter(dx.S)
fitter_w4.weights = [weights]
fitter_w4.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter_w4.fit_histogram(model = mfit.factory_two_gaussians(p1_center=0.1, p2_center=0.4))
S_2peaks_w4 = fitter_w4.params.loc[0, 'p1_center']
dir_ex_S2p_w4 = S_2peaks_w4/(1 - S_2peaks_w4)
print('Fitted direct excitation (na/naa) [2-Gauss]:', dir_ex_S2p_w4)

## KDE
fitter_w4.calc_kde(bandwidth=bandwidth)
fitter_w4.find_kde_max(x_kde, xmin=0, xmax=0.15)
S_peak_w4 = fitter_w4.kde_max_pos[0]
dir_ex_S_kde_w4 = S_peak_w4/(1 - S_peak_w4)
print('Fitted direct excitation (na/naa) [KDE]:    ', dir_ex_S_kde_w4)

fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))

mfit.plot_mfit(fitter_w4, ax=ax[0])
mfit.plot_mfit(fitter_g, ax=ax[0], plot_model=False, plot_kde=False)
#plot_weights(dx.S[0], weights, ax=ax[0])
ax[0].set_title('2-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks_w4*100))

mfit.plot_mfit(fitter_w4, ax=ax[1], plot_model=False, plot_kde=True)
mfit.plot_mfit(fitter_g, ax=ax[1], plot_model=False, plot_kde=False)
#plot_weights(dx.S[0], weights, ax=ax[1])
ax[1].set_title('KDE fit (S_fit = %.2f %%)' % (S_peak_w4*100));

Selection 3
Bursts are here selected according to:
$$n_{aa} - |n_a + n_d| > 30$$

mask = (d.naa[0] - np.abs(d.na[0] + d.nd[0])) > 30
ds_saw = d.select_bursts_mask_apply([mask])
print(ds_saw.num_bursts)

dx = ds_saw

## Weights
weights = None

## 2-Gaussians
fitter_w5 = mfit.MultiFitter(dx.S)
fitter_w5.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter_w5.fit_histogram(model = mfit.factory_two_gaussians(p1_center=0.1, p2_center=0.4))
S_2peaks_w5 = fitter_w5.params.loc[0, 'p1_center']
dir_ex_S2p_w5 = S_2peaks_w5/(1 - S_2peaks_w5)
print('Fitted direct excitation (na/naa) [2-Gauss]:', dir_ex_S2p_w5)

## KDE
fitter_w5.calc_kde(bandwidth=bandwidth)
fitter_w5.find_kde_max(x_kde, xmin=0, xmax=0.15)
S_peak_w5 = fitter_w5.kde_max_pos[0]
S_2peaks_w5_fiterr = fitter_w5.fit_res[0].params['p1_center'].stderr
dir_ex_S_kde_w5 = S_peak_w5/(1 - S_peak_w5)
print('Fitted direct excitation (na/naa) [KDE]:    ', dir_ex_S_kde_w5)

## 2-Asym-Gaussians
fitter_w5a = mfit.MultiFitter(dx.S)
fitter_w5a.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter_w5a.fit_histogram(model = mfit.factory_two_asym_gaussians(p1_center=0.05, p2_center=0.3))
S_2peaks_w5a = fitter_w5a.params.loc[0, 'p1_center']
dir_ex_S2p_w5a = S_2peaks_w5a/(1 - S_2peaks_w5a)
#print(fitter_w5a.fit_obj[0].model.fit_report(min_correl=0.5))
print('Fitted direct excitation (na/naa) [2-Asym-Gauss]:', dir_ex_S2p_w5a)

fig, ax = plt.subplots(1, 3, figsize=(19, 4.5))

mfit.plot_mfit(fitter_w5, ax=ax[0])
mfit.plot_mfit(fitter_g, ax=ax[0], plot_model=False, plot_kde=False)
ax[0].set_title('2-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks_w5*100))

mfit.plot_mfit(fitter_w5, ax=ax[1], plot_model=False, plot_kde=True)
mfit.plot_mfit(fitter_g, ax=ax[1], plot_model=False, plot_kde=False)
ax[1].set_title('KDE fit (S_fit = %.2f %%)' % (S_peak_w5*100));

mfit.plot_mfit(fitter_w5a, ax=ax[2])
mfit.plot_mfit(fitter_g, ax=ax[2], plot_model=False, plot_kde=False)
ax[2].set_title('2-Asym-Gaussians fit (S_fit = %.2f %%)' % (S_2peaks_w5a*100));

sample = data_id
n_bursts_aa = ds_sas.num_bursts[0]

The following string contains the list of variables to be saved. When saving, the order of the variables is preserved.

variables = ('sample n_bursts_aa dir_ex_S1p dir_ex_S_kde dir_ex_S2p dir_ex_S2pa '
             'dir_ex_S2p_w1 dir_ex_S_kde_w1 dir_ex_S_kde_w4 dir_ex_S_kde_w5 dir_ex_S2p_w5 dir_ex_S2p_w5a '
             'S_2peaks_w5 S_2peaks_w5_fiterr\n')

This is just a trick to format the different variables:

variables_csv = variables.replace(' ', ',')
fmt_float = '{%s:.6f}'
fmt_int = '{%s:d}'
fmt_str = '{%s}'
fmt_dict = {**{'sample': fmt_str},
            **{k: fmt_int for k in variables.split() if k.startswith('n_bursts')}}
var_dict = {name: eval(name) for name in variables.split()}
var_fmt = ', '.join([fmt_dict.get(name, fmt_float) % name for name in variables.split()]) + '\n'
data_str = var_fmt.format(**var_dict)

print(variables_csv)
print(data_str)

# NOTE: The file name should be the notebook name but with .csv extension
with open('results/usALEX-5samples-PR-raw-dir_ex_aa-fit-%s.csv' % ph_sel_name, 'a') as f:
    f.seek(0, 2)
    if f.tell() == 0:
        f.write(variables_csv)
    f.write(data_str)

markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code


Prototype pipeline for the analysis of ML arxiv data
We query arxiv to get papers, and then run them against Crossref event data to find social media discussion and Microsoft Academic Knowledge to find institutional affiliations
```
Query Arxiv -> Paper repository -> Analysis -> Topic model -> Classify
                        |               |
                        |               |----> Social network analysis of researchers
                        |               |----> Geocoding of institutions (via GRID?)
                        |
                Extract author data from Google Scholar ----> Geocode institution via Google Places API?
                        |                                                   |
                Enrich paper data with MAK(?)                               |---> Spatial and network analysis
                        |
                Obtain Crossref Event data
```
Preamble

%matplotlib inline

#Some imports
import time
#import xml.etree.ElementTree as etree
from lxml import etree
import feedparser

#Imports
#Key imports are loaded from my profile (see standard_imports.py in src folder).

#Paths

#Paths
top = os.path.dirname(os.getcwd())

#External data (to download the GRID database)
ext_data = os.path.join(top,'data/external')

#Interim data (to place seed etc)
int_data = os.path.join(top,'data/interim')

#Figures
fig_path = os.path.join(top,'reports')

#Models
mod_path = os.path.join(top,'models')

#Get date for saving files
today = datetime.datetime.today()

today_str = "_".join([str(x) for x in [today.day,today.month,today.year]])

#Functions

1. Get Arxiv data about machine learning

Write a APi querier and extract papers with the terms machine learning or artificial intelligence. Get 2000 results... and play nice!

class Arxiv_querier():
    '''
    This class takes as an input a query and the number of results, and returns all the parsed results.
    Includes routines to deal with multiple pages of results.

    '''

    def __init__(self,base_url="http://export.arxiv.org/api/query?"):
        '''
        Initialise
        '''

        self.base_url = base_url

    def query(self,query_string,max_results=100,wait_time=3):
        '''
        Query the base url

        '''
        #Attribute query string

        #Load base URL
        base_url = self.base_url

        #Prepare query string
        processed_query = re.sub(' ','+',query_string)

        self.query_string="_".join(query_string.split(" "))

        start=0
        pages = 0

        #Run the query and store results for as long as the number of results is bigger than the max results
        keep_running = True

        result_store = []

        while keep_running==True:
            pages +=1
            print(pages)

            #Query url (NB the start arg, which will change as we go through different
            #pages)
            query_url = base_url+'search_query=all:{q}&start={s}&max_results={max_res}'.format(
                q=processed_query,s=start,max_res=max_results)

            #Download
            source = requests.get(query_url)

            #Parse the xml and get the entries (papers)
            parsed = feedparser.parse(source.content)

            #Extract entries
            entries = parsed['entries']

            #If the number of entries is bigger than the maximum number of results
            #this means we need to go to another page. We do that by offseting the
            #start with max results

            result_store.append(entries)

            if len(entries)==max_results:
                start+=max_results

            #If we have less than max results this means we have run out of
            #results and we toggle the keep_running switch off.
            if len(entries)<max_results:
                keep_running=False

            time.sleep(wait_time)

        #Save results in a flat list
        self.entry_results = [x for el in result_store for x in el]

    def extract_data(self):
        '''
        Here we extract data from the entries

        '''

        #Load entries
        entries = self.entry_results

        #Create df
        output = pd.concat([pd.DataFrame({
            'query':self.query_string,
            'id':x['id'],
            'link':x['link'],
            'title':x['title'],
            'authors':", ".join([el['name'] for el in x['authors']]),
            'summary':x['summary'],
            'updated':x['updated'],
            'published':x['published'],
            'category':x['arxiv_primary_category']['term'],
            'pdf':str([el['href'] for el in x['links']  if el['type']=='application/pdf'][0]
                     )},index=[0]) for x in entries]).reset_index(drop=True)

        output['year_published'] = [x.split("-")[0] for x in output['published']]

        self.output_df = output

query_terms = ['artificial intelligence','machine learning','deep learning']

#There are some inconsistencies in the number of results so we run the query three times for each
#term and remove duplicated results

def extract_arxiv_data(term,max_results=1000,wait_time=10, tests=3):
    '''
    This function initialises the Arxiv_querier class, extracts the data and outputs it

    '''
    print(term)

    collected = []

    #We collect the data thrice
    for i in np.arange(tests):
        print('run'+ ' ' +str(i))
        initialised = Arxiv_querier()
        initialised.query(term,max_results,wait_time)
        initialised.extract_data()
        out = initialised.output_df
        collected.append(out)

    #We concatenate the dfs and remove the duplicates.

    output = pd.concat(collected)
    output_no_dupes = output.drop_duplicates('id')

    #Return both
    return([output,output_no_dupes])

arxiv_ai_results_three = [extract_arxiv_data(term=q) for q in query_terms]

all_papers = pd.concat([x[1] for x in arxiv_ai_results_three]).drop_duplicates('id').reset_index(drop=True)
print(all_papers.shape)
all_papers.head()

all_papers.to_csv(int_data+'/{today}_ai_papers.csv'.format(today=today_str),index=False)

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize, RegexpTokenizer, PunktSentenceTokenizer
from nltk.stem import WordNetLemmatizer, SnowballStemmer, PorterStemmer
import scipy
import ast
import string as st
from bs4 import BeautifulSoup

import gensim
from gensim.models.coherencemodel import CoherenceModel
from sklearn.feature_extraction.text import TfidfVectorizer
from itertools import product

stopwords_c =  stopwords.words('english')
stemmer = PorterStemmer()
lemmatizer= WordNetLemmatizer()

#Read papers
all_papers = pd.read_csv(int_data+'/19_8_2017_ai_papers.csv'.format(today=today_str))

#Let's begin by looking at years

#When where they published?

#Year distribution
year_pubs = all_papers['year_published'].value_counts()
year_pubs.index = [int(x) for x in year_pubs.index]

fig,ax = plt.subplots(figsize=(10,5))

year_pubs_sorted = year_pubs[sorted(year_pubs.index)]
year_pubs_subset = year_pubs_sorted[year_pubs_sorted.index>1991]

ax.plot(np.arange(1993,2018),year_pubs_subset.cumsum(),color='red')
ax.bar(np.arange(1993,2018),year_pubs_subset)
ax.hlines(xmin=1993,xmax=2017,y=[10000,20000,30000,40000],colors='green',linestyles='dashed',alpha=0.7)

ax.set_title("Papers on AI, ML and DL, total per year (bar) and cumulative (red)",size=14)

#What are the categories of the papers? Are we capturing what we think we are capturing
#Top 20
all_papers['category'].value_counts()[:20]

See here for abbreviations of categories.
In a nutshell, AI is AI, LG is 'Learning', CV is 'Computer Vision', 'CL' is 'computation and language' and NE is 'Neural and Evolutionary computing'. SL.ML is kind of self-explanatory. We seem to be picking up the main things

#NB do we want to remove hyphens?
punct = re.sub('-','',st.punctuation)

def comp_sentence(sentence):
    '''
    Takes a sentence and pre-processes it.
    The output is the sentence as a bag of words

    '''
    #Remove line breaks and hyphens
    sentence = re.sub('\n',' ',sentence)
    sentence = re.sub('-',' ',sentence)

    #Lowercase and tokenise
    text_lowered = [x.lower() for x in sentence.split(" ")]

    #Remove signs and digits
    text_no_signs_digits = ["".join([x for x in el if x not in punct+st.digits]) for
                            el in text_lowered]

    #Remove stop words, single letters
    text_stopped = [w for w in text_no_signs_digits if w not in stopwords_c and
                   len(w)>1]

    #Stem
    text_lemmatised = [lemmatizer.lemmatize(w) for w in text_stopped]

    #Output
    return(text_lemmatised)

#Process text
clean_corpus = [comp_sentence(x) for x in all_papers['summary']]

#We remove rate words
word_freqs = pd.Series([x for el in clean_corpus for x in el]).value_counts()

word_freqs[:30]

rare_words = word_freqs.index[word_freqs<=2]
rare_words[:10]

Lots of the rare words seem to be typos and so forth. We remove them

#Removing rare words
clean_corpus_no_rare = [[x for x in el if x not in rare_words] for el in clean_corpus]

2 NLP (topic modelling & word embeddings)

#Identify 2-grams (frequent in science!)
bigram_transformer = gensim.models.Phrases(clean_corpus_no_rare)

#Train the model on the corpus

#Let's do a bit of grid search

#model = gensim.models.Word2Vec(bigram_transformer[clean_corpus], size=360, window=15, min_count=2, iter=20)

model.most_similar('ai_safety')

model.most_similar('complexity')

model.most_similar('github')

#Create 3 different dictionaries and bows depending on word sizes

def remove_words_below_threshold(corpus,threshold):
    '''
    Takes a list of terms and removes any which are below a threshold of occurrences

    '''
    #Produce token frequencies
    token_frequencies = pd.Series([x for el in corpus for x in el]).value_counts()

    #Identify tokens to drop (below a threshold)
    tokens_to_drop = token_frequencies.index[token_frequencies<=threshold]

    #Processed corpus
    processed_corpus = [[x for x in el if x not in tokens_to_drop] for el in corpus]

    #Dictionary
    dictionary = gensim.corpora.Dictionary(processed_corpus)
    corpus_bow = [dictionary.doc2bow(x) for x in processed_corpus]

    return([dictionary,corpus_bow,processed_corpus])

#Initial model run to see what comes out.

#Transform corpus to bigrams
transformed_corpus = bigram_transformer[clean_corpus]

corpora_to_process = {str(x):remove_words_below_threshold(transformed_corpus,x) for x in [1,2,5,10]}

#Need to turn this into a function.
#Topic modelling

#Parameters for Grid search.
lda_params = list(product([100,200,300],[2,5]))

#Model container
lda_models = []

for x in lda_params:
    #Print stage
    print('{x}_{y}'.format(x=x[0],y=x[1]))

    #Load corpus and dict

    dictionary = corpora_to_process[str(x[1])][0]
    corpus_bow = corpora_to_process[str(x[1])][1]
    corpus = corpora_to_process[str(x[1])][2]

    print('training')
    #Train model
    mod = gensim.models.LdaModel(corpus_bow,num_topics=x[0],id2word=dictionary,
                                 passes=10,iterations=50)

    print('coherence')
    #Extract coherence
    cm = CoherenceModel(mod,texts=corpus,
                             dictionary=dictionary,coherence='u_mass')

    #Get value
    try:
        coherence_value = cm.get_coherence()
    except:
        print('coherence_error')
        coherence_value='error'

    lda_models.append([x,mod,[coherence_value,cm]])

with open(mod_path+'/{t}_ai_topic_models.p'.format(t=today_str),'wb') as outfile:
    pickle.dump(lda_models,outfile)

#Visualiase model performance

model_eval = pd.DataFrame([[x[0][0],x[0][1],x[2][0]] for x in lda_models],columns=['topics','word_lim','coherence'])

fig,ax = plt.subplots(figsize=(10,5))

cols = ['red','green','blue']
legs = []

for num,x in enumerate(set(model_eval['word_lim'])):

    subset = model_eval.loc[[z == x  for z in model_eval['word_lim']],:]

    ax.plot(subset.loc[:,'topics'],subset.loc[:,'coherence'],color=cols[num-1])

    legs.append([cols[num-1],x])

ax.legend(labels=[x[1] for x in legs],title='Min word count')
ax.set_title('Model performance with different parameters')

with open(mod_path+'/19_8_2017_ai_topic_models.p','rb') as infile:
    lda_models = pickle.load(infile)

check_model= lda_models[1][1]

#Explore topics via LDAvis
import pyLDAvis.gensim

pyLDAvis.enable_notebook()
pyLDAvis.gensim.prepare(
    #Insert best model/corpus/topics here
    check_model,
    corpora_to_process[str(5)][1],
    corpora_to_process[str(5)][0])

#Can we extract the relevant terms for the topics as in Sievert and Shirley in order to name them?

#First - create a matrix with top 30 terms per topic
top_30_kws = [check_model.get_topic_terms(topicid=n,topn=1000) for n in np.arange(0,100)]

#Keyword df where the columns are tokens and the rows are topics
top_30_kws_df = pd.concat([pd.DataFrame([x[1] for x in el],
                                        index=[x[0] for x in el]) for el in top_30_kws],
                          axis=1).fillna(0).T.reset_index(drop=True)

#This is the dictionary
selected_dictionary = corpora_to_process[str(5)][0]

#Total number of terms in the document
total_terms = np.sum([vals for vals in selected_dictionary.dfs.values()])

#Appearances of different terms
document_freqs = pd.Series([v for v in selected_dictionary.dfs.values()],
                           index=[k for k in selected_dictionary.dfs.keys()])[top_30_kws_df.columns]/total_terms

#Normalise the terms (divide the vector of probabilities of each keywords in each topic by the totals)
top_30_kws_normalised = top_30_kws_df.apply(lambda x: x/document_freqs,axis=0)

#Now we want to extract, for each topic, the relevance score.

def relevance_score(prob_in_topic,prob_in_corpus,id2word_lookup,lambda_par = 0.6):
    '''
    Combines the probabilities using the definition in Sievert and Shirley and returns the top 5 named
    #terms for each topic
    '''
    #Create dataframe
    combined = pd.concat([prob_in_topic,prob_in_corpus],axis=1)

    combined.columns=['prob_in_topic','prob_in_corpus']

    #Create relevance metric
    combined['relevance'] = lambda_par*combined['prob_in_topic'] + (1-lambda_par)*combined['prob_in_corpus']

    #Top words
    top_ids = list(combined.sort_values('relevance',ascending=False).index[:5])

    #Top words
    top_words = "_".join([id2word_lookup[this_id] for this_id in top_ids])

    return(top_words)

relevance_scores = [relevance_score(top_30_kws_df.iloc[n,:],
                                   top_30_kws_normalised.iloc[n,:],
                                   dictionary.id2token,lambda_par=0.6) for n in np.arange(len(top_30_kws_df))]

%%time
#Create a df with the topic predictions.
paper_preds = check_model[corpora_to_process[str(5)][1]]

paper_topics_df = pd.concat([pd.DataFrame([x[1] for x in el],index=[x[0] for x in el]) for el in paper_preds],
                           axis=1).T

#Replace NAs with zeros and drop pointless index
paper_topics_df.fillna(value=0,inplace=True)
paper_topics_df.reset_index(drop=True,inplace=True)

paper_topics_df.columns = relevance_scores

paper_topics_df.to_csv(int_data+'/{t}_paper_topic_mix.csv'.format(t=today_str),index=False)

#paper_topics_df = pd.read_csv(int_data+'/{t}_paper_topic_mix.csv')

#Quick test of Deep learning papers

#These are papers with a topic that seems to capture deep learning
dl_papers = [x>0.05 for x in paper_topics_df['network_training_model_deep_deep_learning']]

dl_papers_metadata = pd.concat([pd.Series(dl_papers),all_papers],axis=1)

paper_frequencies = pd.crosstab(dl_papers_metadata.year_published,dl_papers_metadata[0])

paper_frequencies.columns=['no_dl','dl']

fig,ax = plt.subplots(figsize=(10,5))

paper_frequencies.plot.bar(stacked=True,ax=ax)
ax.set_title('Number of papers in the DL \'topic\'')
ax.legend(labels=['Not ANN/DL related','NN/DL topic >0.05'])

Some of this is interesting. Doesn't seem to be picking up the policy related terms (safety, discrimination)
Next stages - focus on policy related terms. Can we look for papers in keyword dictionaries identified through the word embeddings?
Obtain Google Scholar data

#How many authors are there in the data? Can we collect all their institutions from Google Scholar

paper_authors = pd.Series([x for el in all_papers['authors'] for x in el.split(", ")])

paper_authors_unique = paper_authors.drop_duplicates()

len(paper_authors_unique)

We have 68,000 authors. It might take a while to get their data from Google Scholar

#Top authors and frequencies

authors_freq = paper_authors.value_counts()

fig,ax=plt.subplots(figsize=(10,3))

ax.hist(authors_freq,bins=30)
ax.set_title('Distribution of publications')

#Pretty skewed distribution!
print(authors_freq.describe())

np.sum(authors_freq>2)

Less than 10,000 authors with 3+ papers in the data

get_scholar_data(

%%time
#Test run
import scholarly

@ratelim.patient(max_calls=30,time_interval=60)
def get_scholar_data(scholarly_object):
    ''''''
    try:
        scholarly_object = next(scholarly_object)
        metadata = {}
        metadata['name']=scholarly_object.name
        metadata['affiliation'] = scholarly_object.affiliation
        metadata['interests'] = scholarly_object.interests
        return(metadata)

    except:
        return('nothing')

#Extract information from each query (it is a generator)
#Get data

#ml_author_gscholar=[]

for num,x in enumerate(paper_authors_unique[1484:]):
    if num % 100 == 0:
        print(str(num)+":"+x)

    result = get_scholar_data(scholarly.search_author(x))
    ml_author_gscholar.append(result)

len(ml_author_gscholar)

markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code


pomegranate and parallelization
pomegranate supports parallelization through a set of built in functions based off of joblib. All computationally intensive functions in pomegranate are implemented in cython with the global interpreter lock (GIL) released, allowing for multithreading to be used for efficient parallel processing. The following functions can be called for parallelization:

fit
summarize
predict
predict_proba
predict_log_proba
log_probability
probability

These functions can all be simply parallelized by passing in n_jobs=X to the method calls. This tutorial will demonstrate how to use those calls. First we'll look at a simple multivariate Gaussian mixture model, and compare its performance to sklearn. Then we'll look at a hidden Markov model with Gaussian emissions, and lastly we'll look at a mixture of Gaussian HMMs. These can all utilize the build-in parallelization that pomegranate has.
Let's dive right in!

%pylab inline
from sklearn.mixture import GaussianMixture
from pomegranate import *
import seaborn, time
seaborn.set_style('whitegrid')

def create_dataset(n_samples, n_dim, n_classes, alpha=1):
    """Create a random dataset with n_samples in each class."""

    X = numpy.concatenate([numpy.random.normal(i*alpha, 1, size=(n_samples, n_dim)) for i in range(n_classes)])
    y = numpy.concatenate([numpy.zeros(n_samples) + i for i in range(n_classes)])
    idx = numpy.arange(X.shape[0])
    numpy.random.shuffle(idx)
    return X[idx], y[idx]

1. General Mixture Models
pomegranate has a very efficient implementation of mixture models, particularly Gaussian mixture models. Lets take a look at how fast pomegranate is versus sklearn, and then see how much faster parallelization can get it to be.

n, d, k = 1000000, 5, 3
X, y = create_dataset(n, d, k)

print "sklearn GMM"
%timeit GaussianMixture(n_components=k, covariance_type='full', max_iter=15, tol=1e-10).fit(X)
print
print "pomegranate GMM"
%timeit GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, k, X, max_iterations=15, stop_threshold=1e-10)
print
print "pomegranate GMM (4 jobs)"
%timeit GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, k, X, n_jobs=4, max_iterations=15, stop_threshold=1e-10)

It looks like on a large dataset not only is pomegranate faster than sklearn at performing 15 iterations of EM on 3 million 5 dimensional datapoints with 3 clusters, but the parallelization is able to help in speeding things up.
Lets now take a look at the time it takes to make predictions using GMMs. Lets fit the model to a small amount of data, and then predict a larger amount of data drawn from the same underlying distributions.

d, k = 25, 2
X, y = create_dataset(1000, d, k)
a = GaussianMixture(k, n_init=1, max_iter=25).fit(X)
b = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, k, X, max_iterations=25)

del X, y
n = 1000000
X, y = create_dataset(n, d, k)

print "sklearn GMM"
%timeit -n 1 a.predict_proba(X)
print
print "pomegranate GMM"
%timeit -n 1 b.predict_proba(X)
print
print "pomegranate GMM (4 jobs)"
%timeit -n 1 b.predict_proba(X, n_jobs=4)

It looks like pomegranate can be slightly slower than sklearn when using a single processor, but that it can be parallelized to get faster performance. At the same time, predictions at this level happen so quickly (millions per second) that this may not be the most reliable test for parallelization.
To ensure that we're getting the exact same results just faster, lets subtract the predictions from each other and make sure that the sum is equal to 0.

print (b.predict_proba(X) - b.predict_proba(X, n_jobs=4)).sum()

Great, no difference between the two.
Lets now make sure that pomegranate and sklearn are learning basically the same thing. Lets fit both models to some 2 dimensional 2 component data and make sure that they both extract the underlying clusters by plotting them.

d, k = 2, 2
X, y = create_dataset(1000, d, k, alpha=2)
a = GaussianMixture(k, n_init=1, max_iter=25).fit(X)
b = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, k, X, max_iterations=25)

y1, y2 = a.predict(X), b.predict(X)

plt.figure(figsize=(16,6))
plt.subplot(121)
plt.title("sklearn clusters", fontsize=14)
plt.scatter(X[y1==0, 0], X[y1==0, 1], color='m', edgecolor='m')
plt.scatter(X[y1==1, 0], X[y1==1, 1], color='c', edgecolor='c')

plt.subplot(122)
plt.title("pomegranate clusters", fontsize=14)
plt.scatter(X[y2==0, 0], X[y2==0, 1], color='m', edgecolor='m')
plt.scatter(X[y2==1, 0], X[y2==1, 1], color='c', edgecolor='c')

It looks like we're getting the same basic results for the two. The two algorithms are initialized a bit differently, and so it can be difficult to directly compare the results between them, but it looks like they're getting roughly the same results.
3. Multivariate Gaussian HMM
Now let's move on to training a hidden Markov model with multivariate Gaussian emissions with a diagonal covariance matrix. We'll randomly generate some Gaussian distributed numbers and use pomegranate with either one or four threads to fit our model to the data.

X = numpy.random.randn(1000, 500, 50)

print "pomegranate Gaussian HMM (1 job)"
%timeit -n 1 -r 1 HiddenMarkovModel.from_samples(NormalDistribution, 5, X, max_iterations=5)
print
print "pomegranate Gaussian HMM (2 jobs)"
%timeit -n 1 -r 1 HiddenMarkovModel.from_samples(NormalDistribution, 5, X, max_iterations=5, n_jobs=2)
print
print "pomegranate Gaussian HMM (2 jobs)"
%timeit -n 1 -r 1 HiddenMarkovModel.from_samples(NormalDistribution, 5, X, max_iterations=5, n_jobs=4)

All we had to do was pass in the n_jobs parameter to the fit function in order to get a speed improvement. It looks like we're getting a really good speed improvement, as well! This is mostly because the HMM algorithms perform a lot more operations than the other models, and so spend the vast majority of time with the GIL released. You may not notice as strong speedups when using a MultivariateGaussianDistribution because BLAS uses multithreaded operations already internally, even when only one job is specified.
Now lets look at the prediction function to make sure the we're getting speedups there as well. You'll have to use a wrapper function to parallelize the predictions for a HMM because it returns an annotated sequence rather than a single value like a classic machine learning model might.

model = HiddenMarkovModel.from_samples(NormalDistribution, 5, X, max_iterations=2, verbose=False)

print "pomegranate Gaussian HMM (1 job)"
%timeit predict_proba(model, X)
print
print "pomegranate Gaussian HMM (2 jobs)"
%timeit predict_proba(model, X, n_jobs=2)

Great, we're getting a really good speedup on that as well! Looks like the parallel processing is more efficient with a bigger, more complex model, than with a simple one. This can make sense, because all inference/training is more complex, and so there is more time with the GIL released compared to with the simpler operations.
4. Mixture of Hidden Markov Models
Let's stack another layer onto this model by making it a mixture of these hidden Markov models, instead of a single one. At this point we're sticking a multivariate Gaussian HMM into a mixture and we're going to train this big thing in parallel.

def create_model(mus):
    n = mus.shape[0]

    starts = numpy.zeros(n)
    starts[0] = 1.

    ends = numpy.zeros(n)
    ends[-1] = 0.5

    transition_matrix = numpy.zeros((n, n))
    distributions = []

    for i in range(n):
        transition_matrix[i, i] = 0.5

        if i < n - 1:
            transition_matrix[i, i+1] = 0.5

        distribution = IndependentComponentsDistribution([NormalDistribution(mu, 1) for mu in mus[i]])
        distributions.append(distribution)

    model = HiddenMarkovModel.from_matrix(transition_matrix, distributions, starts, ends)
    return model

def create_mixture(mus):
    hmms = [create_model(mu) for mu in mus]
    return GeneralMixtureModel(hmms)

n, d = 50, 10
mus = [(numpy.random.randn(d, n)*0.2 + numpy.random.randn(n)*2).T for i in range(2)]

model = create_mixture(mus)
X = numpy.random.randn(400, 150, d)

print "pomegranate Mixture of Gaussian HMMs (1 job)"
%timeit model.fit(X, max_iterations=5)
print

model = create_mixture(mus)
print "pomegranate Mixture of Gaussian HMMs (2 jobs)"
%timeit model.fit(X, max_iterations=5, n_jobs=2)

Looks like we're getting a really nice speed improvement when training this complex model. Let's take a look now at the time it takes to do inference with it.

model = create_mixture(mus)

print "pomegranate Mixture of Gaussian HMMs (1 job)"
%timeit model.predict_proba(X)
print

model = create_mixture(mus)
print "pomegranate Mixture of Gaussian HMMs (2 jobs)"
%timeit model.predict_proba(X, n_jobs=2)

We're getting a good speed improvement here too through parallelization.
Conclusions
Hopefully you'll find pomegranate useful in your work! Parallelization should allow you to train complex models faster than before. Keep in mind though that there is an overhead to using parallel processing, and so it's possible that on some smaller examples it does not work as well. In general, the bigger the dataset, the closer to a linear speedup you'll get with pomegranate.
If you have any interesting examples of how you've used pomegranate in your work, I'd love to hear about them. In addition I'd like to hear any feedback you may have on features you'd like to see. Please shoot me an email. Good luck!

markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown


T81-558: Applications of Deep Neural Networks
Class 4: Classification and Regression
* Instructor: Jeff Heaton, School of Engineering and Applied Science, Washington University in St. Louis
* For more information visit the class website.
Binary Classification, Classification and Regression

Binary Classification - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).
Classification - Classification between more than 2.  The iris dataset (3-way classification).
Regression - Numeric prediction.  How many MPG does a car get?

In this class session we will look at some visualizations for all three.
Feature Vector Encoding
These are exactly the same feature vector encoding functions from Class 3.  They must be defined for this class as well.  For more information, refer to class 3.

from sklearn import preprocessing
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)
def encode_text_dummy(df,name):
    dummies = pd.get_dummies(df[name])
    for x in dummies.columns:
        dummy_name = "{}-{}".format(name,x)
        df[dummy_name] = dummies[x]
    df.drop(name, axis=1, inplace=True)

# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).
def encode_text_index(df,name):
    le = preprocessing.LabelEncoder()
    df[name] = le.fit_transform(df[name])
    return le.classes_

# Encode a numeric column as zscores
def encode_numeric_zscore(df,name,mean=None,sd=None):
    if mean is None:
        mean = df[name].mean()

    if sd is None:
        sd = df[name].std()

    df[name] = (df[name]-mean)/sd

# Convert all missing values in the specified column to the median
def missing_median(df, name):
    med = df[name].median()
    df[name] = df[name].fillna(med)

# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs
def to_xy(df,target):
    result = []
    for x in df.columns:
        if x != target:
            result.append(x)

    # find out the type of the target column.  Is it really this hard? :(
    target_type = df[target].dtypes
    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type

    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.
    if target_type in (np.int64, np.int32):
        # Classification
        return df.as_matrix(result).astype(np.float32),df.as_matrix([target]).astype(np.int32)
    else:
        # Regression
        return df.as_matrix(result).astype(np.float32),df.as_matrix([target]).astype(np.float32)

# Nicely formatted time string
def hms_string(sec_elapsed):
    h = int(sec_elapsed / (60 * 60))
    m = int((sec_elapsed % (60 * 60)) / 60)
    s = sec_elapsed % 60
    return "{}:{:>02}:{:>05.2f}".format(h, m, s)

Toolkit: Visualization Functions
This class will introduce 3 different visualizations that can be used with the two different classification type neural networks and regression neural networks.

Confusion Matrix - For any type of classification neural network.
ROC Curve - For binary classification.
Lift Curve - For regression neural networks.

The code used to produce these visualizations is shown here:

%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc

# Plot a confusion matrix.
# cm is the confusion matrix, names are the names of the classes.
def plot_confusion_matrix(cm, names, title='Confusion matrix', cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(names))
    plt.xticks(tick_marks, names, rotation=45)
    plt.yticks(tick_marks, names)
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Plot an ROC. pred - the predictions, y - the expected output.
def plot_roc(pred,y):
    fpr, tpr, _ = roc_curve(y_test, pred)
    roc_auc = auc(fpr, tpr)

    plt.figure()
    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC)')
    plt.legend(loc="lower right")
    plt.show()

# Plot a lift curve.  pred - the predictions, y - the expected output.
def chart_regression(pred,y):
    t = pd.DataFrame({'pred' : pred.flatten(), 'y' : y_test.flatten()})
    t.sort_values(by=['y'],inplace=True)

    a = plt.plot(t['y'].tolist(),label='expected')
    b = plt.plot(t['pred'].tolist(),label='prediction')
    plt.ylabel('output')
    plt.legend()
    plt.show()

Binary Classification
Binary classification is used to create a model that classifies between only two classes.  These two classes are often called "positive" and "negative".  Consider the following program that uses the wcbreast_wdbc dataset to classify if a breast tumor is cancerous (malignant) or not (benign).  The iris dataset is not binary, because there are three classes (3 types of iris).

import os
import pandas as pd
from sklearn.cross_validation import train_test_split
import tensorflow.contrib.learn as skflow
import numpy as np
from sklearn import metrics

path = "./data/"

filename = os.path.join(path,"wcbreast_wdbc.csv")
df = pd.read_csv(filename,na_values=['NA','?'])

# Encode feature vector
df.drop('id',axis=1,inplace=True)
encode_numeric_zscore(df,'mean_radius')
encode_text_index(df,'mean_texture')
encode_text_index(df,'mean_perimeter')
encode_text_index(df,'mean_area')
encode_text_index(df,'mean_smoothness')
encode_text_index(df,'mean_compactness')
encode_text_index(df,'mean_concavity')
encode_text_index(df,'mean_concave_points')
encode_text_index(df,'mean_symmetry')
encode_text_index(df,'mean_fractal_dimension')
encode_text_index(df,'se_radius')
encode_text_index(df,'se_texture')
encode_text_index(df,'se_perimeter')
encode_text_index(df,'se_area')
encode_text_index(df,'se_smoothness')
encode_text_index(df,'se_compactness')
encode_text_index(df,'se_concavity')
encode_text_index(df,'se_concave_points')
encode_text_index(df,'se_symmetry')
encode_text_index(df,'se_fractal_dimension')
encode_text_index(df,'worst_radius')
encode_text_index(df,'worst_texture')
encode_text_index(df,'worst_perimeter')
encode_text_index(df,'worst_area')
encode_text_index(df,'worst_smoothness')
encode_text_index(df,'worst_compactness')
encode_text_index(df,'worst_concavity')
encode_text_index(df,'worst_concave_points')
encode_text_index(df,'worst_symmetry')
encode_text_index(df,'worst_fractal_dimension')
diagnosis = encode_text_index(df,'diagnosis')
num_classes = len(diagnosis)

# Create x & y for training

# Create the x-side (feature vectors) of the training
x, y = to_xy(df,'diagnosis')

# Split into train/test
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.25, random_state=42)

# Create a deep neural network with 3 hidden layers of 10, 20, 10
classifier = skflow.TensorFlowDNNClassifier(hidden_units=[10, 20, 10], n_classes=num_classes,
    steps=10000)

# Early stopping
early_stop = skflow.monitors.ValidationMonitor(x_test, y_test,
    early_stopping_rounds=200, print_steps=50, n_classes=num_classes)

# Fit/train neural network
classifier.fit(x_train, y_train, early_stop)

# Measure accuracy
score = metrics.accuracy_score(y, classifier.predict(x))
print("Final accuracy: {}".format(score))

Confusion Matrix
The confusion matrix is a common visualization for both binary and larger classification problems.  Often a model will have difficulty differentiating between two classes.  For example, a neural network might be really good at telling the difference between cats and dogs, but not so good at telling the difference between dogs and wolves.  The following code generates a confusion matrix:

import numpy as np

from sklearn import svm, datasets
from sklearn.cross_validation import train_test_split
from sklearn.metrics import confusion_matrix

pred = classifier.predict(x_test)

# Compute confusion matrix
cm = confusion_matrix(y_test, pred)
np.set_printoptions(precision=2)
print('Confusion matrix, without normalization')
print(cm)
plt.figure()
plot_confusion_matrix(cm, diagnosis)

# Normalize the confusion matrix by row (i.e by the number of samples
# in each class)
cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
print('Normalized confusion matrix')
print(cm_normalized)
plt.figure()
plot_confusion_matrix(cm_normalized, diagnosis, title='Normalized confusion matrix')

plt.show()

The above two confusion matrixes show the same network.  The bottom (normalized) is the type you will normally see.  Notice the two labels.  The label "B" means benign (no cancer) and the label "M" means malignant (cancer).  The left-right (x) axis are the predictions, the top-bottom) are the expected outcomes.  A perfect model (that never makes an error) has a dark blue diagonal that runs from top-left to bottom-right.
To read, consider the top-left square.  This square indicates "true labeled" of B and also "predicted label" of B.  This is good! The prediction matched the truth.  The blueness of this box represents how often "B" is classified correct.  It is not darkest blue.  This is because the square to the right(which is off the perfect diagonal) has some color.  This square indicates truth of "B" but prediction of "M". The white square, at the bottom-left, indicates a true of "M" but predicted of "B".  The whiteness indicates this rarely happens.
Your conclusion from the above chart is that the model sometimes classifies "B" as "M" (a false negative), but never mis-classifis "M" as "B".  Always look for the dark diagonal, this is good!
ROC Curves
ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).
Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:

False Positive - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.
False Negative - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.
True Positive - Your test (neural network) correctly identified that the patient had the disease.
True Negative - Your test (neural network) correctly identified that the patient did not have the disease.

Types of errors:

Neural networks classify in terms of probbility of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensative or specific:

The following shows a more sensitive cutoff:

An ROC curve measures how good a model is regardless of the cutoff. The following shows how to read a ROC chart:

The following code shows an ROC chart for the breast cancer neural network.  The area under the curve (AUC) is also an important measure.  The larger the AUC, the better.

pred = classifier.predict_proba(x_test)
pred = pred[:,1] # Only positive cases
# print(pred[:,1])
plot_roc(pred,y_test)

Classification
We've already seen multi-class classification, with the iris dataset.  Confusion matrixes work just fine with 3 classes.  The following code generates a confusion matrix for iris.

import os
import pandas as pd
from sklearn.cross_validation import train_test_split
import tensorflow.contrib.learn as skflow
import numpy as np

path = "./data/"

filename = os.path.join(path,"iris.csv")
df = pd.read_csv(filename,na_values=['NA','?'])

# Encode feature vector
encode_numeric_zscore(df,'petal_w')
encode_numeric_zscore(df,'petal_l')
encode_numeric_zscore(df,'sepal_w')
encode_numeric_zscore(df,'sepal_l')
species = encode_text_index(df,"species")
num_classes = len(species)

# Create x & y for training

# Create the x-side (feature vectors) of the training
x, y = to_xy(df,'species')

# Split into train/test
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.25, random_state=45)
    # as much as I would like to use 42, it gives a perfect result, and a boring confusion matrix!

# Create a deep neural network with 3 hidden layers of 10, 20, 10
classifier = skflow.TensorFlowDNNClassifier(hidden_units=[10, 20, 10], n_classes=num_classes,
    steps=10000)

# Early stopping
early_stop = skflow.monitors.ValidationMonitor(x_test, y_test,
    early_stopping_rounds=200, print_steps=50, n_classes=num_classes)

# Fit/train neural network
classifier.fit(x_train, y_train, early_stop)

import numpy as np

from sklearn import svm, datasets
from sklearn.cross_validation import train_test_split
from sklearn.metrics import confusion_matrix

pred = classifier.predict(x_test)

# Compute confusion matrix
cm = confusion_matrix(y_test, pred)
np.set_printoptions(precision=2)
print('Confusion matrix, without normalization')
print(cm)
plt.figure()
plot_confusion_matrix(cm, species)

# Normalize the confusion matrix by row (i.e by the number of samples
# in each class)
cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
print('Normalized confusion matrix')
print(cm_normalized)
plt.figure()
plot_confusion_matrix(cm_normalized, species, title='Normalized confusion matrix')

plt.show()

See the strong diagonal?  Iris is easy.  See the light blue near the bottom?  Sometimes virginica is confused for versicolor.
Regression
We've already seen regression with the MPG dataset.  Regression uses its own set of visualizations, one of the most common is the lift chart.  The following code generates a lift chart.

import tensorflow.contrib.learn as skflow
import pandas as pd
import os
import numpy as np
from sklearn import metrics
from scipy.stats import zscore

path = "./data/"

filename_read = os.path.join(path,"auto-mpg.csv")
df = pd.read_csv(filename_read,na_values=['NA','?'])

# create feature vector
missing_median(df, 'horsepower')
df.drop('name',1,inplace=True)
encode_numeric_zscore(df, 'horsepower')
encode_numeric_zscore(df, 'weight')
encode_numeric_zscore(df, 'cylinders')
encode_numeric_zscore(df, 'displacement')
encode_numeric_zscore(df, 'acceleration')
encode_text_dummy(df, 'origin')

# Encode to a 2D matrix for training
x,y = to_xy(df,['mpg'])

# Split into train/test
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.25, random_state=42)

# Create a deep neural network with 3 hidden layers of 50, 25, 10
regressor = skflow.TensorFlowDNNRegressor(hidden_units=[50, 25, 10], steps=5000)

# Early stopping
early_stop = skflow.monitors.ValidationMonitor(x_test, y_test,
    early_stopping_rounds=200, print_steps=50)

# Fit/train neural network
regressor.fit(x_train, y_train, early_stop)

pred = regressor.predict(x_test)

chart_regression(pred,y_test)

To generate a lift chart, perform the following activities:

Sort the data by expected output.  Plot the blue line above.
For every point on the x-axis plot the predicted value for that same data point.  This is the green line above.
The x-axis is just 0 to 100% of the dataset.  The expected always starts low and ends high.
The y-axis is ranged according to the values predicted.

Reading a lift chart:
* The expected and predict lines should be close.  Notice where one is above the ot other.
* The above chart is the most accurate on lower MPG.

markdown code markdown code markdown code markdown code markdown code markdown code markdown code markdown


Linear Programming with OR-Tools
In this notebook, we do some basic LP solving with Google's OR-Tools. Problems used will be examples in Hamdy Taha's Operations Research: An Introduction, 9th Edition, which I have in paperback.

from ortools.linear_solver import pywraplp

Reddy Mikks model
Given the following variables:
$\begin{aligned}
x_1 = \textrm{Tons of exterior paint produced daily} \newline
x_2 = \textrm{Tons of interior paint produced daily}
\end{aligned}$
and knowing that we want to maximize the profit, where \$5000 is the profit from exterior paint and \$4000 is the profit from a ton of interior paint, the Reddy Mikks model is:
$$\textrm{Maximize  } z = 5x_1 + 4x_2$$
subject to
$$6x_1 + 4x_2 \le 24$$
$$x_1 + 2x_2 \le 6$$
$$-x_1 + x_2 \le 1$$
$$x_2 \le 2$$
$$x_1, x_2 \ge 0$$

reddymikks = pywraplp.Solver('Reddy_Mikks', pywraplp.Solver.GLOP_LINEAR_PROGRAMMING)

x1 = reddymikks.NumVar(0, reddymikks.infinity(), 'x1')
x2 = reddymikks.NumVar(0, reddymikks.infinity(), 'x2')

reddymikks.Add(6*x1 + 4*x2 <= 24)
reddymikks.Add(x1 + 2*x2 <= 6)
reddymikks.Add(-x1 + x2 <= 1)
reddymikks.Add(x2 <= 2)

profit = reddymikks.Objective()
profit.SetCoefficient(x1, 5)
profit.SetCoefficient(x2, 4)
profit.SetMaximization()

status = reddymikks.Solve()

if status not in [reddymikks.OPTIMAL, reddymikks.FEASIBLE]:
    raise Exception('No feasible solution found')

print("The company should produce",round(x1.solution_value(),2),"tons of exterior paint")
print("The company should produce",round(x2.solution_value(),2),"tons of interior paint")
print("The optimal profit is", profit.Value(), 'thousand USD')

More simple problems
A company that operates 10 hours a day manufactures two products on three sequential processes. The following data characterizes the problem:

import pandas as pd

problemdata = pd.DataFrame({'Process 1': [10, 5], 'Process 2':[6, 20], 'Process 3':[8, 10], 'Unit profit':[20, 30]})
problemdata.index = ['Product 1', 'Product 2']

problemdata

Where there are 10 hours a day dedicated to production. Process times are given in minutes per unit while profit is given in USD.
The optimal mix of the two products would be characterized by the following model:
$\begin{aligned}
x_1 = \textrm{Units of product 1} \newline
x_2 = \textrm{Units of product 2}
\end{aligned}$
$$\textrm{Maximize  } z = 20x_1 + 30x_2$$
subject to
$$\begin{array}{rcl}
10x_1 + 5x_2 \le 600 \newline
6x_1 + 20x_2 \le 600 \newline
8x_1 + 10x_2 \le 600 \newline
x_1, x_2 \ge 0
\end{array}$$
(we will assume that continuous solution values are acceptable for this problem)

simpleprod = pywraplp.Solver('Simple_Production', pywraplp.Solver.GLOP_LINEAR_PROGRAMMING)

x1 = simpleprod.NumVar(0, simpleprod.infinity(), 'x1')
x2 = simpleprod.NumVar(0, simpleprod.infinity(), 'x2')

for i in problemdata.columns[:-1]:
    simpleprod.Add(problemdata.loc[problemdata.index[0], i]*x1 + problemdata.loc[problemdata.index[1], i]*x2 <= 600)

profit = simpleprod.Objective()
profit.SetCoefficient(x1, 20)
profit.SetCoefficient(x2, 30)
profit.SetMaximization()

status = simpleprod.Solve()

if status not in [simpleprod.OPTIMAL, simpleprod.FEASIBLE]:
    raise Exception('No feasible solution found')

print("The company should produce",round(x1.solution_value(),2),"units of product 1")
print("The company should produce",round(x2.solution_value(),2),"units of product 2")
print("The optimal profit is", round(profit.Value(),2), 'USD')

markdown code markdown code markdown code markdown code


Fake Driving School pussy creampie and anal sex for busty milf in threesome - 13 min


Deepthroat Cum Slut Whore Getting Used to Extreme


contact / help

Track Name: burger king (what is this, democracy or uhhh... the other thing)

driving my car. thinking about you. all hopes are laid to rest but we'll all be with them in the end.

Track Name: ...and now it's perfect for yoga!

everyone you love is crumbling down around at your feet. everyone you hate is rising up in success around you. sixteen year old girls with petty problems, stupid drama. i don't give a fuck, i left you long ago. say it's unfair. scream it's unfair. and it's so sad. but i guess that's life. darling, i know we'll make it someday. but until then, we'll have to write it right.

Track Name: turnover

turnover. rollover. i'm so dumb. you're so cute. you call me sweet. i call you beautiful. you are everything i wanted. you are everything to me. you are everything i wanted. you are everything to me.

Track Name: green

fuck this.

Track Name: why can't i be snowing?

and i listened to pump fake 3000 times and wished i could find my place, my generation.
sitting home alone after a fucking shitty show where no one showed up like they said they would.
and now we're grown ups. we're fucking grown up. with a bummed out swan song wasted on closed ears.
fuck this fuck everything no one even cares i wish they would care.
i hope that i die so people might listen to the songs i write, or wrote.

Track Name: intro

fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you fuck you


MILF Tits - This is a real MILF mom showing off her tits.


Chubby Shemale sucks cock and gets cumshot after ass fucking

Link to this video BB code


Fuck Me Daddy, Fuck Me In My Ass I Like It!

Fuck Me Daddy, Fuck Me In My Ass I Like It!


Round MILF Bum

MILF feeling sexy and sticking out her ass.


Best of femdom handjob by troc


Input: [1] Quad core, 1400 MHz, ARM Cortex-A9 [2] Dual core, 1500 MHz, Krait [Hardware]

Output: Processor Processor - The processor is the main computing component of a phone and is a major factor when it comes to the overall speed of the device. Some more powerful smartphones use dual-core and quad-core processors designed to deliver greater performance.

Fujitsu STYLISTIC M532 vs Samsung Galaxy Tab 2 10.1 - Phone specs comparison


Input: [1] Yes [2] Adreno 225 [Hardware]

Output: Graphics processor

Fujitsu STYLISTIC M532 vs Samsung Galaxy Tab 2 10.1 - Phone specs comparison


Input: [1] 1024 MB RAM [2] 1024 MB RAM [Hardware]

Output: System memory System memory - System memory, or RAM memory is the type of memory that the device uses to temporarily store data from the OS or currently-running apps. The more RAM available to the device, the better the performance will be when multiple or heavier programs are running.

Fujitsu STYLISTIC M532 vs Samsung Galaxy Tab 2 10.1 - Phone specs comparison


Input: [1] 32 GB [2] 8 GB [Hardware]

Output: Built-in storage

Fujitsu STYLISTIC M532 vs Samsung Galaxy Tab 2 10.1 - Phone specs comparison


Input: [1] microSD, microSDHC [2] microSD, microSDHC, microSDXC up to 64 GB [Hardware]

Output: Storage expansion

Fujitsu STYLISTIC M532 vs Samsung Galaxy Tab 2 10.1 - Phone specs comparison


Input: [1] Dual core, 1500 MHz [2] Quad core, 1500 MHz, Krait [Hardware]

Output: Processor Processor - The processor is the main computing component of a phone and is a major factor when it comes to the overall speed of the device. Some more powerful smartphones use dual-core and quad-core processors designed to deliver greater performance.

Motorola ELECTRIFY M vs HTC DROID DNA - Phone specs comparison


Input: [1] Yes [2] Adreno 320 [Hardware]

Output: Graphics processor

Motorola ELECTRIFY M vs HTC DROID DNA - Phone specs comparison


Input: [1] 1024 MB RAM [2] 2048 MB RAM [Hardware]

Output: System memory System memory - System memory, or RAM memory is the type of memory that the device uses to temporarily store data from the OS or currently-running apps. The more RAM available to the device, the better the performance will be when multiple or heavier programs are running.

Motorola ELECTRIFY M vs HTC DROID DNA - Phone specs comparison


FIG.4 EFFECT OF GLYCEROL ON FORMALDEHYDE DELIVERY TO SIDESTREAM AND MAINSTREAM SMOKE FORMAL?EEYDE DELIVERY (ug/cig.} Jjoe 180 { a -QQ: @ s m { | i | ! 1 3.4 5.4 288 B ss HHiE ms o GLYCEROL (%) IN FILLER I80ETETOSE


PY: 1995 LA: ENGLISH AB: Pulmonary eosinophilic granuloma is a disorder caused by localized collections of proliferating histiocytes in the lung. Little is known about its etiology except that the majority (38 to 97%) of patients are current or ex-smokers, making the potential etiologic role of tobacco products an important area for research. Tobacco glycoprotein (TGP) is a potent immunostimulator that has been isolated from cigarette smoke. TGP-specific lymphocyte proliferation, and cytokine production in vitro, were measured in three patients with pulmonary eosinopbilic granuloma in remission and in three closely matched normal subjects with similar smoking histories. One patient with eosinophilic granuloma of bone and a matched control subject were also studied. Peripheral blood mononuclear cells were cultured with TGP, the recall antigen streptokinase (SK), and the mitogen concanavalin A (Con A). All three of the patients with pulmonary eosinophilic granuloma exhibited significant decreases in lymphocyte stimulation to TGP, despite normal responses to SK and Con A. In contrast, the response of the patient with eosinophilic granuloma of bone was higher than her matched control. The mean responses of the patients with pulmonary eosinophilic granuloma to TGP was significantly lower than the mean of nondiseased smokers or of normal nonsmokers. Twenty-four-hour culture supernatants were collected and assayed for cytokine levels (IL-1, IL-2, and TL- 6). TGP-stimulated IL-2 production was significantly lower in the patients with pulmonary eosinophilic granuloma than in the normal subjects, confirming the reduced T-cell proliferative response.(ABSTRACT TRUNCATED AT 250 WORDS) MESH: Adult-; Analysis-of-Variance; Cell-Division-drug-effects; Cells,-Cultured; Comparative-Study; Eosinophilic-Granuloma-etiology; Female-; Human-; Interleukins-blood; Lung-Diseases-etiology; Lymphocytes-cytology; Male-; Middle-Age; Smoking-blood; Statistics,-Nonparametric; Support,-Non- U.S.-Gov't; Support,-U.S.-Gov't,-P.H.S. MESH: *Eosinophilic-Granuloma-blood; *Glycoproteins-pharmacology; *Lung-Diseases-blood; *Lymphocytes-drug-effects; *Tobacco- PT: JOURNAL-ARTICLE RN: 0 (Glycoproteins); O (Interleukins) ISSN: 1073-449X CO: BZS UDh: 9504 AN: 95111735 Record 21 of 34 TOXLINE 1/94-12/97 SF: TOXBIB TI: The role of airway mucus in pulmonary toxicology. AU: Samet-JM; Cheng-PW AD: Center for Environmental Medicine and Lung Biology, University of North Carolina at Chapel Hill, SO: Environ-Health-Perspect; VOL 102 Suppl 2, 1994, P89-103 (REF: 178) PY: 1994 Support,-U.S.-Gov't,-P.H.S. MESH: *Air-Pollutants-adverse-effects; *Lung-drug-effects; *Lung-Diseases-chemically-induced; *Mucus-physiclogy LA: ENGLISH AB: Airway mucus is a complex airway secretion whose primary function as part of the mucociliary transport mechanism is to to serve as renewable and transportable barrier against inhaled particulates and toxic agents. The rheologic properties necessary for this function are imparted by glycoproteins, or mucins. Some respiratory disease states, e.g., asthma, cystic fibrosis, and bronchitis, are characterized by quantitative and qualitative changes in mucus biosynthesis that contribute to pulmonary pathology. Similar alterations in various aspects of mucin biochemistry and biophysics, leading to mucus hypersecretion and altered mucus rheology, result from inhalation of certain air pollutants, such as ozone, sulfur dioxide, nitrogen dioxide, and cigarette smoke. The consequences of these pollutant-induced alterations in mucus biology are discussed in the context of pulmonary pathophysiology and toxicology. MESH: Animal-; Carbohydrate-Sequence; Human-; Lung-Diseases-physiopathology; Molecular-Sequence- Data; Mucus-chemistry; Mucus-cytology; Support,-Non-U.S.-Gov't; Support,-U.S.-Gov't,-Non-P.H.S.; 209s2Zreloz


49 CONCEPT AND COMPONENTS OF THE NEW CIGARETTE FIGURE 2.5.2-1 10.8% i N = 4147 % FIBER DIAMETER, MICRONS 5 = 088 2 1 LOWER CELL LIMITS 1 Plot of mean glass fiber diameter from production between November 1987 and March 1988 illustrating the lack of fibers below 3 um. FIGURE 2.5.2- ple size = Mean = Standard deviation CV = Cocfficient of variation X s N = Sam


TREATMENT #6 EMPLOYER OGBURN VERSION LOGOTYPE FOR: R.J. REYNOLDS


If you are trying to understand how Kobid * does not affect fertilization, learn more about the recent study of the effects of Kobid's menstrual cycle.


If the disease is serious, it is unlikely that more people will travel with towels, more likely to get sick in their homes and in their beds? No matter who you sit next to, vaccines will not prevent transmission.


Papa Scepticism stems from the campaign of disinformation. One does not impose like polio or measles. VOCID'T flu 939, 064 dead of him They openly demand, but do not offer real help. They "rekindle chaos." If you do not notice, it was open.


If you don't want others like measles to be unreliable, then maybe you're not trying to destroy people so you don't want another Kobid joke?


JUST you've read it/2022/790,000-- tumor in the neuro secret--up to kovid-19


I watched an interview with an interview, plus interview with the doctor I took off a boo-boo... watch surveillance surveillance surveillance surveillance watch watch watch watch watch watch watch watch watches follow up and follow an interview and fear fear I fear fear fear fear fear fear


Remember that horse? Everyone told you not to use it? The new work on the AMA Daily is much better than the COVID vaccine, and it avoids being killed by COVID.


False. shows that 4% of all covid is reinfection, another study it is 10%, in which 650 000 people used Omicron. I call this not unusual, rarely gets a


It is surprising that so many people like him are congested in this disinformation/cloud on Cowed.


#1 small corded treadmill without remote control </s> sunny health & fitness sf-t1407m foldable manual walking treadmill, gray </s> excellent cardio benefits: this affordable treadmill will help you start your fitness journey! walking can significantly improve mental and physical health. the compact design of the treadmill will get you into shape regardless of home size.compact and ergonomic: this compact, lightweight treadmill is excellent for small spaces. running surface: 42in x 13in. the treadmill is durable and can withstand power walking or light jogging.lcd monitor: track fitness progress on the easy to read digital monitor. treadmill monitor displays time, speed, distance, calories.non-electric: space-saving, treadmill machine requires no electric outlet needed and remains low maintece. easy folding functionality and transportation wheels allow easy mobility and storage.non slip running surface: feel safe and secure on the smooth belt. 220 lbs. maximum user weight </s> sunny </s> gray </s>


#1 small corded treadmill without remote control </s> sunny health & fitness asuna space saving treadmill, motorized with speakers for aux audio connection - 8730g </s> easy-to-use display: watch your workout progress in real-time with the 4-window display. view your time, distance, speed, and calories burned all at the same time. use the integrated pedometer to track your steps during each exercise session.extremely portable: this running treadmill can be easily and securely folded flat with the included folding key. take advantage of the low-profile design, which makes it easy to slide the machine under surfaces and move it from room to room.folding tool: turn the tool counterclockwise in the frame locks to fold the treadmill frame. after it is folded, use the treadmill key to secure the frame in place. keep the tool handy by placing it in its dedicated slot at the base of the treadmill.play your tunes: connect your music device to the onboard 3.5mm headphone port to listen to music through the integrated speakers on the treadmill. mount your favorite devices on the treadmill by using the convenient tablet holder.sturdy build: this treadmill supports a maximum user weight of 220 lbs. reduce impact on your joints as you walk or run when you strike against the shock-absorption-equipped running deck, compared to exercising on concrete or other hard surfaces. </s> sunny health & fitness </s> 8730g </s>


Kelly: Oh! Oh! Can I pick the first question?
Jessica: Sure. Go for it!
Kelly: What's the scariest place you've been to!
Jessica: I'll start: Palermo in Italy.
Mickey: And what's so scary about that? Did you break your nail? :P
Jessica: Shut it, Mickey! No, there are the Capuchin Catacombs with 8000 corpses!
Kelly: Ewwww! Corpses? Rly?
Jessica: Yeah! And you can look at them like museum exhibits. I think they're divided somehow, but have no clue how!
Ollie: That's so cool! Do you get to see the bones or are they covered up?
Jessica: Well, partly. Most of them were exhibited in their clothes. Basically only skulls and hands.
Mickey: I'm writing this one down! That's so precious!
Ollie: Me too!

The scariest place for Jessica was the Capuchin Catacombs in Palermo.


Carrie: Just back from Fantastic Beast :)
Gina: and what do you think?
Carrie: generally good - as usual nice special effect and visuals, an ok plot, a glimpse of the wizarding community in the US.
Alex: Sounds cool. I was thinking of going this weekend with Lane, but I've seen some bad reviews.
Carrie: Depends on what you expect really - I have a lot of sentiment towards Harry Potter so, I'm gonna like everything the do. But seriously the movie was decent. However, if you're expecting to have your mind blown, then no, it's not THAT good.
Gina: I agree. I saw it last week and basically I'm satisfied.
Alex: No spoilers, girls.
Carrie: no worries ;)
Carrie: And Gina, what do you think about Eddie Redmayne as Newt?
Gina: I loved him <3 I loved how introverted and awkward he was and how caring he was towards the animals. And with all that he showed a lot of confidence in his beliefs and was a genuinely compassionate character
Carrie: not your standard protagonist, that's for sure
Gina: and that's what I liked about him
Alex: Maybe I'll go and see it sooner so we can all talk about it.
Carrie: go see it. If' you're not expecting god-knows-what you're going to enjoy it ;)

Carrie and Gina saw "Fantastic Beast" and liked it. Ginna loved Eddie Redmayne as Newt.


Samantha: How are you doing today?
Robyn: better, but I really exaggerated with bier last night
Samantha: was it really only bier?
Robyn: sure, why?
Samantha: I don't know, your eyes, behaviour
Robyn: you want to say I took drugs?
Samantha: I'm only asking, I don't have anything against drugs
Robyn: but I have and I never take them
Samantha: ok, sorry, I didn't intend to offend you
Robyn: I just drank too much, that's eat
Samantha: much too much
Robyn: yes, I lost control a bit
Robyn: I am sorry for that
Samantha: important that you feel better today

Robyn drank too much beer last night and lost control. She is feeling better today.


Peyton: I have been asking you to bring that video game for me
Cameron: Honey, I am not having enough time to come home
Peyton: When would you come home?
Cameron: I will have to stay out of town for another week i guess
Peyton: Cant you just deliver that game through the courier? :P
Cameron: Dont be mean :/
Peyton: Get the job done and come to home then. ASAP :P

Peyton is expecting Cameron to bring the video game. Cameron will probably be out for another week.


Gene: Did you get the package I sent you
Jack: No, when did you send it?
Gene: on Friday
Jack: shit I should have gotten it by now
Jack: send me the tracking umber I;ll check whats up
Gene: 12345678900

Jack has not received yet the package Gene had sent him on Friday. She sent him the tracking number, so he could check the status of the shipment.


Russ: Hi Gurdun, are you feeling better?
Gurdun: Hi Ross, a little bit better, yes, thank you.
Russ: Do you think you'll be back at the uni this week?
Gurdun: Not tomorrow for sure, but I hope Wednesday maybe.
Russ: Oh, that's fine.
Gurdun: Yeah. I'm feeling a bit weak still
Russ: I guess that's pretty normal at this stage.
Gurdun: I believe so. And the muscle pain.
Russ: I know what you mean. I had flu like that three months ago.
Gurdun: Luckily the headache's gone. It was absolutely awful.
Russ: True enough. I had the same.
Gurdun: Anyway, how are things with you?
Russ: Same old, same old, you know. Not much happening in the middle of term.
Gurdun: And the team?
Russ: Oh yeah, we won the last match. It was a good game.
Gurdun: Great. Listen Russ, I need to run. Catch up later.
Russ: OK, talk to you later.

Gurdun has the flu. He is feeling better. Gurdun hopes he'll be back at the uni on Wednesday. Russ and his team won the last match.


Susan: Will you be home tomorrow?
Rob: Why do you ask?
Susan: We could have dinner together if you have time after work
Rob: Sure! Sounds great
Susan: Fantastic! I will cook something special then!

Rob will be home tomorrow. Rob and Susan will have dinner together after Rob's work. Susan will cook something special.


Evan: Hi Dennis, I need to cancel my lesson tomorrow, sorry.
Dennis: Fine, what's up?
Evan: Oh, got the flu, been off college since Tuesday. Should be ok for next Thurs.
Dennis: OK, just tell me in good time if you can't make it. Hope you feel better soon.
Evan: Bye, Den! Thanks.

Evan can't come to his lesson tomorrow, because he's got the flu.


Lorena: Hi, can you help me with something?
Martin: Well, I can try.
Martin: Depends what it is?
Lorena: I got a new desk and it comes with assembly instructions but I give up, I just can't do it.
Lorena: I'm illiterate when it comes to instructions, haha.
Lorena: So I could really use some help...
Martin: Hmm, I can't today, but how about tomorrow? Shouldn't take long anyway.
Lorena: Yeah, it's fine! I just need it for the weekend, but tomorrow's great! Sorry for troubling you again.
Martin: Don't mention it, we've known each other for so long it's almost like we're siblings, haha.
Lorena: Thanks a lot! I should be home by 6, so let me know when you can.

Martin is going to help Lorena assemble a new desk. He is coming to her house tomorrow.


Joanna: They are sending emails about Lewandowska.
Merve: What happened?
Joanna: <file_photo>
Merve: Wooow!
Joanna: She is hospitalized because she has measles.
Merve: She had what?
Joanna: Anyone who had contact with her within the last couple of days must get vaccinated.
Merve: Luckily I didn't see her since the last semester...
Joanna: I did, she is my thesis mentor :(
Merve: What will you do?
Joanna: They are organizing vaccinations in the main building from 17th until 19th.
Merve: You have to go!
Joanna: I know... And I just started working so I really don't have a lot of time.
Merve: Come on, this is really important.
Joanna: I will try to do it before work on 18th, hopefully I won't lose the entire day...

Lewandowska has measles. There are vaccinations in the main building from 17th until 19th for everyone who had contact with her.


n, m, l = map(int, input().split())
a = [list(map(int, input().split())) for _ in range(n)]
b = [list(map(int, input().split())) for _ in range(m)]
c = [[0 for i in range(l)] for j in range(n)]

for i in range(n):
    for j in range(l):
        for k in range(m):
            c[i][j] += a[i][k] * b[k][j]
for i in range(n):
    print(" ".join(map(str, c[i])))

def main():

    N, D = map(int, input().split())
    ans = 0
    for _ in range(N):
        x, y = map(int, input().split())
        if x*x + y* y <= D*D:
            ans += 1
    return ans

if __name__ == '__main__':
    print(main())


import math
alpha = []
while True:
    n = float(raw_input())
    if n == 0:
        break
    nums_str = raw_input().split()
    nums = []
    for s in nums_str:
        nums.append(float(s))
    ave = sum(nums)/n
    n_sum = 0.0
    for num in nums:
        n_sum += (num-ave)**2
    alpha.append(n_sum/n)
for a in alpha:
    print math.sqrt(a)

import math

def main():
    while True:
        n = int(input())
        if n == 0:
            break

        nums = list(map(int, input().split()))
        mean = sum(nums) / n
        var = sum((s - mean) ** 2 for s in nums) / n
        sd = math.sqrt(var)

        print(sd)

if __name__ == "__main__":
    main()


H = int(input())

def attack(h):
    if h == 1:
        return 0
    r = 2
    h //= 2
    ans = 2
    while h > 1:
        h //= 2
        r *= 2
        ans += r
    return ans

print(1 + attack(H))

h = int(input())

ans = 0
n_enemy = 1
while h > 1:
    h //= 2
    ans += n_enemy
    n_enemy *= 2

if h == 1:
    ans += n_enemy

print(ans)


def lcm(a, b):
    from fractions import gcd

    return a // gcd(a, b) * b

def main():
    _ = int(input())
    a = list(map(int, input().split()))
    x = 1

    for e in a:
        x = lcm(x, e)

    ans = 0
    for e in a:
        ans += x // e

    print(ans % int(1e9 + 7))

main()

suits = ['S', 'H', 'C', 'D']
table = [[False] * 14 for i in range(4)]

N = int(input())
for _i in range(N):
    mark, num = input().split()
    num = int(num)

    if mark == 'S':
        table[0][num] = True
    if mark == 'H':
        table[1][num] = True
    if mark == 'C':
        table[2][num] = True
    if mark == 'D':
        table[3][num] = True

for i in range(4):
    for j in range(1, 14):
        if not table[i][j]:
            print(f'{suits[i]} {j}')


a1, a2, a3 = map(int, input().split(" "))
tmp = a1 + a2 + a3
if tmp >= 22:
  print("bust")
else:
  print("win")

#!/usr/bin/env python3

import sys
import math
from bisect import bisect_right as br
from bisect import bisect_left as bl
sys.setrecursionlimit(2147483647)
from heapq import heappush, heappop,heappushpop
from collections import defaultdict
from itertools import accumulate
from collections import Counter
from collections import deque
from operator import itemgetter
from itertools import permutations
mod = 10**9 + 7
inf = float('inf')
def I(): return int(sys.stdin.readline())
def LI(): return list(map(int,sys.stdin.readline().split()))

a = LI()
s = sum(a)
if s >= 22:
    print('bust')
else:
    print('win')


import sys
import fractions
input = sys.stdin.readline

mod = 10 ** 9 + 7
N = int(input().strip())
A = list(map(int, input().strip().split(" ")))

lcm = 1
for a in A:
    lcm = a // fractions.gcd(a, lcm) * lcm

print(sum([lcm // a for a in A]) % mod)

def gcd (a, b):
    while b != 0:
        a, b = b, a%b
    return a

def lcm(a, b,mod):
    return a*b//gcd(a,b)

N = int(input())
A = list(map(int,input().split()))
mod = 10**9+7
b = 1
for i in range(N):
    a = A[i]
    b = lcm(a,b,mod)
ans = 0
for a in A:
    ans += b//a
print(ans%mod)


n = int(input())
xlist = list(map(int,input().split()))
sum = 0
for i in range(n):
  for j in range(i+1,n):
    sum+=xlist[i]*xlist[j]
print(sum)

import sys
import time
import random
import math
from collections import deque
import heapq
import itertools
from decimal import Decimal
import bisect
from operator import itemgetter
MAX_INT = int(10e18)
MIN_INT = -MAX_INT
mod = 1000000000+7
sys.setrecursionlimit(1000000)
def IL(): return list(map(int,input().split()))
def SL(): return input().split()
def I(): return int(sys.stdin.readline())
def S(): return input()

def tami(ans):
  res = 0
  LastDay = [0]*26
  for i in range(D):
    res += s[i][ans[i]]
    for j in range(26):
      if ans[i] == j:
        continue
      res -= (i+1 - LastDay[j]) * c[j]
    LastDay[ans[i]] = i+1
  return res

def nasu():
  res = []
  LastDay = [0]*26
  for d in range(1,D+1):
    tmp = ["", 0]
    for i in range(26):
      if tmp[1] < s[d-1][i] + (d - LastDay[i]) * c[i]:
        tmp = [i, s[d-1][i] + (d - LastDay[i]) * c[i]]
    res.append(tmp[0])
    LastDay[tmp[0]] = d
  return res

def nbo(ans, score, num):
  tmp_ans = ans[:]
  for _ in range(num):
    random_date = random.randint(0,D-1)
    random_val = random.randint(0,25)
    tmp_ans[random_date] = random_val
  scoscore = tami(tmp_ans)
  if scoscore > score:
    return (True, tmp_ans, scoscore)
  else:
    return (False, tmp_ans, scoscore)

start = time.time()
D = I()
c = IL()
s = [IL() for i in range(D)]

ans = nasu()
#print(ans)

score = tami(ans)
#print(score)

end = time.time()
num = 4
while (time.time() - start) < 1.8:
  judge, anans, scoscore = nbo(ans, score, num)
  if judge == True:
    ans, score = anans, scoscore
    #print(ans)
    #print(tami(ans))
for i in ans:
  print(i+1)


INF=1000000000+7

s=int(input())
dp=[0]*(2000+1);

dp[0]=dp[1]=dp[2]=0
dp[3]=1

for i in range(4,s+1):
	temp=1
	for j in range(3,i-2):
		temp+=dp[j]%INF;
	temp%=INF;
	dp[i]=temp
print(dp[s])

from sys import stdin,stdout
l,r,d=list(map(int,stdin.readline().strip().split()))
c=0
if l%d==0 and r%d==0:
    stdout.write(str(int(r/d)-int(l/d)+1)+'\n')
else:
    stdout.write(str(int(r/d)-int(l/d))+'\n')


ans = []
while True:
  arr = map(int, raw_input().split())
  if arr[0] == 0 and arr[1] == 0:
    break
  else:
    arr.sort()
    ans.append(arr)

for x in ans:
  print(" ".join(map(str, x)))

while 1:
    x,y=map(int,raw_input().split())
    if x==y==0: exit()
    else: print min(x,y),max(x,y)


Q:

PHP - latex format of function with regex

Is it possible to write a regex which would take input like 'sqrt(2 * (2+2)) + sin(pi/6)' and transform it into '\sqrt{2 \cdot (2+2)} + \sin(\pi/6)'?

The problem is the 'sqrt' and parentheses in it.

It is obvious I can't simply use something like this:
/sqrt\((.?)\)/ -> \\sqrt{$1}

because this code would create something like this '\sqrt{2 \cdot (2+2)) + \sin(\pi/6}'.

A:

My solution: it simply go throw the string converted to char array and tests if a current substring starts with $latex, if it does second for-cycle go from this point in different direction and by parentheses decides where the function starts and ends. (

startsWith function)
Code:
public static function formatFunction($function, $latex, $input) {
    $input = preg_replace("/" .

$input);
    $arr = str_split($input);

    $inGap = false;
    $gap = 0;

    for ($i = count($arr) - 1; $i >= 0; $i--) {
        if (startsWith(substr($input, $i), $latex)) {
            for ($x = $i; $x < count($arr); $x++) {
                if ($arr[$x] == "(" || $arr[$x] == "{") { $gap++; $inGap = true; }
                else if ($arr[$x] == ")" || $arr[$x] == "}") {  $gap--; }

                if ($inGap && $gap == 0) {
                    $arr[$x] = "}";
                    $inGap = false;
                    break;
                }
            }
        }
        $gap = 0;
    }

    return implode($arr);
}

Use:
self::formatFunction("sqrt", "\\sqrt",
"sqrt(25 + sqrt(16 - sqrt(49)) + (7 + 1)) + sin(pi/2)");

Output:
\sqrt{25+\sqrt{16-\sqrt{49}}+(7+1)}+\sin (\pi/2)

Note: sin and pi aren't formated by this code, it's only str_replace function...


php

namespace Egulias\EmailValidator\Warning;

class CFWSNearAt extends Warning
{
    const CODE = 49;

    public function __construct()
    {
        $this->message = "Deprecated folding white space near @";
    }
}


Static Site Substitution: Jekyll to Jigsaw + Netlify

A couple of years ago I created this blog with Jekyll.

I'm not a Ruby guy, but Jekyll had certain advantages that were attractive at the time.

It had templating which would save me from repeating HTML and I could have it automatically build and get free hosting thanks to GitHub Pages.

As time passed, a few problems started popping up that made even tiny edits become tedious.

I found that I only ever install Ruby to update my blog, on Windows Jekyll started requiring me to install additional dependencies, and I used Bower and Gulp to install and compile dependencies back when I started.

I did have the idea to refresh the blog for some time but I kept putting it off.

It wasn't until I heard of Jigsaw and Netlify that I got really motivated to do so.

Jigsaw

When I started using Jigsaw, there were several advantages that were immediately apparent.

I already had the necessary development environment, I got to use the Blade templating engine which I was comfortable with, and it already came with a Laravel Mix script for compiling assets.

Truly a static site generator made by Laravel people for Laravel people.

Netlify

GitHub Pages is a good static site host, but to me Netlify has one significant advantage over it, flexibility.

Using anything other than Jekyll on GitHub Pages is fine, but you lose the convenience of the automatic build step.

And then there's Netlify, which is so flexible that you can configure it to play nice with any static site generator, even the lesser known ones.

On top of that, Netlify has a wide range of features to either enhance your site or make managing it easier.

You can even add the equally flexible Netlify CMS to your site, you just need to take some time to configure it properly.

Final Thoughts

Jekyll has been a great option for building static sites, having lots of users and resources.

However, Jigsaw allows me to build with a familiar toolset and Netlify has allowed me to keep the same quick and easy deployment process even though I'm using a somewhat obscure static site generator.

I'm really happy that I made these changes.


The enhanced aerosol performance of salbutamol from dry powders containing engineered mannitol as excipient.

The aim of the present study was to investigate the effect of crystallising mannitol from different binary mixtures of acetone/water on the resultant physical properties and to determine the effects of any changes on in vitro aerosolisation performance, when the different mannitol crystals were used as a carrier in dry powder inhaler formulations containing salbutamol sulphate.

Mannitol particles were crystallised under controlled conditions by dissolving the sugar in water and precipitating the sugar using binary mixtures of acetone/water in different percentages as anti-solvent media.

For comparison purposes the physical properties and deposition behaviour of commercially available mannitol were also studied.

SEM showed that all crystallised mannitol particles were more elongated than the commercial mannitol.

Solid state studies revealed that commercial mannitol and mannitol crystallised using acetone in the presence of 10-25% v/v water as anti-solvent was beta-polymorphic form whereas mannitol crystallised in the presence of a small amount of water (0-7.5%) was the alpha-form.

All the crystallised mannitol samples showed poor flowability.

Nevertheless, the powdered crystallised mannitol and commercial samples were blended with salbutamol in the ratio 67.5:1.

The aerosolisation performance of the formulations containing the engineered mannitol (evaluated using Multi Stage Liquid Impinger) was considerably better than that of the commercial mannitol formulation (the fine particle fraction was increased from 15.42% to 33.07-43.99%, for the formulations containing crystallised mannitol).

Generally, carriers having a high tapped density and high fraction of fine carrier particles produced a high FPF.

The improvement in the DPI performance could be attributed to the presence of elongated carrier particles with smooth surfaces since these are believed to have less adhesive forces between carrier and the drug resulting in easier detachment of the drug during the inhalation.


The Role of Neuroimaging in the Assessment of the Cognitively Impaired Elderly.

This article reviews the current diagnostic tools that are available for structural, functional, and molecular imaging of the brain, summarizing some of the key findings that have been reported in individuals diagnosed with Alzheimer disease, mild cognitive impairment, prodromal AD, or other prevalent dementias.

Given recent advances in the development of amyloid PET tracers, current guidelines for the use of amyloid PET imaging in patients with cognitive complaints are reviewed.

In addition, data addressing the potential value of amyloid PET imaging in the clinical setting are highlighted.


Phoenix coach Ernie Merrick has blasted Football Federation Australia for not providing sideline dugouts across all A-League venues.

Under pressure Sydney FC coach Frank Farina had a drink thrown on him by a disgruntled fan last Saturday during the 3-0 home loss to Adelaide United at Allianz Stadium.

Merrick has coached in the A-League since its inception in 2005 and, without going into specifics, said it was naive to think the Farina incident was the first time the safety of a coach or player had been threatened.

Merrick said Westpac Stadium was the only venue that uses dugouts and they were doing so against FFA orders.

Other clubs use uncovered chairs on the sideline.

Merrick said most A-League supporters were fantastic but "there's always one or two that are out of order".

"And we should be not be exposed, any coaches, doctors, physios, players, shouldn't be exposed to sitting two metres from the fans, with some of their behaviours," Merrick said.

"It's a security issue and on top of that an environmental issue as well.

I mean there's nothing more ridiculous than us sitting on $5 white plastic BBQ chairs, in the howling wind and rain, with umbrellas over our heads, trying to coach a game or being prepared to come on.

I mean we'd be the only country in the world that's doing that."

The FFA has said that all A-League venues will have dugouts from next year but Merrick said the safety issue had been dodged since 2006.

"Let's be sensible, a little bit of security and safety and a shelter from the environment is such a good thing.

But let's not pretend that one incident is the only incident that occurred, against Frank Farina.

And Frank shouldn't have to deal with that, none of us should."

Meanwhile, key Phoenix midfielder Albert Riera will miss Sunday's home game against Melbourne Heart as he awaits further scans on his groin tear.

Riera also missed last Sunday's 3-2 win over the Jets in Newcastle.

"He's unable to play this week and I think he'll be out for a couple of weeks, but we don't know the details yet."

Leo Bertos is also likely to miss the Heart match with his own groin injury but Michael Boxall and Kenny Cunningham are set to return from suspension to the starting XI.


Hillary Clinton takes "responsibility" for loss, but says others contributed

allsides_1019_408_0_former Secretary of State Hillary Clinton_8_15


Clinton also looked to social media, saying sites like Facebook and Twitter have been "victimized by deliberate efforts to shape the conversation and push it towards conspiracies, lies, false information."

allsides_1019_408_9_former Secretary of State Hillary Clinton_0_7


Zombie Blender

[{'id': 2, 'description': 'Single-player'}]

[{'id': '1', 'description': 'Action'}, {'id': '25', 'description': 'Adventure'}, {'id': '4', 'description': 'Casual'}, {'id': '23', 'description': 'Indie'}, {'id': '70', 'description': 'Early Access'}]


PLACEFRONT

[{'id': '25', 'description': 'Adventure'}, {'id': '4', 'description': 'Casual'}, {'id': '23', 'description': 'Indie'}, {'id': '28', 'description': 'Simulation'}]


CodeWalkers

[{'id': 2, 'description': 'Single-player'}, {'id': 1, 'description': 'Multi-player'}, {'id': 49, 'description': 'PvP'}, {'id': 36, 'description': 'Online PvP'}]

[{'id': '1', 'description': 'Action'}, {'id': '28', 'description': 'Simulation'}]


Sky Garden

[{'id': 2, 'description': 'Single-player'}, {'id': 21, 'description': 'Downloadable Content'}]

[{'id': '4', 'description': 'Casual'}, {'id': '28', 'description': 'Simulation'}, {'id': '54', 'description': 'Education'}]


Wind Angel

[{'id': '25', 'description': 'Adventure'}, {'id': '4', 'description': 'Casual'}, {'id': '23', 'description': 'Indie'}]


i start to feel emotional


i believe feeling duality suffering soul growth tells of an ending or a decline or a change of direction often one associated with emotions and it offers one possible response to that decline or change moving on


i hostage negotiator on her case has her feeling hopeful about her future


im feeling a little dirty


i can cope with his presence without feeling distressed if i can force myself into a quiet and resigned friendship


i had the feeling that it might not have been taken as the truthful and sincere compliment it would have been


i asked some girls what it meant to them to be valued and for the most part the response was that they felt valued when the people around them made them feel valued and treated them in a loving and caring manner


i feel if i completely hated things i d exercise my democratic right speak my mind in what ever ways possible and try to enact a change


i could almost feel her gentle touch in the moonbeam she sent to shine over me he added touching his face dreamily


i woke up very early this morning feeling joyful