data science python data cleaning

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
2
down vote

favorite












I am preparing a dataset for a model, but somehow the code just doesn't run well.



The major error is:




File "/Users/liangjulia/Desktop/UW DS Certificate Learning Material/untitled6.py", line 61
 'income2' = pd.to_numeric(Adult.income, errors='coerce')
 ^
SyntaxError: can't assign to literal



Code:



# import statement
import numpy as np
import pandas as pd
import csv
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
# loading dataset, it is a combination of categorical and numerical data
hp = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",header=0,sep=',')
hp.columns = ['age','workclass','income','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country','salary-range'] 
# dataset basics
hp.head()
hp.shape
hp.dtypes

# account for all value '?'
hp.replace('?','na')
hp.isnull().sum()

#Remove onsolete data point in income
hp('income').dropna()

# replace all aberrant values
hp.replace('nan', 0)
hp.replace('NULL', 0)

# change data type of certain data point to numerical
number = LabelEncoder()
hp['income'] = number.fit_transform(hp['income'.astype('str')])
hp['capital-gain'] = number.fit_transform(hp['capital-gain'.astype('str')])
hp['capital-loss'] = number.fit_transform(hp['capital-loss'.astype('str')])

# Choose the datapoint 'income' to perform the data cleaning and remove outliers
LimitHi=np.mean('income') + 2*np.std('income')
LimitLo=np.mean('income') + 2*np.std('income')
BadIncome = ('income' > LimitHi) & ('income' < LimitLo)

# Replace outliars
RightIncome = ~BadIncome
x[BadIncome] = np.mean(x[RightIncome])

# normalize the Income Column using numpy
#'income2' = pd.to_numeric(Adult.income, errors='coerce')
minmaxscaled =('income' - min('income'))/(max('income') - min('income'))

# bin age data into several ranges
hp['bin'] = pd.cut(hp['age'], [15,30,45,60,75,90])

# construct new categorical data point with existing data point 
hp['EvalonInvestment'] = 'zzz'
hp.loc[(hp['capital-gain'] >= 50000), 'loc2'] = 'investmentking'
hp.loc[(hp['capital-gain'] > 10000) & (hp['capital-gain'] < 50000), 'loc2'] = 'good-investment'
hp.loc[(hp['capital-gain'] > 0) & (hp['capital-gain'] <= 10000), 'loc2'] = 'ok-investment'

print(hp)



python data-cleaning data




share|improve this question

























    up vote
    2
    down vote

    favorite












    I am preparing a dataset for a model, but somehow the code just doesn't run well.



    The major error is:




    File "/Users/liangjulia/Desktop/UW DS Certificate Learning Material/untitled6.py", line 61
     'income2' = pd.to_numeric(Adult.income, errors='coerce')
     ^
    SyntaxError: can't assign to literal



    Code:



    # import statement
    import numpy as np
    import pandas as pd
    import csv
    import matplotlib.pyplot as plt
    from sklearn.preprocessing import LabelEncoder
    # loading dataset, it is a combination of categorical and numerical data
    hp = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",header=0,sep=',')
    hp.columns = ['age','workclass','income','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country','salary-range'] 
    # dataset basics
    hp.head()
    hp.shape
    hp.dtypes

    # account for all value '?'
    hp.replace('?','na')
    hp.isnull().sum()

    #Remove onsolete data point in income
    hp('income').dropna()

    # replace all aberrant values
    hp.replace('nan', 0)
    hp.replace('NULL', 0)

    # change data type of certain data point to numerical
    number = LabelEncoder()
    hp['income'] = number.fit_transform(hp['income'.astype('str')])
    hp['capital-gain'] = number.fit_transform(hp['capital-gain'.astype('str')])
    hp['capital-loss'] = number.fit_transform(hp['capital-loss'.astype('str')])

    # Choose the datapoint 'income' to perform the data cleaning and remove outliers
    LimitHi=np.mean('income') + 2*np.std('income')
    LimitLo=np.mean('income') + 2*np.std('income')
    BadIncome = ('income' > LimitHi) & ('income' < LimitLo)

    # Replace outliars
    RightIncome = ~BadIncome
    x[BadIncome] = np.mean(x[RightIncome])

    # normalize the Income Column using numpy
    #'income2' = pd.to_numeric(Adult.income, errors='coerce')
    minmaxscaled =('income' - min('income'))/(max('income') - min('income'))

    # bin age data into several ranges
    hp['bin'] = pd.cut(hp['age'], [15,30,45,60,75,90])

    # construct new categorical data point with existing data point 
    hp['EvalonInvestment'] = 'zzz'
    hp.loc[(hp['capital-gain'] >= 50000), 'loc2'] = 'investmentking'
    hp.loc[(hp['capital-gain'] > 10000) & (hp['capital-gain'] < 50000), 'loc2'] = 'good-investment'
    hp.loc[(hp['capital-gain'] > 0) & (hp['capital-gain'] <= 10000), 'loc2'] = 'ok-investment'

    print(hp)



    python data-cleaning data




    share|improve this question























      up vote
      2
      down vote

      favorite









      up vote
      2
      down vote

      favorite











      I am preparing a dataset for a model, but somehow the code just doesn't run well.



      The major error is:




      File "/Users/liangjulia/Desktop/UW DS Certificate Learning Material/untitled6.py", line 61
       'income2' = pd.to_numeric(Adult.income, errors='coerce')
       ^
      SyntaxError: can't assign to literal



      Code:



      # import statement
      import numpy as np
      import pandas as pd
      import csv
      import matplotlib.pyplot as plt
      from sklearn.preprocessing import LabelEncoder
      # loading dataset, it is a combination of categorical and numerical data
      hp = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",header=0,sep=',')
      hp.columns = ['age','workclass','income','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country','salary-range'] 
      # dataset basics
      hp.head()
      hp.shape
      hp.dtypes

      # account for all value '?'
      hp.replace('?','na')
      hp.isnull().sum()

      #Remove onsolete data point in income
      hp('income').dropna()

      # replace all aberrant values
      hp.replace('nan', 0)
      hp.replace('NULL', 0)

      # change data type of certain data point to numerical
      number = LabelEncoder()
      hp['income'] = number.fit_transform(hp['income'.astype('str')])
      hp['capital-gain'] = number.fit_transform(hp['capital-gain'.astype('str')])
      hp['capital-loss'] = number.fit_transform(hp['capital-loss'.astype('str')])

      # Choose the datapoint 'income' to perform the data cleaning and remove outliers
      LimitHi=np.mean('income') + 2*np.std('income')
      LimitLo=np.mean('income') + 2*np.std('income')
      BadIncome = ('income' > LimitHi) & ('income' < LimitLo)

      # Replace outliars
      RightIncome = ~BadIncome
      x[BadIncome] = np.mean(x[RightIncome])

      # normalize the Income Column using numpy
      #'income2' = pd.to_numeric(Adult.income, errors='coerce')
      minmaxscaled =('income' - min('income'))/(max('income') - min('income'))

      # bin age data into several ranges
      hp['bin'] = pd.cut(hp['age'], [15,30,45,60,75,90])

      # construct new categorical data point with existing data point 
      hp['EvalonInvestment'] = 'zzz'
      hp.loc[(hp['capital-gain'] >= 50000), 'loc2'] = 'investmentking'
      hp.loc[(hp['capital-gain'] > 10000) & (hp['capital-gain'] < 50000), 'loc2'] = 'good-investment'
      hp.loc[(hp['capital-gain'] > 0) & (hp['capital-gain'] <= 10000), 'loc2'] = 'ok-investment'

      print(hp)



      python data-cleaning data




      share|improve this question













      I am preparing a dataset for a model, but somehow the code just doesn't run well.



      The major error is:




      File "/Users/liangjulia/Desktop/UW DS Certificate Learning Material/untitled6.py", line 61
       'income2' = pd.to_numeric(Adult.income, errors='coerce')
       ^
      SyntaxError: can't assign to literal



      Code:



      # import statement
      import numpy as np
      import pandas as pd
      import csv
      import matplotlib.pyplot as plt
      from sklearn.preprocessing import LabelEncoder
      # loading dataset, it is a combination of categorical and numerical data
      hp = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",header=0,sep=',')
      hp.columns = ['age','workclass','income','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country','salary-range'] 
      # dataset basics
      hp.head()
      hp.shape
      hp.dtypes

      # account for all value '?'
      hp.replace('?','na')
      hp.isnull().sum()

      #Remove onsolete data point in income
      hp('income').dropna()

      # replace all aberrant values
      hp.replace('nan', 0)
      hp.replace('NULL', 0)

      # change data type of certain data point to numerical
      number = LabelEncoder()
      hp['income'] = number.fit_transform(hp['income'.astype('str')])
      hp['capital-gain'] = number.fit_transform(hp['capital-gain'.astype('str')])
      hp['capital-loss'] = number.fit_transform(hp['capital-loss'.astype('str')])

      # Choose the datapoint 'income' to perform the data cleaning and remove outliers
      LimitHi=np.mean('income') + 2*np.std('income')
      LimitLo=np.mean('income') + 2*np.std('income')
      BadIncome = ('income' > LimitHi) & ('income' < LimitLo)

      # Replace outliars
      RightIncome = ~BadIncome
      x[BadIncome] = np.mean(x[RightIncome])

      # normalize the Income Column using numpy
      #'income2' = pd.to_numeric(Adult.income, errors='coerce')
      minmaxscaled =('income' - min('income'))/(max('income') - min('income'))

      # bin age data into several ranges
      hp['bin'] = pd.cut(hp['age'], [15,30,45,60,75,90])

      # construct new categorical data point with existing data point 
      hp['EvalonInvestment'] = 'zzz'
      hp.loc[(hp['capital-gain'] >= 50000), 'loc2'] = 'investmentking'
      hp.loc[(hp['capital-gain'] > 10000) & (hp['capital-gain'] < 50000), 'loc2'] = 'good-investment'
      hp.loc[(hp['capital-gain'] > 0) & (hp['capital-gain'] <= 10000), 'loc2'] = 'ok-investment'

      print(hp)




      python data-cleaning data





      share|improve this question












      share|improve this question




      share|improve this question








      edited 18 hours ago









      Stephen Rauch

      1,29341028




      1,29341028









      asked 18 hours ago









      user633599

      111




      111




















          2 Answers
          2






          active

          oldest

          votes

















          up vote
          2
          down vote













          It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string






          share|improve this answer




























            up vote
            0
            down vote













            What line 61 should read is



            income2 = pd.to_numeric(hp['income'], errors = 'coerce')


            Let's break this down.



            You want to assign a new variable income2 to the output of a method in the pd class, to_numeric.



            to_number takes two arguments: arg : list, tuple, 1-d array, or Series, a choice on how to treat errors, and an optional downcast operator. You find this out by 



            help(pd.to_numeric)


            Line 61 originally provided the arg Adult.Income, which could be a list, tuple, 1-d array, or Series, but it hasn't been defined, so it can't be a valid arg.



            Try



            type(hp['income'])



            and you get



            pandas.core.series.Series


            which is a valid argument.



            This line is not the only problem in your code. I suggest that you use iPython so that you can see all of the other errors. Or just use the command line and copy in a few lines at a time and you'll find new error messages to resolve.



            Errors in code are like cockroaches -- they seldom travel alone.



            Good luck!






            share|improve this answer





















              Your Answer




              StackExchange.ifUsing("editor", function ()
              return StackExchange.using("mathjaxEditing", function ()
              StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
              StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
              );
              );
              , "mathjax-editing");

              StackExchange.ready(function()
              var channelOptions =
              tags: "".split(" "),
              id: "557"
              ;
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function()
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled)
              StackExchange.using("snippets", function()
              createEditor();
              );

              else
              createEditor();

              );

              function createEditor()
              StackExchange.prepareEditor(
              heartbeatType: 'answer',
              convertImagesToLinks: false,
              noModals: false,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: null,
              bindNavPrevention: true,
              postfix: "",
              noCode: true, onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              );



              );








               

              draft saved


              draft discarded


















              StackExchange.ready(
              function ()
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36496%2fdata-science-python-data-cleaning%23new-answer', 'question_page');

              );

              Post as a guest

















              2 Answers
              2






              active

              oldest

              votes








              2 Answers
              2






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes








              up vote
              2
              down vote













              It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string






              share|improve this answer

























                up vote
                2
                down vote













                It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string






                share|improve this answer























                  up vote
                  2
                  down vote










                  up vote
                  2
                  down vote









                  It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string






                  share|improve this answer













                  It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string







                  share|improve this answer













                  share|improve this answer



                  share|improve this answer











                  answered 16 hours ago









                  Omkaar.K

                  1406




                  1406




















                      up vote
                      0
                      down vote













                      What line 61 should read is



                      income2 = pd.to_numeric(hp['income'], errors = 'coerce')


                      Let's break this down.



                      You want to assign a new variable income2 to the output of a method in the pd class, to_numeric.



                      to_number takes two arguments: arg : list, tuple, 1-d array, or Series, a choice on how to treat errors, and an optional downcast operator. You find this out by 



                      help(pd.to_numeric)


                      Line 61 originally provided the arg Adult.Income, which could be a list, tuple, 1-d array, or Series, but it hasn't been defined, so it can't be a valid arg.



                      Try



                      type(hp['income'])



                      and you get



                      pandas.core.series.Series


                      which is a valid argument.



                      This line is not the only problem in your code. I suggest that you use iPython so that you can see all of the other errors. Or just use the command line and copy in a few lines at a time and you'll find new error messages to resolve.



                      Errors in code are like cockroaches -- they seldom travel alone.



                      Good luck!






                      share|improve this answer

























                        up vote
                        0
                        down vote













                        What line 61 should read is



                        income2 = pd.to_numeric(hp['income'], errors = 'coerce')


                        Let's break this down.



                        You want to assign a new variable income2 to the output of a method in the pd class, to_numeric.



                        to_number takes two arguments: arg : list, tuple, 1-d array, or Series, a choice on how to treat errors, and an optional downcast operator. You find this out by 



                        help(pd.to_numeric)


                        Line 61 originally provided the arg Adult.Income, which could be a list, tuple, 1-d array, or Series, but it hasn't been defined, so it can't be a valid arg.



                        Try



                        type(hp['income'])



                        and you get



                        pandas.core.series.Series


                        which is a valid argument.



                        This line is not the only problem in your code. I suggest that you use iPython so that you can see all of the other errors. Or just use the command line and copy in a few lines at a time and you'll find new error messages to resolve.



                        Errors in code are like cockroaches -- they seldom travel alone.



                        Good luck!






                        share|improve this answer























                          up vote
                          0
                          down vote










                          up vote
                          0
                          down vote









                          What line 61 should read is



                          income2 = pd.to_numeric(hp['income'], errors = 'coerce')


                          Let's break this down.



                          You want to assign a new variable income2 to the output of a method in the pd class, to_numeric.



                          to_number takes two arguments: arg : list, tuple, 1-d array, or Series, a choice on how to treat errors, and an optional downcast operator. You find this out by 



                          help(pd.to_numeric)


                          Line 61 originally provided the arg Adult.Income, which could be a list, tuple, 1-d array, or Series, but it hasn't been defined, so it can't be a valid arg.



                          Try



                          type(hp['income'])



                          and you get



                          pandas.core.series.Series


                          which is a valid argument.



                          This line is not the only problem in your code. I suggest that you use iPython so that you can see all of the other errors. Or just use the command line and copy in a few lines at a time and you'll find new error messages to resolve.



                          Errors in code are like cockroaches -- they seldom travel alone.



                          Good luck!






                          share|improve this answer













                          What line 61 should read is



                          income2 = pd.to_numeric(hp['income'], errors = 'coerce')


                          Let's break this down.



                          You want to assign a new variable income2 to the output of a method in the pd class, to_numeric.



                          to_number takes two arguments: arg : list, tuple, 1-d array, or Series, a choice on how to treat errors, and an optional downcast operator. You find this out by 



                          help(pd.to_numeric)


                          Line 61 originally provided the arg Adult.Income, which could be a list, tuple, 1-d array, or Series, but it hasn't been defined, so it can't be a valid arg.



                          Try



                          type(hp['income'])



                          and you get



                          pandas.core.series.Series


                          which is a valid argument.



                          This line is not the only problem in your code. I suggest that you use iPython so that you can see all of the other errors. Or just use the command line and copy in a few lines at a time and you'll find new error messages to resolve.



                          Errors in code are like cockroaches -- they seldom travel alone.



                          Good luck!







                          share|improve this answer













                          share|improve this answer



                          share|improve this answer











                          answered 14 hours ago









                          Richard Careaga

                          1213




                          1213






















                               

                              draft saved


                              draft discarded


























                               


                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function ()
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f36496%2fdata-science-python-data-cleaning%23new-answer', 'question_page');

                              );

                              Post as a guest
































































                              Comments

                              Post a Comment

                              Popular posts from this blog

                              What is the equation of a 3D cone with generalised tilt?

                              Image
                              Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite What is the equation of a 3D cone with generalized tilt? I've noticed that in most equations given to represent a cone, there is no parameter which defines the tilt of the cone in 3D space and that most of them have their point at the origin $(0,0,0)$ - I was wondering if anyone could give me a more generalized cone equation for a cone in any position in 3D space 3d share | cite | improve this question edited Jul 25 '16 at 0:05 ervx 9,425 3 13 36 asked Jul 24 '16 at 15:38 Charlene 11 2 2 Welcome to Math.SE! Could you please explain a few things to help people give an answer useful to you: 1. What do you mean by "generalized tilt"? 2. Are you asking about a right circular cone? Does the angle at the vertex matter? 3. What form of answer (implicit, parametric...?) are you looking for? 4. If this is homework, what tools do you have available, an...
                              Read more

                              Color the edges and diagonals of a regular polygon

                              Image
                              Clash Royale CLAN TAG #URR8PPP up vote 5 down vote favorite 1 Here is the problem: For what $n$ is it possible to color the edges and diagonals of an $n$-side regular polygon with $dfracbinomn23$ colors, such that you use every color exactly three times and for every color the three segments (edges or diagonals) with that color form a triangle? Trivially $nequiv 0 text or 1 pmod3$. I can also prove that if the statement is true for $k$ than it is true for $3k$ as well. How to finish? Please help! Thanks combinatorial-geometry share | cite | improve this question edited Aug 6 at 21:57 alcana 118 4 asked Aug 6 at 21:15 Leo Gardner 356 11 Even assuming "polyhpn" in the title is a typo for polygon , it is unclear what you mean by "the edges and sides". Aren't the side of a regular polygon the same as the edges? A regular $n$-sided polygon has $n$ edges. – hardmath Aug 6 at 21:20 3 $n$ ne...
                              Read more

                              Relationship between determinant of matrix and determinant of adjoint?

                              Image
                              Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite We are studying adjoints in class, and I was curious if there is a relationship between the determinant of matrix A, and the determinant of the adjoint of matrix A? I assume there would be a relationship because finding the adjoint requires creating a cofactor matrix and then transposing it. linear-algebra determinant adjoint-operators share | cite | improve this question asked Nov 17 '17 at 17:32 CluelessCoder 32 5 For a square matrix $A$ of order $n$, we have $A(operatornameadj A)=(operatornameadj A)A=|A|I_n$ where $I_n$ is the identity matrix of order $n$. This should tell you about the determinant of the adjoint in terms of that of $A$ (use multiplicative property and other elementary properties of determinants). – Prasun Biswas Nov 17 '17 at 17:35 1 @CluelessCoder: see here: math.stackexchange.com/questions/516127/…...
                              Read more