Skip to content
Search
Generic filters
Exact matches only

How To Load Machine Learning Data in Python

Last Updated on August 21, 2019

You must be able to load your data before you can start your machine learning project.

The most common format for machine learning data is CSV files. There are a number of ways to load a CSV file in Python.

In this post you will discover the different ways that you can use to load your machine learning data in Python.

Discover how to prepare data with pandas, fit and evaluate models with scikit-learn, and more in my new book, with 16 step-by-step tutorials, 3 projects, and full python code.

Let’s get started.

  • Update March/2017: Change loading from binary (‘rb’) to ASCII (‘rt).
  • Update March/2018: Added alternate link to download the dataset as the original appears to have been taken down.
  • Update March/2018: Updated NumPy load from URL example to work wth Python 3.

How To Load Machine Learning Data in Python

How To Load Machine Learning Data in Python
Photo by Ann Larie Valentine, some rights reserved.

Considerations When Loading CSV Data

There are a number of considerations when loading your machine learning data from CSV files.

For reference, you can learn a lot about the expectations for CSV files by reviewing the CSV request for comment titled Common Format and MIME Type for Comma-Separated Values (CSV) Files.

CSV File Header

Does your data have a file header?

If so this can help in automatically assigning names to each column of data. If not, you may need to name your attributes manually.

Either way, you should explicitly specify whether or not your CSV file had a file header when loading your data.

Comments

Does your data have comments?

Comments in a CSV file are indicated by a hash (“#”) at the start of a line.

If you have comments in your file, depending on the method used to load your data, you may need to indicate whether or not to expect comments and the character to expect to signify a comment line.

Delimiter

The standard delimiter that separates values in fields is the comma (“,”) character.

Your file could use a different delimiter like tab (“t”) in which case you must specify it explicitly.

Quotes

Sometimes field values can have spaces. In these CSV files the values are often quoted.

The default quote character is the double quotation marks “””. Other characters can be used, and you must specify the quote character used in your file.

Need help with Machine Learning in Python?

Take my free 2-week email course and discover data prep, algorithms and more (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

Start Your FREE Mini-Course Now!

Machine Learning Data Loading Recipes

Each recipe is standalone.

This means that you can copy and paste it into your project and use it immediately.

If you have any questions about these recipes or suggested improvements, please leave a comment and I will do my best to answer.

Load CSV with Python Standard Library

The Python API provides the module CSV and the function reader() that can be used to load CSV files.

Once loaded, you convert the CSV data to a NumPy array and use it for machine learning.

For example, you can download the Pima Indians dataset into your local directory (download from here).

All fields are numeric and there is no header line. Running the recipe below will load the CSV file and convert it to a NumPy array.

# Load CSV (using python)
import csv
import numpy
filename = ‘pima-indians-diabetes.data.csv’
raw_data = open(filename, ‘rt’)
reader = csv.reader(raw_data, delimiter=’,’, quoting=csv.QUOTE_NONE)
x = list(reader)
data = numpy.array(x).astype(‘float’)
print(data.shape)

# Load CSV (using python)

import csv

import numpy

filename = ‘pima-indians-diabetes.data.csv’

raw_data = open(filename, ‘rt’)

reader = csv.reader(raw_data, delimiter=’,’, quoting=csv.QUOTE_NONE)

x = list(reader)

data = numpy.array(x).astype(‘float’)

print(data.shape)

The example loads an object that can iterate over each row of the data and can easily be converted into a NumPy array. Running the example prints the shape of the array.

For more information on the csv.reader() function, see CSV File Reading and Writing in the Python API documentation.

Load CSV File With NumPy

You can load your CSV data using NumPy and the numpy.loadtxt() function.

This function assumes no header row and all data has the same format. The example below assumes that the file pima-indians-diabetes.data.csv is in your current working directory.

# Load CSV
import numpy
filename = ‘pima-indians-diabetes.data.csv’
raw_data = open(filename, ‘rt’)
data = numpy.loadtxt(raw_data, delimiter=”,”)
print(data.shape)

# Load CSV

import numpy

filename = ‘pima-indians-diabetes.data.csv’

raw_data = open(filename, ‘rt’)

data = numpy.loadtxt(raw_data, delimiter=”,”)

print(data.shape)

Running the example will load the file as a numpy.ndarray and print the shape of the data:

This example can be modified to load the same dataset directly from a URL as follows:

Note: This example assumes you are using Python 3.

# Load CSV from URL using NumPy
from numpy import loadtxt
from urllib.request import urlopen
url = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv’
raw_data = urlopen(url)
dataset = loadtxt(raw_data, delimiter=”,”)
print(dataset.shape)

# Load CSV from URL using NumPy

from numpy import loadtxt

from urllib.request import urlopen

url = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv’

raw_data = urlopen(url)

dataset = loadtxt(raw_data, delimiter=”,”)

print(dataset.shape)

Again, running the example produces the same resulting shape of the data.

For more information on the numpy.loadtxt() function see the API documentation (version 1.10 of numpy).

Load CSV File With Pandas

You can load your CSV data using Pandas and the pandas.read_csv() function.

This function is very flexible and is perhaps my recommended approach for loading your machine learning data. The function returns a pandas.DataFrame that you can immediately start summarizing and plotting.

The example below assumes that the ‘pima-indians-diabetes.data.csv‘ file is in the current working directory.

# Load CSV using Pandas
import pandas
filename = ‘pima-indians-diabetes.data.csv’
names = [‘preg’, ‘plas’, ‘pres’, ‘skin’, ‘test’, ‘mass’, ‘pedi’, ‘age’, ‘class’]
data = pandas.read_csv(filename, names=names)
print(data.shape)

# Load CSV using Pandas

import pandas

filename = ‘pima-indians-diabetes.data.csv’

names = [‘preg’, ‘plas’, ‘pres’, ‘skin’, ‘test’, ‘mass’, ‘pedi’, ‘age’, ‘class’]

data = pandas.read_csv(filename, names=names)

print(data.shape)

Note that in this example we explicitly specify the names of each attribute to the DataFrame. Running the example displays the shape of the data:

We can also modify this example to load CSV data directly from a URL.

# Load CSV using Pandas from URL
import pandas
url = “https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv”
names = [‘preg’, ‘plas’, ‘pres’, ‘skin’, ‘test’, ‘mass’, ‘pedi’, ‘age’, ‘class’]
data = pandas.read_csv(url, names=names)
print(data.shape)

# Load CSV using Pandas from URL

import pandas

url = “https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv”

names = [‘preg’, ‘plas’, ‘pres’, ‘skin’, ‘test’, ‘mass’, ‘pedi’, ‘age’, ‘class’]

data = pandas.read_csv(url, names=names)

print(data.shape)

Again, running the example downloads the CSV file, parses it and displays the shape of the loaded DataFrame.

To learn more about the pandas.read_csv() function you can refer to the API documentation.

Summary

In this post you discovered how to load your machine learning data in Python.

You learned three specific techniques that you can use:

  • Load CSV with Python Standard Library.
  • Load CSV File With NumPy.
  • Load CSV File With Pandas.

Your action step for this post is to type or copy-and-paste each recipe and get familiar with the different ways that you can load machine learning data in Python.

Do you have any questions about loading machine learning data in Python or about this post? Ask your question in the comments and I will do my best to answer it.

Discover Fast Machine Learning in Python!

Master Machine Learning With Python

Develop Your Own Models in Minutes

…with just a few lines of scikit-learn code

Learn how in my new Ebook:
Machine Learning Mastery With Python

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, modeling, tuning, and much more…

Finally Bring Machine Learning To
Your Own Projects

Skip the Academics. Just Results.

See What’s Inside

error: Content is protected !!