How to Install Scrapy (Python install best practice)

[darrenfu]

In OSX EI Capitan

There are quite a few limitations for OpenSSL, cryptography, and pip in OSX EI Capitan. Here is the one-key install command to install scrapy via pip:

sudo -H \
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" \
pip --no-cache-dir \
--default-timeout=100 \
install --install-option '--install-data=/usr/local' \
--upgrade --ignore-installed \
cryptography==1.0.2 scrapy

If you already install higher version of cryptography, please run sudo -H pip uninstall cryptography, and reinstall cryptography==1.0.2.
pypa/pip#3165
pyca/cryptography#2350
pyca/cryptography#2468

In CentOS 6

Note that Python 2.6 is default python version and required in CentOS 6 for yum.
Install Python 2.7 first: http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/

sudo yum groupinstall "Development tools"
sudo yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
# Python 2.7.11:
wget http://python.org/ftp/python/2.7.11/Python-2.7.11.tar.xz
tar xf Python-2.7.11.tar.xz
cd Python-2.7.11
sudo ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
sudo make && sudo make altinstall
# First get the setup script for Setuptools:
wget https://bootstrap.pypa.io/ez_setup.py
# Then install it for Python 2.7:
sudo /usr/local/bin/python2.7 ez_setup.py
# Now install pip using the newly installed setuptools:
sudo /usr/local/bin/easy_install-2.7 pip

Alternatively, if you are behind firewall and find easy_install not working for you, you might try get-pip.py:

wget https://bootstrap.pypa.io/get-pip.py
sudo /usr/local/bin/python2.7 get-pip.py --proxy="[user:passwd@]proxy.server:port"

Then install a few library dependencies for cryptography and lxml:

# Fix a cryptography error
sudo yum install gcc libffi-devel python-devel openssl-devel
# Fix a lxml error
sudo yum install libxslt-devel libxml2-devel libxml2 libxslt

Then we are ready to install Scrapy behind firewall:

sudo -E /usr/local/bin/pip2.7 --no-cache-dir install --upgrade --ignore-installed scrapy

OR

sudo /usr/local/bin/pip2.7 --proxy http://proxy.server:port --no-cache-dir install --upgrade --ignore-installed scrapy

Last but not the least, you may consider to use virtualenv to create a sandbox to switch between Python 2.6 and Python 2.7:

# Install virtualenv for Python 2.7 and create a sandbox called my27project:
sudo -E /usr/local/bin/pip2.7 install virtualenv
virtualenv mypy27
# Check the system Python interpreter version:
python --version
# This will show Python 2.6.6
# Activate the my27project sandbox and check the version of the default Python interpreter in it:
source mypy27/bin/activate
python --version
# This will show Python 2.7.11
deactivate

In CentOS 7

If you are behind firewall and find easy_install not working for you, you might try get-pip.py:

export http_proxy=...
export https_proxy=...
wget https://bootstrap.pypa.io/get-pip.py

export http_proxy=...
export https_proxy=...
sudo su -
python get-pip.py

Note: all the commands below should be executed under root user.

We need to install a few library dependencies for cryptography and lxml:

# Fix a cryptography error
yum install gcc libffi-devel python-devel openssl-devel
# Fix a lxml error
yum install libxslt-devel libxml2-devel libxml2 libxslt

Everything is ready! Install incremental and Scrapy:

pip install incremental
pip install scrapy

http://stackoverflow.com/a/22210069
http://stackoverflow.com/a/34839358