Home Assistant: upgrade to Python 3.8

Here is how to upgrade a Home Assistant instance to python 3.8.6 using python virtual environments. This is my own experience, with some minor problems that I will explain later, yours could be different. I encourage to take a full backup of your Home Assistant, configuration and database before starting. Anyway I have tried to do the upgrade in the least possible intrusive way, so we can rollback to our last state if something goes wrong.

With the announce of the 0.116 version we have seen that python 3.7 will be deprecated, probably in version 0.118. This procedure is for those that use Home Assistant core installed in a virtual environment and do not have python >= 3.8 provided by their distribution, Raspberry PI OS and Debian Buster included. Those distributions provide python 3.7.3 at a maxim. This procedure has been tested with both. You could upgrade to python 3.9 as well only changing the sources to download.

Python 3.8.6 install

The idea is install our new python version using altinstall, so we maintain our main python version to not broke any dependencies on our system. All commands are meant to be executed as a regular user except those executed with sudo.

First we will install some required packages. For Debian use this command:

apt-get install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev libffi-dev curl libbz2-dev libmysqlclient-dev libjpeg-dev libtiff5 libopenjp2-7

If you are on Raspberry PI OS, use this one:

apt-get install -y build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev libjpeg-dev default-libmysqlclient-dev libtiff5 libopenjp2-7

Then, download, compile and install python 3.8.6 from source.

wget https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tar.xz
tar xf Python-3.8.6.tar.xz
cd Python-3.8.6
./configure --enable-optimizations
# compile using number of cpus available, ex. 2
make -j 2
sudo make altinstall

New virtual environment

Next we will create a new Home Assistant virtual environment with the upgraded python 3.8 version we have just compiled from source. Then we will install home assistant and check its logs while it is booting, searching for errors. This assumes a standard HA installation with the virtual environment located in ‘/srv/homeassistant’ and configuration in the home of the homeassistant user, adapt this to your settings. Commented commands I used are below.

# stop home assistant and related services
sudo service hass stop

# cd into /srv directory
cd /srv/

# move your old virtualenv folder and create an empty new one
sudo mv homeassistant homeassistant.3.7
sudo mkdir homeassistant

# assign necessary permissions
sudo chown homeassistant:homeassistant homeassistant

# become homeassistant user and cd into your new created directory
sudo -u homeassistant -H -s
cd /srv/homeassistant

# create your new virtual environmet in the current folder 
# with the new python version you compiled before
python3.8 -m venv .

# activate the new viltual environment
source bin/activate

# install necesary python module and homeassistant
python3 -m pip install wheel
pip3 install homeassistant

# exit the virtual environment and
# start home assistant and check logs for errors
deactivate
sudo service hass start
# check logs

Troubleshooting

These are some of the problems I faced while updating/reinstalling. This is an error collection I got from different HA installs, you can find some or hopefully none of them.

• Modules not installed: During Home Assistant boot, some errors about missing modules that were not installed as part of homeassistant. You should reinstall them from inside the new virtual environment. That should solve the problem.

• Pillow installation: During pillow module installation it complains about some necessary files not found. In mi case installing ‘zlib1g-dev’ and ‘libjpeg8-dev’ with apt solved the issue.

• Permission problems: Some ‘Permission denied’ errors arose during installation. The error lines were like this:

error: [Errno 13] Permission denied: '/usr/local/lib/python3.8/lib2to3/Grammar3.8.6.final.0.pickle'       
error: [Errno 13] Permission denied: '/usr/local/lib/python3.8/lib2to3/PatternGrammar3.8.6.final.0.pickle'

This was because those files were created with 640 permissions. Chmoding them to 644 was the fix.

• Package versions: some packages complain about not satisfied dependencies like

aiohttp 3.6.2 requires multidict<5.0,>=4.5, but you'll have multidict 5.0.0 which is incompatible.

installing the proper package within the venv fixed it, in this case

pip install --upgrade --force-reinstall multidict==4.7.6

• Other modules installation issues: Depending which modules you had previously installed, some of them throw errors, ej. mysqllclient, pymysql, broadlink, homeassistant-pyozw, … I installed some of them without issues from within the virtual environment. Others needed some dev libraries installed system-wide with apt, ej. libudev-dev, libopenzwave1.5-dev, libjpeg-dev, zlib1g-dev, libmysqlclient-dev, default-libmysqlclient-dev, … You should google it to find the required ones for your system.

Rollback

In case things go very bad and your Home Assistant become unstable or broken, you should go back to your old virtual environment. Find the commands below to rollback to your old setup.

# stop home assistant and related services
sudo service hass stop

# cd into /srv directory
cd /srv/

# move the failed venv and restore the old one
sudo mv homeassistant homeassistant.bad
sudo mv homeassistant.3.7 homeassistant

# start home assistant
sudo service hass start

Conclusion

We have seen how to upgrade our Home Assistant virtual environment to python 3.8.6. Even though this is not a very difficult procedure, depending on your system and your Home Assistant installation you can face from none to several problems. I encourage you to do it and tell me how it was. I hope this post has been useful to you.