Home Assistant: upgrade to Python 3.8

home assistant upgraded to python 3.8

Here is how to upgrade a Home Assistant instance to python 3.8.6 using python virtual environments. This is my own experience, with some minor problems that I will explain later, yours could be different. I encourage to take a full backup of your Home Assistant, configuration and database before starting. Anyway I have tried to do the upgrade in the least possible intrusive way, so we can rollback to our last state if something goes wrong.

With the announce of the 0.116 version we have seen that python 3.7 will be deprecated, probably in version 0.118. This procedure is for those that use Home Assistant core installed in a virtual environment and do not have python >= 3.8 provided by their distribution, Raspberry PI OS and Debian Buster included. Those distributions provide python 3.7.3 at a maxim. This procedure has been tested with both. You could upgrade to python 3.9 as well only changing the sources to download.

Python 3.8.6 install

The idea is install our new python version using altinstall, so we maintain our main python version to not broke any dependencies on our system. All commands are meant to be executed as a regular user except those executed with sudo.

First we will install some required packages. For Debian use this command:

apt-get install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev libffi-dev curl libbz2-dev libmysqlclient-dev libjpeg-dev libtiff5 libopenjp2-7

If you are on Raspberry PI OS, use this one:

apt-get install -y build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev libjpeg-dev default-libmysqlclient-dev libtiff5 libopenjp2-7

Then, download, compile and install python 3.8.6 from source.

wget https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tar.xz
tar xf Python-3.8.6.tar.xz
cd Python-3.8.6
./configure --enable-optimizations
# compile using number of cpus available, ex. 2
make -j 2
sudo make altinstall

New virtual environment

Next we will create a new Home Assistant virtual environment with the upgraded python 3.8 version we have just compiled from source. Then we will install home assistant and check its logs while it is booting, searching for errors. This assumes a standard HA installation with the virtual environment located in ‘/srv/homeassistant’ and configuration in the home of the homeassistant user, adapt this to your settings. Commented commands I used are below.

# stop home assistant and related services
sudo service hass stop

# cd into /srv directory
cd /srv/

# move your old virtualenv folder and create an empty new one
sudo mv homeassistant homeassistant.3.7
sudo mkdir homeassistant

# assign necessary permissions
sudo chown homeassistant:homeassistant homeassistant

# become homeassistant user and cd into your new created directory
sudo -u homeassistant -H -s
cd /srv/homeassistant

# create your new virtual environmet in the current folder 
# with the new python version you compiled before
python3.8 -m venv .

# activate the new viltual environment
source bin/activate

# install necesary python module and homeassistant
python3 -m pip install wheel
pip3 install homeassistant

# exit the virtual environment and
# start home assistant and check logs for errors
deactivate
sudo service hass start
# check logs

Troubleshooting

These are some of the problems I faced while updating/reinstalling. This is an error collection I got from different HA installs, you can find some or hopefully none of them.

  • Modules not installed: During Home Assistant boot, some errors about missing modules that were not installed as part of homeassistant. You should reinstall them from inside the new virtual environment. That should solve the problem.
  • Pillow installation: During pillow module installation it complains about some necessary files not found. In mi case installing ‘zlib1g-dev’ and ‘libjpeg8-dev’ with apt solved the issue.
  • Permission problems: Some ‘Permission denied’ errors arose during installation. The error lines were like this:
error: [Errno 13] Permission denied: '/usr/local/lib/python3.8/lib2to3/Grammar3.8.6.final.0.pickle'       
error: [Errno 13] Permission denied: '/usr/local/lib/python3.8/lib2to3/PatternGrammar3.8.6.final.0.pickle'

This was because those files were created with 640 permissions. Chmoding them to 644 was the fix.

  • Package versions: some packages complain about not satisfied dependencies like
aiohttp 3.6.2 requires multidict<5.0,>=4.5, but you'll have multidict 5.0.0 which is incompatible.

installing the proper package within the venv fixed it, in this case

pip install --upgrade --force-reinstall multidict==4.7.6
  • Other modules installation issues: Depending which modules you had previously installed, some of them throw errors, ej. mysqllclient, pymysql, broadlink, homeassistant-pyozw, … I installed some of them without issues from within the virtual environment. Others needed some dev libraries installed system-wide with apt, ej. libudev-dev, libopenzwave1.5-dev, libjpeg-dev, zlib1g-dev, libmysqlclient-dev, default-libmysqlclient-dev, … You should google it to find the required ones for your system.

Rollback

In case things go very bad and your Home Assistant become unstable or broken, you should go back to your old virtual environment. Find the commands below to rollback to your old setup.

# stop home assistant and related services
sudo service hass stop

# cd into /srv directory
cd /srv/

# move the failed venv and restore the old one
sudo mv homeassistant homeassistant.bad
sudo mv homeassistant.3.7 homeassistant

# start home assistant
sudo service hass start

Conclusion

We have seen how to upgrade our Home Assistant virtual environment to python 3.8.6. Even though this is not a very difficult procedure, depending on your system and your Home Assistant installation you can face from none to several problems. I encourage you to do it and tell me how it was. I hope this post has been useful to you.

19 thoughts on “Home Assistant: upgrade to Python 3.8”

  1. Great post! Thanks for the well explained steps! In my case I had to install pymysql and broadlink modules to get rid of the reported error in hass.
    Now I have python 3.8.6 on my debian buster!

  2. Thanks for the instructions but my upgrade did not go so well. After re-start my HACS and corresponding integrations were gone. I’m going to have to roll back.
    Any pointers on what might have happened with HACS?
    I was running 2020.12.1 prior to trying the Python upgrade.

    1. Hi, sorry to hear that. Unfortunately I do not use hacs right now but I will try to help you. Providing you have already updated your Home Assistant startup script to reflect the new virtual environment, restarted Home Assistant and deleted your browser cache, this issue you are facing could be caused probably because of missing installed python packages from within the new virtual environment.
      Hacs files used to be stored in Home Assistant configuration folder, usually in /home/homeassistant/.homeassistant/custom_components or similar. This procedure do not touch any of these files so they should be still there.
      I am not 100% sure but I believe that when you install hacs, or a hacs integration, some packages are installed from the Home Assistant virtual environment (e.g. hacs-frontend, …). These packages are the ones that should be missing in the new venv. I would probably check which packages are installed in the old venv and not installed in the new one and manually install them, starting with the hacs-frontend one. If this does not work, I would try to reinstall hacs with their official installation script and restart Home Assistant. Missing packages from integrations should be installed with this restart.
      If none of the above work, I do not really know the way to go, as I do not know exactly how hacs internally works, I am sorry. You can get a list of packages in a virtual environment with a command like '/venv_dir/bin/python -m pip freeze --local'.
      Do not forget to keep a backup of your old virtual environment folder (e.g. /srv/homeassistant.3.7) and your home assistant configuation folder (e.g. /home/homeassistant/.homeassistant) so in case things go wrong you will always be able to rollback to your original state.
      Any way, which python version were you running before the upgrade? and what operating system are you using? are there any error in home assistant log that could point to the issue?
      Please let me know if you were able to solve the issue this way or if you have any other question about this. Regards.

  3. Appreciate the pointers on troubleshooting, I will try those steps. As for my environment: I am running on RPI-4 with Raspbian 10 and Python version 3.7.3.

    1. Comparing the packages installed there’s a big delta there (not just HACS related). For instance, I use zigby devices in my hass but there are a whole bunch of zigpy_* packages missing in the new /srv/homeassistant/lib/python3.8/site-packages directory. Do these get installed when I add the integration from hass? If that’s the case, rather than go through re-installation of integrations from GUI can I compile a list of missing packages and write a quick bash script to install the missing packages? If yes what’s the correct command to install (say) zigpy_xbee package (as an example)? Thanks again for your help.

      1. On researching this further looks like starting with a fresh install *will* mean that all integrations will need to be re-installed, the easiest way to do that is to do a diff of packages between the old and new installation using “python -m pip freeze –local > list_of_packages”, store the diff in another file with just the name of packages and then do a “python -m pip install -r diff.packages” to get the new packages installed. One more thing to keep in mind is that if your database file is based on an old version of hass then during the startup it may need to upgrade the version of the database so it could take a long time. I just deleted the old db since I did not care about the history that much and this sped up the process.

        1. Yes, as stated in my previous comment I also think the best way to do this is the freeze method. I hope that you could get all your integrations and stuff working again with the new virtual environment. As the config is untouched it should work as expected. Maybe some dependency problem may arise due to the new python/packages version but it should not be difficult to solve.
          Regarding the database, I have observed in some hass updates it took very long to update as you said. If you are using a sqlite file as database it is not recommended to keep too much history as this can also degrade the overall performance of the installation.
          Thanks for all you comments, I hope they could help others to fix the issues when upgrading the python version.
          Please let me know if you have any problem or if I can help you further.

        2. True, all ‘delta’ packages installed after your first hass installation, included those installed by integrations should be reinstalled back in the new virtual environment. I do not know any better way to reinstall those packages than creating a requirements file from the old environment using pip freeze and restoring it on the new venv with ‘pip install -r requeriments_file’.

      2. I was able to skip installing tk-dev on Raspbian where I’m not using any GUI direct on the device. It’s probably not strictly needed for Python 3.8 for Home Assistant and can save a little space and time in the Python compilation.

        Thank you very much for putting this together!

        1. Hi, glad it helped you. Thanks for the hint, If you are not using any GUI you can omit the package tk-dev on raspberry pi installations. Hope this can help others to save space and time as well. Thanks for your comment.

      3. I rarely ever comment on blogs like these, but I must give you my thanks for a very well written tutorial. You saved me a lot of hassle in the my upgrade process and now I successfully run HA on Python 3.9.1.
        One question/comment though. Wouldn’t you save some HA downtime by first creating a new server folder, e.g. …/homeassistant3.9.1/, set up your venv and install HA. Then stop HA service, rename old server folder, rename new folder and start HA.

        Thanks for your time and effort writing this tutorial. I especially appreciate the follow-up on all the possible warnings afterwards. Most other tutorials are just sunny day scenarios but we all know that never happens irl. 😀

        1. Hi, thanks for your comment, glad it helped you 🙂

          One question/comment though. Wouldn’t you save some HA downtime by first creating a new server folder, e.g. …/homeassistant3.9.1/, set up your venv and install HA. Then stop HA service, rename old server folder, rename new folder and start HA.

          Sure! You can avoid some downtime doing the procedure this way. Hope this advise can help others in case they need to reduce the HA downtime. Thanks again.

      4. Hi, i followed the procedure on my “Raspbian GNU/Linux 10 (buster)”, but i had these errors
        ./python -E -S -m sysconfig –generate-posix-vars ;\
        if test $? -ne 0 ; then \
        echo “generate-posix-vars failed” ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
        fi
        Could not find platform independent libraries
        Could not find platform dependent libraries
        Consider setting $PYTHONHOME to [:]
        Python path configuration:
        PYTHONHOME = (not set)
        PYTHONPATH = (not set)
        program name = ‘./python’
        isolated = 0
        environment = 0
        user site = 1
        import site = 0
        sys._base_executable = ‘/home/pi/Python-3.8.6/python’
        sys.base_prefix = ‘/usr/local’
        sys.base_exec_prefix = ‘/usr/local’
        sys.executable = ‘/home/pi/Python-3.8.6/python’
        sys.prefix = ‘/usr/local’
        sys.exec_prefix = ‘/usr/local’
        sys.path = [
        ‘/usr/local/lib/python38.zip’,
        ‘/usr/local/lib/python3.8’,
        ‘/usr/local/lib/lib-dynload’,
        ]
        Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
        Python runtime state: core initialized
        ModuleNotFoundError: No module named ‘encodings’
        Could someone give me any advice how to solve them?

        Thanks.

        1. Hi, sorry to hear that. I have never faced a similar issue. It could be due to several reasons, from dependencies issue to gcc version or any other I can not imagine now.
          Only thing I can advise you is to try to configure before compile without the '--enable-optimizations' option, i.e. just run ./configure with no options before compiling as the error seems related to this. If it does not solve it, you can try then to upgrade your gcc version to the latest available. Hope this helps. Regards.

      5. Thanks for the writeup! I have homeassistant running on Raspberry OS with python 3.8 but 3.9 is giving me trouble. I get this in the homeassistant log –

        Unable to load auth provider homeassistant: libffi.so.7: cannot open shared object file: No such file or directory

        Any suggestions?

        1. Hi, I am no sure but this is probably because your operating system is not providing the libffi.so.7 library that python 3.9 needs. Probably you have version 6 on your system. The cleanest workaround I can think would be installing the libffi7 package which will provide the required library from a newer OS version (e.g. if you are in buster installing libffi7 package from bullseye) and rebuilding python3.9/hass. Other workarounds (symlinking version 7 to 6) could also work but can cause unexpected behaviour. Let me know if this helped.

Leave a Reply

Your email address will not be published.