Troubleshooting

Installation

I got a Python 2 error during installation. Now what?

When installing Kur, you may encounter this error:

============================================================

                           ERROR

             Kur requires Python 3.4 or later.
        See our troubleshooting page to get started:

 https://kur.deepgram.com/troubleshooting.html#installation

============================================================

Uh, oh. There was an error. Look up there ^^^^ and you'll be
            training awesome models in no time!

This is because your version of Python is too old:

$ python --version
Python 2.7.12

Kur only supports Python 3.4 or higher (our reasoning is here). If you thought you had Python 3 installed, then you should set up a virtual environment to prevent your computer from getting confused about which version of Python it should use. If you create a virtualenv using the -p /usr/bin/python3 option (as we show in our Installing Kur guide), then the virtual environment will automatically use Python 3. Of course, you need to make sure you remember to activate the virtual environment before running Kur!

Running Kur

Kur doesn’t look like it is doing anything.

Chances are, the model is still compiling. These deep learning models are highly optimized in order to execute quickly. The backends are responsible for calling low-level compilers (CUDA, GCC, etc.) in order to convert your Kur model into something your processor or GPU knows how to use. For large models, this can take a long time. Try running Kur with increased verbosity (kur -v ...) and see if, in fact, Kur says that it is waiting for the model to finish compiling.

Kur takes a very long before it starts training.

See this answer.

Theano is complaining about BLAS.

Theano might throw an exception that looks like this:

AssertionError: AbstractConv2d Theano optimization failed: there is no
implementation available supporting the requested options. Did you exclude
both "conv_dnn" and "conv_gemm" from the optimizer? If on GPU, is cuDNN
available and does the GPU support it? If on CPU, do you have a BLAS
library installed Theano can link against?

There are a number of ways to solve this.

  • If you are trying to run on a CPU, do one of the following.

    • Switch another backend. For example, if you are currently using the Keras backend with Theano (the default), try switching to the TensorFlow backend:

      settings:
        backend:
          name: keras
          backend: tensorflow
      
    • If you want to use the Keras backend with Theano, then you can add the optimizer: no setting to your specification file:

      settings:
        backend:
          name: keras
          backend: theano
          optimizer: no
      
    • If you are using the Keras backend with Theano programmatically through the Python API, you can pass the Keras backend constructor an additional parameter:

      backend = KerasBacked(optimizer=False)
      
    • Install a linear algebra library. This depends a little on your platform. For Ubuntu, you can do this:

      sudo apt-get install libblas-dev liblapack-dev gfortran-4.9
      
    • Disable optimizer in Theano globally. Edit your ~/.theanorc file and make sure these lines are present:

      [global]
      optimizer = None
      
  • If you are trying to run on an NVIDIA GPU
    • Install cuDNN from NVIDIA’s website.

Theano complains with error “Optimization failure due to: constant_folding”

See this answer.

Theano complains about an ImportError: ... file too short

This can be caused if the Theano cache becomes corrupt. Try moving (or, if you are brave, deleting) the ~/.theano directory someplace else and trying again.

TensorFlow throws an ImportError about GLIBC not found.

This is an unfortunate problem in TensorFlow that boils down to this: your operating system’s version of libc (the C standard library) is too old. On some platforms, you can easily upgrade the OS or libc and fix the problem; on other platforms, it isn’t as easy. For example, we’ve seen this bug crop up on Ubuntu 12.04 (Precise Pangolin), but upgrading the distribution to Ubuntu 14.04 (Trusty Tahr) resolved the problem.

If you don’t know how to upgrade your system, or if you just don’t want to, then the easiest workaround is to simply not use TensorFlow, and instead use a backend based on, e.g., Theano instead. If you are using Keras as a backend to Kur, then you can request that Keras use Theano behind the scenes by putting this in your specification:

settings:
  backend:
    name: keras
    backend: theano

Couldn’t find ffmpeg or avconv

So, you want to do some speech-based learning. Great! In order to handle the wide variety of audio and video formats that your training set might need, we use ffmpeg as a format-conversion tool. This is not a native Python package, so we can’t make it trivially use with pip install. Instead, you need to install the appropriate package for your operating system:

  • macOS / OS X. Make sure you have Homebrew installed (see our installation instructions for some guidance). Then you can install it with:

    $ brew install ffmpeg
    
  • Ubuntu 14.04 (Trusty Tahr). Ubuntu switched to avconv briefly as a replacement for ffmpeg. So you can install a similar tool with:

    $ sudo apt-get install libav-tools
    
  • Ubuntu 16.04 (Xenial Xerus). ffmpeg made a return in Ubuntu 15.04, so you can install it with:

    $ sudo apt-get install ffmpeg
    
  • Arch Linux. Pretty simple:

    $ sudo pacman -S ffmpeg
    

fatal error: Python.h: No such file or directory

This is caused by not having the Python development headers installed. Try the following:

  • Ubuntu

    sudo apt-get install python3-dev
    

TypeError: Expected int32, got list containing Tensors of type ‘_Message’ instead.

If you are using TensorFlow <= 0.12 (see pip freeze), upgrade to TensorFlow >= 1.0:

pip install --upgrade tensorflow      # And/or tensorflow-gpu

You forced the use of gpu device ‘gpu’, but nvcc was not found.

If you don’t have a GPU, then remove the device: gpu line from your Kurfile. If you do have a GPU, read on.

First, do you actually have CUDA installed? If not, make sure you have the latest NVIDIA drivers for your GPU, and then install CUDA.

Next, we need to get nvcc on your PATH. It’s probably in /usr/local/cuda/bin, but you can also directly search for it:

sudo find / -name nvcc -type f 2>/dev/null

Finally, we need to update your PATH. Depending on your platform and shell, the command may look like:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc

Now, either close your terminal and then bring it back up, or just activate the new PATH using:

source ~/.bashrc

ValueError: Invalid value (“true”) for configuration variable “dnn.enabled”

Theano has changed the format for its configuration file, which by default is at ~/.theanorc. Change lines like this:

[dnn]
enabled = true

to this:

[dnn]
enabled = True

Plotting

I get an error from matplotlib saying “Python is not installed as a framework.”

Did you use pip to install matplotlib (you should!). The trick is to tell matplotlib which “backend” it should use for its plotting. Do this:

echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc

There, that should do it.

Plots from matplotlib don’t appear, or I get errors about backends.

See this answer.